LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 07-26-2010, 09:40 AM   #1
tuxtutorials
Member
 
Registered: Dec 2008
Location: New York
Distribution: RedHat, Solaris
Posts: 68

Rep: Reputation: 16
Performance issues causing process to crash


Hello all, we are currently running a squid proxy server that has a commercial web filtering service. Lately a service within the web filtering software appears to randomly crash throughout the day and I am having a tough time to weed out what could be going on. I do have one lead and believe it is related to memory deprivation causing the service to crash.

I wrote a monitor script to alert me when the process goes down and automatically restart the service but would like to resolve the issue rather than kludge it. Here are the times I get alerts when the service goes down:

3:50PM
3:15PM
2:45PM

12:44PM
10:38AM
10:12AM

Here my findings on sar.


Code:
Linux 2.6.9-78.0.8.ELsmp (box.example.com)     07/21/2010

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
12:10:01 AM    963960   3183728     76.76    160516   1941064   8289292       208      0.00         0
12:20:01 AM    962224   3185464     76.80    161704   1941176   8289292       208      0.00         0
12:30:01 AM    961920   3185768     76.81    162820   1940840   8289292       208      0.00         0
12:40:01 AM    963744   3183944     76.76    163928   1937132   8289292       208      0.00         0
12:50:01 AM    963104   3184584     76.78    165032   1937328   8289292       208      0.00         0
01:00:01 AM    962208   3185480     76.80    166136   1937264   8289292       208      0.00         0
01:10:01 AM    963128   3184560     76.78    167408   1935472   8289292       208      0.00         0
01:20:01 AM    961984   3185704     76.81    168564   1935356   8289292       208      0.00         0
01:30:01 AM    960824   3186864     76.83    169664   1935556   8289292       208      0.00         0
01:40:01 AM    957504   3190184     76.91    170912   1937428   8289292       208      0.00         0
01:50:01 AM    960320   3187368     76.85    172000   1933480   8289292       208      0.00         0
02:00:01 AM    959232   3188456     76.87    173128   1933392   8289292       208      0.00         0
02:10:01 AM    955776   3191912     76.96    174400   1935500   8289292       208      0.00         0
02:20:01 AM    958720   3188968     76.89    175436   1931344   8289292       208      0.00         0
02:30:01 AM    957504   3190184     76.91    176572   1931508   8289292       208      0.00         0
02:40:01 AM    955904   3191784     76.95    177576   1931544   8289292       208      0.00         0
02:50:01 AM    958400   3189288     76.89    178584   1928196   8289292       208      0.00         0
03:00:01 AM    958328   3189360     76.89    179564   1927736   8289292       208      0.00         0
03:10:01 AM    956680   3191008     76.93    180544   1927796   8289292       208      0.00         0
03:20:01 AM    954360   3193328     76.99    181700   1929760   8289292       208      0.00         0
03:30:01 AM    957056   3190632     76.93    182796   1925804   8289292       208      0.00         0
03:40:01 AM    956040   3191648     76.95    183876   1925764   8289292       208      0.00         0
03:50:01 AM    952968   3194720     77.02    185008   1927492   8289292       208      0.00         0
04:00:01 AM    952000   3195688     77.05    186000   1927800   8289292       208      0.00         0
04:10:01 AM    938872   3208816     77.36    187252   1938768   8289292       208      0.00         0
04:20:01 AM    936392   3211296     77.42    188080   1940540   8289292       208      0.00         0
04:30:01 AM    935688   3212000     77.44    188832   1940828   8289292       208      0.00         0
04:40:01 AM    934144   3213544     77.48    189516   1940664   8289292       208      0.00         0
04:50:01 AM    936448   3211240     77.42    190320   1938560   8289292       208      0.00         0
05:00:01 AM    935744   3211944     77.44    190952   1938708   8289292       208      0.00         0
05:10:01 AM    935120   3212568     77.45    191636   1938544   8289292       208      0.00         0
05:20:01 AM    932608   3215080     77.51    192396   1940384   8289292       208      0.00         0
05:30:01 AM    931568   3216120     77.54    193156   1940664   8289292       208      0.00         0
05:40:01 AM    932080   3215608     77.53    194204   1938836   8289292       208      0.00         0
05:50:01 AM    929040   3218648     77.60    195272   1940108   8289292       208      0.00         0
06:00:01 AM    923984   3223704     77.72    197576   1941964   8289292       208      0.00         0
06:10:01 AM    917032   3230656     77.89    199720   1946060   8289292       208      0.00         0
06:20:01 AM    909616   3238072     78.07    203384   1949156   8289292       208      0.00         0
06:30:01 AM    902664   3245024     78.24    206364   1952676   8289292       208      0.00         0
06:40:01 AM    888608   3259080     78.58    209488   1962552   8289292       208      0.00         0
06:50:01 AM    878080   3269608     78.83    212648   1970052   8289292       208      0.00         0
07:00:01 AM    872624   3275064     78.96    214796   1970244   8289292       208      0.00         0
07:10:01 AM    868840   3278848     79.05    217012   1971148   8289292       208      0.00         0
07:20:01 AM    854160   3293528     79.41    219744   1982976   8289292       208      0.00         0
07:30:01 AM    814272   3333416     80.37    224520   2015380   8289292       208      0.00         0
07:40:01 AM    792560   3355128     80.89    228952   2031488   8289292       208      0.00         0
07:50:01 AM    772904   3374784     81.37    232100   2045760   8289292       208      0.00         0
08:00:01 AM    762816   3384872     81.61    234752   2052728   8289292       208      0.00         0
08:10:01 AM    756720   3390968     81.76    236940   2056260   8289292       208      0.00         0
08:20:01 AM    740984   3406704     82.14    239920   2067320   8289292       208      0.00         0
08:30:01 AM    736456   3411232     82.24    241844   2069296   8289292       208      0.00         0
08:40:01 AM    715400   3432288     82.75    245048   2084552   8289292       208      0.00         0
08:50:01 AM    700600   3447088     83.11    248156   2094964   8289292       208      0.00         0
09:00:01 AM    684232   3463456     83.50    251172   2107028   8289292       208      0.00         0
09:10:01 AM    648112   3499576     84.37    255432   2136048   8289292       208      0.00         0
09:20:01 AM    620032   3527656     85.05    258436   2158784   8289292       208      0.00         0
09:30:01 AM    610704   3536984     85.28    260116   2164644   8289292       208      0.00         0
09:40:01 AM    605440   3542248     85.40    261352   2167828   8289292       208      0.00         0
09:50:01 AM    601176   3546512     85.51    262564   2170776   8289292       208      0.00         0
10:00:01 AM    589864   3557824     85.78    264376   2179884   8289292       208      0.00         0
10:10:01 AM    578560   3569128     86.05    266780   2187620   8289292       208      0.00         0
10:20:01 AM    300288   3847400     92.76    270616   2608624   8289292       208      0.00         0
10:30:01 AM    280600   3867088     93.23    269456   2623304   8289292       208      0.00         0
10:40:01 AM    267912   3879776     93.54    267716   2639604   8289292       208      0.00         0
10:50:01 AM    259512   3888176     93.74    265500   2645460   8289292       208      0.00         0
11:00:01 AM    246512   3901176     94.06    267244   2652296   8289292       208      0.00         0
11:10:01 AM    226616   3921072     94.54    265868   2671092   8289292       208      0.00         0
11:20:01 AM    211192   3936496     94.91    262596   2688144   8289292       208      0.00         0
11:30:01 AM    198744   3948944     95.21    258412   2701948   8289292       208      0.00         0
11:40:01 AM    176928   3970760     95.73    256876   2719084   8289292       208      0.00         0
11:50:01 AM    144656   4003032     96.51    253028   2751012   8289292       208      0.00         0
12:00:01 PM    127616   4020072     96.92    255384   2765556   8289292       208      0.00         0
12:10:01 PM     96760   4050928     97.67    252128   2796372   8289292       208      0.00         0
12:20:01 PM     77912   4069776     98.12    248344   2815756   8289292       208      0.00         0
12:30:01 PM     60448   4087240     98.54    247548   2833712   8289292       208      0.00         0
12:40:01 PM     52120   4095568     98.74    246308   2842232   8289292       208      0.00         0
12:50:01 PM     50360   4097328     98.79    245948   2856632   8289292       208      0.00         0
01:00:01 PM     35520   4112168     99.14    248456   2866864   8289292       208      0.00         0
01:10:01 PM     24272   4123416     99.41    237484   2891356   8289292       208      0.00         0
01:20:01 PM     24568   4123120     99.41    212732   2910908   8289292       208      0.00         0
01:30:01 PM     23128   4124560     99.44    187372   2932888   8289292       208      0.00         0
01:40:01 PM     21976   4125712     99.47    170316   2940324   8289292       208      0.00         0
01:50:01 PM     34944   4112744     99.16    167328   2930572   8289292       208      0.00         0
02:00:01 PM     47872   4099816     98.85    171236   2914964   8289292       208      0.00         0
02:10:01 PM     62720   4084968     98.49    177376   2898944   8289292       208      0.00         0
02:20:01 PM     87656   4060032     97.89    177852   2873768   8289292       208      0.00         0
02:30:01 PM     82088   4065600     98.02    181896   2864264   8289292       208      0.00         0
02:40:01 PM     62256   4085432     98.50    183880   2868260   8289292       208      0.00         0
02:50:02 PM     74848   4072840     98.20    190316   2877164   8289292       208      0.00         0
03:00:01 PM     37624   4110064     99.09    195208   2906852   8289292       208      0.00         0
03:10:01 PM     35840   4111848     99.14    196584   2906776   8289292       208      0.00         0
03:20:01 PM     64952   4082736     98.43    195196   2869944   8289292       208      0.00         0
03:30:01 PM     48968   4098720     98.82    198732   2877328   8289292       208      0.00         0
03:40:01 PM     45400   4102288     98.91    199844   2879596   8289292       208      0.00         0
03:50:01 PM    341384   3806304     91.77    197292   2877208   8289292       208      0.00         0
04:00:01 PM     55896   4091792     98.65    196016   2867564   8289292       208      0.00         0
04:10:01 PM     57576   4090112     98.61    193272   2865888   8289292       208      0.00         0
04:20:01 PM     55864   4091824     98.65    194496   2864664   8289292       208      0.00         0
04:30:01 PM     59144   4088544     98.57    193960   2861820   8289292       208      0.00         0
04:40:01 PM     50080   4097608     98.79    197020   2870200   8289292       208      0.00         0
04:50:01 PM     53992   4093696     98.70    195680   2867900   8289292       208      0.00         0
05:00:01 PM     64280   4083408     98.45    195408   2858032   8289292       208      0.00         0
05:10:01 PM     58328   4089360     98.59    197516   2860344   8289292       208      0.00         0
05:20:01 PM     75144   4072544     98.19    199764   2864336   8289292       208      0.00         0
05:30:01 PM     72696   4074992     98.25    201416   2864764   8289292       208      0.00         0
05:40:01 PM     83240   4064448     97.99    199496   2856544   8289292       208      0.00         0
05:50:01 PM     80888   4066800     98.05    201564   2857076   8289292       208      0.00         0
06:00:01 PM     90872   4056816     97.81    199932   2850128   8289292       208      0.00         0
06:10:01 PM     89328   4058360     97.85    201108   2849992   8289292       208      0.00         0
06:20:01 PM     88120   4059568     97.88    202476   2850704   8289292       208      0.00         0
06:30:01 PM     96312   4051376     97.68    200324   2842456   8289292       208      0.00         0
06:40:01 PM     94904   4052784     97.71    201596   2842484   8289292       208      0.00         0
06:50:01 PM     93816   4053872     97.74    202744   2842636   8289292       208      0.00         0
07:00:01 PM    103648   4044040     97.50    200136   2835884   8289292       208      0.00         0
07:10:01 PM    102264   4045424     97.53    201260   2835800   8289292       208      0.00         0
07:20:01 PM    100856   4046832     97.57    202416   2835944   8289292       208      0.00         0
07:30:01 PM     99576   4048112     97.60    203552   2835848   8289292       208      0.00         0
07:40:01 PM    108216   4039472     97.39    201196   2830664   8289292       208      0.00         0
07:50:01 PM    107000   4040688     97.42    202344   2830556   8289292       208      0.00         0
08:00:02 PM    106040   4041648     97.44    203572   2830628   8289292       208      0.00         0
08:10:01 PM    117304   4030384     97.17    201068   2821692   8289292       208      0.00         0
08:20:01 PM    116344   4031344     97.19    202144   2821916   8289292       208      0.00         0
08:30:01 PM    114016   4033672     97.25    203384   2821716   8289292       208      0.00         0
08:40:01 PM    113968   4033720     97.25    204748   2821912   8289292       208      0.00         0
08:50:01 PM    116720   4030968     97.19    202164   2813316   8289292       208      0.00         0
09:00:01 PM    107376   4040312     97.41    203152   2813368   8289292       208      0.00         0
09:10:01 PM    117680   4030008     97.16    204180   2813380   8289292       208      0.00         0
09:20:01 PM    117168   4030520     97.18    205260   2813340   8289292       208      0.00         0
09:30:01 PM    126192   4021496     96.96    202636   2807644   8289292       208      0.00         0
09:40:01 PM    125024   4022664     96.99    203704   2807616   8289292       208      0.00         0
09:50:01 PM    126448   4021240     96.95    204656   2807444   8289292       208      0.00         0
10:00:01 PM    125400   4022288     96.98    205600   2807540   8289292       208      0.00         0
10:10:01 PM    132432   4015256     96.81    203496   2802884   8289292       208      0.00         0
10:20:01 PM    131384   4016304     96.83    204452   2802708   8289292       208      0.00         0
10:30:01 PM    129568   4018120     96.88    205380   2802820   8289292       208      0.00         0
10:40:01 PM    129240   4018448     96.88    206420   2802820   8289292       208      0.00         0
10:50:01 PM    136080   4011608     96.72    206044   2796956   8289292       208      0.00         0
11:00:01 PM    135008   4012680     96.74    206936   2796844   8289292       208      0.00         0
11:10:01 PM    134288   4013400     96.76    207872   2796948   8289292       208      0.00         0
11:20:01 PM    133504   4014184     96.78    208684   2796916   8289292       208      0.00         0
11:30:01 PM    132688   4015000     96.80    209456   2796924   8289292       208      0.00         0
11:40:01 PM    145872   4001816     96.48    210296   2785944   8289292       208      0.00         0
11:50:01 PM    144920   4002768     96.51    211104   2785916   8289292       208      0.00         0
Average:       433069   3714619     89.56    207635   2466292   8289292       208      0.00         0

From the sar reports. It looks like every time this process dies the system appears to be at 99% memory utilization. Am I correct on my assumptions that this is a memory issue and the process is terminating when memory usage hits 100%? If so why is swap not being used instead? Please let me know if I am on the right path and if there are any other stats I should investigate.

Thanks

Last edited by tuxtutorials; 07-26-2010 at 09:42 AM. Reason: improper grammar
 
Old 07-27-2010, 05:49 PM   #2
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,790

Rep: Reputation: 651Reputation: 651Reputation: 651Reputation: 651Reputation: 651Reputation: 651
Seeing how the web filtering software is commercial you could probably raise a ticket with the vendor...
 
Old 07-28-2010, 12:44 AM   #3
saifkhan123
Member
 
Registered: Apr 2009
Distribution: Red Hat/CentOS
Posts: 108

Rep: Reputation: 19
Well! it can't be said that its a memory issue until you investigate completely. You have mentioned the times when the service crashes, I think you should look into the logs at those times, may be you can find some clue if look closely.
A complete list of all the things you should look for at this point would be large. Initially, consider the following things
  • Use command "Vmstat 3 15". The vmstat utility will show how many processes are in the run queue. If the vmstat utility shows many processes in the blocked statistic (.b. under .procs.), then the system may be I/O bound. This could be because of slow disks or slow Ethernet cards.
  • Confirm that you are running the latest update to your operating system.
  • Are there networking or other issues that may have been overlooked? (for example, are there any errors reported in ifconfig?)
  • Make sure nothing suspicious is in /var/log/messages. This may include hard disk or other hardware errors that have been overlooked.
 
Old 07-28-2010, 10:10 AM   #4
tuxtutorials
Member
 
Registered: Dec 2008
Location: New York
Distribution: RedHat, Solaris
Posts: 68

Original Poster
Rep: Reputation: 16
Thanks kbp and saifkhan123. I'll continue to look at the issue I already opened a case up with the webfilter guys and they were clueless. I'll continue to look for the smoking gun. Thanks for the tips.
 
Old 07-28-2010, 11:42 AM   #5
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191
Quote:
Originally Posted by tuxtutorials View Post
I do have one lead and believe it is related to memory deprivation
I think you are misinterpreting the data. I don't have a guess of what is causing the crash, but the data you posted makes any memory shortage look very unlikely.

Quote:
From the sar reports. It looks like every time this process dies the system appears to be at 99% memory utilization.
Memory utilization is nearly meaningless for this purpose.

Quote:
Am I correct on my assumptions that this is a memory issue and the process is terminating when memory usage hits 100%?
No.

Quote:
If so why is swap not being used instead?
As the memory demands grow, the system takes memory away from the least recently used buffers and cache. So you can see (in the stats you posted) buffers and cache going down as the system's memory load increases.

If all the buffers and cache were recently used, then the system would swap instead of taking data from buffers and cache. Since that isn't happening, we can conclude that much of the buffers an cache is old data just sitting there, because some chance the data will be reused in cache is better than zero chance if freed.

Even once the system is swapping, old anonymous data goes out first, so swapping wouldn't necessarily imply any significant memory pressure. In this case, the lack of swapping clearly shows there is no significant memory pressure.

But is this a 32 bit system or 64 bit?


Everything I said above refers to system wide memory considerations. But in a 32 bit system, you also must worry about per process virtual limits.

If most of the memory use is inside one big 32 bit process, that process might be crashing by hitting its own internal 32 bit limits. You would need to look at its /proc/pid/smaps for details as it grows toward failure to get a better understanding.

Last edited by johnsfine; 07-28-2010 at 11:46 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Flash V9 causing Firefox to crash girish_hilage Linux - Networking 3 08-12-2008 04:50 AM
Wine causing KDE to crash/logout Jayla Linux - Software 3 11-18-2007 07:12 AM
Using two Ultra100 TX2 causing Linux to crash almvtb Linux - Hardware 3 03-21-2006 03:45 PM
pppd causing kernel crash lukebeales Linux - Software 10 06-05-2005 09:25 AM
Browsers causing KDE to crash juanr0 Linux - Newbie 0 02-16-2005 12:53 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 04:37 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration