LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
Reload this Page Sudden death of SSO web server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 08-05-2008, 03:02 PM   #1
tanveer
Member
 
Registered: Feb 2004
Location: e@rth
Distribution: RHEL-3/4/5,Gloria,opensolaris
Posts: 525

Rep: Reputation: 39
Sudden death of SSO web server

Hi all,
I have a SSO web server with apache,php & mysql. It was working great for last almost one year. But suddenly it hanged and found these in /var/log/messages file. After a restart it became online again.
How come it eats up all Swap space all of a sudden and what does these message means for troubleshooting.

PHP Code:
Aug  1 12:40:05 server1 kernel: oom-killer: gfp_mask=0xd2
Aug  1 12:40:05 server1 kernel: Mem-info:
Aug  1 12:40:05 server1 kernel: DMA per-cpu:
Aug  1 12:40:05 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Aug  1 12:40:05 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Aug  1 12:40:05 server1 kernel: cpu 1 hot: low 2, high 6, batch 1
Aug  1 12:40:05 server1 kernel: cpu 1 cold: low 0, high 2, batch 1
Aug  1 12:40:05 server1 kernel: cpu 2 hot: low 2, high 6, batch 1
Aug  1 12:40:05 server1 kernel: cpu 2 cold: low 0, high 2, batch 1
Aug  1 12:40:05 server1 kernel: cpu 3 hot: low 2, high 6, batch 1
Aug  1 12:40:05 server1 kernel: cpu 3 cold: low 0, high 2, batch 1
Aug  1 12:40:05 server1 kernel: Normal per-cpu:
Aug  1 12:40:10 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug  1 12:40:10 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug  1 12:40:10 server1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug  1 12:40:11 server1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug  1 12:40:11 server1 kernel: cpu 2 hot: low 32, high 96, batch 16
Aug  1 12:40:11 server1 kernel: cpu 2 cold: low 0, high 32, batch 16
Aug  1 12:40:12 server1 kernel: cpu 3 hot: low 32, high 96, batch 16
Aug  1 12:40:12 server1 kernel: cpu 3 cold: low 0, high 32, batch 16
Aug  1 12:40:12 server1 kernel: HighMem per-cpu:
Aug  1 12:40:13 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug  1 12:40:13 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug  1 12:40:13 server1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug  1 12:40:13 server1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug  1 12:40:13 server1 kernel: cpu 2 hot: low 32, high 96, batch 16
Aug  1 12:40:13 server1 kernel: cpu 2 cold: low 0, high 32, batch 16
Aug  1 12:40:14 server1 kernel: cpu 3 hot: low 32, high 96, batch 16
Aug  1 12:40:14 server1 crond(pam_unix)[23892]: session opened for user root by (uid=0)
Aug  1 12:40:14 server1 crond(pam_unix)[23891]: session opened for user root by (uid=0)
Aug  1 12:40:14 server1 kernel: cpu 3 cold: low 0, high 32, batch 16
Aug  1 12:40:14 server1 kernel: 
Aug  1 12:40:14 server1 kernel: Free pages:       14012kB (512kB HighMem)
Aug  1 12:40:15 server1 kernel: Active:271231 inactive:230139 dirty:0 writeback:0 unstable:0 free:3503 slab:4751 mapped:501131 pagetables:452
1
Aug  1 12:40:15 server1 kernel: DMA free:12588kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:122809 all
_unreclaimable? yes
Aug  1 12:40:15 server1 kernel: protections[]: 0 0 0
Aug  1 12:40:15 server1 kernel: Normal free:912kB min:928kB low:1856kB high:2784kB active:73468kB inactive:777992kB present:901120kB pages_sc
anned:1768239 all_unreclaimable? yes
Aug  1 12:40:25 server1 kernel: protections[]: 0 0 0
Aug  1 12:40:29 server1 kernel: HighMem free:512kB min:512kB low:1024kB high:1536kB active:1011840kB inactive:142180kB present:1179596kB page
s_scanned:3094975 all_unreclaimable? yes
Aug  1 12:40:30 server1 kernel: protections[]: 0 0 0
Aug  1 12:40:30 server1 kernel: DMA: 3*4kB 4*8kB 4*16kB 2*32kB 4*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12588kB
Aug  1 12:40:30 server1 kernel: Normal: 0*4kB 14*8kB 2*16kB 8*32kB 2*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 912kB
Aug  1 12:40:31 server1 kernel: HighMem: 2*4kB 1*8kB 3*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 512kB
Aug  1 12:40:31 server1 kernel: Swap cache: add 4121947, delete 4121245, find 28269186/28669404, race 0+1784
Aug  1 12:40:32 server1 crond(pam_unix)[23892]: session closed for user root
Aug  1 12:40:32 server1 kernel: 0 bounce buffer pages

Aug  1 12:40:33 server1 kernel: Free swap:            0kB
Aug  1 12:40:33 server1 kernel: 524275 pages of RAM
Aug  1 12:40:33 server1 kernel: 294899 pages of HIGHMEM
Aug  1 12:40:36 server1 kernel: 5534 reserved pages
Aug  1 12:40:36 server1 kernel: 79666 pages shared
Aug  1 12:40:37 server1 kernel: 724 pages swap cached
Aug  1 12:40:37 server1 kernel: Out of Memory: Killed process 23487 (httpd). 
Thanks in advance.
 
Old 08-05-2008, 08:09 PM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,359

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
You need to examine the various logfiles to see what was going on in the preceding eg 30 mins to see what was running just before it called the oom-killer. Just because the httpd server was killed doesn't necessarily mean that was the culprit. oom-killer is not that smart.
 
Old 08-05-2008, 08:50 PM   #3
trickykid
LQ Guru
 
Registered: Jan 2001
Posts: 24,149

Rep: Reputation: 269Reputation: 269Reputation: 269
More details would be helpful with your setup, amount of RAM, etc. But usually when the system or kernel starts killing processes, that means you either have a memory leak somewhere or your actual memory is going bad, from my experience at least. OOM might not be smart but the OS will start killing off processes that it feels are not as important as the ones to keep the OS running.
 
Old 08-06-2008, 01:53 AM   #4
tanveer
Member
 
Registered: Feb 2004
Location: e@rth
Distribution: RHEL-3/4/5,Gloria,opensolaris
Posts: 525

Original Poster
Rep: Reputation: 39
It's a HP Proliant DL 380 G4 server with 2GB RAM and swap of 6GB with Intel(R) Xeon(TM) CPU 3.00GHz running php,apache,mysql,winbind for authentication.
Running Kernel version-2.6.9-22 (32bit) though it supports 64bit.

After reboot the scenario is like this

PHP Code:
[root@phobos ~]# free -lm
             total       used       free     shared    buffers     cached
Mem:          2026       1307        718          0         33       1114
Low:           874        157        717
High:         1151       1150          1
-/+ buffers/cache:        159       1866
Swap:         6143          0       6143 
In log files nothing significant found though. Confused why it happened all on a sudden!!

Thanks for all your help.
 
Old 08-06-2008, 09:44 AM   #5
trickykid
LQ Guru
 
Registered: Jan 2001
Posts: 24,149

Rep: Reputation: 269Reputation: 269Reputation: 269
Quote:
Originally Posted by tanveer View Post
It's a HP Proliant DL 380 G4 server with 2GB RAM and swap of 6GB with Intel(R) Xeon(TM) CPU 3.00GHz running php,apache,mysql,winbind for authentication.
Running Kernel version-2.6.9-22 (32bit) though it supports 64bit.

After reboot the scenario is like this

PHP Code:
[root@phobos ~]# free -lm
             total       used       free     shared    buffers     cached
Mem:          2026       1307        718          0         33       1114
Low:           874        157        717
High:         1151       1150          1
-/+ buffers/cache:        159       1866
Swap:         6143          0       6143 
In log files nothing significant found though. Confused why it happened all on a sudden!!

Thanks for all your help.
When was the last reboot before it started killing processes?
 
Old 08-09-2008, 10:19 AM   #6
tanveer
Member
 
Registered: Feb 2004
Location: e@rth
Distribution: RHEL-3/4/5,Gloria,opensolaris
Posts: 525

Original Poster
Rep: Reputation: 39
sorry for late reply. I was out of the town.
It will be almost a year since last reboot.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Which is easier to install on Linux PGP, x.509/PKCS, SAML, or SSO Web Services sdonohue Linux - Software 2 09-26-2006 04:12 PM
Mail server sudden slow-down. (Sendmail) LWillmann Linux - Software 4 09-14-2006 04:21 PM
OpenSuSE - Sudden Internet Death madgenius Linux - Networking 1 04-10-2006 04:34 AM
OpenLDAP for web application SSO integration mytto Linux - Networking 3 02-15-2006 11:23 AM
Enable SSO in Applications Aman25 Linux - Security 1 07-30-2004 09:43 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 06:56 AM.

Contact Us - Advertising Info - Rules - Privacy - LQ Merchandise - Donations - Contributing Member - LQ Sitemap -
Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration