No Free Swap? Kernel Messages Abound.

ECRocker · 09-13-2010, 07:46 AM

I saw my backups failed on my LAMP Wordpress server this weekend, but thought it could have waited till Monday. It looks like the server was down all weekend with a message (attached as jpeg) on the console referring to "Free Swap = 0kb". I couldn't log in by any means, switching terminals only threw up more system messages in my face. After a reboot, I grabbed this repeating message:

Code:

Sep  9 11:44:36 LAMPWord kernel: Call Trace:
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff800c65e9>] out_of_memory+0x8e/0x2f3
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8000f487>] __alloc_pages+0x245/0x2ce
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff80023441>] alloc_page_interleave+0x3d/0x74
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8003c5e2>] __get_free_pages+0xe/0x71
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8001ef08>] copy_process+0xc6/0x15db
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8009ddda>] alloc_pid+0x1f6/0x292
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff800312be>] do_fork+0x69/0x1c1
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8009fe9f>] keventd_create_kthread+0x0/0xc4
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8005df3d>] kernel_thread+0x81/0xeb
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8009fe9f>] keventd_create_kthread+0x0/0xc4
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff80032852>] kthread+0x0/0x132
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8009febc>] keventd_create_kthread+0x1d/0xc4
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8004d8ed>] run_workqueue+0x94/0xe4
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8004a12f>] worker_thread+0x0/0x122
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8004a21f>] worker_thread+0xf0/0x122
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8008c86c>] default_wake_function+0x0/0xe
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff80032950>] kthread+0xfe/0x132
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff80032852>] kthread+0x0/0x132
Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Sep  9 11:44:36 LAMPWord kernel:
Sep  9 11:44:36 LAMPWord kernel: Mem-info:
Sep  9 11:44:36 LAMPWord kernel: Node 0 DMA per-cpu:
Sep  9 11:44:36 LAMPWord kernel: cpu 0 hot: high 0, batch 1 used:0
Sep  9 11:44:36 LAMPWord kernel: cpu 0 cold: high 0, batch 1 used:0
Sep  9 11:44:36 LAMPWord kernel: Node 0 DMA32 per-cpu:
Sep  9 11:44:36 LAMPWord kernel: cpu 0 hot: high 186, batch 31 used:13
Sep  9 11:44:36 LAMPWord kernel: cpu 0 cold: high 62, batch 15 used:58
Sep  9 11:44:36 LAMPWord kernel: Node 0 Normal per-cpu: empty
Sep  9 11:44:36 LAMPWord kernel: Node 0 HighMem per-cpu: empty
Sep  9 11:44:36 LAMPWord kernel: Free pages:        4800kB (0kB HighMem)
Sep  9 11:44:36 LAMPWord kernel: Active:35880 inactive:32760 dirty:0 writeback:0 unstable:0 free:1200 slab:10810 mapped-file:1301 mapped-anon:71747 pagetables:36766
Sep  9 11:44:36 LAMPWord kernel: Node 0 DMA free:2012kB min:56kB low:68kB high:84kB active:0kB inactive:72kB present:10556kB pages_scanned:50094 all_unreclaimable? yes
Sep  9 11:44:36 LAMPWord kernel: lowmem_reserve[]: 0 489 489 489
Sep  9 11:44:36 LAMPWord kernel: Node 0 DMA32 free:2788kB min:2800kB low:3500kB high:4200kB active:143520kB inactive:130968kB present:500896kB pages_scanned:1413555 all_unreclaimable? yes
Sep  9 11:44:36 LAMPWord kernel: lowmem_reserve[]: 0 0 0 0
Sep  9 11:44:36 LAMPWord kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep  9 11:44:36 LAMPWord kernel: lowmem_reserve[]: 0 0 0 0
Sep  9 11:44:36 LAMPWord kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep  9 11:44:36 LAMPWord kernel: lowmem_reserve[]: 0 0 0 0
Sep  9 11:44:36 LAMPWord kernel: Node 0 DMA: 1*4kB 3*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2012kB
Sep  9 11:44:36 LAMPWord kernel: Node 0 DMA32: 23*4kB 23*8kB 11*16kB 1*32kB 2*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2788kB
Sep  9 11:44:36 LAMPWord kernel: Node 0 Normal: empty
Sep  9 11:44:36 LAMPWord kernel: Node 0 HighMem: empty
Sep  9 11:44:36 LAMPWord kernel: 1613 pagecache pages
Sep  9 11:44:36 LAMPWord kernel: Swap cache: add 261612673, delete 261612396, find 152881959/201026151, race 117+2338
Sep  9 11:44:36 LAMPWord kernel: Free swap  = 0kB
Sep  9 11:44:36 LAMPWord kernel: Total swap = 2064376kB
Sep  9 11:44:36 LAMPWord kernel: Free swap:            0kB
Sep  9 11:44:36 LAMPWord kernel: 131072 pages of RAM
Sep  9 11:44:36 LAMPWord kernel: 3490 reserved pages
Sep  9 11:44:36 LAMPWord kernel: 44005 pages shared
Sep  9 11:44:36 LAMPWord kernel: 277 pages swap cached
Sep  9 11:44:36 LAMPWord kernel: audispd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

I can't figure out what it's telling me, and I've never seen anything like this before. Any hints?

Thanks!

AlucardZero · 09-13-2010, 08:32 AM

Code:

Sep  9 11:44:36 LAMPWord kernel:  [<ffffffff800c65e9>] out_of_memory+0x8e/0x2f3
Sep  9 11:44:36 LAMPWord kernel: audispd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

You're completely Out Of Memory. Reduce your memory usage / add more RAM / as a last resort, add more swap.

tommylovell · 09-13-2010, 10:10 AM

You may also have a memory leak. If you have sar installed, you may be able to see something in the sar data prior to the failure. It may show you a gradual increase. As sar is (by default) on a ten minute interval, and if the rise in memory usage was quick, it may show nothing (the last interval may not have been written out due to the oom).

You may have been subject to a DOS attack, or an intrusion attempt.

Did you add any new functionality recently? Has usage changed? If higher memory usage is the new norm, buy some more memory as suggested. Memory is cheap. Adding swap as a safety net will allow you to avoid the oom, or at least delay it, but performance will suffer.

ECRocker · 09-13-2010, 09:11 PM

This blog server gets attacked by the Chinese daily, however I'm more concerned about a rouge web developer / honorary mysql dba :P
It's a virtual guest that does use very little resources, so any increase in memory usage should be very noticeable.

I just couldn't believe what I was seeing in those messages... out of memory messages are always reserved for the windows servers in my datacenter.

Coburn64 · 09-14-2010, 05:38 AM

Find the IP Address range where the Chinese are attacking from and go whammy on them (aka firewall block/drop rule). Maybe also install a firewall to block DoS attacks. My server got hit by a few DoS attacks, it came out undefeated, it routed all the attack packets to /dev/null

ECRocker · 09-15-2010, 02:26 PM

I have both a hardware firewall and a software firewall, plus fail2ban

If you haven't tried it, fail2ban is a cool application that dynamically modifies firewall rules to ban/unban ip address that make repeated failed connections.

Ginola · 09-28-2012, 07:17 AM

I saw this post whilst investigating a very similar problem.

http://linux-mm.org/OOM_Killer

The article tells you how to immunized against the OOM Killer!!

It helped me, so I thought I'd share.