apache2: page allocation failure. order:0, mode:0x20
 

Crashed my server. :(
Linux ubuntu 2.6.15-26-amd64-server
Do you have any idea where should I start? Do you believe I have to tune apache conf or look somewhere else?
Here is the syslog (same thing in the messages) at the time server crashed (they repeated for a couple of times):


Mar 23 23:18:11 server kernel: [310187.230949] apache2: page allocation failure. order:0, mode:0x20
Mar 23 23:18:11 server kernel: [310187.230953]
Mar 23 23:18:11 server kernel: [310187.230954] Call Trace: <IRQ> <ffffffff8016e281>{__alloc_pages+721} <ffffffff88194229>{:e1000:e1000_alloc_rx_buffers_ps+233}
Mar 23 23:18:11 server kernel: [310187.230981] <ffffffff88199e09>{:e1000:e1000_clean_rx_irq_ps+1753}
Mar 23 23:18:14 server kernel: [310187.230991] <ffffffff88199637>{:e1000:e1000_intr+167} <ffffffff80166b3c>{handle_IRQ_event+44}
Mar 23 23:18:21 server kernel: [310187.231005] <ffffffff80166c22>{__do_IRQ+178} <ffffffff80112e7f>{do_IRQ+47}
Mar 23 23:18:21 server kernel: [310187.231013] <ffffffff8011031c>{ret_from_intr+0} <EOI> <ffffffff80204ff8>{deadline_set_request+56}
Mar 23 23:18:21 server kernel: [310187.231019] <ffffffff801ff4b7>{get_request+455} <ffffffff8030e459>{.text.lock.spinlock+0}
Mar 23 23:18:21 server kernel: [310187.231027] <ffffffff80181b4d>{page_lock_anon_vma+29} <ffffffff8018257d>{page_referenced+93}
Mar 23 23:18:21 server kernel: [310187.231034] <ffffffff80175af5>{shrink_zone+565} <ffffffff80192fde>{__find_get_block+382}
Mar 23 23:18:21 server kernel: [310187.231043] <ffffffff881201c7>{:jbd:do_get_write_access+1335} <ffffffff8019302f>{__getblk+47}
Mar 23 23:18:21 server kernel: [310187.231056] <ffffffff88137bdf>{:ext3:ext3_getblk+223} <ffffffff80191c49>{ll_rw_block+169}
Mar 23 23:18:28 server kernel: [310187.231072] <ffffffff80176f35>{try_to_free_pages+357} <ffffffff80296933>{release_sock+19}
Mar 23 23:18:28 server kernel: [310187.231083] <ffffffff8016e17d>{__alloc_pages+461} <ffffffff8016fa4a>{__do_page_cache_readahead+218}
Mar 23 23:18:28 server kernel: [310187.231091] <ffffffff801a7583>{dput+35} <ffffffff8019e5ae>{__link_path_walk+3454}
Mar 23 23:18:28 server kernel: [310187.231100] <ffffffff801542e0>{autoremove_wake_function+0} <ffffffff8016fefd>{blockable_page_cache_readahead+109}
Mar 23 23:18:28 server kernel: [310187.231108] <ffffffff8017013b>{page_cache_readahead+299} <ffffffff801699c3>{do_generic_mapping_read+291}
Mar 23 23:18:28 server kernel: [310187.231115] <ffffffff80167d00>{file_send_actor+0} <ffffffff80169cb7>{generic_file_sendfile+71}
Mar 23 23:18:28 server kernel: [310187.231125] <ffffffff8018ee98>{do_sendfile+488} <ffffffff8018efcc>{sys_sendfile64+92}
Mar 23 23:18:28 server kernel: [310187.231133] <ffffffff8010fd7a>{system_call+126}
Mar 23 23:18:28 server kernel: [310187.231141] Mem-info:
Mar 23 23:18:28 server kernel: [310187.231142] Node 0 DMA per-cpu:
Mar 23 23:18:28 server kernel: [310187.231145] cpu 0 hot: low 0, high 0, batch 1 used:0
Mar 23 23:18:28 server kernel: [310187.231147] cpu 0 cold: low 0, high 0, batch 1 used:0
Mar 23 23:18:28 server kernel: [310187.231149] cpu 1 hot: low 0, high 0, batch 1 used:0
Mar 23 23:18:32 server kernel: [310187.231151] cpu 1 cold: low 0, high 0, batch 1 used:0
Mar 23 23:18:32 server kernel: [310187.231153] cpu 2 hot: low 0, high 0, batch 1 used:0
Mar 23 23:18:32 server kernel: [310187.231155] cpu 2 cold: low 0, high 0, batch 1 used:0
Mar 23 23:18:32 server kernel: [310187.231157] cpu 3 hot: low 0, high 0, batch 1 used:0
Mar 23 23:18:32 server kernel: [310187.231159] cpu 3 cold: low 0, high 0, batch 1 used:0
Mar 23 23:18:32 server kernel: [310187.231161] Node 0 DMA32 per-cpu:
Mar 23 23:18:32 server kernel: [310187.231163] cpu 0 hot: low 0, high 186, batch 31 used:32
Mar 23 23:18:32 server kernel: [310187.231165] cpu 0 cold: low 0, high 62, batch 15 used:56
Mar 23 23:18:32 server kernel: [310187.231168] cpu 1 hot: low 0, high 186, batch 31 used:39
Mar 23 23:18:32 server kernel: [310187.231170] cpu 1 cold: low 0, high 62, batch 15 used:53
Mar 23 23:18:32 server kernel: [310187.231172] cpu 2 hot: low 0, high 186, batch 31 used:24
Mar 23 23:18:32 server kernel: [310187.231174] cpu 2 cold: low 0, high 62, batch 15 used:31
Mar 23 23:18:32 server kernel: [310187.231176] cpu 3 hot: low 0, high 186, batch 31 used:33
Mar 23 23:18:32 server kernel: [310187.231178] cpu 3 cold: low 0, high 62, batch 15 used:38
Mar 23 23:18:32 server kernel: [310187.231180] Node 0 Normal per-cpu:
Mar 23 23:18:32 server kernel: [310187.231182] cpu 0 hot: low 0, high 186, batch 31 used:30
Mar 23 23:18:32 server kernel: [310187.231184] cpu 0 cold: low 0, high 62, batch 15 used:20
Mar 23 23:18:32 server kernel: [310187.231186] cpu 1 hot: low 0, high 186, batch 31 used:20
Mar 23 23:18:32 server kernel: [310187.231188] cpu 1 cold: low 0, high 62, batch 15 used:18
Mar 23 23:18:32 server kernel: [310187.231191] cpu 2 hot: low 0, high 186, batch 31 used:129
Mar 23 23:18:32 server kernel: [310187.231193] cpu 2 cold: low 0, high 62, batch 15 used:14
Mar 23 23:18:32 server kernel: [310187.231195] cpu 3 hot: low 0, high 186, batch 31 used:134
Mar 23 23:18:32 server kernel: [310187.231197] cpu 3 cold: low 0, high 62, batch 15 used:35
Mar 23 23:18:32 server kernel: [310187.231199] Node 0 HighMem per-cpu: empty
Mar 23 23:18:32 server kernel: [310187.231202] Free pages: 17360kB (0kB HighMem)
Mar 23 23:18:32 server kernel: [310187.231206] Active:651852 inactive:319736 dirty:1391 writeback:17 unstable:0 free:4340 slab:16915 mapped:908715 pagetables:11576
Mar 23 23:18:32 server kernel: [310187.231208] Node 0 DMA free:12348kB min:20kB low:8kB high:20kB active:0kB inactive:0kB present:11988kB pages_scanned:57906 all_unreclaimable? yes
Mar 23 23:18:32 server kernel: [310187.231213] lowmem_reserve[]: 0 3508 4013 4013
Mar 23 23:18:32 server kernel: [310187.231216] Node 0 DMA32 free:4644kB min:7076kB low:3536kB high:7076kB active:2267916kB inactive:1145452kB present:3593072kB pages_scanned:72642 all_unreclaimable? no
Mar 23 23:18:32 server kernel: [310187.231222] lowmem_reserve[]: 0 0 505 505
Mar 23 23:18:32 server kernel: [310187.231224] Node 0 Normal free:368kB min:1016kB low:508kB high:1016kB active:339492kB inactive:133492kB present:517120kB pages_scanned:13002 all_unreclaimable? no
Mar 23 23:18:32 server kernel: [310187.231230] lowmem_reserve[]: 0 0 0 0
Mar 23 23:18:32 server kernel: [310187.231232] Node 0 HighMem free:0kB min:128kB low:32kB high:64kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Mar 23 23:18:32 server kernel: [310187.231236] lowmem_reserve[]: 0 0 0 0
Mar 23 23:18:32 server kernel: [310187.231238] Node 0 DMA: 3*4kB 6*8kB 4*16kB 4*32kB 5*64kB 0*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12348kB
Mar 23 23:18:32 server kernel: [310187.231247] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 4644kB
Mar 23 23:18:32 server kernel: [310187.231255] Node 0 Normal: 0*4kB 0*8kB 11*16kB 2*32kB 0*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 368kB
Mar 23 23:18:32 server kernel: [310187.231262] Node 0 HighMem: empty
Mar 23 23:18:32 server kernel: [310187.231266] Swap cache: add 2436401, delete 2434327, find 49396016/49456873, race 0+249
Mar 23 23:18:32 server kernel: [310187.231268] Free swap = 0kB
Mar 23 23:18:32 server kernel: [310187.231270] Total swap = 3903784kB
Mar 23 23:18:32 server kernel: [310187.231271] Free swap: 0kB
Mar 23 23:18:32 server kernel: [310187.248907] 1179648 pages of RAM
Mar 23 23:18:32 server kernel: [310187.248909] 167197 reserved pages
Mar 23 23:18:32 server kernel: [310187.248910] 362573 pages shared
Mar 23 23:18:32 server kernel: [310187.248912] 2077 pages swap cached
Mar 23 23:18:32 server kernel: [310187.248926] apache2: page allocation failure. order:0, mode:0x20

hi,

i have the same problem...

since 24/03

solution found ?

maybe...
 

I tuned mysql and a client fixed some loop in a script eating all my resources.
I think that is a low memory issue (apache couldn't find any memory available).

k thx,

me, i will change server :)


bye