Hi all

I have a problem, the first thing I see is the kernel error in the
subject, looking deeper into the logs I find a whole load of errors.

Machine details, anything else I can give?

root@quota log]# uname -a
Linux quota.unseen.org 2.6.9-11.EL #1 Wed Jun 8 16:59:52 CDT 2005 i686
i686 i386 GNU/Linux

[root@quota etc]# cat /etc/redhat-release
CentOS release 4.1 (Final)

A full dmesg with all the boot info is at www unseen org/~kevin/tmp/dmesg

The ext3_orphan_cleanup in the above dmesg is due to the last time this
error happened.

A few log lines before the issue:

application vmware-vmx uses obsolete OSS audio interface
/dev/vmnet: open called by PID 20000 (vmware-vmx)
device eth0 entered promiscuous mode
bridge-eth0: enabled promiscuous mode
/dev/vmnet: port on hub 0 successfully opened
/dev/vmmon[20008]: host clock rate change request 0 -> 19
/dev/vmmon[20008]: host clock rate change request 19 -> 83
vortex: IRQ fifo error

I'm pretty sure the "vortex: IRQ fifo error" isnt an issue, that occours a
few random times in the log.

The Vmware thing may be related, the virtual disk is on hdb1 (see error
below), and it does vaguly seem to be related to me starting vmware but
not consistantly enough to say at the mo.

kswapd0: page allocation failure. order:0, mode:0x850
[<c0146dff>] __alloc_pages+0x28b/0x29d
[<c0146e29>] __get_free_pages+0x18/0x24
[<c014a2b6>] kmem_getpages+0x15/0x94
[<c014af73>] cache_grow+0x107/0x233
[<c014b296>] cache_alloc_refill+0x1f7/0x227
[<c014b7cb>] __kmalloc+0x6b/0x7d
[<f0852520>] __jbd_kmalloc+0x16/0x17 [jbd]
[<f0849833>] journal_get_undo_access+0x58/0x122 [jbd]
[<f087ed09>] ext3_try_to_allocate_with_rsv+0x40/0x35c [ext3]
[<c0164786>] bh_wake_function+0x0/0x29
[<f087f30d>] ext3_new_block+0x260/0x581 [ext3]
[<c01455fe>] mempool_alloc+0x7b/0x1f9
[<f08814e8>] ext3_alloc_block+0x9/0xb [ext3]
[<f08817fa>] ext3_alloc_branch+0x4a/0x25a [ext3]
[<f0881d31>] ext3_get_block_handle+0x1b7/0x276 [ext3]
[<f0881e54>] ext3_get_block+0x64/0x6c [ext3]
[<c0166a0d>] __block_write_full_page+0xd8/0x2ae
[<f0881df0>] ext3_get_block+0x0/0x6c [ext3]
[<c0167de3>] block_write_full_page+0xa4/0xad
[<f0881df0>] ext3_get_block+0x0/0x6c [ext3]
[<f088275b>] ext3_ordered_writepage+0xce/0x13a [ext3]
[<f088266f>] bget_one+0x0/0x6 [ext3]
[<c014d884>] pageout+0x88/0xc5
[<c014daca>] shrink_list+0x209/0x4ea
[<c0303838>] common_interrupt+0x18/0x20
[<c014cb62>] __pagevec_release+0x15/0x1d
[<c014dfaa>] shrink_cache+0x1ff/0x454
[<c014e964>] shrink_zone+0x8f/0x9e
[<c014ecf8>] balance_pgdat+0x188/0x2b5
[<c014eede>] kswapd+0xb9/0xbb
[<c011debc>] autoremove_wake_function+0x0/0x2d
[<c0303612>] ret_from_fork+0x6/0x14
[<c011debc>] autoremove_wake_function+0x0/0x2d
[<c014ee25>] kswapd+0x0/0xbb
[<c01041d9>] kernel_thread_helper+0x5/0xb

Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu: empty

Free pages: 0kB (0kB HighMem)
Active:129492 inactive:26729 dirty:108 writeback:0 unstable:0 free:0
slab:6346 mapped:137376 pa getables:1094
DMA free:0kB min:16kB low:32kB high:48kB active:1212kB inactive:8228kB
present:16384kB pages_sc anned:99 all_unreclaimable? no
protections[]: 0 0 0
Normal free:0kB min:864kB low:1728kB high:2592kB active:516756kB
inactive:98688kB present:77004 8kB pages_scanned:198 all_unreclaimable? no
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB
present:0kB pages_scann ed:0 all_unreclaimable? no
protections[]: 0 0 0
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 0kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 0 kB

HighMem: empty
Swap cache: add 17737, delete 16749, find 656/979, race 0+0
Free swap: 2037900kB
196608 pages of RAM
0 pages of HIGHMEM
2939 reserved pages
230767 pages shared
988 pages swap cached
journal_get_undo_access: No memory for committed data
ext3_try_to_allocate_with_rsv: aborting transaction: Out of memory in
__ext3_journal_get_undo_a ccess
EXT3-fs error (device hdb1) in ext3_new_block: Out of memory
Aborting journal on device hdb1.
ext3_abort called.
EXT3-fs error (device hdb1): ext3_journal_start_sb: Detected aborted
journal
Remounting filesystem read-only
EXT3-fs error (device hdb1) in ext3_ordered_writepage: Out of memory
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
EXT3-fs error (device hdb1) in start_transaction: Journal has aborted
[snip]

Any ideas? Any ways to get more info?

Cheers
Kev

I'm getting the exact same error. I'm running VmWare 5.0 on Redhat AS4 on an EXT3 filesystem. The system seems to run fine if I'm only running one guest OS. Once I get two or three gues OS's going, it doesn't seem to last more than 24 hours before I get a stack trace exactly like the one you posted. The very first indication in the logfile that this is happening is:

Jul 28 04:41:04 avalanche kernel: kswapd0: page allocation failure. order:0, mode:0x50
Jul 28 04:41:04 avalanche kernel: [<c013f1ab>] __alloc_pages+0x28b/0x298
Jul 28 04:41:04 avalanche kernel: [<c013baa2>] find_lock_page+0x96/0x9d
Jul 28 04:41:04 avalanche kernel: [<c013bae2>] find_or_create_page+0x39/0x72
Jul 28 04:41:04 avalanche kernel: [<c015749b>] grow_dev_page+0x26/0x102
Jul 28 04:41:04 avalanche kernel: [<c0157646>] __getblk_slow+0xcf/0xf3
Jul 28 04:41:04 avalanche kernel: [<c015793d>] __getblk+0x3f/0x49
Jul 28 04:41:04 avalanche kernel: [<c0157982>] __bread+0x9/0x1e
Jul 28 04:41:04 avalanche kernel: [<f88660ae>] read_block_bitmap+0x29/0x4d [ext3]
Jul 28 04:41:04 avalanche kernel: [<f886706e>] ext3_new_block+0x232/0x49e [ext3]
Jul 28 04:41:04 avalanche kernel: [<f8868fb4>] ext3_alloc_block+0x9/0xb [ext3]
Jul 28 04:41:04 avalanche kernel: [<f88692c6>] ext3_alloc_branch+0x4a/0x25c [ext3]
Jul 28 04:41:04 avalanche kernel: [<f8869804>] ext3_get_block_handle+0x1bc/0x27d [ext3]
Jul 28 04:41:05 avalanche kernel: [<f8869929>] ext3_get_block+0x64/0x6c [ext3]
Jul 28 04:41:05 avalanche kernel: [<c0157d1c>] __block_write_full_page+0x107/0x2ea
Jul 28 04:41:06 avalanche kernel: [<f88698c5>] ext3_get_block+0x0/0x6c [ext3]
Jul 28 04:41:06 avalanche kernel: [<c015916c>] block_write_full_page+0xc5/0xce
Jul 28 04:41:06 avalanche kernel: [<f88698c5>] ext3_get_block+0x0/0x6c [ext3]
Jul 28 04:41:06 avalanche kernel: [<f886a23b>] ext3_ordered_writepage+0xce/0x13a [ext3]
Jul 28 04:41:06 avalanche kernel: [<f886a14d>] bget_one+0x0/0x7 [ext3]
Jul 28 04:41:06 avalanche kernel: [<c01446d5>] pageout+0x8d/0xcc
Jul 28 04:41:06 avalanche kernel: [<c014491c>] shrink_list+0x208/0x3ee
Jul 28 04:41:06 avalanche kernel: [<c0143bc0>] __pagevec_release+0x15/0x1d
Jul 28 04:41:07 avalanche kernel: [<c0144cdf>] shrink_cache+0x1dd/0x34d
Jul 28 04:41:07 avalanche kernel: [<c014539d>] shrink_zone+0xa7/0xb6
Jul 28 04:41:07 avalanche kernel: [<c0145740>] balance_pgdat+0x1b6/0x2f8
Jul 28 04:41:07 avalanche kernel: [<c011e7fd>] prepare_to_wait+0x12/0x4c
Jul 28 04:41:07 avalanche kernel: [<c014594c>] kswapd+0xca/0xcc
Jul 28 04:41:07 avalanche kernel: [<c011e8d2>] autoremove_wake_function+0x0/0x2d
Jul 28 04:41:07 avalanche kernel: [<c02c5fca>] ret_from_fork+0x6/0x14
Jul 28 04:41:07 avalanche kernel: [<c011e8d2>] autoremove_wake_function+0x0/0x2d
Jul 28 04:41:07 avalanche kernel: [<c0145882>] kswapd+0x0/0xcc
Jul 28 04:41:07 avalanche kernel: [<c01041f1>] kernel_thread_helper+0x5/0xb

Which is immediately followed by:

Jul 28 04:41:07 avalanche kernel: journal_get_undo_access: No memory for committed data
Jul 28 04:41:07 avalanche kernel: ext3_try_to_allocate_with_rsv: aborting transaction: Out of memory in __ext3_journal_get_undo_access
Jul 28 04:41:07 avalanche kernel: EXT3-fs error (device hda7) in ext3_new_block: Out of memory
Jul 28 04:41:07 avalanche kernel: Aborting journal on device hda7.
Jul 28 04:41:07 avalanche kernel: ext3_abort called.
Jul 28 04:41:07 avalanche kernel: EXT3-fs error (device hda7): ext3_journal_start_sb: Detected aborted journal
Jul 28 04:41:07 avalanche kernel: Remounting filesystem read-only

The machine has 1G RAM and 2G swap, both of which are plenty free when this happens (over 600G of used memory is file cache, and swap is only using 8M or so). Note that once it flips the filesystem into read-only above, the only fix is to reboot (attempts to remount rw fail with disk write-protected error).

I've run up to 5 simultaneous VM's on a similarly configured FC3 machine and never encountered this issue.

Hi,

I was having this same problem and I came across this conversation on VMWare website:

http://www.vmware.com/community/thre...ssageID=249823

-----
1) Recommended. Edit /etc/sysctl.conf and set "vm.min_free_kbytes = <nnnn>"
2) "echo <nnnn> > /proc/sys/vm/min_free_kbytes" in /etc/rc.local or some other place in the boot process.
-----

<nnnn> should be 5120 or 10240 (or higher).

I have not had time to verify this yet, but it seemed to work for the problems they were having.

Br,
Sami

Hi

Brilliant, thanks for pointing that out, I made the change above and I have now had a win2k3 instance running in vmware and it has been stable for a week compared to less than 48 hours previously.

Cheers
Kev