I'm having issues lately where the machine is getting unresponsive, system keeps reporting "XFS: possible memory allocation deadlock in kmem_alloc (mode:0x2d0)" and I keep getting irq handler error :


Jun 22 03:59:51 kernel: low stack detected by irq handler
Jun 22 03:59:51 kernel: [<c0107a57>] do_IRQ+0x121/0x1f2
Jun 22 03:59:51 kernel: =======================
Jun 22 03:59:51 kernel: [<c02de850>] common_interrupt+0x18/0x20
Jun 22 03:59:51 kernel: [<c02dbccd>] schedule+0x885/0x8f3
Jun 22 03:59:51 kernel: [<c01496d4>] kmem_cache_alloc+0x22/0x57
Jun 22 03:59:51 kernel: [<f8983d59>] kmem_zone_alloc+0x3b/0x7f [xfs]
Jun 22 03:59:51 kernel: [<f898268a>] xfs_buf_get_flags+0x2e/0x110 [xfs]
Jun 22 03:59:51 kernel: [<f898277f>] xfs_buf_read_flags+0x13/0x76 [xfs]
Jun 22 03:59:51 kernel: [<f896fad0>] xfs_trans_read_buf+0x1ba/0x2bf [xfs]
Jun 22 03:59:51 kernel: [<f892c439>] xfs_alloc_read_agfl+0x81/0x9c [xfs]
Jun 22 03:59:51 kernel: [<f892dd3b>] xfs_alloc_fix_freelist+0x2b2/0x338 [xfs]
Jun 22 03:59:51 kernel: [<f892e462>] xfs_alloc_vextent+0x2ec/0x46a [xfs]
Jun 22 03:59:51 kernel: [<f893b556>] xfs_bmap_alloc+0x15db/0x18c2 [xfs]
Jun 22 03:59:51 kernel: [<c0145894>] buffered_rmqueue+0x17d/0x1a5
Jun 22 03:59:51 kernel: [<f893e821>] xfs_bmapi+0x935/0x1390 [xfs]
Jun 22 03:59:51 kernel: [<f893cf00>] xfs_bmap_do_search_extents+0x2bb/0x384 [xfs]
Jun 22 03:59:51 kernel: [<c014528c>] free_pages_bulk+0x1cb/0x1d7
Jun 22 03:59:51 kernel: [<c02dc386>] __cond_resched+0x14/0x39
Jun 22 03:59:51 kernel: [<f8963889>] xlog_grant_push_ail+0xce/0x101 [xfs]
Jun 22 03:59:51 kernel: [<f897a504>] xfs_iomap_write_allocate+0x2a6/0x455 [xfs]
Jun 22 03:59:51 kernel: [<c022adf1>] __generic_unplug_device+0x2b/0x2d
Jun 22 03:59:51 kernel: [<c022c3b6>] __make_request+0x452/0x46c
Jun 22 03:59:51 kernel: [<f8979602>] xfs_iomap+0x36c/0x40a [xfs]
Jun 22 03:59:51 kernel: [<f898073e>] xfs_bmap+0x1a/0x1e [xfs]
Jun 22 03:59:51 kernel: [<f897ab15>] xfs_map_blocks+0x27/0x54 [xfs]
Jun 22 03:59:51 kernel: [<f897b786>] xfs_page_state_convert+0x2eb/0x552 [xfs]
Jun 22 03:59:51 kernel: [<c014528c>] free_pages_bulk+0x1cb/0x1d7
Jun 22 03:59:51 kernel: [<c011d5c2>] recalc_task_prio+0x128/0x133
Jun 22 03:59:51 kernel: [<f897bed0>] linvfs_writepage+0x91/0xc6 [xfs]
Jun 22 03:59:51 kernel: [<c014b969>] pageout+0x8d/0xcc
Jun 22 03:59:51 kernel: [<c014bbe4>] shrink_list+0x23c/0x422
Jun 22 03:59:51 kernel: [<c02de850>] common_interrupt+0x18/0x20
Jun 22 03:59:51 kernel: [<c014bfa7>] shrink_cache+0x1dd/0x34d
Jun 22 03:59:51 kernel: [<c014c71c>] shrink_zone+0x101/0x116
Jun 22 03:59:51 kernel: [<c014c78c>] shrink_caches+0x5b/0x66
Jun 22 03:59:51 kernel: [<c014c8ff>] try_to_free_pages+0xef/0x1ef
Jun 22 03:59:51 kernel: [<c0145a83>] __alloc_pages+0x1c7/0x2b1
Jun 22 03:59:51 kernel: [<c0145b85>] __get_free_pages+0x18/0x24
Jun 22 03:59:51 kernel: [<c01487c8>] kmem_getpages+0x1c/0xbb
Jun 22 03:59:51 kernel: [<c0149316>] cache_grow+0xab/0x138
Jun 22 03:59:51 kernel: [<c0149508>] cache_alloc_refill+0x165/0x19d
Jun 22 03:59:51 kernel: [<c01498dc>] __kmalloc+0x76/0x88
Jun 22 03:59:51 kernel: [<f8983c04>] kmem_alloc+0x50/0xad [xfs]
Jun 22 03:59:51 kernel: [<f895ddb9>] xfs_iread_extents+0x73/0xc4 [xfs]
Jun 22 03:59:51 kernel: [<f893e1ee>] xfs_bmapi+0x302/0x1390 [xfs]
Jun 22 03:59:51 kernel: [<c0120f01>] autoremove_wake_function+0x0/0x2d
Jun 22 03:59:51 kernel: [<c022bc51>] part_round_stats+0x48/0x63
Jun 22 03:59:51 kernel: [<c01448ba>] mempool_alloc+0x7b/0x135
Jun 22 03:59:51 kernel: [<c0120f01>] autoremove_wake_function+0x0/0x2d
Jun 22 03:59:51 last message repeated 2 times
Jun 22 03:59:51 kernel: [<c022bc51>] part_round_stats+0x48/0x63
Jun 22 03:59:51 kernel: [<c0229625>] __elv_add_request+0x4f/0x87
Jun 22 03:59:51 kernel: [<c022c385>] __make_request+0x421/0x46c
Jun 22 03:59:51 kernel: [<c022c3b6>] __make_request+0x452/0x46c
Jun 22 03:59:51 kernel: [<f893cfbd>] xfs_bmap_do_search_extents+0x378/0x384 [xfs]
Jun 22 03:59:51 kernel: [<c0142142>] find_lock_page+0x96/0x9d
Jun 22 03:59:51 kernel: [<c0142161>] find_or_create_page+0x18/0x72
Jun 22 03:59:51 kernel: [<f89794d8>] xfs_iomap+0x242/0x40a [xfs]
Jun 22 03:59:51 kernel: [<c0141e15>] wake_up_page+0x9/0x29
Jun 22 03:59:51 kernel: [<f8982424>] _pagebuf_lookup_pages+0x23d/0x26a [xfs]
Jun 22 03:59:51 kernel: [<f898254c>] _pagebuf_find+0x8a/0x19a [xfs]
Jun 22 03:59:51 kernel: [<f898073e>] xfs_bmap+0x1a/0x1e [xfs]
Jun 22 03:59:51 kernel: [<f897ba89>] __linvfs_get_block+0x9c/0x290 [xfs]
Jun 22 03:59:51 kernel: [<c02dc386>] __cond_resched+0x14/0x39
Jun 22 03:59:51 kernel: [<c02dc386>] __cond_resched+0x14/0x39
Jun 22 03:59:51 kernel: [<c015f9d5>] set_bh_page+0x2c/0x34
Jun 22 03:59:51 kernel: [<f897bc90>] linvfs_get_block+0x13/0x17 [xfs]
Jun 22 03:59:51 kernel: [<c0160024>] __block_prepare_write+0x165/0x40b
Jun 22 03:59:51 kernel: [<c016096b>] block_prepare_write+0x16/0x23
Jun 22 03:59:51 kernel: [<f897bc7d>] linvfs_get_block+0x0/0x17 [xfs]
Jun 22 03:59:51 kernel: [<f897bfa8>] linvfs_prepare_write+0x12/0x16 [xfs]
Jun 22 03:59:51 kernel: [<f897bc7d>] linvfs_get_block+0x0/0x17 [xfs]
Jun 22 03:59:51 kernel: [<c01439d2>] generic_file_buffered_write+0x1e9/0x501
Jun 22 03:59:51 kernel: [<c0127674>] current_fs_time+0x44/0x4c
Jun 22 03:59:51 kernel: [<f89802f2>] xfs_write+0x700/0xaee [xfs]
Jun 22 03:59:51 kernel: [<c01721a4>] dput+0x34/0x1a7
Jun 22 03:59:51 kernel: [<c016a0de>] __link_path_walk+0xb6d/0xc25
Jun 22 03:59:51 kernel: [<c02dc386>] __cond_resched+0x14/0x39
Jun 22 03:59:51 kernel: [<f897ca00>] linvfs_write+0xcc/0xe5 [xfs]
Jun 22 03:59:51 kernel: [<c01713bc>] fcntl_setlk+0x2f8/0x302
Jun 22 03:59:51 kernel: [<c0120f01>] autoremove_wake_function+0x0/0x2d
Jun 22 03:59:51 kernel: [<c02dc386>] __cond_resched+0x14/0x39
Jun 22 03:59:51 kernel: [<c015cf78>] generic_file_llseek+0x30/0xcb
Jun 22 03:59:51 kernel: [<c015cf78>] generic_file_llseek+0x30/0xcb
Jun 22 03:59:51 kernel: [<c015d673>] vfs_write+0xb6/0xe2
Jun 22 03:59:51 kernel: [<c015d73d>] sys_write+0x3c/0x62
Jun 22 03:59:51 kernel: [<c02dde77>] syscall_call+0x7/0xb
Jun 22 03:59:51 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x2d0)

Any ideas are appreciated.

A "low stack detected by irq handler" error is usually a driver bug; the driver simply uses way too much stack space (for some reason), and this is detected by the kernel.

I see you're using XFS. How big is your file system and how much memory do you have? Is your system under heavy I/O load (disk and/or network)?

Edit: Oh, and which kernel version and architecture are you using?

1 members found this post helpful.

Ser Olmy - thanks for your reply. The filesystem is about 250 gb. the server has 4gb of RAM and kernel is 2.6.9.89. I updated the kernel to 2.6.9-103 and I'm no longer getting the irq handler error but I'm still getting the XFS deadlock from XFS and yes, the server is under heavy I/O load during this time since we're running a full metal bare back up.

Is there any way I can pin point what causing the deadlock and so I can correct it?