[SOLVED] malloc-64 memory leak/trashing

Sheridan · 03-02-2011, 05:39 PM

Hi dear fellows,

I'm afraid I have a huge issue with my newest Fedora 14 server. I recently migrated to Fedora 14 from Centos 5, which was very stable, but had ancient packages and libraries and my users were revolting...

The machine is a HP ProLiant 360 G7, with 12G RAM and 6 SAS drives in RAID 5.

After I migrated to Fedora 14, I noticed that for some reason, during the course of about 24 hours, all usable RAM "disappears" and applications are forced down to swap space. Needless to say I didn't have this issue on CentOS. The server does heavy IO as per it's function (it's a heavily loaded file processing server and user simulation computing station among other things, which causes lots of random IO), so I thought it may be the cache, but then I realized it cannot be - because obviously Linux will use onyl "unused" RAM for caching and frees it up as soon as an app need it.

Then, I thought to check the "slabtop" to see what's going on in Kernel memory. Unfortunately I don't have the screenshot from the time just before the latest crash, but there's a certain value displayed by slabtop, which slowly, byte-from-byte creeps over all available RAM, eventually forcing applications down to the swap. This is malloc-64, and as you can see from the bellow copy-paste, it's building up again even now...

Code:

Active / Total Objects (% used)    : 9118075 / 9153600 (99.6%)
 Active / Total Slabs (% used)      : 152157 / 152157 (100.0%)
 Active / Total Caches (% used)     : 75 / 94 (79.8%)
 Active / Total Size (% used)       : 704083.01K / 718307.90K (98.0%)
 Minimum / Average / Maximum Object : 0.01K / 0.08K / 11.19K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
8176704 8176427  99%    0.06K 127761       64    511044K kmalloc-64
554814 553957  99%    0.10K  14226       39     56904K buffer_head
 82824  79807  96%    0.55K   2856       29     45696K radix_tree_node
 73836  66062  89%    0.19K   1758       42     14064K dentry
 23940  19264  80%    0.19K    570       42      4560K kmalloc-192
 22016  21908  99%    0.01K     43      512       172K kmalloc-8
 19737  19737 100%    0.08K    387       51      1548K sysfs_dir_cache
......

By this time tomorrow, all 12GB RAM will just about be eaten up by kmalloc-64, and even a simple SSH login won't be possible.

Nothing I tried to force out reclaimin of this memory was successful. Among other things, I tried to reload X, change runlevels, restarted all kinds of processes... and I went as far as trying ugly things like:

echo 1 > zone_reclaim_mode
echo 150 > vfs_cache_pressure
sync; echo 3 > drop_caches

... to no avail whatsoever. Nothing I did had any effect on this thing short of a clean reboot.

I thought that maybe some user app they are running on the box maybe causing this... so I denied access for everyone for one day, and allowed only the regular file processing load to go on. The problem persisted, even the most basic system daemons were forced into swap eventually, then the box was dead.

FYI my kernel := "Linux gepard.unixhosting.local 2.6.35.11-83.fc14.x86_64 #1 SMP Mon Feb 7 07:06:44 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux"

Obviously I need some help on either: 1) how to force-reclaim this space periodically, or better still, 2) find out what's causing this if this is my fault and get this over with...

Please, anyone who has a clue on what could be going on, help me. I've exhausted my Google options and I'm completely stuck.

Sheridan · 03-03-2011, 03:59 PM

It's getting worse... Please look bellow:

Code:

for slab in /sys/kernel/slab/* ; do echo $(cat $slab/objects) in $slab ; done|sort -n

0 in /sys/kernel/slab/pid_namespace
0 in /sys/kernel/slab/posix_timers_cache
0 in /sys/kernel/slab/:t-0000176
0 in /sys/kernel/slab/:t-0000248
0 in /sys/kernel/slab/:t-0000312
0 in /sys/kernel/slab/:t-0000368
0 in /sys/kernel/slab/:t-0000424
0 in /sys/kernel/slab/:t-0001064
0 in /sys/kernel/slab/:t-0001152
0 in /sys/kernel/slab/:t-0002112
0 in /sys/kernel/slab/:t-0002176
0 in /sys/kernel/slab/:t-0002240
0 in /sys/kernel/slab/:t-0002608
0 in /sys/kernel/slab/:t-0011456
0 in /sys/kernel/slab/tw_sock_TCPv6
0 in /sys/kernel/slab/UDP-Lite
0 in /sys/kernel/slab/UDPLITEv6
30 N0=30 in /sys/kernel/slab/numa_policy
30 N0=30 in /sys/kernel/slab/:t-0000136
36 N0=36 in /sys/kernel/slab/mqueue_inode_cache
39 N1=39 in /sys/kernel/slab/rpc_inode_cache
52 N0=39 N1=13 in /sys/kernel/slab/blkdev_queue
52 N0=39 N1=13 in /sys/kernel/slab/:t-0002376
53 N1=53 in /sys/kernel/slab/hugetlbfs_inode_cache
84 N0=42 N1=42 in /sys/kernel/slab/fuse_inode
92 N1=92 in /sys/kernel/slab/:at-0000088
92 N1=92 in /sys/kernel/slab/ext3_xattr
92 N1=92 in /sys/kernel/slab/ext4_xattr
105 N0=7 N1=98 in /sys/kernel/slab/bip-256
105 N0=7 N1=98 in /sys/kernel/slab/:t-0004224
138 N0=55 N1=83 in /sys/kernel/slab/kmalloc-8192
138 N0=55 N1=83 in /sys/kernel/slab/:t-0008192
256 N1=256 in /sys/kernel/slab/:at-0000016
256 N1=256 in /sys/kernel/slab/jbd2_revoke_table
256 N1=256 in /sys/kernel/slab/revoke_table
333 N0=333 in /sys/kernel/slab/tw_sock_TCP
340 N0=187 N1=153 in /sys/kernel/slab/TCPv6
350 N0=94 N1=256 in /sys/kernel/slab/kmem_cache_node
408 N0=306 N1=102 in /sys/kernel/slab/fuse_request
408 N0=306 N1=102 in /sys/kernel/slab/:t-0000632
419 N0=173 N1=246 in /sys/kernel/slab/TCP
480 N0=288 N1=192 in /sys/kernel/slab/UDPv6
511 N0=292 N1=219 in /sys/kernel/slab/kvm_pte_chain
511 N0=292 N1=219 in /sys/kernel/slab/:t-0000056
511 N0=292 N1=219 in /sys/kernel/slab/uhci_urb_priv
511 N0=292 N1=219 in /sys/kernel/slab/utrace_engine
561 N0=297 N1=264 in /sys/kernel/slab/:t-0000488
561 N0=297 N1=264 in /sys/kernel/slab/xfs_da_state
672 N0=336 N1=336 in /sys/kernel/slab/:at-0000144
672 N0=336 N1=336 in /sys/kernel/slab/ext4_alloc_context
716 N0=461 N1=255 in /sys/kernel/slab/ip6_dst_cache
716 N0=461 N1=255 in /sys/kernel/slab/ndisc_cache
716 N0=461 N1=255 in /sys/kernel/slab/nfsd4_delegations
716 N0=461 N1=255 in /sys/kernel/slab/:t-0000320
735 N0=420 N1=315 in /sys/kernel/slab/cfq_queue
735 N0=420 N1=315 in /sys/kernel/slab/:t-0000232
741 N0=390 N1=351 in /sys/kernel/slab/bdev_cache
760 N0=440 N1=320 in /sys/kernel/slab/:t-0000400
760 N0=440 N1=320 in /sys/kernel/slab/xfs_efd_item
779 N0=492 N1=287 in /sys/kernel/slab/:t-0000392
779 N0=492 N1=287 in /sys/kernel/slab/xfs_efi_item
796 N0=439 N1=357 in /sys/kernel/slab/biovec-256
796 N0=439 N1=357 in /sys/kernel/slab/kmalloc-4096
796 N0=439 N1=357 in /sys/kernel/slab/names_cache
796 N0=439 N1=357 in /sys/kernel/slab/sgpool-128
796 N0=439 N1=357 in /sys/kernel/slab/:t-0004096
828 N0=720 N1=108 in /sys/kernel/slab/:t-0000224
828 N0=720 N1=108 in /sys/kernel/slab/xfs_buf_item
864 N0=468 N1=396 in /sys/kernel/slab/:at-0000112
864 N0=468 N1=396 in /sys/kernel/slab/jbd2_journal_head
864 N0=468 N1=396 in /sys/kernel/slab/journal_head
879 N0=700 N1=179 in /sys/kernel/slab/task_struct
879 N0=701 N1=178 in /sys/kernel/slab/:t-0005952
897 N0=429 N1=468 in /sys/kernel/slab/UDP
901 N0=454 N1=447 in /sys/kernel/slab/mm_struct
901 N0=454 N1=447 in /sys/kernel/slab/:t-0000896
936 N0=546 N1=390 in /sys/kernel/slab/fsnotify_event
936 N0=546 N1=390 in /sys/kernel/slab/:t-0000104
936 N0=546 N1=390 in /sys/kernel/slab/:t-0000208
936 N0=546 N1=390 in /sys/kernel/slab/xfs_btree_cur
944 N0=742 N1=202 in /sys/kernel/slab/sighand_cache
960 N0=600 N1=360 in /sys/kernel/slab/:t-0000200
960 N0=600 N1=360 in /sys/kernel/slab/xfs_log_ticket
968 N0=440 N1=528 in /sys/kernel/slab/file_lock_cache
1092 N0=763 N1=329 in /sys/kernel/slab/blkdev_requests
1092 N0=763 N1=329 in /sys/kernel/slab/:t-0000344
1127 N0=539 N1=588 in /sys/kernel/slab/:t-0000328
1127 N0=539 N1=588 in /sys/kernel/slab/taskstats
1139 N0=587 N1=552 in /sys/kernel/slab/files_cache
1139 N0=587 N1=552 in /sys/kernel/slab/:t-0000704
1199 N0=732 N1=467 in /sys/kernel/slab/idr_layer_cache
1199 N0=732 N1=467 in /sys/kernel/slab/:t-0000544
1199 N0=732 N1=467 in /sys/kernel/slab/xfs_trans
1224 N0=612 N1=612 in /sys/kernel/slab/kvm_mmu_page_header
1224 N0=612 N1=612 in /sys/kernel/slab/sigqueue
1224 N0=612 N1=612 in /sys/kernel/slab/:t-0000160
1301 N0=693 N1=608 in /sys/kernel/slab/RAW
1301 N0=693 N1=608 in /sys/kernel/slab/:t-0000832
1301 N0=693 N1=608 in /sys/kernel/slab/UNIX
1404 N0=925 N1=479 in /sys/kernel/slab/:t-0000216
1404 N0=925 N1=479 in /sys/kernel/slab/xfs_ili
1680 N0=1103 N1=577 in /sys/kernel/slab/proc_inode_cache
1695 N0=596 N1=1099 in /sys/kernel/slab/shmem_inode_cache
1732 N0=873 N1=859 in /sys/kernel/slab/sock_inode_cache
1742 N0=1159 N1=583 in /sys/kernel/slab/blkdev_integrity
1742 N0=1159 N1=583 in /sys/kernel/slab/dnotify_mark_entry
1742 N0=1159 N1=583 in /sys/kernel/slab/inotify_inode_mark_entry
1742 N0=1159 N1=583 in /sys/kernel/slab/:t-0000112
1742 N0=1159 N1=583 in /sys/kernel/slab/task_delay_info
1742 N0=1159 N1=583 in /sys/kernel/slab/xfs_ioend
1752 N0=949 N1=803 in /sys/kernel/slab/:at-0000056
1752 N0=949 N1=803 in /sys/kernel/slab/ext4_free_block_extents
2038 N0=1072 N1=966 in /sys/kernel/slab/nf_conntrack_ffffffff81e3fc60
2048 N0=1024 N1=1024 in /sys/kernel/slab/:at-0000032
2048 N0=1024 N1=1024 in /sys/kernel/slab/jbd2_revoke_record
2048 N0=1024 N1=1024 in /sys/kernel/slab/revoke_record
2324 N0=1188 N1=1136 in /sys/kernel/slab/kmalloc-512
2331 N0=1206 N1=1125 in /sys/kernel/slab/xfrm_dst_cache
2336 N0=1208 N1=1128 in /sys/kernel/slab/task_xstate
2338 N0=1209 N1=1129 in /sys/kernel/slab/:t-0000512
2341 N0=1210 N1=1131 in /sys/kernel/slab/skbuff_fclone_cache
2350 N0=1211 N1=1139 in /sys/kernel/slab/sgpool-16
3324 N0=2060 N1=1264 in /sys/kernel/slab/kmalloc-1024
3325 N0=2084 N1=1241 in /sys/kernel/slab/:t-0001024
3326 N0=2062 N1=1264 in /sys/kernel/slab/jbd2_1k
3326 N0=2074 N1=1252 in /sys/kernel/slab/biovec-64
3327 N0=2070 N1=1257 in /sys/kernel/slab/sgpool-32
3330 N0=2078 N1=1252 in /sys/kernel/slab/signal_cache
3331 N0=2074 N1=1257 in /sys/kernel/slab/RAWv6
3978 N0=1428 N1=2550 in /sys/kernel/slab/dm_io
3978 N0=1428 N1=2550 in /sys/kernel/slab/ksm_stable_node
3978 N0=1428 N1=2550 in /sys/kernel/slab/kvm_rmap_desc
3978 N0=1428 N1=2550 in /sys/kernel/slab/:t-0000040
3980 N0=2417 N1=1563 in /sys/kernel/slab/bip-16
3980 N0=2417 N1=1563 in /sys/kernel/slab/dm_rq_target_io
3980 N0=2417 N1=1563 in /sys/kernel/slab/ip_dst_cache
3980 N0=2417 N1=1563 in /sys/kernel/slab/kioctx
3980 N0=2417 N1=1563 in /sys/kernel/slab/:t-0000384
3980 N0=2417 N1=1563 in /sys/kernel/slab/xfs_buf
4080 N0=2040 N1=2040 in /sys/kernel/slab/:at-0000024
4080 N0=2040 N1=2040 in /sys/kernel/slab/jbd2_journal_handle
4080 N0=2040 N1=2040 in /sys/kernel/slab/journal_handle
4273 N0=1245 N1=3028 in /sys/kernel/slab/ext4_inode_cache
5030 N0=3185 N1=1845 in /sys/kernel/slab/flow_cache
5030 N0=3185 N1=1845 in /sys/kernel/slab/kmalloc-96
5030 N0=3185 N1=1845 in /sys/kernel/slab/:t-0000096
5573 N0=1982 N1=3591 in /sys/kernel/slab/biovec-128
5579 N0=1971 N1=3608 in /sys/kernel/slab/sgpool-64
5579 N0=1976 N1=3603 in /sys/kernel/slab/kmalloc-2048
5580 N0=1987 N1=3593 in /sys/kernel/slab/:t-0002048
5582 N0=1973 N1=3609 in /sys/kernel/slab/rpc_buffers
5610 N0=2210 N1=3400 in /sys/kernel/slab/dm_snap_tracked_chunk
5610 N0=2210 N1=3400 in /sys/kernel/slab/dm_target_io
5610 N0=2210 N1=3400 in /sys/kernel/slab/fsnotify_event_holder
5610 N0=2210 N1=3400 in /sys/kernel/slab/fstrm_item
5610 N0=2210 N1=3400 in /sys/kernel/slab/scsi_data_buffer
5610 N0=2210 N1=3400 in /sys/kernel/slab/:t-0000024
5610 N0=2210 N1=3400 in /sys/kernel/slab/xfs_bmap_free_item
5610 N0=2210 N1=3400 in /sys/kernel/slab/xfs_dabuf
6611 N0=3314 N1=3297 in /sys/kernel/slab/Acpi-Namespace
6611 N0=3314 N1=3297 in /sys/kernel/slab/dm_exception
6611 N0=3314 N1=3297 in /sys/kernel/slab/dnotify_struct
6611 N0=3314 N1=3297 in /sys/kernel/slab/inotify_event_private_data
6611 N0=3314 N1=3297 in /sys/kernel/slab/ip_fib_alias
6611 N0=3314 N1=3297 in /sys/kernel/slab/kmalloc-32
6611 N0=3314 N1=3297 in /sys/kernel/slab/sd_ext_cdb
6611 N0=3314 N1=3297 in /sys/kernel/slab/:t-0000032
6611 N0=3314 N1=3297 in /sys/kernel/slab/xfs_mru_cache_elem
7367 N0=3943 N1=3424 in /sys/kernel/slab/:t-0000128
7370 N0=3943 N1=3427 in /sys/kernel/slab/request_sock_TCP
7374 N0=3947 N1=3427 in /sys/kernel/slab/iommu_domain
7374 N0=3947 N1=3427 in /sys/kernel/slab/ip6_mrt_cache
7374 N0=3947 N1=3427 in /sys/kernel/slab/ip_mrt_cache
7374 N0=3947 N1=3427 in /sys/kernel/slab/nfsd4_stateids
7375 N0=3948 N1=3427 in /sys/kernel/slab/kmalloc-128
7376 N0=3946 N1=3430 in /sys/kernel/slab/bip-1
7376 N0=3946 N1=3430 in /sys/kernel/slab/btree_node
7376 N0=3946 N1=3430 in /sys/kernel/slab/cfq_io_context
7376 N0=3946 N1=3430 in /sys/kernel/slab/eventpoll_epi
7376 N0=3949 N1=3427 in /sys/kernel/slab/pid
7377 N0=3946 N1=3431 in /sys/kernel/slab/uid_cache
7902 N0=5115 N1=2787 in /sys/kernel/slab/inode_cache
8817 N0=4665 N1=4152 in /sys/kernel/slab/anon_vma
9373 N0=5010 N1=4363 in /sys/kernel/slab/Acpi-Parse
9394 N0=5010 N1=4384 in /sys/kernel/slab/anon_vma_chain
9523 N0=5093 N1=4430 in /sys/kernel/slab/fasync_cache
9523 N0=5093 N1=4430 in /sys/kernel/slab/ksm_mm_slot
9523 N0=5093 N1=4430 in /sys/kernel/slab/nsproxy
9523 N0=5093 N1=4430 in /sys/kernel/slab/shared_policy_node
9523 N0=5093 N1=4430 in /sys/kernel/slab/:t-0000048
9889 N0=4134 N1=5755 in /sys/kernel/slab/mnt_cache
9892 N0=4144 N1=5748 in /sys/kernel/slab/biovec-16
9892 N0=4147 N1=5745 in /sys/kernel/slab/arp_cache
9898 N0=4142 N1=5756 in /sys/kernel/slab/kiocb
9898 N0=4142 N1=5756 in /sys/kernel/slab/kmalloc-256
9900 N0=4152 N1=5748 in /sys/kernel/slab/sgpool-8
9900 N0=4152 N1=5748 in /sys/kernel/slab/:t-0000256
9901 N0=4152 N1=5749 in /sys/kernel/slab/skbuff_head_cache
9902 N0=4152 N1=5750 in /sys/kernel/slab/rpc_tasks
11173 N0=4870 N1=6303 in /sys/kernel/slab/Acpi-Operand
11173 N0=4870 N1=6303 in /sys/kernel/slab/Acpi-ParseExt
11173 N0=4870 N1=6303 in /sys/kernel/slab/blkdev_ioc
11173 N0=4870 N1=6303 in /sys/kernel/slab/eventpoll_pwq
11173 N0=4870 N1=6303 in /sys/kernel/slab/ip_fib_hash
11173 N0=4870 N1=6303 in /sys/kernel/slab/nfsd4_files
11173 N0=4870 N1=6303 in /sys/kernel/slab/selinux_inode_security
11173 N0=4870 N1=6303 in /sys/kernel/slab/:t-0000072
14482 N0=7837 N1=6645 in /sys/kernel/slab/biovec-1
14482 N0=7837 N1=6645 in /sys/kernel/slab/dm_rq_clone_bio_info
14482 N0=7837 N1=6645 in /sys/kernel/slab/kmalloc-16
14482 N0=7837 N1=6645 in /sys/kernel/slab/:t-0000016
17015 N0=8556 N1=8459 in /sys/kernel/slab/vm_area_struct
17020 N0=8561 N1=8459 in /sys/kernel/slab/:t-0000184
18788 N0=11812 N1=6976 in /sys/kernel/slab/xfs_inode
19625 N0=10766 N1=8859 in /sys/kernel/slab/Acpi-State
19625 N0=10766 N1=8859 in /sys/kernel/slab/dm_snap_pending_exception
19625 N0=10766 N1=8859 in /sys/kernel/slab/sysfs_dir_cache
19625 N0=10766 N1=8859 in /sys/kernel/slab/:t-0000080
22524 N0=12284 N1=10240 in /sys/kernel/slab/kmalloc-8
22524 N0=12284 N1=10240 in /sys/kernel/slab/:t-0000008
31652 N0=15715 N1=15937 in /sys/kernel/slab/:at-0000192
31652 N0=15715 N1=15937 in /sys/kernel/slab/dentry
38110 N0=20794 N1=17316 in /sys/kernel/slab/filp
38117 N0=20772 N1=17345 in /sys/kernel/slab/cred_jar
38131 N0=20809 N1=17322 in /sys/kernel/slab/bio-0
38149 N0=20820 N1=17329 in /sys/kernel/slab/key_jar
38151 N0=20823 N1=17328 in /sys/kernel/slab/:t-0000192
38155 N0=20833 N1=17322 in /sys/kernel/slab/bip-4
38168 N0=20822 N1=17346 in /sys/kernel/slab/request_sock_TCPv6
38192 N0=20846 N1=17346 in /sys/kernel/slab/kmalloc-192
75421 N0=37119 N1=38302 in /sys/kernel/slab/radix_tree_node
566447 N0=222840 N1=343607 in /sys/kernel/slab/:at-0000104
566447 N0=222840 N1=343607 in /sys/kernel/slab/buffer_head
566447 N0=222840 N1=343607 in /sys/kernel/slab/ext4_prealloc_space
45600478 N0=27619140 N1=17981338 in /sys/kernel/slab/secpath_cache
45600504 N0=27619180 N1=17981324 in /sys/kernel/slab/biovec-4
45600540 N0=27619217 N1=17981323 in /sys/kernel/slab/ksm_rmap_item
45600549 N0=27619234 N1=17981315 in /sys/kernel/slab/kmalloc-64
45600554 N0=27619193 N1=17981361 in /sys/kernel/slab/avc_node
45600554 N0=27619247 N1=17981307 in /sys/kernel/slab/inet_peer_cache
45600561 N0=27619254 N1=17981307 in /sys/kernel/slab/fs_cache
45600565 N0=27619258 N1=17981307 in /sys/kernel/slab/iommu_devinfo
45600567 N0=27619260 N1=17981307 in /sys/kernel/slab/io
45600591 N0=27619284 N1=17981307 in /sys/kernel/slab/fib6_nodes
45600600 N0=27619293 N1=17981307 in /sys/kernel/slab/iommu_iova
45600612 N0=27619257 N1=17981355 in /sys/kernel/slab/:t-0000064
45600717 N0=27619317 N1=17981400 in /sys/kernel/slab/utrace
45600752 N0=27619392 N1=17981360 in /sys/kernel/slab/tcp_bind_bucket
45600762 N0=27619326 N1=17981436 in /sys/kernel/slab/xfs_ifork

This is getting out of control. I'm not sure what to do or how to investigate this.

Sheridan · 03-05-2011, 06:39 AM

Today I tried to update NIC drivers, controller drivers... nothing helps :-(

Sheridan · 03-06-2011, 05:28 PM

Sorry to bump this post again... but today I tried to remove all but the absolutely essential kernel modules and installed all standard HP drivers.... still the same issue. I swear if I don't get any replies now I'll say no more on this matter. But if you can suggest anything... Thank you.

Sheridan · 03-09-2011, 06:38 PM

SOLVED !!!! HEUREKA !!!!

If you have Fedora 14 + HP ProLiant DL360 G7 X5650 2P or similar and you have the above issue, add teh following to grub.conf kernel line and it will possibly fix this for you:

apm=off acpi=off

But... as for WHY APM and ACPI tangles with this particular Proliant... I have no clue.

Hope this helps someone.