[SOLVED] RAM usage by unknown sources not traceable via regular tools (meminfo, smem)

sanjibdas · 02-06-2021, 02:00 AM

Hi all,

I do see lot of memory (RAM) usage by some unknown source & is not listed by any of these cached, buffer, slab, kernel stack, vmallocUsed, anon or applications. I used smem tool so far & also enabled kmemleak. But not much luck yet.

Total RAM size is 16G & we carve out around 2.3G for a different purpose (not available to linux).

I used these fields to calculate total usage, which typically give closer value to MemTotal in normal cases. Here I see ~6.5G is missing.

MemTotal = MemFree + Buffers + Cached + AnonPages + Slab + KernelStack + PageTables + VmallocUsed

MemTotal: 13535936 kB (16G - 2.3G)
MemFree: 782080 kB
Buffers: 615744 kB
Cached: 2748416 kB
SwapCached: 0 kB
Active: 4932800 kB
Inactive: 845376 kB
Active(anon): 4148928 kB
Inactive(anon): 0 kB
Active(file): 783872 kB
Inactive(file): 845376 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2621312 kB
SwapFree: 2621312 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 2404544 kB
Mapped: 828288 kB
Shmem: 1739776 kB
Slab: 280768 kB
SReclaimable: 56128 kB
SUnreclaim: 224640 kB
KernelStack: 33792 kB
PageTables: 39296 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 9389248 kB
Committed_AS: 6923008 kB
VmallocTotal: 35184367894528 kB
VmallocUsed: 4480 kB
VmallocChunk: 35184367857920 kB

My linux kernel is 2.6.32.11 (comparatively old) & I am clueless how to track this GAP. It would be a great help if you could share some pointers in this context. Any specific open source tool which could be useful in my case?

Thanks in advance.
Sanjib

teckk · 02-06-2021, 08:36 AM

A few examples.

Code:

ps -e | grep xterm
226455 ?        00:00:00 xterm
226570 ?        00:00:00 xterm
 
pmap -x 226570
226570:   xterm -class UXTerm -title uxterm -u8
Address           Kbytes     RSS   Dirty Mode  Mapping
0000557e18f26000     136     136       0 r---- xterm
0000557e18f48000     468     468       0 r-x-- xterm
0000557e18fbd000     192     136       0 r---- xterm
0000557e18fed000      28      28      28 r---- xterm
0000557e18ff4000      36      36      36 rw--- xterm
0000557e18ffd000      76       8       8 rw---   [ anon ]
0000557e197e8000    6208    6072    6072 rw---   [ anon ]
00007f1e9aa5d000     248     224       0 r---- DejaVuSansMono-BoldOblique.ttf
....

cat /proc/226570/status
Name:   xterm
Umask:  0022
State:  S (sleeping)
Tgid:   226570
Ngid:   0
Pid:    226570
...

cat /proc/226570/smaps

Also found this

Code:

ps -eo size,pid,user,command --sort -size | \
    awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |\
    cut -d "" -f2 | cut -d "-" -f1

Also:
Valgrind
Exmap
ps_mem
top
htop

sanjibdas · 02-06-2021, 11:20 PM

Thanks for your reply teckk. I did memory usage calculation for all my processes, but nothing adds up to the total usage (at least closer to that).
Other than Slab & VmallocUsed, is there any other parameter which could give some hint about kernel memory leak?

ps_mem seems to be quite useful, I will keep this running at frequent intervals.

sanjibdas · 03-09-2021, 03:19 AM

This link was really helpful (check for Memory black hole).
https://titanwolf.org/Network/Articl...2954#gsc.tab=0

Memory allocation via alloc_pages() without proper book-keeping could lead to such situation where we cannot track
memory usage from /proc/meminfo dump.

pan64 · 03-09-2021, 07:59 AM

www.linuxatemyram.com

syg00 · 03-09-2021, 04:19 PM

Sounds scarily similar to this recent thread