What is consuming almost my whole RAM (Ubuntu 15.10 + CUDA)?

sacherus · 05-18-2016, 08:56 AM

Hi guys,
I'm not sure is it good place to put this thread, but I have "computing server" which I'm using for CUDA computation (with some framework). After a some computation and killing all computing processes I get 6.7GB/32GB free memory. After some point I'm not able to make anything that needs more memory. Any ideas?

free:

Code:

             total       used       free     shared    buffers     cached
Mem:           31G        25G       5.7G        73M        26M       983M
-/+ buffers/cache:        24G       6.7G
Swap:           0B         0B         0B

ps_mem.py:

Code:

(...)
 12.9 MiB +   1.8 MiB =  14.7 MiB       bash (4)
 14.9 MiB +  16.2 MiB =  31.0 MiB       systemd-journald
---------------------------------
                        119.5 MiB

After restart:

Code:

             total       used       free     shared    buffers     cached
Mem:           31G       463M        30G       9.6M        33M       210M
-/+ buffers/cache:       218M        31G
Swap:           0B         0B         0B

Code:

(...)
  8.4 MiB +  70.0 KiB =   8.4 MiB       dhclient
  8.0 MiB +   1.8 MiB =   9.8 MiB       NetworkManager
---------------------------------
                         88.3 MiB

Emerson · 05-18-2016, 09:11 AM

Check this out.

sacherus · 05-18-2016, 09:24 AM

I'm "only" caching 1GB for disk, where the rest went?

Mitt Green · 05-18-2016, 09:28 AM

Try htop.

sacherus · 05-18-2016, 09:40 AM

Quote:

Originally Posted by Mitt Green

Try htop.

I don't have screenshot, but htop showed exactly this same amount of RAM as free and no processes consuming memory.

Emerson · 05-18-2016, 09:44 AM

24 GB shown as used by buffers is actually free, it will be freed as soon as it is needed.

sacherus · 05-18-2016, 09:55 AM

Quote:

Originally Posted by Emerson

24 GB shown as used by buffers is actually free, it will be freed as soon as it is needed.

Code:

             total       used       free     shared    buffers     cached
Mem:           31G        25G       5.7G        73M        26M       983M -> taken by disk cache
-/+ buffers(used-cached)<-24G       6.7G -> free memory (5.7G) + cached (1G)
Swap:           0B         0B         |
                               this is real free value

Am I right?

rknichols · 05-18-2016, 11:00 AM

Quote:

Originally Posted by Emerson

24 GB shown as used by buffers is actually free, it will be freed as soon as it is needed.

Look at those numbers again. The buffer and cache sizes are in megabytes, not gigabytes, and don't add up to anything like 24 GB. This sounds more like shared memory that was not released when the processes using it were killed. Maybe a huge tmpfs that is chewing up space?

Emerson · 05-18-2016, 11:02 AM

24 GB is free for use as soon as it is needed?

rknichols · 05-18-2016, 11:06 AM

Quote:

Originally Posted by Emerson

24 GB is free for use as soon as it is needed?

No, it is not. Even after subtracting the buffer and cache sizes, there is still 24GB used and not available.

syg00 · 05-18-2016, 06:50 PM

Quote:

Originally Posted by rknichols

This sounds more like shared memory that was not released when the processes using it were killed. Maybe a huge tmpfs that is chewing up space?

tmpfs defaults to half RAM, so would have to be several, but I agree it's some along the shared lines. Lets see what /proc/meminfo has to say.

sacherus · 05-19-2016, 06:25 AM

Quote:

Originally Posted by syg00

tmpfs defaults to half RAM, so would have to be several, but I agree it's some along the shared lines. Lets see what /proc/meminfo has to say.

current situation - some more data:

Code:

             
             total       used       free     shared    buffers     cached
Mem:           31G        21G       9.7G        33M       3.5G        11G
-/+ buffers/cache:       6.2G        25G
Swap:           0B         0B         0B

htop: 6384MB

df -h:

Code:

Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G   34M  3.1G   2% /run
/dev/sda5       109G   96G  7.6G  93% /
tmpfs            16G   84K   16G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/sda2        96M   23M   74M  24% /boot/efi
/dev/sdb1       234G  8.0M  234G   1% /media/sacherus/DATA1
/dev/sdb5       233G   37G  196G  16% /media/sacherus/DATA2
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           3.2G     0  3.2G   0% /run/user/1000

/proc/meminfo

Code:

MemTotal:       32843624 kB
MemFree:        10189532 kB
MemAvailable:   26631168 kB
Buffers:         3622652 kB
Cached:         12495188 kB
SwapCached:            0 kB
Active:          6717624 kB
Inactive:        9592220 kB
Active(anon):     194028 kB
Inactive(anon):    32424 kB
Active(file):    6523596 kB
Inactive(file):  9559796 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        192020 kB
Mapped:            87120 kB
Shmem:             34452 kB
Slab:             607404 kB
SReclaimable:     444020 kB
SUnreclaim:       163384 kB
KernelStack:        5776 kB
PageTables:         7384 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16421812 kB
Committed_AS:    1474488 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      324136 kB
VmallocChunk:   34358945788 kB
HardwareCorrupted:     0 kB
AnonHugePages:     81920 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     5901200 kB
DirectMap2M:    18112512 kB
DirectMap1G:    11534336 kB

sudo python ~/ps_mem.py

Code:

 Private  +   Shared  =  RAM used       Program
184.0 KiB +  87.5 KiB = 271.5 KiB       atd
196.0 KiB +  81.0 KiB = 277.0 KiB       agetty
364.0 KiB + 106.5 KiB = 470.5 KiB       irqbalance
400.0 KiB + 133.0 KiB = 533.0 KiB       systemd-timesyncd
364.0 KiB + 238.0 KiB = 602.0 KiB       cron
496.0 KiB + 106.5 KiB = 602.5 KiB       cgmanager
396.0 KiB + 225.0 KiB = 621.0 KiB       kerneloops
504.0 KiB + 340.0 KiB = 844.0 KiB       byobu-select-se
864.0 KiB + 110.0 KiB = 974.0 KiB       systemd-logind
404.0 KiB + 712.0 KiB =   1.1 MiB       avahi-daemon (2)
  1.1 MiB +  27.5 KiB =   1.1 MiB       mount.exfat-fuse
860.0 KiB + 343.5 KiB =   1.2 MiB       dnsmasq
  1.0 MiB + 242.0 KiB =   1.3 MiB       sudo
  1.3 MiB +  54.5 KiB =   1.3 MiB       mount.ntfs
752.0 KiB + 668.0 KiB =   1.4 MiB       dbus
844.0 KiB + 663.0 KiB =   1.5 MiB       polkitd
  1.2 MiB + 306.5 KiB =   1.5 MiB       dbus-daemon
904.0 KiB + 707.0 KiB =   1.6 MiB       (sd-pam)
  1.3 MiB + 375.0 KiB =   1.7 MiB       smartd
  1.8 MiB + 387.5 KiB =   2.2 MiB       wpa_supplicant
  1.4 MiB + 817.0 KiB =   2.2 MiB       accounts-daemon
  2.0 MiB + 178.5 KiB =   2.2 MiB       systemd-udevd
  1.4 MiB + 850.0 KiB =   2.2 MiB       cupsd
  1.4 MiB +   1.5 MiB =   2.9 MiB       cups-browsed
  3.2 MiB + 338.0 KiB =   3.5 MiB       hamachid
  3.7 MiB + 111.0 KiB =   3.8 MiB       rsyslogd
  1.7 MiB +   2.4 MiB =   4.2 MiB       systemd (2)
  2.6 MiB +   1.9 MiB =   4.4 MiB       whoopsie
  3.7 MiB + 843.5 KiB =   4.5 MiB       ModemManager
  3.9 MiB + 870.0 KiB =   4.8 MiB       tmux (2)
  1.5 MiB +   3.7 MiB =   5.2 MiB       sshd (3)
  6.4 MiB + 975.0 KiB =   7.4 MiB       bash (2)
  8.4 MiB +  67.0 KiB =   8.4 MiB       dhclient
  8.0 MiB +   1.8 MiB =   9.7 MiB       NetworkManager
 10.0 MiB +  16.5 MiB =  26.5 MiB       systemd-journald
---------------------------------
                        112.9 MiB
=================================

sacherus · 05-19-2016, 06:43 AM

tmpfs:
I didn't know that such thing like tmpfs exists, but I don't think that tmpfs has to do anything:
http://unix.stackexchange.com/questi...oes-it-compete:
"tmpfs only uses the amount of memory that is taken, not the full 50%. So, if only 10mb of those 1gb are taken, your tmpfs instance only occupies those 10mb"

So tmpfs in my case should occupy at most ~100kB

shared memory:
ps_mem & free is reporting this same amount of memory used by shared libraries (is this the same as shared libraries?) ~32MB

/proc/meminfo
I was trying to read it before, but I gave up.

syg00 · 05-19-2016, 07:05 AM

Irreconcilable numbers. That last lot is what we would expect to see.
We need the same displays when you reckon you have a problem. uneditted.

sacherus · 05-19-2016, 07:17 AM

Quote:

Originally Posted by syg00

Irreconcilable numbers. That last lot is what we would expect to see.
We need the same displays when you reckon you have a problem. uneditted.

There is problem cause I don't have any processes running and it's 6GB. After server starts tt's consuming ~350 MB. So there is a gap.
I'm restarting server every 2 days, cause I need all memory. I'm able to create situation with higher memory consumption, but it will take some time (1 week?), because it's random process.

IMHO ps_mem should report this same amount of memory as free.
On my notebook:

Code:

(...)
130.7 MiB +  34.8 MiB = 165.5 MiB	evince (2)
221.8 MiB +   3.2 MiB = 225.0 MiB	compiz
  2.1 GiB +  80.2 MiB =   2.1 GiB	chromium-browser [updated] (22)
---------------------------------
                          3.8 GiB
=================================

             total       used       free     shared    buffers     cached
Mem:          7.7G       6.5G       1.2G       826M       120M       2.6G
-/+ buffers/cache:       3.8G       3.9G
Swap:         7.6G       5.9M       7.6G