LinuxQuestions.org - [SOLVED] Virtual Memory limit exhausted at 1TB? malloc/mmap failures even with free memory

- Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)

- - Virtual Memory limit exhausted at 1TB? malloc/mmap failures even with free memory (https://www.linuxquestions.org/questions/linux-server-73/virtual-memory-limit-exhausted-at-1tb-malloc-mmap-failures-even-with-free-memory-4175427414/)

Virtual Memory limit exhausted at 1TB? malloc/mmap failures even with free memory

Curious,

even with ulimit for virtual mem set to unlimited or 0, I get malloc failures after about 1 TB virtual is exhausted, with malloc size > 128kB that should do a private mmap. Disk-supported mmap still seem to work beyond one terabyte.

Anybody encountered similar? Could it be my testing distro is a "desktop" instead of an enterprise version? Many thanks.

SuSE Linux Kernel 2.6.37.6-0.20-desktop #1 SMP PREEMPT 2011-12-19 23:39:38 +0100 x86_64 x86_64 x86_64 GNU/Linux

24 GB RAM + 36 GB Swap, 1020 GB can be allocated (of course not used). mmap against some 1TB+ disks in raid, malloc fails with ENOMEM after approx 1TB with 0% swap and 30% ram real usage.

Just wondering if anybody has some ideas.

I thought the virtual limit to be 128TB, sometimes 64TB, on Intel distros, not just 1TB.....

Thanks and Cheers, Martin

I would be suspecting something like control block shortage. Keep an eye on /proc/meminfo and /proc/slabinfo ("cat" them to disk for later perusal).
There is also slabtop which can be handy in "normal" storage shortages, but don't use it when the system (i.e. kernel controlled) storage is in crisis. Your system might just appear to "go away" - maybe for several hours.

Thanks, in this case the mem chunks are quite big since I increased them beyond reasonable sizes, so to eliminate control structure shortages.

And found the solution: turn vm overcommit checks completely off, apparently the default heuristics stop at the reasonable size of 1 TB:

Code:

echo 1 > /proc/sys/vm/overcommit_memory

Now I can overcommit well beyond 1 TB, such that TOP displays a queston mark, is that a bug maybe, VIRT=???

TOP output

Code:

top - 09:13:06 up  1:00,  3 users,  load average: 10.34, 7.08, 5.32

Tasks: 191 total,  1 running, 190 sleeping,  0 stopped,  0 zombie

Cpu(s): 99.0%us,  0.4%sy,  0.0%ni,  0.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:    24152M total,    15088M used,    9064M free,      34M buffers

Swap:    36863M total,        0M used,    36863M free,      513M cached



  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

 6453 martin    20  0    ?  11g 2392 S 1190 50.4  61:41.11 mchess

Nevertheless, immediate problem solved.

Cheers, Martin