my server run out of memory
 

Dear every body,

My server is almost run out of memory. here is the out put of free command:

# free -h
total used free shared buffers cached
Mem: 7.8G 7.6G 214M 0B 798M 6.4G
-/+ buffers/cache: 467M 7.4G
Swap: 11G 384K 11G

my server is running backuppc

and here the out put of top command sort my memory (M)


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21117 backuppc 20 0 65184 13m 1868 S 0 0.2 0:01.15 BackupPC
2478 root 20 0 52244 9664 1224 S 0 0.1 0:11.81 munin-node
29301 root 0 -20 19472 7196 2712 S 0 0.1 0:04.12 atop
21124 backuppc 20 0 45596 7016 2276 S 0 0.1 0:00.06 BackupPC_trashC
8214 www-data 20 0 364m 4872 1896 S 0 0.1 0:00.03 apache2
8193 root 20 0 83528 4728 2432 S 0 0.1 0:00.21 apache2
8213 www-data 20 0 428m 4444 1704 S 0 0.1 0:00.03 apache2
9167 root 20 0 79716 3824 2984 S 0 0.0 0:00.00 sshd
20947 root 20 0 79724 3812 2968 S 0 0.0 0:00.02 sshd
8198 www-data 20 0 82756 2924 680 S 0 0.0 0:00.00 apache2


I can not find out any process that take up my server memory. can any one give me some idea about this.

Thanks,
Vannath

have you checked this: http://www.linuxatemyram.com/ ?

coo, not seen that link for years. a decade ago we were posting it daily.

Thank you so much for this useful link. but why my other server is fine and i always get alarm memory critical so is there any way to prevent this?

Thanks,
Vannath

actually it looks like your swap is not in use, therefore I think it is ok. Probably the config of the alarm is not ok, or not similar to the others...

Here my configuration to monitor the server memory. I check if the memory reach 90% of its total memory it will be critical.

command[check_memory]=/usr/local/nagios/libexec/check_memory.sh -w 80 -c 90

and the server memory is always 90% used.

Memory: Critical Total: 7997 MB - Used: 6986 MB - 92% used!

Thanks
Vannath

Maybe you are facing a memory leak in kernel space. You won't see anything with "free" in that case. Can you check these files :
# cat /proc/meminfo
# cat /proc/slabinfo

also here a few usefull commands which can help you (for debian) :

user space usage :
# ps -e o rsz,vsz,pid,command --sort -rsz

processes memory space :
# pmap -x `ps ax | cut -d" " -f1`

displays peak memory usage :
# grep ^VmSize /proc/*/status | sort -n -k+2 | tail

Here the out put of commane "cat /proc/meminfo"

Here is the out put of command "pmap -x `ps ax | cut -d" " -f1` "

Here is the out put of command ""

I understand nothing about those above out put command.

Here the out put of top command sort by memory usaged (shift + m)


and here is the the out put of free command


I really can not find out what process is taking up my memory.

and how can fix the memory leak. and with disk caching how can i make my memory work as usaul.

Thanks with best regards,
Vannath

Now I fix it. thank you so much for all you help.

root@backup2:~# free -m
total used free shared buffers cached
Mem: 7997 7765 231 0 567 6738
-/+ buffers/cache: 460 7537
Swap: 11443 0 11443
root@backup2:~# echo 3 | sudo tee /proc/sys/vm/drop_caches
3
root@backup2:~#
root@backup2:~#
root@backup2:~# free -m
total used free shared buffers cached
Mem: 7997 174 7823 0 1 14
-/+ buffers/cache: 158 7838
Swap: 11443 0 11443
root@backup2:~#

Good! Could you share with the rest of us what you did to fix it?

I use this command

#echo 3 | sudo tee /proc/sys/vm/drop_caches
3

Oops! My bad! I missed that in your post!

Thanks!

that sounds good, however you only dropped the caches... I fear that your problem may occur again in the future.
/proc/meminfo shows a lot amount of caches but it's not an issue. I suggest that you monitor the /proc/slabinfo file if the problem occures again.

Yes, it actually happen again after 9 to 12 hour later. I really tried command "cat /proc/slabinfo" but i really can not understand the out put of this command. what information that i can get from this command. and could you tell me any solution that i can fix this problem permanently?

Thanks,
Vannath

Well I think that you should monitor the evolution of /proc/meminfo (and drop the caches when the system becomes unsafe). This can be done with a simple loop like that for example :

# mkdir /root/tmp
# while true; do cp /proc/meminfo /root/tmp/meminfo.`date +%Y%m%d%H%M`; sleep 3600; done

After a while, you can try to find out which part of the memory is concerned. For exemple for Slab cache :
# grep Slab root/tmp/meminfo.*

you can "export" the result to excel or something like that to check the evolution, that can give you a first clue.
If the part of memory is the cache, then you should have a look at the slab cache because it can help you to find out which process is leaking.
You can monitor the amount of memory allocated inside the cache like that :
# cat /proc/slabinfo | awk -F" " '{printf "$1 : %10.0f\n", ($3*$4) }'

use this to drop the process which are not using any memory :
# cat /proc/slabinfo | awk -F" " '{printf $1" : %10.0f\n", ($3*$4) }' | grep -ve " 0$"

after that, you can create the same kind of loop to monitor the slab cache :
# while true; do cat /proc/slabinfo | awk -F" " '{printf $1" : %10.0f\n", ($3*$4) }' | grep -ve " 0$" > /root/tmp/slab.`date +%Y%m%d%H%M` ; sleep 3600; done

and chech the evolution of the process. cred_jar for example :
# grep cred_jar root/tmp/slab.*

Based on what you will find, you may be able to google about a memory leak in that process...

good luck !