I decided to make this a new post since the previous post (Does free( ) immediately free the memory?) has been inactive for a while now. After many hours of debugging a fork'd c program, Valgrind reports that I have 0 memory leaks and 0 memory issues. I have ran the application for over 3days processing over 100 thousand files, and the reports still never showed any leaks. However, when I use the top command to check the free and available memory, it still lists
8097678K Total, 7968xxxK used, 1000-2000K free. Top reports the memory is at the 7GB used mark after the first day, but will never go over the 8GB used mark. The swap space is never used up beyond 160KB, and the application is not taking a speed hit (I have built in speed checking). Is there some issue with running top in reporting the memory, does it report the memory that is in the de-allocation pool as being used? Is there a better utility, other that top, for checking the memory usage?

I am really not sure if I have a memory problem now, I let the program run over this weekend to see if it terminates or crashes the system, but I don't believe it will. Any further suggestions would be greatly appreciated.

Thanks in advance!

Look into /proc/meminfo and /proc/vmstat. They provide much more detailed info. In fact, short form of that info you can get after running 'free' command. My guess is that you have huge amount in the 'buffers' and 'cached' column, what means that the systems keeps that memory.

You are correct, the buffered and cached memory keeps increasing. Does this effect the performance of the system? As I understood, Red Hat keeps the memory buffers for quicker access on sucessive mallocs. Is there a way to periodically release this memory from the buffers/cache? If not, will using realloc instead of malloc help?

Thanks

Ravyen, are you observing any problem? What you report sounds like everything is working right. Maybe you make an implicit assumption about memory management and your observations don't confirm it?

Unix processes traditionally never shrink. When you allocate memory (e.g., malloc), eventually memory management will need to call brk(2) and increase the end of the data segment. This increases the "used memory" visible from outside, e.g., top. When you later free() memory, your process keeps it for subsequent allocation. The size of your process does not decrease. At least this has been the Unix model for decades -- if I'm outdated, please someone correct me! There are some ways around this (if you need to make some of the space available for other processes), but I haven't used any.

Don't worry about buffer/cache. Linux puts unused memory to use in buffers and caches, to give you better performance. The kernel will reclaim it as needed. The credo is "Free memory is wasted memory". I don't think you'd want to release this memory from the buffers/cache.

No, no problem is observed, I was just assuming that when you free'd the memory, that the OS would then reduce the number that was reported as being "used". As long as Linux will know that other programs will need to utilize this memory in the future, I am ok with things! Thanks for the great response!!

Normally, free'd memory is only available within the process. Other programs won't be able to utilize it until the process exits. But this doesn't have directly to do with the small amount of free memory -- it's typical to see a small figure here. When a big process exits, you can see that free memory temporarily goes up correspondingly, but slowly it gets put to "better use" in caches and buffers.

That said, there appear to be mechanisms by which a process can give memory back to the OS -- I've seen emacs's total size go down after killing a big buffer. Not sure how that's accomplished.

This has already been discussed and explained in the afore mentioned previous thread.