kernel q re: user-space memory handling
This is probably a q for kernel cognoscenti.
Gist: "top" summary line for Mem shows "used" constantly rises to near-total, then drops, then rises again, repeat; meanwhile, application being monitored gets slower & slower. I'm trying to track down cause of slowdown in app; behavior suggests quadratic algorithm in that the larger the database is creates, the slower it gets, however I think I've ruled that out and was suspecting heap fragmentation, so... I'm running top -p <app process id> while the app is running, I thought I'd see if its mem consumption grows as its progress report slows down. But: VIRT, RES, SHR, CODE, DATA %MEM don't seem to change. Meanwhile, in the Mem summary line, I see the "used" number slowly but consistently, monotonically increase until it reaches something very close to the 11.5G total. Then it drops down and repeats this behavior. (Also, nFLT and nDRT remain 0 all the time.) I can't find any documentation on the meaning of the fields in that summary line, for one thing. I'm guessing that (perhaps since I have little else running on this machine (8 cores with hyper-threading -> 16 threads total), the kernel is willing to allocate more and more phys memory to the process until it reaches a threshold, perhaps? Can a kernel guru tell me what this data from top might indicate is going on in the kernel? Another thing I noticed is that as time goes on, more and more often top shows that all threads in the process are sleeping but one, which is in log_wait. The only thing I have found about log_wait is that it's a synchro variable for syslog, but none of *my* code is using syslog. Aside from the possibility that some library code is using syslog (no messages show up in the logs, by the way), is there something in the kernel that would necessarily wait on log_wait for an app that does almost entirely these kernel- or near-kernel operations: pread, write pthread_* operations I haven't yet found any wikis or tutorials that would shed light on the matter, but if anyone knows of anything that might, please send a pointer my way. Thanks for any ideas... |
Top :: Memory field breakdown
The following fields are:
VIRT = Virtual Memory available = Real Memory + Swap Area. RES = Residential Memory = Amount of memory existing in core memory. SHR = Shared Memory = Code data that is shared (read only) across more than one process. CODE = Code Memory = code/executable memory. DATA = Data Memory = memory holding variables. %MEM = % of (total usable) memory currently available. So, I wouldn't expect VIRT, RES,SHR,CODE,or %MEM to change much at all for only one type of CPU bound process. Maybe DATA would move minimally as new variables are acquired/used. You can fins more information on these labels/meaning by looking into the top source code, inside the linux doc system "aka: man top", as well as in /usr/src/kernel/Docs, or goto URL http://linux.die.net/man/1/top As you correctly pointed out, the linux kernel will usually use all of available free memory rather than waste time cleaning up (freeing memory allocation tables) or initialing (possibly zeroing) used memory - it is more efficient this way (as measured by total execution time) for each process/thread execution. You can see how the memory management will handle memory requests by the memory size block allocations as per slab in /proc fs. Have a look i.e. cat /proc/slabinfo and cat /proc/meminfo for more information of memory relationships. Hope this assists you. Cheers. Grahame |
Thanks; I had already exhausted the doc for "top", alas.
What didn't make sense to me was to see the summary info for memory grow while the %MEM for the process didn't. As for the wchan, I figured out that it was truncating the symbol, so it wasn't really log_wait; it was log_wait_something_else_that_i_dont_recall in fact. Arg. In the old days, I'd have just read the kernel source, but it's not as simple as it was in 1983... |
All times are GMT -5. The time now is 02:21 PM. |