This is probably a q for kernel cognoscenti.

Gist: "top" summary line for Mem shows "used" constantly rises to
near-total, then drops, then rises again, repeat;
meanwhile, application being monitored gets slower & slower.

I'm trying to track down cause of slowdown in app; behavior suggests
quadratic algorithm in that the larger the database is creates, the
slower it gets, however I think I've ruled that out and was suspecting
heap fragmentation, so...

I'm running top -p <app process id> while the app is running, I thought
I'd see if its mem consumption grows as its progress report slows down.

But:
VIRT, RES, SHR, CODE, DATA %MEM don't seem to change.
Meanwhile, in the Mem summary line, I see the "used" number
slowly but consistently, monotonically increase until it reaches
something very close to the 11.5G total. Then it drops down
and repeats this behavior.
(Also, nFLT and nDRT remain 0 all the time.)

I can't find any documentation on the meaning of the fields in that
summary line, for one thing.
I'm guessing that (perhaps since I have little else running on this
machine (8 cores with hyper-threading -> 16 threads total), the
kernel is willing to allocate more and more phys memory to the process
until it reaches a threshold, perhaps?

Can a kernel guru tell me what this data from top might indicate is
going on in the kernel?


Another thing I noticed is that as time goes on, more and more often
top shows that all threads in the process are sleeping but one,
which is in log_wait.
The only thing I have found about log_wait is that it's a
synchro variable for syslog, but none of *my* code is using
syslog. Aside from the possibility that some library code is using
syslog (no messages show up in the logs, by the way), is there
something in the kernel that would necessarily wait on log_wait
for an app that does almost entirely these kernel- or near-kernel
operations:
pread, write
pthread_* operations

I haven't yet found any wikis or tutorials that would shed light
on the matter, but if anyone knows of anything that might, please send
a pointer my way.

Thanks for any ideas...

The following fields are:
VIRT = Virtual Memory available = Real Memory + Swap Area.
RES = Residential Memory = Amount of memory existing in core memory.
SHR = Shared Memory = Code data that is shared (read only) across more than one process.
CODE = Code Memory = code/executable memory.
DATA = Data Memory = memory holding variables.
%MEM = % of (total usable) memory currently available.

So, I wouldn't expect VIRT, RES,SHR,CODE,or %MEM to change much at all for only one type of CPU bound process. Maybe DATA would move minimally as new variables are acquired/used.

You can fins more information on these labels/meaning by looking into the top source code, inside the linux doc system "aka: man top", as well as in /usr/src/kernel/Docs, or goto URL http://linux.die.net/man/1/top

As you correctly pointed out, the linux kernel will usually use all of available free memory rather than waste time cleaning up (freeing memory allocation tables) or initialing (possibly zeroing) used memory - it is more efficient this way (as measured by total execution time) for each process/thread execution.

You can see how the memory management will handle memory requests by the memory size block allocations as per slab in /proc fs. Have a look i.e. cat /proc/slabinfo and cat /proc/meminfo for more information of memory relationships.

Hope this assists you.
Cheers. Grahame

Thanks; I had already exhausted the doc for "top", alas.
What didn't make sense to me was to see the summary info for memory grow
while the %MEM for the process didn't.


As for the wchan, I figured out that it was truncating the symbol, so it wasn't
really log_wait; it was log_wait_something_else_that_i_dont_recall
in fact.
Arg.

In the old days, I'd have just read the kernel source, but it's not as simple
as it was in 1983...