[SOLVED] Existeth a way to 'clear out' all the old/cached crap from RAM/SWAP to free up space?

GrapefruiTgirl · 07-22-2010, 02:17 PM

Hey john,

I haven't yet logged out (always getting sidetracked by something or another!) but before I do I will save that file and save a new one upon logging back in.

In fact, so I don't forget, here it currently is:

Code:

sasha@reactor: cat /proc/slabinfo
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
nv_stack_t            41     44  12288    2    8 : tunables    0    0    0 : slabdata     22     22      0
kmalloc_dma-512       16     16    512   16    2 : tunables    0    0    0 : slabdata      1      1      0
kcopyd_job             0      0    368   22    2 : tunables    0    0    0 : slabdata      0      0      0
dm_uevent              0      0   2608   12    8 : tunables    0    0    0 : slabdata      0      0      0
dm_rq_target_io        0      0    376   21    2 : tunables    0    0    0 : slabdata      0      0      0
dm_io                  0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
cfq_io_context       133    150    136   30    1 : tunables    0    0    0 : slabdata      5      5      0
cfq_queue            113    124    264   31    2 : tunables    0    0    0 : slabdata      4      4      0
bsg_cmd                0      0    312   26    2 : tunables    0    0    0 : slabdata      0      0      0
mqueue_inode_cache      1     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
udf_inode_cache        0      0    616   26    4 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    592   27    4 : tunables    0    0    0 : slabdata      0      0      0
fat_inode_cache       26     26    624   26    4 : tunables    0    0    0 : slabdata      1      1      0
fat_cache            102    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
journal_handle       340    340     24  170    1 : tunables    0    0    0 : slabdata      2      2      0
journal_head          73    108    112   36    1 : tunables    0    0    0 : slabdata      3      3      0
revoke_table         260    512     16  256    1 : tunables    0    0    0 : slabdata      2      2      0
revoke_record        256    256     32  128    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_inode_cache   29339  29340    888   18    4 : tunables    0    0    0 : slabdata   1630   1630      0
ext4_free_block_extents    146    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_alloc_context     56     56    144   28    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_prealloc_space     78     78    104   39    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_system_zone       0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
ext2_inode_cache      22     44    728   22    4 : tunables    0    0    0 : slabdata      2      2      0
ext3_inode_cache       0      0    744   22    4 : tunables    0    0    0 : slabdata      0      0      0
ext3_xattr             0      0     88   46    1 : tunables    0    0    0 : slabdata      0      0      0
configfs_dir_cache      1     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
kioctx                 0      0    320   25    2 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark_entry    288    288    112   36    1 : tunables    0    0    0 : slabdata      8      8      0
posix_timers_cache     29     56    144   28    1 : tunables    0    0    0 : slabdata      2      2      0
UDP-Lite               0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
ip_dst_cache          63     63    384   21    2 : tunables    0    0    0 : slabdata      3      3      0
RAW                   26     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
UDP                   42     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP           21     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                   43     63   1536   21    8 : tunables    0    0    0 : slabdata      3      3      0
blkdev_queue          29     30   2176   15    8 : tunables    0    0    0 : slabdata      2      2      0
blkdev_requests       82     96    336   24    2 : tunables    0    0    0 : slabdata      4      4      0
fsnotify_event_holder    340    340     24  170    1 : tunables    0    0    0 : slabdata      2      2      0
fsnotify_event        78     78    104   39    1 : tunables    0    0    0 : slabdata      2      2      0
sock_inode_cache     206    250    640   25    4 : tunables    0    0    0 : slabdata     10     10      0
skbuff_fclone_cache     36     36    448   18    2 : tunables    0    0    0 : slabdata      2      2      0
file_lock_cache       44     44    184   22    1 : tunables    0    0    0 : slabdata      2      2      0
net_namespace          0      0   1472   22    8 : tunables    0    0    0 : slabdata      0      0      0
shmem_inode_cache    873    880    736   22    4 : tunables    0    0    0 : slabdata     40     40      0
Acpi-ParseExt       1590   1624     72   56    1 : tunables    0    0    0 : slabdata     29     29      0
Acpi-Parse           170    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
proc_inode_cache   11272  11367    592   27    4 : tunables    0    0    0 : slabdata    421    421      0
sigqueue              50     50    160   25    1 : tunables    0    0    0 : slabdata      2      2      0
bdev_cache            37     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
sysfs_dir_cache    10212  10251     80   51    1 : tunables    0    0    0 : slabdata    201    201      0
inode_cache         1680   1680    544   30    4 : tunables    0    0    0 : slabdata     56     56      0
dentry             46187  46200    192   21    1 : tunables    0    0    0 : slabdata   2200   2200      0
radix_tree_node     7251   8526    560   29    4 : tunables    0    0    0 : slabdata    294    294      0
buffer_head        59760  59760    112   36    1 : tunables    0    0    0 : slabdata   1660   1660      0
vm_area_struct      6043   6144    168   24    1 : tunables    0    0    0 : slabdata    256    256      0
files_cache          253    276    704   23    4 : tunables    0    0    0 : slabdata     12     12      0
signal_cache         216    247    832   19    4 : tunables    0    0    0 : slabdata     13     13      0
sighand_cache        135    150   2112   15    8 : tunables    0    0    0 : slabdata     10     10      0
task_struct          164    220   1488   22    8 : tunables    0    0    0 : slabdata     10     10      0
anon_vma            1882   2048     32  128    1 : tunables    0    0    0 : slabdata     16     16      0
idr_layer_cache      353    390    544   30    4 : tunables    0    0    0 : slabdata     13     13      0
kmalloc-8192          42     44   8192    4    8 : tunables    0    0    0 : slabdata     11     11      0
kmalloc-4096         737    760   4096    8    8 : tunables    0    0    0 : slabdata     95     95      0
kmalloc-2048         270    288   2048   16    8 : tunables    0    0    0 : slabdata     18     18      0
kmalloc-1024         379    400   1024   16    4 : tunables    0    0    0 : slabdata     25     25      0
kmalloc-512          866    976    512   16    2 : tunables    0    0    0 : slabdata     61     61      0
kmalloc-256         3201   3360    256   16    1 : tunables    0    0    0 : slabdata    210    210      0
kmalloc-128         1003   1216    128   32    1 : tunables    0    0    0 : slabdata     38     38      0
kmalloc-64          2172   6016     64   64    1 : tunables    0    0    0 : slabdata     94     94      0
kmalloc-32          3867   4864     32  128    1 : tunables    0    0    0 : slabdata     38     38      0
kmalloc-16          2040   2304     16  256    1 : tunables    0    0    0 : slabdata      9      9      0
kmalloc-8           3549   3584      8  512    1 : tunables    0    0    0 : slabdata      7      7      0
kmalloc-192         3028   3108    192   21    1 : tunables    0    0    0 : slabdata    148    148      0
kmalloc-96           744    924     96   42    1 : tunables    0    0    0 : slabdata     22     22      0
sasha@reactor:

I realize this alone doesn't tell us much relatively speaking, but I'll compare it with the new one after logging back in later..

johnsfine · 07-22-2010, 02:22 PM

Is there anyone reading this expert enough to interpret the important smaps info in post #11?

It certainly looks like X is doing something very much like "leaking" (that could be accumulating memory inside X in response to some resource leak in a client of X. It isn't unambiguously a leak in X itself).

Edit: I had a theory posted here, but checked some details on my own system with results that I think eliminates that theory.

BTW, forget about slabinfo. That showed ext4_inode_cache is consuming 26MB, which is not enough to be interesting for the current problem. Everything else in slabinfo is even smaller.

Zero Angel · 07-22-2010, 02:23 PM

Sorry to hijack the thread with my question but does logging out actually restart X? Or does one have to CTRL+Bkspace or killall -9 Xorg to actually restart it?

GrapefruiTgirl · 07-22-2010, 02:28 PM

@ Zero Angel,

Normally logging out will not kill X, and on my machine IIRC I have CTRL-ALT-BKSPC disabled.

But, in my case, as I'm not using a login manager like XDM or KDM or whatever, but instead I use `startx` which starts my window manager via xinitrc, when I log out (exit) from my desktop session, both my window manager and X both are stopped.

Then I'll `startx` fresh which will start a fresh window manager session & a new X.

Not a hijack - actually a good point to make.

johnsfine · 07-22-2010, 02:58 PM

Those advising you to stop X and restart it assumed you know better than we do the best way to stop and restart X on your own system. I'm glad to hear that is correct.

I'm confident that stopping and restarting X will fix most or all of the problem. But that still leaves us all wondering why X accumulated so much memory.

I have certainly read about memory leak bugs in X, but I never understood the detail. I never saw the situation myself. I thought such issues had been corrected.

GrapefruiTgirl · 07-22-2010, 03:17 PM

Quote:

Originally Posted by johnsfine

I have certainly read about memory leak bugs in X, but I never understood the detail. I never saw the situation myself. I thought such issues had been corrected.

Indeed; and It's probably fair to say that fixing memory leaks is a good portion of the work actively done on Xorg.

For the record, my current version of X is still a few versions behind the current "Slackware64-current" version of X, for reasons outlined here but I doubt that a GIANT improvement in the 'memory leakage department' would be seen in the current version, over my current version.

One day very soon I'll be revisiting that thread and that problem, and see if the current version of X in Slackware64-current actually works for me yet.

johnsfine · 07-22-2010, 03:30 PM

If you still haven't restarted X, how many threads does that process have? Then after restart, how many threads?

nlwp in the field list of ps (where you have pid,user,rss, etc.) is one way to get the count of threads.

Code:

ps -eo pid,user,rss,nlwp,comm --sort=-rss | head -n 10

GrapefruiTgirl · 07-22-2010, 03:42 PM

Looks to me like X is the only thread:

Code:

sasha@reactor: ps -eo pid,user,rss,nlwp,comm --sort=-rss | head -n 10
  PID USER       RSS NLWP COMMAND
  818 root     2210388  1 X
16710 sasha    183280  11 firefox-bin
  483 sasha    43312    1 kwrite
12339 sasha    16752    1 xterm
14653 root     15940    1 kded4
32150 sasha    15936    1 knotify4
17739 sasha    12224    2 kmix
14650 root      9152    1 klauncher
18837 sasha     8256    1 kio_file
sasha@reactor:

And from top (including processes from the time I ran `startx`:

Code:

  801 sasha     20   0  8944   932   928 S  0.0  0.0  0:00.00  |   `- /bin/sh /usr/bin/startx
  817 sasha     20   0 15588   668   664 S  0.0  0.0  0:00.00  |       `- xinit /usr/lib64/X11/xinit/xinitrc -- /usr/bin/X :0 -auth /home/sasha/.serverauth.801
  823 sasha     20   0 41172  3724  1376 S  0.0  0.1 12:06.34  |           `- /usr/bin/i3 --force-xinerama -V -d all -a
  818 root      20   0 3159M 2158M  3996 S  1.0 54.5 13h25:00  |           `- /usr/bin/X :0 -auth /home/sasha/.serverauth.801

johnsfine · 07-22-2010, 03:52 PM

Pending a correction to the likely memory leak in X, you might be a lot better off if you increase the size of your swap partition (or temporarily add a swap file).

The leaked memory in X apparently goes stale (never accessed) so it would get swapped out as the system needs ram for other processes. Now stale data from X apparently has filled the entire small swap partition so no more gets swapped out.

If your swap partition were bigger than the current anonymous memory use of X, the performance consequences of the memory leak would probably be undetectable. The leaked memory would just waste space in the swap partition. (This is one of the advantages on 64 bit. In 32 bit a similar scale memory leak in X would cause much worse problems.)

Your current performance should be pretty bad with the memory leak in X taking up all of the swap file plus around half of physical memory.

After you restart X, I don't know how long the performance would stay good. But with a much bigger swap partition it would be much longer.

Since you seem hesitant about restarting X now, you could add a temporary swap file and fix the performance without restarting X.

BTW, now that we see that X has only one thread, my flawed theory about the nature of the leak inside X (that I edited out above) would look even stupider. I know nothing about the internals of X, so my speculations based on your smaps output were unsound.

GrapefruiTgirl · 07-22-2010, 05:02 PM

Quote:

Originally Posted by johnsfine

Pending a correction to the likely memory leak in X, you might be a lot better off if you increase the size of your swap partition (or temporarily add a swap file).

The leaked memory in X apparently goes stale (never accessed) so it would get swapped out as the system needs ram for other processes. Now stale data from X apparently has filled the entire small swap partition so no more gets swapped out.

If your swap partition were bigger than the current anonymous memory use of X, the performance consequences of the memory leak would probably be undetectable. The leaked memory would just waste space in the swap partition. (This is one of the advantages on 64 bit. In 32 bit a similar scale memory leak in X would cause much worse problems.)

By the same token, maybe I'd be better off with no swap space at all. Then this stale/anonymous space-wasting junk would have no choice but to be "thrown away" whatever that might mean.

By rights, with 4Gib of RAM, I should have no need of a swap space at all really, and I wonder if making it bigger would only make the problem worse in the long run -- in that the kernel would eventually have a (let's say) 4 Gib swap space filled to the brim with old crap, PLUS the 4Gib of RAM filled to capacity too...

Quote:

Your current performance should be pretty bad with the memory leak in X taking up all of the swap file plus around half of physical memory.

Yeah, I dunno.. "Pretty bad" might be a bit strong - but definitely, noticeably sluggish compared to a fresh startup.

Quote:

After you restart X, I don't know how long the performance would stay good. But with a much bigger swap partition it would be much longer.

Well, I would think that, if let's say I made a 4Gib swap space, then at the time which 1Gib of swap PLUS the RAM got filled to the level that they are at this moment now, performance would be identically bad as it is right now; then, as the remaining 3Gib of swap gradually got filled, performance would continue to deteriorate. By the time 4Gib of swap got filled and 4Gib of RAM are filled, I'd be like working on a VIC-20 :/

Quote:

Since you seem hesitant about restarting X now, you could add a temporary swap file and fix the performance without restarting X.

Nah, not really hesitant - just doing a half-dozen other things (including non-computer things) during the day, and simply haven't done it. I'mma go log out right now. Back shortly.

EDIT: Oops, forgot I have slackpkg updating some packages - must wait till that's done. Will update later when it's done and I've logged out & back in, and post some of the outputs I've posted above for comparison.

EDIT 2: For the record, I have restarted Firefox a couple times and adjusted my sysctl settings, and I have somehow managed to free some swap. See here:

Code:

root@reactor: sysctl -p /etc/sysctl.conf
vm.swappiness = 10
vm.vfs_cache_pressure = 50
vm.drop_caches = 3
kernel.core_pattern = core.%e.%p
root@reactor: sync
root@reactor: free
             total       used       free     shared    buffers     cached
Mem:       4057996    2692488    1365508          0       4268      62108
-/+ buffers/cache:    2626112    1431884
Swap:      1020088     890824     129264
root@reactor:

phil.d.g · 07-22-2010, 05:18 PM

Quote:

Originally Posted by GrapefruiTgirl

By the same token, maybe I'd be better off with no swap space at all. Then this stale/anonymous space-wasting junk would have no choice but to be "thrown away" whatever that might mean.

It doesn't work quite like that, it isn't the same as the buffers or caches that can be reclaimed. The only way the space can be reclaimed is by killing the process that 'owns' the space. The kernel will start killing the most memory hungry processes once the all of RAM and swap has been consumed.

johnsfine · 07-22-2010, 05:36 PM

Quote:

Originally Posted by GrapefruiTgirl

Then this stale/anonymous space-wasting junk would have no choice but to be "thrown away" whatever that might mean.

What that means is the OOM killer. Nothing other than shutting down X will "throw away" that memory. If you don't shut down X then eventually the OOM killer will kill X. It doesn't know/care that by doing so it would trash all the applications you have open under X.

I think swapping out the stale memory is much better. Not leaking it at all would be best. But if you can't stop the leak, having swap space big enough to hold the leak is next best.

Quote:

By rights, with 4Gib of RAM, I should have no need of a swap space at all really,

I don't want to trigger a big debate on that. I have explained in many other threads why I think what you said there is not correct.

Quote:

I wonder if making it bigger would only make the problem worse in the long run

I can't see it as worse.

Without swap it slows down sooner and triggers the OOM killer sooner.

Quote:

in that the kernel would eventually have a (let's say) 4 Gib swap space filled to the brim with old crap, PLUS the 4Gib of RAM filled to capacity too.

But that would take twice as long to happen, giving you twice as much time to get around to restarting X.

Say it takes three weeks to reach OOM kill stage without a swap partition. From the very beginning of those three weeks the system is slowing down (very little at first but some slowing). By the end of three weeks it is very slow. Then it abruptly crashes (X and all applications under X are suddenly gone. That is a "crash" from the user's perspective even if not from the Kernel's perspective).

Same use but with a 4GB swap partition. For the first three weeks there is no performance loss. The swap space gradually fills. Then the second three weeks act much like the only three weeks of the previous scenario.

I already like the second choice better.

Now lets say you restart X after 2 weeks without a swap partition and after 4 weeks with a swap partition. Second choice is now much better. Three out of four weeks have no performance loss and the last week of the four is much better than the last week of the (no swap partition) two. Plus you only need to restart X half as often. How can you say that isn't better?

Quote:

I would think that, if let's say I made a 4Gib swap space, then at the time which 1Gib of swap PLUS the RAM got filled to the level that they are at this moment now, performance would be identically bad as it is right now

Wrong because (unless you keep messing around unwisely with swappiness) it would never do that. The stale memory from the memory leak would fill the swap space before it would take any memory away from active use.

I don't know how much of your 4GB of ram ever gets to "active use". So the memory leak may take significant ram before taking any swap, but only if the loss of that ram doesn't affect performance. As soon as the ram matters to performance, the stale data from the memory leak would go into swap space.

Since your swap space was fully used, we know you reached that state and we can assume you went beyond it and the leak used more ram after the swap was full.

Zero Angel · 07-22-2010, 06:55 PM

Having a bigger swap partition would only delay the problem and give her a few more days before needing a restart of X. 30 days of uptime is still not bad.

syg00 · 07-23-2010, 12:30 AM

From my experience, X is usually a victim (well, maybe willing accomplice) - think of excessive storage consumption in X as a symptom of something else going wrong. F/F is the problem in all likelihood.
X needs to maintain all the bitmaps that F/F (in this case) wants to render on the desktop somewhere. It (X) probably keeps them even when F/F is shutdown, as it probably doesn't know they can be discarded.
F/F is a pig - I tend to use Opera in preference these days.

The smaps output is merely to confirm what we already know; X is the big consumer - of swap as well as main storage. As intimated earlier, swap is probably holding all that stuff that nobody wants to see anymore - F/F cache for instance. Those entries are storage allocations - the ones with 00:00 (major, minor) are anonymous malloc'd or mmap'd. Candidates for swapping - as can be seen.

jiml8 · 07-23-2010, 01:00 AM

You haven't specified your desktop. KDE3 had some serious memory leaks, but KDE4 is a lot better.

Ktorrent has a gross memory leak if you have it set up to do IP filtering and the IP filter database is out of date.

I agree with the others; Xorg is likely a victim or willing accomplice, but most likely something else is primarily responsible.