[SOLVED] Existeth a way to 'clear out' all the old/cached crap from RAM/SWAP to free up space?

GrapefruiTgirl · 07-23-2010, 04:46 AM

@ jiml8 - you're right, initially I didn't mention my desktop, but then somewhere up there near the top, I indicated my window manager is the very tiny i3 (I don't use a DE per se) and that the only KDE things I use are Kwrite and Kmix, one or both of which do start up all sorts of daemons and services that don't go away when the application closes, leaving me to kill them when I'm done with Kwrite/Kmix. I've never used Ktorrent, and any other KDE apps I may ever use, are used very rarely. I try to avoid them

Basically it seems you & most others above are right - X is the main culprit, or at very least the scapegoat in this situation.

Quote:

Originally Posted by johnsfine

Wrong because (unless you keep messing around unwisely with swappiness) it would never do that.

@ johnsfine - if you'd care to qualify this remark with something helpful about vm.swappiness, that'd be great. We've gotten this far in a thread without snide/unqualified interjections, and I'm grateful for your input thus far. FWIW, my vm.swappiness has been =50 for about 2 years now; I changed it yesterday for the first time.

@ syg00 - I believe you're right about things like FF images/bitmaps. I definitely see the performance degradation increase after a large amount of image-filled websites have been viewed. What would be nice in this regard is some sort of settable time limit, that would say "after XYZ image has been lying around in memory/swap unused for ABC amount of time, let's dump it and free that space rather than continue to store it indefinitely."

So... It took MUCH longer than anticipated for slackpkg to update my system last night (due to my ^$#&@*% wireless internet connection being near unusable) I STILL have not logged out, but will be doing so shortly. Even though I doubt we'll see anything surprising (that we aren't expecting) when I run some of the commands I have run above, after logging back in with a fresh X, I'll post the outputs anyhow, so we can see the baseline of a fresh X session - OR, if that's not much different, I'll reboot and do it then.

Thanks for the input everyone!

johnsfine · 07-23-2010, 06:27 AM

Quote:

Originally Posted by GrapefruiTgirl

if you'd care to qualify this remark with something helpful about vm.swappiness

Something helpful about swappiness: Leave it at the default.

Quote:

We've gotten this far in a thread without snide/unqualified interjections

No insult was intended. I just wanted to qualify my own statement about the helpfulness of adding more swap space for your problem. That depends on a reasonable value in swappiness. The value you lowered it to is approaching unreasonable for that purpose.

Quote:

my vm.swappiness has been =50 for about 2 years now

Great.

Quote:

I changed it yesterday for the first time.

Not so great.

GrapefruiTgirl · 07-23-2010, 06:31 AM

So, I've just logged out of X and back in. Memory consumption shows as minimal, and swap appears to have been 'refreshed'. Here's some outputs comparable to those I posted earlier when everything was maxed out:

Code:

root@reactor: free
             total       used       free     shared    buffers     cached
Mem:       4057996     478128    3579868          0      41036     126000
-/+ buffers/cache:     311092    3746904
Swap:      1020088       5560    1014528
root@reactor:

root@reactor: cat /proc/slabinfo
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
nv_stack_t            39     40  12288    2    8 : tunables    0    0    0 : slabdata     20     20      0
kmalloc_dma-512       16     16    512   16    2 : tunables    0    0    0 : slabdata      1      1      0
kcopyd_job             0      0    368   22    2 : tunables    0    0    0 : slabdata      0      0      0
dm_uevent              0      0   2608   12    8 : tunables    0    0    0 : slabdata      0      0      0
dm_rq_target_io        0      0    376   21    2 : tunables    0    0    0 : slabdata      0      0      0
dm_io                  0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
cfq_io_context       103    150    136   30    1 : tunables    0    0    0 : slabdata      5      5      0
cfq_queue             94    124    264   31    2 : tunables    0    0    0 : slabdata      4      4      0
bsg_cmd                0      0    312   26    2 : tunables    0    0    0 : slabdata      0      0      0
mqueue_inode_cache      1     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
udf_inode_cache        0      0    616   26    4 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    592   27    4 : tunables    0    0    0 : slabdata      0      0      0
fat_inode_cache       26     26    624   26    4 : tunables    0    0    0 : slabdata      1      1      0
fat_cache            102    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
journal_handle       340    340     24  170    1 : tunables    0    0    0 : slabdata      2      2      0
journal_head          73    108    112   36    1 : tunables    0    0    0 : slabdata      3      3      0
revoke_table         260    512     16  256    1 : tunables    0    0    0 : slabdata      2      2      0
revoke_record        256    256     32  128    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_inode_cache    2354   4212    888   18    4 : tunables    0    0    0 : slabdata    234    234      0
ext4_free_block_extents    146    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_alloc_context     56     56    144   28    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_prealloc_space     78     78    104   39    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_system_zone       0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
ext2_inode_cache      22     44    728   22    4 : tunables    0    0    0 : slabdata      2      2      0
ext3_inode_cache       0      0    744   22    4 : tunables    0    0    0 : slabdata      0      0      0
ext3_xattr             0      0     88   46    1 : tunables    0    0    0 : slabdata      0      0      0
configfs_dir_cache      1     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
kioctx                 0      0    320   25    2 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark_entry    199    252    112   36    1 : tunables    0    0    0 : slabdata      7      7      0
posix_timers_cache     29     56    144   28    1 : tunables    0    0    0 : slabdata      2      2      0
UDP-Lite               0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
ip_dst_cache          42     42    384   21    2 : tunables    0    0    0 : slabdata      2      2      0
RAW                   26     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
UDP                   42     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP           21     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                   42     42   1536   21    8 : tunables    0    0    0 : slabdata      2      2      0
blkdev_queue          29     30   2176   15    8 : tunables    0    0    0 : slabdata      2      2      0
blkdev_requests       82     96    336   24    2 : tunables    0    0    0 : slabdata      4      4      0
fsnotify_event_holder    340    340     24  170    1 : tunables    0    0    0 : slabdata      2      2      0
fsnotify_event        78     78    104   39    1 : tunables    0    0    0 : slabdata      2      2      0
sock_inode_cache     162    225    640   25    4 : tunables    0    0    0 : slabdata      9      9      0
skbuff_fclone_cache     36     36    448   18    2 : tunables    0    0    0 : slabdata      2      2      0
file_lock_cache       44     44    184   22    1 : tunables    0    0    0 : slabdata      2      2      0
net_namespace          0      0   1472   22    8 : tunables    0    0    0 : slabdata      0      0      0
shmem_inode_cache    873    880    736   22    4 : tunables    0    0    0 : slabdata     40     40      0
Acpi-ParseExt       1578   1624     72   56    1 : tunables    0    0    0 : slabdata     29     29      0
Acpi-Parse           170    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
proc_inode_cache     656    675    592   27    4 : tunables    0    0    0 : slabdata     25     25      0
sigqueue              50     50    160   25    1 : tunables    0    0    0 : slabdata      2      2      0
bdev_cache            37     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
sysfs_dir_cache    10223  10251     80   51    1 : tunables    0    0    0 : slabdata    201    201      0
inode_cache          226    270    544   30    4 : tunables    0    0    0 : slabdata      9      9      0
dentry              5525   7077    192   21    1 : tunables    0    0    0 : slabdata    337    337      0
radix_tree_node     2419   7424    560   29    4 : tunables    0    0    0 : slabdata    256    256      0
buffer_head        12018  12132    112   36    1 : tunables    0    0    0 : slabdata    337    337      0
vm_area_struct      2410   2424    168   24    1 : tunables    0    0    0 : slabdata    101    101      0
files_cache          169    207    704   23    4 : tunables    0    0    0 : slabdata      9      9      0
signal_cache         174    247    832   19    4 : tunables    0    0    0 : slabdata     13     13      0
sighand_cache        119    150   2112   15    8 : tunables    0    0    0 : slabdata     10     10      0
task_struct          136    198   1488   22    8 : tunables    0    0    0 : slabdata      9      9      0
anon_vma             983   1280     32  128    1 : tunables    0    0    0 : slabdata     10     10      0
idr_layer_cache      283    390    544   30    4 : tunables    0    0    0 : slabdata     13     13      0
kmalloc-8192          39     40   8192    4    8 : tunables    0    0    0 : slabdata     10     10      0
kmalloc-4096         722    744   4096    8    8 : tunables    0    0    0 : slabdata     93     93      0
kmalloc-2048         252    272   2048   16    8 : tunables    0    0    0 : slabdata     17     17      0
kmalloc-1024         279    320   1024   16    4 : tunables    0    0    0 : slabdata     20     20      0
kmalloc-512          836    848    512   16    2 : tunables    0    0    0 : slabdata     53     53      0
kmalloc-256          955   1136    256   16    1 : tunables    0    0    0 : slabdata     71     71      0
kmalloc-128          904   1120    128   32    1 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-64          2065   6080     64   64    1 : tunables    0    0    0 : slabdata     95     95      0
kmalloc-32          3823   3840     32  128    1 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-16          2249   2304     16  256    1 : tunables    0    0    0 : slabdata      9      9      0
kmalloc-8           3518   3584      8  512    1 : tunables    0    0    0 : slabdata      7      7      0
kmalloc-192         2058   2331    192   21    1 : tunables    0    0    0 : slabdata    111    111      0
kmalloc-96           729    756     96   42    1 : tunables    0    0    0 : slabdata     18     18      0

sasha@reactor: ps -eo pid,user,rss,nlwp,comm --sort=-rss | head -n 10
  PID USER       RSS NLWP COMMAND
19406 root     125128   1 X
20478 sasha    121596  13 firefox-bin
22585 sasha     4232    1 bash
22476 sasha     4140    1 xterm
25776 sasha     3116    1 bash
19478 sasha     2924    1 dzen2
19481 sasha     2920    1 dzen2
19487 sasha     2480    1 xscreensaver
  620 82        2060    1 hald

Unless someone specifically wants to see the output of the script I used earlier in post#11, I won't post that, since it just expands upon the known fact that X appears to be the main consumer.

GrapefruiTgirl · 07-23-2010, 06:38 AM

Thanks johnsfine for the clarifications re: swappiness. I've moved it back to 50 since when my automatic daily backup began this morning, the desktop became near unusable :P -- putting swappiness back where it had been for so long, instantly revived the desktop.

For the time being I'm not going to increase the swap space - I'll save that experiment for (A) after I look at the HDD and see what space there is to expand it into, and (B) after I examine (C) below:

(C) Next plan: try to upgrade my Xorg systems to the current Slackware64-current version. If that actually works, I'll begin to examine this entire situation anew, and see how the gradual consumption of RAM/SWAP compares (roughly) to how it is on my present version of Xorg. I don't expect any miracle, but who knows.. Maybe they fixed some leaks

If the newest Slack64 Xorg does NOT work still for me, I'll look at increasing swap space. LOL, stay tuned for the next few months -- I'll need another 30-90 days of uptime for a comparison

johnsfine · 07-23-2010, 07:29 AM

Quote:

Originally Posted by GrapefruiTgirl

I've moved it back to 50 since when my automatic daily backup began this morning, the desktop became near unusable :P -- putting swappiness back where it had been for so long, instantly revived the desktop.

That is not at all what I expected. Enough so that I wonder if there was some coincidence that something else happened at the moment you changed swappiness.

Your swap partition size is low enough that an increase in swappiness should not be able to make any dramatic improvement.

A daily backup is the kind of operation where I would expect the system to feel more responsive with a lower value of swappiness.

The major impact of swappiness is on the relative memory priority of anonymous vs. non anonymous memory. Simplistically: the anonymous memory is the stack and heap of all your processes, and non anonymous memory is the code of all your processes plus the files recently read or written by your processes.

Normally anonymous has a moderate priority over non anonymous. If you set swappiness lower, that relative priority increases, so that stale anonymous data (such as that accumulating inside X) has priority over the code of running programs. At a normal swappiness setting the stale vs. active difference overwhelms the moderate relative priority of anonymous.

A daily backup might be expected to read and write a lot of files, so recently copied files would be active non annonymous data and would take up memory forcing inactive anonymous data to the swap partition. But some of that inactive anonymous data is in GUI programs that are waiting for the next mouse click. When that next mouse click happens such programs would take longer to get back to normal speed than they would have with a low swappiness setting. That is the reason many people prefer a low swappiness setting. Overall it makes performance and throughput worse, but in that specific very noticeable case it makes responsiveness better.

GrapefruiTgirl · 07-23-2010, 07:35 AM

Quote:

That is not at all what I expected. In fact I think there was some coincidence that something else happened at the moment you changed swappiness.
...
A daily backup is the kind of operation where I would expect the system to feel more responsive with a lower value of swappiness.

Sentence 2 above is precisely what I was thinking when I tried to change it to 10; I had hoped for a "general" improvement in responsiveness (EDIT: That's not to say that responsiveness is generally a problem, but I thought what the heck, let's see what this does! ).

Maybe there was just some coincidence at the same time, but I don't know what it may have been..

1 - I was using browser.
2 - mouse cursor stopped responding for seconds at a time, browser wouldn't scroll, I thought ??wth??
3 - noticed the time of day & the backup drive was mounted - Aha! That's the problem, but *normally* it doesn't affect desktop responsiveness.
4 - zapped vm.swappiness back to 50
5 - instant recovery back to normal responsiveness.

FWIW - The backup started before I logged out and cleaned out all the junk; and, the backup happens via rsync, if that makes any difference.

markseger · 07-23-2010, 08:45 AM

I didn't want to disappoint 'syg00', so I'll chime in a little

To see the top 10 processes sorted by rss and show ALL the types of memory usage try this:

collectl --top rss,10 -c1 --procopt m

20282 root S 179M 0 21816K 37908K 332K 32K 12320K 0 15 /usr/sbin/snmpd
11493 root R 109M 0 17948K 16136K 96K 12K 3820K 0 22 /usr/bin/perl
3440 root S 111M 0 17404K 16404K 88K 12K 3860K 0 0 /usr/bin/perl
28833 root T 108M 0 7216K 760K 328K 1556K 5164K 0 0 emacs
3490 root S 24960K 0 5812K 11156K 84K 1572K 10408K 0 0 /opt/hp/vcagent/bin/vcagentd
2507 ntp S 23384K 23384K 5020K 788K 84K 460K 3388K 0 0 ntpd
3312 haldaemo S 30984K 0 3908K 2376K 84K 244K 3564K 0 0 hald
2763 hpsmdb S 47912K 0 3896K 668K 92K 3064K 4612K 0 0 /opt/hpsmdb/bin/postmaster
23140 root S 88220K 0 3380K 752K 84K 364K 6308K 0 0 sshd:
22965 root S 88220K 0 3372K 752K 84K 364K 6308K 0 0 sshd:

of course if you leave off the count switch, -c1, it will run continously and if you leave off the option to display memory usage, --procopt m, you'll see 'standard' top format.

If people are interested in slabs, you can use --top with a sort field that shows slab data (see collectl --showtopopt) or just run:

collectl -sY -i:1 --slabopt S

to see slabs as they change (and ONLY those that change). for more options see "collectl --showsubopt"

syg00 - happy now?

-mark

GrapefruiTgirl · 07-29-2010, 12:46 PM

Since last posting in this thread, I have been using drop_caches = 3 to dump all the crap when memory gets full. I have it as a setting in /etc/sysctl.conf but it doesn't seem to actively make a difference during the day, just from being there. I have to manually apply it any time I want to 'clear the crap'. No big deal - I figure I'll stick an entry into a daily (or nightly, when nothing's going on) cron script, as follows:

Code:

#!/bin/sh
sync; sync;
sysctl -w vm.drop_caches=3

And that will give me a fresh start each day.

At this time I don't think I'll get too involved with collectl - but will keep it in mind for future use, should I find a need for it.

Thanks for everyone's input!