[SOLVED] Background Marathon Runner

Rather than add/edit my last post, here's the top P/O when both Iceweasel and LibreOffice are closed.

Why is more than half of the KiB Mem (288/505 or 52%) 'used'?

top - 12:43:39 up 2 days,  7:56,  4 users,  load average: 0.31, 0.84, 1.11
Tasks: 129 total,   1 running, 128 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.3 us,  1.0 sy,  0.0 ni, 94.0 id,  2.7 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    505432 total,   288468 used,   216964 free,     4176 buffers
KiB Swap:   498008 total,    97588 used,   400420 free,   180432 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                      
 3538 rooster   20   0  115m  13m 9620 S   1.0  2.7  17:41.64 konsole                      
 2432 root      20   0 34488  12m 4880 S   0.3  2.5  67:52.91 Xorg                         
 3448 rooster   20   0  319m  11m 4976 S   0.7  2.2  20:46.82 gnome-panel                  
 3488 rooster   20   0 69076 7832 2184 S   0.0  1.5   0:05.00 tracker-store                
 3439 rooster   20   0  357m 5608 3552 S   0.0  1.1   4:55.62 metacity                     
 3475 rooster   39  19 68164 4340 2284 S   0.0  0.9   0:02.60 tracker-miner-f              
 3481 rooster   20   0  275m 4220 3144 S   0.0  0.8   0:01.64 notification-da              
 3472 rooster   20   0  514m 3980 2832 S   0.0  0.8   0:00.86 nm-applet                    
 3404 rooster   20   0  152m 3916 2816 S   0.0  0.8   1:09.82 gnome-settings-              
 3473 rooster   20   0  334m 3900 2800 S   0.0  0.8   0:01.01 gnome-sound-app              
 3419 rooster    9 -11 99168 3592 2432 S   0.0  0.7   3:19.34 pulseaudio                   
 4543 rooster   20   0  195m 2880 2220 S   0.0  0.6   0:04.14 knotify4                     
 4308 rooster   20   0 99540 2760 2136 S   0.0  0.5   0:06.64 kded4                        
 3521 rooster   20   0  276m 2608 2132 S   0.0  0.5   0:10.99 gnome-screensav              
12438 root      20   0  7860 2600 2108 S   0.0  0.5   0:00.02 sshd                         
 3487 rooster   20   0  276m 2284 1920 S   0.0  0.5   0:00.74 polkit-gnome-au              
 3484 rooster   20   0 39328 2160 1460 S   0.0  0.4   0:09.85 system-config-p              
 3341 rooster   20   0 49304 2128 1668 S   0.0  0.4   0:01.09 x-session-manag              
 3471 rooster   20   0  274m 1752 1504 S   0.0  0.3   0:00.38 bluetooth-apple              
 3476 rooster   20   0 59828 1728 1476 S   0.0  0.3   0:00.80 evolution-alarm              
rooster@royrogers:~$

Originally Posted by r00ster Obs 1: Memory; @ Dec 5.13 'top' shows Iceweasel using 35% and LibreOffice 16%. Total 51%. Total Mem: 505432 total, 453688 used, 51744 free, 5796 buffers This shows 90% of mem is 'used'. What is using the 39% (90% - 51%)?

Well there's a lot (probably) of other processes running on the system. Make sure 'top' or 'htop' isn't limited to displayed only your processes.

Thanks to your (mturn) prompt in the previous post... Obs 2: ~$ ps -efL | grep firef | grep -v grep | wc -l 0

That command line snippet was supposed to count the Firefox threads. I'm not surprised it returned "0" if you're using IceWeasel. Try changing the first 'grep' to: "grep weasel" (or whatever IceWeasel" shows up as in 'ps' when it's running) and see what is returned when you've got your normal set of tabs open.

Obs 3: After writing to LibreOffice for an hour or so, its Mem dropped to 6%. while Iceweasel went from 35% to 40%. Although I don't know what the output means: rooster@royrogers:~$ ps -efL | grep libreoffice | grep -v grep | wc -l 8 rooster@royrogers:~$ ps -efL | grep soffice | grep -v grep | wc -l 5

Beats me. To know that, you'd probably have to dig around on the developer forums for those applications and look for how they use threads. My original intent with that set of piped commands was to find a metric that I could use to predict when Firefox performance was likely to start getting bad. I'm not sure it correlates really well with FF tab count or jumpy scrolling except that as the count goes up, performance tends to get worse. But... I have noticed something about one web site that I frequent: it never seems to finish loading -- it would stay in a "loading" state with the little orange-red swirly icon spinning forever -- until I get fed up and hit Esc. Until I do that FF scrolling lags quite a bit and is very jumpy. I haven't been able to figure that one out yet. I noticed that as I have been typing this reply, the thread count for FF had been fluctuating and I haven't opened or closed any new tabs. There are, apparently, things happening in the background that I don't have much control over that are causing that.

Obs 4: Mem: Icedove. [snip] Significance? Obs 5: 'top' time columns for Iceweasel, ksysguard and xorg seem out-of-whack. Current session 'uptime' , 1h 3min. Iceweasel shows 2h 25m. Ksysguard shows 3h 52m. 'xorg shows 3h 6min. Significance?

No idea.

Obs 6: the number of system freezes has diminished in the last month or so. If memory serves, a significant percentage of sys freezes involved scrolling both in Iceweasel and LibreOffice. A smaller, but noticeable percentage occurred during start-up while Iceweasel and LibreOffice were loading. Googling I found... Obs 7 Even after closing Iceweasel (not killing) I see: KiB Mem: 505432 total, 498476 used, 6956 free, 1504 buffers 3502 36.6 1061m 36m 180m 104 658m 20m 48k 0 S 20 0 0.0 iceweasel

So you've exited and it's still in memory (as far as top is showing)? Does it eventually disappear from the list of processes?

Note 2: I have 2 disks in the box. 80GB W-XP LAN Disabled (Not connected to Internet) 160GB HD Separate Partitions for Squeeze & Wheezy. Note 3: Backups: I BU my DOCS, Boot and 'etc' folders to the 80GB HD and to Removable DVD.

No comment other than to say: a.) my compliments on keeping yourself safe when running Windows by not connecting it to the Internet and b.) since I no longer have Windows on my systems I know what I'd use that 80GB disk for.

--
Rick

Rick;

Originally Posted by mturn Well there's a lot (probably) of other processes running on the system. Make sure 'top' or 'htop' isn't limited to displayed only your processes.

You'll have to clarify what you mean by “running”. As you can see, my 'top' PO only quants 1 – 4 tasks; depending. But no more than that. Looking over the entire 'top' output in standard mode (not the 4 window analytical mode) showing all the apps, they all show less than 1% mem. Nowhere nearly enough to account for the 288468 KiB Mem usage.

'htop' isn't a recognized command on my sytem.

Conceding that my brain still thinks in analog mode, it seems to me a bunch of stuff might be getting loaded into memory during startup that perhaps shouldn't be. Seeing 52% mem used when only a couple of bash windows are 'running' makes me think my problem might be what processes/applications etc., are being loaded at start-up.

It would be interesting to compare what %Mem is being used by other Wheezy/Deb7 desktoppers when only 1 bash/konsole window is running.

Originally Posted by rnturn Try changing the first 'grep' to: "grep weasel"...

~$ ps -efL | grep iceweasel | grep -v grep | wc -l
33

Originally Posted by rnturn So you've exited and it's still in memory (as far as top is showing)? Does it eventually disappear from the list of processes?

Doh! My bad. I didn't realize I had 'interrupted' that 'top window.

Originally Posted by rnturn ...since I no longer have Windows on my systems I know what I'd use that 80GB disk for.

Yeah. I have friends and relatives who use MS. Also gov agencies and other institutions can be obstreperous WRT what program(s) they accept. The doc converter in Deb does a poor job; esp when forms and images are involved. When sending/attaching docs, it's easier for them if I use MS Office. 'Chacun à son goût'. Also, since I no longer run anti-malware & all the etc. programs associated with security, XP runs like Usain Bolt with a tail wind.

Added by edit:
With Iceweasel “killed”, whopping great drop in Mem used.

KiB Mem:    505432 total,   158700 used,   346732 free,      896 buffers
KiB Swap:   498008 total,   223260 used,   274748 free,    71604 cached

Within a minute, without doing anything except watching 'top', Mem used went (back) up to:

KiB Mem:    505432 total,   254800 used,   250632 free,     1948 buffers
KiB Swap:   498008 total,   204848 used,   293160 free,   140228 cached

That's almost 100 MB being used by something without any input.
When I restarted Iceweasel, Mem usage went back up to:

KiB Mem:    505432 total,   473736 used,    31696 free,    10440 buffers
KiB Swap:   498008 total,   193976 used,   304032 free,   222452 cached

That's almost 220 MB increase over when Iceweasel wasn't 'running' or 315 MB over and above when Iceweasel was a fresh 'kill'.

FYI:

~$ ps -efL | grep iceweasel | grep -v grep | wc -l
27

I'm thinking some application or process is getting loaded into Mem that is 'supposed' to get reniced; i.e., freed up; but isn't.

rooster

R00ster,

Here's an article that you might find interesting.

http://l3net.wordpress.com/2013/12/0...ry-comparison/

It's not exactly the same situaton as you but it does seem to show an ugly trend. Not too encouraging for folks with low resource (esp., memory) systems.

--
Rick

I came across another shepherd having their RAM being shorn. They cited a blog that informed me about the command:

~$ cat /proc/meminfo

I ran it a couple of times and noticed behaviour at 2 lines in the list that bear interpretation by someone smarter than me.

First run gave:

CommitLimit:   750724 kB
Committed_AS:   51560 kB

Second run, not having done anything significant ASAIK. gave:

CommitLimit:     750724 kB
Committed_AS:   1792720 kB

Does this give any indication as to what is going on with %Mem usage?

FYI, throwing spaghetti at the wall to see if it sticks, I also ran:

~$ vmstat -s
       505432 K total memory
       489164 K used memory
       219192 K active memory
       232948 K inactive memory
        16268 K free memory
         1316 K buffer memory
       108532 K swap cache
       498008 K total swap
       167020 K used swap
       330988 K free swap
      2850847 non-nice user cpu ticks
         5001 nice user cpu ticks
       951762 system cpu ticks
     41073667 idle cpu ticks
       435675 IO-wait cpu ticks
           14 IRQ cpu ticks
        27218 softirq cpu ticks
            0 stolen cpu ticks
     15535867 pages paged in
     13964752 pages paged out
       772295 pages swapped in
       760545 pages swapped out
     60002953 interrupts
    199320347 CPU context switches
   1386247615 boot time
        25219 forks

Further FYI: Running 'atop -m', the following line presents in RED. The letters 'SWP' are flashing. This would seem to indicate a problem; eh?

SWP |  tot   486.3M  |  free  251.3M  |                 |  vmcom   1.8G  |  vmlim 733.1M

I also have Output from 'cat proc/slabinfo'. It's too large to include here. But if anyone thinks they can interpret it, or if they think it would be useful, we can make arrangements; I'm sure.

Originally Posted by r00ster ~$ cat /proc/meminfo I ran it a couple of times and noticed behaviour at 2 lines in the list that bear interpretation by someone smarter than me. [snip] Does this give any indication as to what is going on with %Mem usage?

I'm not familiar with the quantities being displayed in `meminfo'. Perhaps digging^Wgrepping around in the kernel sources and the documentation would provide some insight. That's sometimes helped me understand where unusual error messages came from in the past.

Further FYI: Running 'atop -m', the following line presents in RED. The letters 'SWP' are flashing. This would seem to indicate a problem; eh? SWP | tot 486.3M | free 251.3M | | vmcom 1.8G | vmlim 733.1M

I don't have an `atop' command so I can't tell you what that means. Maybe the man page for atop (or the developer's web site) explain why they decided to make that blink.

--
Rick

Rick:

Originally Posted by rnturn I'm not familiar with the quantities being displayed in `meminfo'. Perhaps digging^Wgrepping around in the kernel sources and the documentation would provide some insight.

That's a 'con trail' to me. (Way over my head) I'd need command strings. AFAICT, the "quantities" are in kB.
/proc/meminfo

Originally Posted by rnturn I don't have an `atop' command so I can't tell you what that means.

Originally Posted by man 'atop' <SNIP>When atop is started, it switches on the process accounting mechanism in the kernel. This forces the kernel to write a record with accounting information to the accounting file whenever a process ends. Whenever the last incarnation of atop stops (either by pressing `q' or by `kill-15'), it switches off the process accounting mechanism again. You should never terminate atop by `kill -9', because then it has no chance to stop process accounting; as a result the accounting file may consume a lot of disk space after a while. With the environment variable ATOPACCT the name of a specific process accounting file can be specified (accounting should have been activated on beforehand). Whenthis environment variable is present but its contents is empty, process accounting will not be used at all. Notice that root-privileges are required to switch on process accounting in the kernel. You can start atop as root or specify setuid-root privileges to the executable file. In the latter case, atop switches on process accounting and immediately drops the root-privileges again.<SNIP> <SNIP>For the resource consumption on system level, atop uses colors to indicate that a critical occupation percentage has been (almost) reached. A critical occupation percentage means that is likely that this load causes a noticable negative performance influence for applications using this resource. The critical percentage depends on the type of resource: e.g. the performance influence of a disk with a busy percentage of 80% might be more noticable for applications/user than a CPU with a busy percentage of 90%.<SNIP>

Overwhelmed I am at the power of both 'atop' and 'htop'. I'm struggling to understand how I can use them WRT my current predicament; ... at least well enough to ask intelligent questions. In an ideal world (mine that is) someone would ask for the output of a command string or file and that would lead to an understanding as to what my options are.

rooster

Originally Posted by r00ster AFAICT, the "quantities" are in kB. /proc/meminfo

I'm not sure about you but I got less than I hoped from reading that description.

I'm sure there's a tutorial out there (somewhere) that tells you how to tune the kernel to adjust those meminfo parameters to fit your workload. Here's a link to an IBM Linux monitoring/tuning guide that's not too bad:

http://www.redbooks.ibm.com/redpapers/pdfs/redp4285.pdf

Might be of some help, especially sections 3.3 and 4.5. Downside: It's a little dated. And it tends to refer to parameters that it doesn't define or even reference in the index. And, it's not a cookbook. You won't find a simple ``change this to achieve memory use nirvana'' command. As with any tuning, you make small changes (10X is pretty much never appropriate), measure, evaluate, and, maybe, change again. Repeat until the evaluation shows little to no improvement (or maybe even a degradation; then you back the parameter value off a bit). Then do it all over again when you change the system.

Bottom line: You're running some applications that are known to be pretty large memory users (LO/OO, FF, etc.). Short of shutting them down after each use to recover RAM, I suspect your best bet is to bite the bullet and add extra RAM. The only concern I might have in this situation is beefing up a system that's on the old side. I'd go for it just because I tend to find other uses for older systems (server in the home network, etc.) but I would still struggle a bit on spending the cash on memory for an older system.

Aside: One thing you might find useful is to grab the output of `vmstat' (or values ut of /proc but this takes a more complex script) for a time window during which you run some application. Give enough time before and after you run the application to give you an idea of what the unloaded system looks like. Then you can extract the various parameters and plot them (I'm not as good at picking out behavior from several long columns of numbers as I used to be) to see what happens when you run a particular application. I used to write custom extract scripts and plot using `gnuplot' but have been using `R' for this nowadays. There are almost certainly other nifty plotting routines out there you could use.

someone would ask for the output of a command string or file and that would lead to an understanding as to what my options are.

As with many UNIX/Linux things, it takes a good deal of knowledge of the underlying system/subsystems and how they work to ferret out why a piece of code is acting the way it is. I've resorted to running applications via `strace' (or `truss') in order to figure out just what file was not being found when the only clue was a generic `File not found' message (or even nothing at all) or to learn that the manpage was out-of-date and that my config file was in the wrong directory so that my changes were never to be reflected in the way the darned app ran. Can't blame these misunderstanding on Linux programmers; the commercial UNIX vendors are just as bad if not worse.

--
Rick

Rick;

Originally Posted by rnturn As with many UNIX/Linux things, it takes a good deal of knowledge of the underlying system/subsystems and how they work to ferret out why a piece of code is acting the way it is. I've resorted to running applications via `strace'...<SNIP>

Indeed. At time of writing, I'm trying to get a handle on things by studying:
http://lwn.net/Articles/387202/
and
http://www.atoptool.nl/download/case_leakage.pdf

Some 'data-bath' for sure. Using the above links, I'm trying to reconcile the output of 2 'top' tools WRT Mem.

From 'atop -r':

MEM |  tot   493.6M  |  free  391.8M  |   cache  48.6M  |  buff   26.7M  |  slab   10.1M 
SWP |  tot   486.3M  |  free  486.3M  |                 |  vmcom  84.2M  |  vmlim 733.1M

From 'top':

3:Mem - 14:24:30 up  2:55,  3 users,  load average: 0.11, 0.45, 0.53
Tasks: 131 total,   1 running, 130 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.3 us,  7.0 sy,  0.0 ni, 79.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    505432 total,   475508 used,    29924 free,     2420 buffers
KiB Swap:   498008 total,   186004 used,   312004 free,   157408 cached

'atop -r' shows 391.8M/493.6M as 'free'.
'top' shows 29924/505432 free.

Time for 2 fingers of McClelland's; ...eh?

'Och! aye the nonny noo, brough laddie. Pray bide the while...' It's going to take a few days to assimilate what I can from the info I have.

Y'All;

I've learned as much from this thread as I have from anything I've queried before. Thank y'all very much.

Using 'top', 'atop' and 'htop' and analyzing their outputs over a few days led me to conclude my best option is to invest in a 'up-to-date' desktop. I can mitigate the 'Mem' issue by adding memory, but the 'CPU' doesn't have the horsepower to keep up with traffic on today's Internet highway. Just running a HD
Youtube video maxes it out. And 'google earth' is sooo out of the question.

Of course, it's been made clear to me this means I'll have to make some choices; e.g., between a new computer and the missus. I'm going to miss her cooking.

Best of the Season to y'all.

rod

Originally Posted by r00ster Just running a HD Youtube video maxes it out. And 'google earth' is sooo out of the question.

Yeah I find the same thing. Even with hardware acceleration I have trouble watching videos unless I opt for the less-than-HD versions. My biggest trouble is due to the buffering needed because of our less than stellar bandwidth; playback is too jerky. I tend to use 'youtube-dl' to grab the video in th4e background and avoid the buffering but it doesn't work for all sites.

Of course, it's been made clear to me this means I'll have to make some choices; e.g., between a new computer and the missus. I'm going to miss her cooking.

Don't do anything hasty, now...

Best of the Season to y'all.

Back at ya.