Attachment 1200

The above file is twice as long as the post limit, so I've attached it in case anyone would like to have a look.

System:

Slackware64-current
Ext4 filesystem
XFCE desktop, using KDE stuff like Konq, Kwrite, etc.
Tainted: (probably) the nVidia driver (it's the only one).
Kernel: 2.6.30.4
Memory: 4Gib installed
VM_Swappiness: 50%


Scenario:

Was checking/reading some logs in /var/log/ using Konqueror, and decided to open one that was about 1Gib in size. Generally this is not a problem, never has been anyways on Slack 11.

It was seeming to be laggy & slow, trying to scroll down the log file. I decided to close the Kwrite session, highlight the log file, and proceeded to Edit > Delete the file.

At this point, Firefox died (OOM-killed) and my XFCE-Panel died too (OOM-killed) and as you may see in the attached file, most of what was running, including system processes, got killed, one after another. Meanwhile, the HDD was running full tilt.

From this point on, system got slower and slower... I clicked over to an empty VT, which took about 3 minutes to complete. Another few minutes to get a prompt after logging in. Another few minutes to get `top` running, upon which I noticed `kwrite` process consuming ~95% of the CPU, and Konq consuming about 10% of the CPU.
I killed both processes with `kill -9` after which the system returned to normal. The HDD light went out.

Then I ran `dmesg` and copied everything from the "normal" state, to the end of the "borked" state, and placed it in the file I have included here.

I then went back to VT7, and found a `Kwrite` running, but with no window border, and an error Pop-up from XFCE telling me that "The Task-Panel has lost it's contents" which indeed it had; there was no panel at all. I had to log out and back in to restore normalcy to XFCE.

Myself, I don't know enough to make heads nor tails of the stuff in the attached file. I haven't a clue where I might send it. I mean, can anyone tell if it's a kernel issue, an app issue, a KDE/XFCE issue, or who might be interested in this back-trace sort of thing? Since a whack of stuff got killed, I don't know what initially caused the problem.

If more information is required for any interested party to try to figure out what happened, just ask..


Sasha

Here's a possibly related tidbit:

I'm used to seeing (when I look in kinfocenter) a large amount of memory being used as "disk cache" which apparently means the system is well configured, according to the pop-up one gets when hovering over the "Physical Memory" block of kinfocenter.

Looking at kinfocenter right now, I see:

4.13 Gib free of Total Memory
3.18 Gib free of Physical Memory
0.978 Gib free of swap (98% free)

Why is this? And might there be a configuration issue here, whereby my memory isn't being used properly?

Sasha

Try to open the same size, or close to it, file with kwrite from fluxbox through xterm, with the nv driver. (Rules out nvidia, konq, xfce and all the other stuff )

Has happened to me a few times. I've gone back to editing large files with nano.

Sitting idle Kwrite uses 8MB of ram. Opening a 17MB log file Kwrite uses 70MB of ram, start to scroll, move around, or edit the file, and the foot print starts to balloon. It peak at 312MB before I closed it. 1024 / 17 = ~60. A 1gig file might need up-to 312MB * 60, or somewhere around 18,720MB of ram.

That's BRUTAL! I will try this and report back. It'll take a bit of time, I may need to alter my xorg.conf (or temporarily *delete* it) to use the `nv` driver, as I use multiple monitors, and `nv` doesn't like that much in my experience.

Will try though.
Sasha

OK.. Here's what's happening:


Running Fluxbox & the `nv` driver.
Made a 1.1Gib text file.
Opened an Xterm, ran `nano gigtest.txt` and waited about 10 seconds or so for it to fully load, which it did successfully.

I watched in `top` while nano was loading the file; it took about 90% CPU while it was loading, then dropped to 0% when loaded.

I am currently here on LQ using Firefox from this Fluxbox session (kinda cool, minimally)..

Before I logged out, I tested Konq/Kwrite again: I loaded a 212Mib file into Kwrite. Then, I copied & pasted the same file onto itself, as a brand new file; and before the file was even fully pasted into Kwrite, the system started getting laggy, windows were not updating, etc.. So I closed Kwrite.

Anyhow, here I am in Fluxbox now; Here's the full stats:

Driver=nv
Session=Fluxbox
Editor=nano
File=1.1Gib text.

`top` shows:
Kinfocenter shows:Sasha

Looks like Kwrite is a pig.

Your second Kinfo seems ok (but what do I know ) On your initial kinfo post, also seems normal with a recently relaunched system, or one that hasn't had much activity. Normally after 6 or 7 days, my disc cache runs 90%+ of available ram. After the odd restart/Xorg crash, it goes back to ~10% cache until I began to work again.

Well.. Still, I can understand *some* overhead for the editor, but if any of that `top` info is accurate at all, even nano is still using ~twice the amount of resources as the actual size of the file loaded.

Why so much overhead?

It's the nature of what happens when an editor displays text files. It needs to use ram to display the actual text, load commands .....

If you can imagine gzipping a text file, once compressed it's quite small. A text file is not an uncompressed document to the text editor. A text editor needs to uncompress the file to display the contents. Each character takes a little bit of space. The more characters, the more space. Even if all of the lines are not displayed at once, quite a bit is buffered.

Think of looking at a single jpeg picture. A 282kb picture needs more than that much ram to actually draw the picture. If you open 100 282kb jpeg's it will take a heck of alot more ram than 30MB. Granted text files don't have near the overhead of pictures - but you get the idea.

Using an X editor will use even more RAM than nano, pico, elvis ... because it has to use a font rendering engine which then draws the glyphs on screen.

Thank you for your insight & help here, disturbed1, and also for giving me a reason to try Fluxbox (again) -- I have been trying to pry myself off of KDE for some time (thinking about it, you know ) and since S64-current, have been using XFCE because I don't care for the new KDE (yet?) but Fluxbox is nice and small and quick, so maybe I'll spend some more time with it and see if I can get comfy with it.

If anyone else has input about this all, please feel welcome to add.

Meanwhile, (after supper methinks) I'll get my real xorg.conf back in action, and see how Fluxbox likes the nVidia driver & a couple monitors. I'm feeling isolated here with only one screen!

Sasha

Add a little eye candy to Flux
xcompmgr -c -r5 -F -f -D5 -o0.75

Some KDE apps are resource hogs for sure. Dolphin is the main one for me, and it has replaced Konqueror (although konqueror is still available). Opening /usr/bin in Dolphin, it eats up over 70% of the cpu, and stays that way while trying to scroll through. Thunar on the other hand, just used a maximum of 30% for a few seconds, then gave it up after loading /usr/bin. I was then able to scroll forward and back without a performance hit.

I really like KDE. However, it still needs a bit of work. I like the look of Dolphin, but I do not like it's functionality. Maybe 4.3 will fix that. If not, I will continue to use Thunar or Konqueror when its necessary.

I use emacs now for large files. Both Kate and Kwrite locked up and barfed on the same file.

Points noted; thank you mlangdn -- I feel the same about dolphin vs konqeror. Plus, the bold section reminds me of the `old days` when I used some other OS and *everything* locked up and barfed regularly.

Sasha

Scratch that thought for Konqueror - it acts the same way that Dolphin does for /usr/bin. Thunar will be the choice for me, even if it is a bit drab.

I have now begun using Thunar and will make an effort to use it instead of Konq. It may be "slightly drab" but then, how un-drab need a file manager be!?

It acts light & responsive, as does it's associated editor 'Mousepad' (though Mousepad appears to be somewhat feature-less).

Something I used to do on Slack 11 (KDE 3.5.4) was make the background color of Konq when run as root to be a shade of red. The KDE4 Konq does not seem to like or offer such a setting, and always has the same colors regardless who I run it as.

Thunar, on the other hand (and Mousepad), has IMHO a handy red Warning bar when run as root. I like it!

Sasha

PS - I've for now marked the thread as [solved] because the issue seems to have stemmed from simple HOGGYNESS and while I don't agree that I should not be able to open a 1 or 2 Gib file, despite having 4Gib of RAM installed, there's not much I can do about Kwrite's functionality.
If I discover that Mousepad is just as greedy, I'll post about it (haven't tested a giant file in Mousepad yet).