Problem with excessive RAM usage and no obvious culprit
 

I have run into a problem with my desktop using roughly 50% RAM (w/o buffers or cache) while running a limited set of applications (fbterm, tmux, weechat, ncmpc, rtorrent) on the command line. This usage only increases roughly 5-10% when starting X (an addition of xcompmgr, awesome wm, zim, parcellite, 2x conky (one replacing root-tails functionality), plus firefox and other apps that may or may not be running from time to time). (h)top is reporting programs only using roughly .1-.2% per proccess and roughly 100 processes (current look at top shows 120 processes, only 32 of which are registering any usage over 0.0%)

The RAM usage when in the console (which I will add is about 150MB after boot) is totally unreasonable and I need some direction on trying to find out what is using all of this RAM.

System:
Distro: Arch Linux
RAM: 2G
CPU: AMD 64 x2 4800+
HDD: 3x WD Black 750G (RAID 5 on partition 2 (swap) and 3 (root), RAID 1 on partition 1 (boot). LVM over root partition)
GPU: Nvidia 8400 GS

I'm not sure what you mean by "w/o buffers or cache", so I'm not sure your issue isn't the usual one (user confusion about how to interpret the output of free, rather than an actual Linux problem).

Anyway, you haven't given enough info for anyone to diagnose the problem

At least post the output of free (and tell us the conditions under which that was done).

The output of cat /proc/meminfo would give a bit more detail.

Free output:

Meminfo:

ps aux output:
All run within a few seconds of each other (free -> ps aux -> cat /proc/meminfo).

Just rebooted and grabbed all the info from the fresh system (text-only enviroment)

free:
meminfo:
ps aux:

The situation that brought up my posting is that last night I noticed a rather high RAM (~50%) usage while doing very little on the system (fbterm, tmux, weechat-curses, ncmpc) and after rebooting (and seeing a RAM usage around the 150-160MB range) left the system sit overnight unused to return and see that RAM usage was nearly 50% yet again.

That looks more informative than what I have on my systems. I guess you're using a newer kernel than I have. I wish I knew where those extra details were documented, because I'm not 100% sure what each line means.

But the important part is pretty obvious:

In case you don't know, those "slab" statistics describe memory used inside the kernel. There are several kinds of cache with slab memory. The fact that almost all your slab use is "Reclaimable" indicates that it is some kind(s) of cache. Any caching in slab memory is not included when free reports the amount of "cache".

I don't know how reclaimable that "Reclaimable" slab memory actually is. I assume that with memory pressure it could be reclaimed for use as other kinds of slab memory. But can it be reclaimed all the way out of the kernel and used as user memory? I don't know.

You could gain some understanding by looking in /proc/slabinfo and see which use is absurdly high.

As I understand slabinfo, to get the size reported by any line, you multiply the last number before the first colon by the second to last number at the end, and multiply that by 4KB. For example on my system (look at the two numbers I marked in red in this line):
That shows the ext3_inode_cache on my system is using 2386248KB

It would be interesting to run a program that makes heavy use of memory and see if that program forces some of the reclaimable slab memory all the way out of the kernel and into use space.

Use slabtop - much easier.

Kernel is 2.6.38.6, just so you know.

Here is slab info.

Looks like ext4_inode_cache is the chief culprit.

As I said, use "slabtop -o" and dump it into a file.
I don't use ext4 a lot anymore, especially on arch, but on this F14 system which has a mix of ext3, ext4 (/home) and btrfs, the ext4 slabs hardly register.
But it's an older kernel.

Thanks. After a quick look at man slaptop, I'll refine your suggestion. Use:

Edit: I tried -o with slabtop and can't get useful output (maybe a more current version is more useful). Whether redirected to a file or not, it puts in ESC sequences that delete the text from the screen immediately when displayed. So cat the resulting file erases what it displays and you see nothing. less the file lets you see it but garbled.

If this inode cache releases memory all the way to user mode if necessary under memory pressure, then it is not a "culprit". The "culprit" is free not including this kind of reclaimable memory in its -/+ buffer/cache computation.

As long as the memory isn't being used for anything else, using it for inode caching is a very desirable behavior.

There is truth to this. I will have to look into how the kernel (and probably ext4 specifically) handles memory pressure.

Just for fun, slabtop -o output. slabtop -sc > file.txt never exits and such makes it difficult to dump to a file.

I can saw with certainty that the inode_cache is cleared quite efficently. Tested using firefox and an apparent bug in it that managed to suck up not only most of my RAM, but started into swap.

Thanks for the help folks.

From your first "ps" output I was going to mention firefox was a hog, but how is that news to anyone ?.
Allocating truckloads of (userspace) memory shouldn't necessarily translate into inordinate slab cache usage. You'd have to try real hard to have that happen. Lots (and I mean *lots*) of (temporary ?) files maybe.
With the slub allocator, slab pages are consolidated/released much more efficiently, so that would explain the clean-up once firefox went away.

Note that free (or any other userspace tool) can't report these slab caches - they aren't exposed except via slabinfo. They are not (directly) part of the (page) cache.