When I log onto a server used at my company and type the command top, this is what I see at the top

Tasks: 291 total, 1 running, 290 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.8%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32855928k total, 25785840k used, 7070088k free, 10566472k buffers
Swap: 39321596k total, 54784k used, 39266812k free, 11925436k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4163 labfw 20 0 11.5g 169m 6792 S 0.3 0.5 10:41.11 java
1 root 20 0 23596 1332 1116 S 0.0 0.0 0:02.39 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:12.30 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 2:05.73 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.86 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:16.48 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1
9 root 20 0 0 0 0 S 0.0 0.0 2:04.02 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.84 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 0:17.15 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/2
13 root 20 0 0 0 0 S 0.0 0.0 2:00.47 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:00.74 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:17.42 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/3
17 root 20 0 0 0 0 S 0.0 0.0 1:57.65 ksoftirqd/3
18 root RT 0 0 0 0 S 0.0 0.0 0:00.75 watchdog/3
19 root RT 0 0 0 0 S 0.0 0.0 0:17.74 migration/4
20 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/4
21 root 20 0 0 0 0 S 0.0 0.0 1:48.39 ksoftirqd/4
22 root RT 0 0 0 0 S 0.0 0.0 0:00.74 watchdog/4
23 root RT 0 0 0 0 S 0.0 0.0 0:17.48 migration/5
24 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/5
25 root 20 0 0 0 0 S 0.0 0.0 1:48.00 ksoftirqd/5
26 root RT 0 0 0 0 S 0.0 0.0 0:00.73 watchdog/5
27 root 20 0 0 0 0 S 0.0 0.0 1:51.70 events/0
28 root 20 0 0 0 0 S 0.0 0.0 0:21.71 events/1
29 root 20 0 0 0 0 S 0.0 0.0 0:21.92 events/2
30 root 20 0 0 0 0 S 0.0 0.0 0:31.05 events/3
31 root 20 0 0 0 0 S 0.0 0.0 0:21.16 events/4
32 root 20 0 0 0 0 S 0.0 0.0 0:24.14 events/5
33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/0
34 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/1
35 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/2
36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/3
37 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/4
38 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/5
39 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/0
40 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/1
41 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/2
42 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/3
43 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/4
44 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/5
45 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
46 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
47 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
48 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
49 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
50 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
51 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup
52 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
53 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns


This is when the server is not being used to run any jobs. So if I look at the top output every row says %MEM is 0.0.

Right now the problem we are having is that when we run 5 jobs on the server the memory seems to be overloaded. I don't really understand why, because when I run a single process it shows memory usage to be something like maximum 3%. I also understand that so much memory is shown as used even though nothing is being run because of cache and buffers, but it seems like that memory isn't being relinquished to processes when they need it because we keep having memory overloads. Also having that much memory being marked as used because of cache and buffers seems excessive.

I am very confused...Any ideas?

I would have hoped someone who would might have more detailed info. than I might, would have answered your question by now. But since they haven't, a few quick observations:

1) top says 291 processes total, but you have listed only a fraction of them; perhaps the memory usage on the hundreds of processes which are not shown, is more substantial.
2) I'm tempted to say that by default top sorts by CPU usage, not memory usage. If you haven't already, I'd suggest you tell top to sort by memory usage.
3) even though from what you've said you may have some awareness of this, it seems fair to say that Linux more or less uses a JIT ( Just In Time ) memory-reclamation algorithm; that almost can't be emphasized enough.
4) I can more or less understand the seeming contradiction from what you've said about running one process and seeing 3% maximum memory usage. But the top output could potentially be more intuitive if you make sure that you are having it sort by memory usage, and double check exactly what's included in the %MEM column on the specific distro. and version that you are using; for example, how does top contrast resident set size, virtual memory, etc., with %MEM?
5) If it's possible for you to reboot the machine, and run only one instance of the process that may be using a substantial amount of memory, with top sorting my memory usage, what is shown then?

Not so: PID 4163 of user "labfw" is consuming 11.5 GB RAM virtual memory, plus you don't tell us what the specs are of the other processes specs you try to run...

Normally I would start by pointing you at linuxatemyram - well worth a read. But that buffer number is very odd. Post the content of /proc/meminfo.

Explain in (much) more detail what you mean by "when we run 5 jobs on the server the memory seems to be overloaded."

As the referenced (excellent!) web site makes clear, it is normal for Linux to use "the rest of" memory for cache and/or buffers.

It will do this, literally, "so that the RAM is always good for something." For instance, if you finish running a particular program, there's an excellent chance that you'll run the same program again very soon. So, Linux tries to keep it around. When you read a file sequentially, Linux anticipates this and reads more data than you requested this time, so that it won't have to perform another I/O operation next time.

But, when memory starts to become pressured, cache and buffers are "the low man on the totem-pole." As a discretionary use of memory, they get kicked-out to make memory available for mandatory use ("because some program is actually requesting it").

What's actually happening in your case is that you don't have enough RAM to run five instances of that big program at the same time. "Chips are Cheap" these days, so the first thing that you should do is to invest in "as much RAM as that mother(board) will hold."

Your observation that "cache and buffer usage is 'very high'," while true, is actually a red herring.