So lately I was given a machine to work on which is RHEL 5.5. The apps it runs is 32bit so that is what it is and was installed with the PAE kernel so it can use all 8GB of ram.

The problem is the box is randomly crashing.
So looking through the logs I see things like:

And then more saying it invoked oom_killer to kill the process.

I tried dumping commands to logs to see if I can find which process is using the memory up but all of them stayed below 1.4%. So reading up a little bit I learned that:

"An OOM (Out Of Memory) error is what happens when the kernel runs out of memory in its own internal pools and is unable to reclaim memory from any other sources."

And that the kernel is suppose to use almost all of your physical ram. I already assumed that because that is how it has always been but now it's confirmed.

Anyway, so I looked at a working box's /proc/meminfo and it says:

Then I looked at the crashing box and saw:

Which doesn't seem normal to me.

I'm not quite sure how to proceed on troubleshooint / fixing the problem. I'm guessing running a non PAE kernel would probably keep it from crashing but will limit the ram. I may test this while waiting on replies. The other option is running RHEL 64bit and installing the 32bit libraries to run the software.

Thanks,

How much swap space does the system have? If it has any, how much is it using?

There is a good chance you are describing symptoms of having too little swap space. Increasing the amount of swap space should be a very easy fix.

32bit with PAE should be a very good environment for running 32bit apps on an 8GB system. A 64bit kernel might be better.

Where did you read that? It doesn't sound correct to me.

Are we talking about the same thing here? What the kernel uses for itself? In a 32 bit system that is generally limited to 1GB.

With just 8GB of ram, exceeding kernel virtual memory limits would take either a very strange workload or some kind of resource leak.

I think it is more likely that kernel virtual memory is OK and you just need more swap space.

But that was after some processes had been killed, correct?

No. Whatever the problem is, that would just make it fail sooner.

If some resource leak were managing to exhaust kernel virtual memory, that would give you a lot more time to figure it out while the system is degrading rather than failing.

There are a lot of places to look to help figure out the problem, but you generally need to look after the system is sick but before it is killing processes. I can't guess how hard it will be to find the right time to look.

It has 4GB of swap 4GB free.

From here http://linux-mm.org/OOM.

I just remember reading somewhere that almost all of your physical ram should be in use.

Fresh boot, nothing killed, but also nothing running.

I guess I was overreacting to the word "own" in
When considering memory issues for a 32 bit Linux system with large physical ram, you need to clearly distinguish between issues with the memory the kernel is using for itself vs. memory the kernel is managing for use by ordinary processes.

Despite the word "own" in that phrase, that article is about memory the kernel is managing for use by ordinary processes, not about memory the kernel is using for itself.

After enough accesses have been made to different parts of the filesystem (since the last reboot) it is typical to have almost all ram that is not used for more important things used for cache, so very little is free.

So relative to your comment about "ram should be in use", that doesn't apply because not much has happened since last reboot.

Relative to my comment about processes killed, all the processes have been killed since the last time memory was filled up, so none of the stats tell you about what filled memory.

The stats in /proc/meminfo (most of them together, not just the top two alone) would tell you a lot about the nature of the problem, but only if you look at those stats after the problem has started and before the problem has gotten bad enough to kill processes. I don't know enough about your use of your system nor about your problem to give you any better advice on how you grab those stats at a useful moment. I could help you interpret them if you did grab them at a useful moment.

I assume you mean 4GB of swap is free when the system hasn't done any significant work since the last reboot. That means nothing. What matters is how much is free when the system has a realistic load on it.

But anyway, how hard is it for you to increase the swap space? I still think a shortage of swap space is closely related to your problem. If you had more swap space that might completely fix the problem. If the problem can't be fixed by more swap space, it is still very likely that more swap space will extend the time from when the problem could be understood from /proc/meminfo contents to the time the failure occurs. So more swap space would increase the chances that you are able to look at /proc/meminfo at the right time.

Linux kernels divide the memory into zones - this is the likely problem. See the comment on the linked article re "low memory usage" (same applies to zone DMA). The oom messages should show all the zone usage(s) at the time of the error. Maybe also try the script at the bottom of the same article - looks like it saves everything relevant (run it as root).

If you can go with a 64-bit kernel do so - will save a lot of angst.

I'm currently working with Red Hat support. They are thinking I need to increase my lower memory size and send me some good values to use. I'll let you guys know if this solves the issue.

When I was watching the memory on the box during a crash, it was hardly used. Kinda threw me off being it was giving and out of memory error. However, since the memory is split I understand it as higher is for application and lower is for the kernel. Red Hat confirmed that the oom_killer gets invoked when the kernel's memory pool is running out of memory which is probably why they are having me adjust the lower memory. We'll see how it goes.

While I'm working with Red Hat I'm going to increase the swap as per you suggestion and run a script to dump meminfo so hopefully we can spot what is going on.

As a Centos user, I'm very glad Red Hat exists. But when I read things like the above, I'm also glad I don't use Red Hat.

Unless you gave them very different info than you gave us, their diagnosis is unlikely to be correct (even though syg00 seems to be reaching roughly the same diagnosis as Red Hat support).

Good plan. Please remember to tell us how it works out.

While I'm waiting on the box to crash, I thought I'd share what Red Hat told me to do:

Add the following to /etc/sysctl.conf:

If the system is getting significant use, you should also post a full /proc/meminfo to this thread as a baseline (what it looks like after significant use has started but before problems appear to have started).

Ok I was lucky enough to get the /proc/meminfo information during one of the crashes. I grabbed the information once per second. I used xargs to put it all on one line because I was pulling it into a splunk server at the same time.

I am still waiting on an OK to change the swap size btw.

You'll notice there is an hour difference between the last two entries. That is when the box froze and then I restarted it.

Thanks again.

nomb

Attached Files

memory.txt (233.9 KB, 31 views)

I think that indicates the problem is in the memory the kernel is using itself (as opposed to the memory the kernel is managing for other processes).

It also indicates I was wrong about swap space. More swap space won't help. It won't even slow the problem down to get a better look.

A 64 bit kernel would slow the problem down a LOT and maybe fix it. But switching to a 64 bit kernel is a big change.

The info that will tell you more details about the problem is in /proc/slabinfo
That is bigger and harder to understand than /proc/meminfo, but the same general concept applies: Capture it as the system builds toward the problem and you can learn more about the nature of the problem.

The memory problem inside the kernel is almost certainly caused by some kind of resource leak in some application. Watching /proc/slabinfo may give some idea of what kind or resource is leaking, but you still would need to figure out which application is leaking resources and where in the application source code is the bug making it leak resources.

That one is much bigger than the other one so will probably be hard for me to get it in the same way. What do you think I might be looking for?

nomb

I'm watching with slabinfo.

names_cache and size-32 keep going up. But names_cache is way higher. Right now it is at:

Not really concerned with size-32 it is pretty small. But names_cache keeps growing. For instance now it is: 241356K.
I am not having much luck finding out what that is on google, but does that seem problematic? I'm just sorting by size the problem could be something different.

On a 64bit box, the names_cache is only 300K.

nomb

** update **

Currently:
Box just hung:
Everything else has stayed 4056K or below.

I didn't have much luck with "names_cache" on google either.

names_cache is absolutely at the heart of your problem. You can't allow it to grow anywhere near that big (and still have hopes of a stable 32 bit system). That resource leak seems too severe for even a 64 bit kernel to survive for long.

But I can't help you much with what resource is actually leaking.

Do you know yet which process is driving the resource leak? Your earlier result seems to indicate that the kernel associates the problem resource with a process and when that process is killed, the resource is released.

Hopefully someone who knows more will jump into this thread to help.


Also, despite my generally low opinion of customer support organizations including Red Hat, you should try them as well. You have a key new piece of information that you didn't have before (run away allocation of names_cache seen in slabinfo). Pass that along to them and see what they can make of it.

My limited understanding of the limited info I found via google (so don't trust me on this) says the names_cache is roughly what you might expect, a collection of filenames that the kernel is currently working on. But does that mean one of your applications is stressing the kernel by making it work on absurdly long filenames? I don't know the limits on filename length. More importantly, are the limits enforced by the filesystem after the kernel has already cached a copy of the name in its own address space? Or are limits on name length enforced before the kernel makes a copy of the name? Normally, one would expect something like a name cache to be abused by too many names rather than by names that are individually too long. That might be the case this time. But then how is the name cache so far away bigger than any other kernel memory use? A resource leak involving too many names, each of reasonable length, ought to also involve a larger amount of other kernel memory use. I saw what you said about "size-32" and that probably is an associated memory use, but it isn't big enough to make sense as the associated use if the names are individually reasonable size.

If you are refering to the memory.txt I posted, it had stayed hung. I had to reboot the box.
I just rebooted another box, same image, into the non-PAE kernel. It's names_cache is also increasing but at a much much slower rate. Perhaps it will hit a point where it gets stable.

Does anyone happen to know what names_cache is? Or how you can find out what is causing it to leak?

I've been keeping the RedHat thread updated as well.

Thanks,
nomb