System crashing under load due to glibc memory paging error??
Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
System crashing under load due to glibc memory paging error??
I have seen this bug under multiple kernels on this particular machine. The machine is a 486 class with 32 megs of memory. Under heavy loads it crashes regularly with virtual memory paging errors. The last time this happened just before it died, it returned this error for ANY prog I tried to run (ls, ps, df, ifconfig)
error while loading shared libraries: /lib/libc.so.6: symbol _dl_out_of_memory, version GLIBC_2.2 not defined in file ld-linux.so.2 with link time reference
I have 500 megs of swap, but it never seems to use much of it. It is always extremely low on free mem, but the amounts that it has paged are always very low (it does page some..)
I've tried different 2.4 kernels, but I don't believe I've ever used any other glibc then RH 7.1 standard: 2.2.2-10
This latest error about the undefined symbol leads me to believ that perhaps it is a memory allocation bug in glibc, not the kernel.
This machine is console only. No X Windows. I would never dream of running X on a 486 with a 2 meg non accelerated graphics card and 32 megs of mem. This machine is running as a headless server. I maintain it through telnet. (Yes I know telnet's security problems, I'm the only user on the 2 computer network)
RedHat is a production system and it's requirements lists are massively inflated. Linux can be stripped down (even from an RH distro) to run on the most meager systems! There are many embedded systems that run with 4 and 8 megs of ram.
See this howto: http://eddie.cis.uoguelph.ca/~tburge...ll-Memory.html
I realize that when paging memory from the swap, performance drops to hell, but under no circumstances should this make ANY kernel unstable! This is not windows, and there is no excuse for crashing under any amount of load when the swap file is barely even used. I have a 500 Meg swap file and 1 min before crashing, maybe only 1 or 2 megs of swap were actually being used.
This machine uses old DIMMS and I don't care to look for any more for it. If it will swap correctly, that performance will be good enough for me.
command line linux should in no way require that much mem. paging faults should slow things down considerably, but shouldn't crash.
if you've stripped things down from a redhat 7.2 - then the problem could be in the glibc compiles. if it's possible, try recompiling the glibc. generally tho, i wouldn't recommend this to anyone who doesn't know what they're doing - it would probably help to read any lfs stuff you can to help with the glibc.
Thanks A LOT for the reassurance! Currently, I'm updating the glibc. Redhat maintains an update ftp site for the different versions. I'm assuming the only reason they would post updates is to fix bugs while breaking a minimum of dependencies. On this assumption I'm changing from glibc 2.2.2-10 to glibc 2.2.4-19.
I'm kind of worried that the kernel will not work with a diff version of glibc, so I'm also dling there updated precompiled kernel. I've gotten no serious problems with it yet, but I don't know enough to know if the old kernel is fine with the new glibc, or what.
Also, after reading quite a bit about swap, I found that according to /proc/swaps the priority of my swap file was -1!!! I read that valid values are from 1 to ..... After defining my swap as priority 1 and confirming the change in /proc/swaps, I IMMEDIATELY noticed that linux was using the swap quite a bit more via free.
I take this to mean that linux was being starved of swap because the priority of the swap device was fined up. This was how RH was stock, but it is designed for more mem, so this may be a screw up on their end.
(If the new glibc and the swap correction doesn't work, I'll try recompiling the glibc and kernel.)
Well, I'm running the new glibc with the old kernel (because I hadn't gotten the new one dled yet) and I changed the swap priority to 1.
I tried to overload it as usual. I tried multiple samba writes from my windows box. They all timed out, but the linux box seemed unconcerned. Samba and the kernel kept on running. I tried reading and writing simultaneously and starting a buncha telnets with no adverse effects. According to free, the machine is running with less then one meg free mem, but the swap usage increases nicely with load maintaining that amount of free mem.
The thing looks stable to me, I'm not risking it to figure out whether it was glibc, or the screwey -1 swap priority. For now, I'm happy with it. Now all I have to do is trim that mem usage. It's using 12 megs on boot! I turned off the big services that I know I don't need like sendmail...
I've got a 486 server running too and I checked several things on my system to compare. I have 16 megs of ram in it so it should choke faster on heavy loads. So far I've only seen it go really slow with heavy loads. It never crashed on me. On initial install it was mandrake 7.2 but I've tweaked and replaced so much I'm not sure if it's still mandrake. Still using libc 2.1.3 though since that's one thing I haven't replaced. My swap priority is also still set to -1. Usually a value like that would mean it's not set. Meaning the priority wouldn't be used. Don't know if it will really improve things if I set the priority higher. Right now with hardly and load 1MB of the 400MB swap is being used. 15MB of the 16MB ram is being used. Which seems pretty normal to me. When the load gets heavier more swap gets used and I've never tried maxing out the swap space. But I think the whole system would get really slow.
As far as I know the kernel doesn't use libc. So you can use it fine with the old kernel. It's libc that gets built using some of the kernel headers.
Thanks alot! It could very well be that the version of glibc that shipped with RH 7.1 has a bug in it that is causing this crash. As far as I know, very few ppl would use RH 7.1 on an old 486 or any machine with very low mem so it would be obscured. They did post an update also, so anyone who runs up2date would get all the updated versions.
The updated version may very well have a bug fix in it for this, although RH doesn't say much of anything about why there is an update. I don't care for RH after all this. They seem to have custom versions of a lot of things (the gcc compiler!!!!) which is disconcerting to me, their updates aren't usually to the latest stable version, some of their latest stable packages depend on unstables such as gcc 3.0.3! They just don't seem very logical in how they run their packages. I suppose when I get some time, I'll throw together an lfs, but the machine works now so I'm not fooling with it till I get bored with it.
if changing the swap priority fixes things, then don't mess around with the glibc. i kinda thought it was a long-shot for a fix anyway ... i need to read up on the swap priority... i've never heard of that before.
Well crap.. I can't say I'm surprised, yet I am disappointed. It crashed again. I did put in a new glib.. just used RH's rpm.
I'd like to clear up a few things:
The kernel is not linked to the glibc, but it does use the calls from the glibc when a kernel is compiled. True?
If the glibc is recompiled, you need some source code, but it has no connection with the kernel source does it?
When I did upgrade the glibc, isn't there a possibility that other packages would screw up because the libs are a different version then what they were compiled with?
Any opinions on which gcc and which glibc is the most stable? I'm looking for something that I can really hammer in low mem conditions. I guess I'm going with an lfs system. Before I do, I'm going to upgrade my RH to rawhide's kernel, glibc, and gcc to see how it goes.