Server frozen, caused by kernel panic?
Hello there,
Today my server completely froze and required a hard reboot. The /var/log/messages log have the following traces: Code:
Feb 26 14:45:39 decatur kernel: SKB BUG: Invalid truesize (488) len=16384, sizeof(sk_buff)=232 The server is running Fedora Core 5 with Apache 2.2, and is a 64-bit machine. Can someone please recommend some further steps to take to further diagnose this issue? Running ksymoops seems like an option, but from what I understand, that is for soft kernel panics, and this one definitely seems like a hard one (machine totally frozen). Any suggestions would be most appreciated. Thanks, TWR |
According to that second line, it looks like Apache is the program that is segfaulting, not the kernel itself.
Try shutting Apache down, and see if you still see the log filling up with those error messages. If not, you will at least know where to start your search for the problem. |
Thanks for the quick reply, MS3FGX.
Is it actually possible that a segfaulting Apache could bring the whole machine down, causing it to freeze as I mentioned? If not, how about a skb bug (problem with the Linux network buffers, from what I understand)? I'd like to target the freezing culprit first, and then tackle the remaining issue(s) afterward. Thanks, TWR |
While Linux is generally very stable, it is still possible for a malfunctioning application to bring the whole machine down. Or at least run the CPU usage so high that the server is for all intents and purposes unable to function and must be powered down manually.
Or it could be that the SKB bug is actually what is causing Apache to segfault in the first place, and there is actually nothing wrong with Apache. That sounds like would could be happening when you said: Quote:
You may also want to try running the machine with a live CD for a few hours (if you can manage the downtime for the server) to see if the error shows up there. That could help rule out a hardware issue at least. |
Again, thanks for the quick reply.
I thought of something else that may have some bearing: Today we're doing a pretty good amount of traffic (approx 20MB/sec). Is it possible that Apache getting more requests than available threads could cause the machine to completely lock up (I've already bumped up the MaxClients and ThreadsPerChild just in case)? By the way, CPU and memory are doing alright, with lots left of each. Thanks, TWR |
All times are GMT -5. The time now is 02:24 PM. |