Linux locks up periodically - where do I look for clues?
 

Howdy,

My question is about how to start debugging a machine that locks up hard and has to be powered off.

I installed Redhat 9 on a Compaq Proliant 1600 with a 450Mhz PII and 512M of memory. The machine has been running NT4.0 for four years or so with no problems so I don't suspect a hardware problem.

I have installed all recent patches (only lack the two that came out today). It runs postfix, amavis-new, and spamassassin and almost nothing else. The last two times it locked up the last thing I did was wake it up in the morning. I saw the log on prompt but did not log on. I have the machine on a miniview KVM switch and I switched over to my Windows box to do some work. When I switched back, nothing. It does not routinely lock up when I switch between machines, so I don't suspect that is the cause.

The box locks up on me every few days and I don't know where to start looking for the problem. By looking at the Cron log I see that it was running at 10:00am today, then nothing until I rebooted at 13:07. I can't find any clues in any of the other logs I find in /var/log

When I say it locks up I mean no response on keyboard or monitor it will not answer a ping, will not process email sent to it, will not respond to ssh session, nothing. Also, the machine is mostly idle. I'm testing it as a spam filter so it only has work to do a few hours a day while I test.

So finally the question, how do I start debugging this?

Thanks,

Joe

Here are a few guesses of areas where you could start looking.

Is swap initialized and working OK? When a machine is idle for a while swap will sometimes utilize the idle machine time to write a large process to the swap file as a head start for when things get busy. If swap is not set up you could conceivably hang.

Do you have a screensaver enabled? There is the ocassional bug that when a screensaver starts, the machine freezes.

Do you run cron jobs periodically during the day? You could have a screwed up cron job that hangs the machine every time it automatically starts.

"I can't find any clues in any of the other logs I find in /var/log"

You also might take a look at the size of the log files. Is there something that is spewing out large amounts of innocuous error messages?

___________________________________
Be prepared. Create a LifeBoat CD.
http://users.rcn.com/srstites/LifeBo...home.page.html

Steve Stites

i had a problem a month or two ago with a net card sharing an interrupt with anything else. had to swap cards around on the bus until the bios assigned it to it's own irq. very erratic freezes, didn't follow any sort of pattern. might be worth a look.

Thanks for the suggestions. I don't have a screensaver (don't really use the GUI on this machine). The video card was asleep when this problem started so that's a possibility.

How do I check the paging setup? I did a quick search on the Web and got lots of hits but no useful info.

Are there any utilities out there to tell me if I have an IRQ conflict?

Thanks again,

Joe

you can do a quick cat /proc/interrupts in a root term, if everything is on it's own irq than you can prolly rule that out.

"How do I check the paging setup?"

Issue the free command:
free

Look in /etc/fstab to see if there is an entry for swap. Play around with the swapon and swapoff commands. See:
man swapon
man swapoff
man mkswap

If necessary run mkswap.

Check your boot log to make sure that swap is being started during boot.

"Are there any utilities out there to tell me if I have an IRQ conflict?"

lspci might help. See;
man lspci

___________________________________
Be prepared. Create a LifeBoat CD.
http://users.rcn.com/srstites/LifeBo...home.page.html

Steve Stites

OK, I checked the interrupts using cat /proc/interrupts and lspci and I don't see any obvious conflicts. I read the man pages for swapon, swapoff, mkswap, and free and I see that I do have a swap area defined. I turned it on manually and see that the system is not using it. That's not surprising since the box is sitting idle right now.

The man page for swapon said it's designed to be started by a startup script like /etc/rc but that doesn't look like a good place put it to me. Where is the proper startup script for it? Does it need to start before other processes? Reading the man page it looks to me like "swapon -a" is the right way to call it.

Thanks again to all for thjis help.

Joe