ext3 related kernel panic (keeps occurring)
We are running CentOS 4.4 (64 bit) kernel 2.6.9-34.ELsmp. There are 2 RAID arrays on the box 1 72 gig array root partition( / ) at RAID level 1 and a 1.8 Tera byte partition at RAID level 5.
The SCSI and RAID controllers in the box are: SCSI 03:01.0 SCSI storage controller: Adaptec AIC-7892A U160/m (rev 02) Subsystem: Adaptec 29160 Ultra160 SCSI Controller RAID 03:02.0 RAID bus controller: Adaptec AAC-RAID (Rocket) (rev 02) Subsystem: Adaptec ASR-2820SA Approximately every 2-3 weeks we get a kernel panic. Originally we were having drive issues but have since swapped out to a newer drives and according to the RAID controller the arrays are optimal now but the kernel panic keeps occuring. The box does a lot of I/O as it is a production web backup machine. I have not yet had the chance to setup a netdump server so I don't have the whole console but I do have a screen shot of the final lines. It appears to be a syncing issue within ext3. BTW LVM is in use if that is important. Here is a link to the console image. Any help is GREATLY appreciated. Even if it is directing me to a different place to ask the question. http://i211.photobucket.com/albums/b...or-console.jpg |
From what I can see, it looks like a paging error.
1- Are your swap areas properly initialized? 2- Did you run out of swap? If you get a complete panic output, you should submit it to your distribution vendor's bugzilla. |
Quote:
Quote:
Quote:
thanks |
Quote:
Code:
swapoff -a Quote:
Code:
while true; do swapon -s | grep -v Filename | awk '{print $4}' | echo "`date` `cat -`" ; sleep 15 ; done Quote:
Code:
kernel.sysrq = 1 Code:
echo "1" > /proc/sys/kernel/sysrq If the magic sysreq function was able to get control, at the end of that sequence the system will reboot cleanly. With any luck, you will have your panic in /var/log/messages, or at least some more data to go on. |
Thank you very much. Never heard of that magic sequence before I will give it a try. I reinitialized swap. I will setup a monitoring script and record the info somewhere so I can the next time it crashes if swap is full or not.
thanks again o wizard! Doug |
Mmmm - if swap filling was the issue you'd probably hear from the oom_killer. I suspect you have other problems.
Centos should have systat installed - see if you have that running; sar will give you all the info you'll need, and a good history. Running vmstat in background is another option in addition to the commands above. |
Quote:
From memory, the bug was fixed around 2.6.10, and it wouldn't let you install 64 bit/SATA at all, so may not be relevant in any way to your circumstance, but I'd still want to spend a few minutes checking into it. |
Quote:
|
Quote:
thanks Doug |
All times are GMT -5. The time now is 09:53 PM. |