LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
LinkBack Search this Thread
Old 10-11-2007, 10:16 AM   #1
cygnus-x1
Member
 
Registered: Jul 2006
Location: Crisfield
Distribution: Slackware, CentOS, RedHat
Posts: 159

Rep: Reputation: 15
Exclamation ext3 related kernel panic (keeps occurring)


We are running CentOS 4.4 (64 bit) kernel 2.6.9-34.ELsmp. There are 2 RAID arrays on the box 1 72 gig array root partition( / ) at RAID level 1 and a 1.8 Tera byte partition at RAID level 5.

The SCSI and RAID controllers in the box are:

SCSI
03:01.0 SCSI storage controller: Adaptec AIC-7892A U160/m (rev 02)
Subsystem: Adaptec 29160 Ultra160 SCSI
Controller

RAID
03:02.0 RAID bus controller: Adaptec AAC-RAID (Rocket) (rev 02)
Subsystem: Adaptec ASR-2820SA

Approximately every 2-3 weeks we get a kernel panic. Originally we were having drive issues but have since swapped out to a newer drives and according to the RAID controller the arrays are optimal now but the kernel panic keeps occuring.

The box does a lot of I/O as it is a production web backup machine.

I have not yet had the chance to setup a netdump server so I don't have the whole console but I do have a screen shot of the final lines. It appears to be a syncing issue within ext3. BTW LVM is in use if that is important.


Here is a link to the console image.

Any help is GREATLY appreciated. Even if it is directing me to a different place to ask the question.
 
Old 10-11-2007, 04:52 PM   #2
macemoneta
Senior Member
 
Registered: Jan 2005
Location: Manalapan, NJ
Distribution: Fedora x86 and x86_64, Debian PPC and ARM, Android
Posts: 4,593
Blog Entries: 2

Rep: Reputation: 326Reputation: 326Reputation: 326Reputation: 326
From what I can see, it looks like a paging error.

1- Are your swap areas properly initialized?
2- Did you run out of swap?

If you get a complete panic output, you should submit it to your distribution vendor's bugzilla.
 
Old 10-11-2007, 09:21 PM   #3
cygnus-x1
Member
 
Registered: Jul 2006
Location: Crisfield
Distribution: Slackware, CentOS, RedHat
Posts: 159

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by macemoneta View Post
From what I can see, it looks like a paging error.

1- Are your swap areas properly initialized?
Not quite sure I know how to tell. I let CentOS handle that configuration when I installed it. It has 2 gigs of swap space. The swap space is part of LVM however. Not sure if that is a good idea or not.

Quote:
Originally Posted by macemoneta View Post
2- Did you run out of swap?
How would I know that? Is there something I can use to monitor swap usage? When I do a swapon -s I see that I have only used 160 however it is running fine now. The machine locks down hard when it panics and you have to power cycle it so postmortem info is hard to come by. I am really not that familiar with knowing what to do in this case anyway.
Quote:
Originally Posted by macemoneta View Post
If you get a complete panic output, you should submit it to your distribution vendor's bugzilla.
I will try and get a netdump server configured and a whole punched through the firewall so I can get the full console when in it crashes.

thanks
 
Old 10-11-2007, 09:50 PM   #4
macemoneta
Senior Member
 
Registered: Jan 2005
Location: Manalapan, NJ
Distribution: Fedora x86 and x86_64, Debian PPC and ARM, Android
Posts: 4,593
Blog Entries: 2

Rep: Reputation: 326Reputation: 326Reputation: 326Reputation: 326
Quote:
Originally Posted by cygnus-x1 View Post
Not quite sure I know how to tell. I let CentOS handle that configuration when I installed it. It has 2 gigs of swap space. The swap space is part of LVM however. Not sure if that is a good idea or not.
If there was an interruption during the install and you restarted, the installer may think the swap is already initialized and not complete that step if you re-run the install. In that case, you need to manually re-initialize the swap space:

Code:
swapoff -a
mkswap swapdevice
swapon -a
Quote:
How would I know that? Is there something I can use to monitor swap usage? When I do a swapon -s I see that I have only used 160 however it is running fine now. The machine locks down hard when it panics and you have to power cycle it so postmortem info is hard to come by. I am really not that familiar with knowing what to do in this case anyway.
There are many ways to monitor swap. The simplest is to just open a terminal and run:

Code:
while true; do swapon -s | grep -v Filename | awk '{print $4}' | echo "`date` `cat -`" ; sleep 15 ; done
Quote:
I will try and get a netdump server configured and a whole punched through the firewall so I can get the full console when in it crashes.
You can also force the system to reboot cleanly. Some one-time preparation is required. Add to /etc/sysctl.conf:

Code:
kernel.sysrq = 1
and also run (as root):

Code:
echo "1" > /proc/sys/kernel/sysrq
This will enable the magic sysrq function. The next time the system panics, press and hold Alt and Sys Req (usually the same key as PrtSc). While holding those down, press the following keys in this sequence, with about a 5 second delay between keys: reisub

If the magic sysreq function was able to get control, at the end of that sequence the system will reboot cleanly. With any luck, you will have your panic in /var/log/messages, or at least some more data to go on.

Last edited by macemoneta; 10-11-2007 at 09:56 PM.
 
Old 10-11-2007, 10:09 PM   #5
cygnus-x1
Member
 
Registered: Jul 2006
Location: Crisfield
Distribution: Slackware, CentOS, RedHat
Posts: 159

Original Poster
Rep: Reputation: 15
Smile

Thank you very much. Never heard of that magic sequence before I will give it a try. I reinitialized swap. I will setup a monitoring script and record the info somewhere so I can the next time it crashes if swap is full or not.

thanks again o wizard!

Doug
 
Old 10-11-2007, 10:33 PM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 11,792

Rep: Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923
Mmmm - if swap filling was the issue you'd probably hear from the oom_killer. I suspect you have other problems.
Centos should have systat installed - see if you have that running; sar will give you all the info you'll need, and a good history. Running vmstat in background is another option in addition to the commands above.
 
Old 10-12-2007, 02:57 AM   #7
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,860

Rep: Reputation: 768Reputation: 768Reputation: 768Reputation: 768Reputation: 768Reputation: 768Reputation: 768
Quote:
We are running CentOS 4.4 (64 bit) kernel 2.6.9-34.ELsmp.
Back then there was a bug in SATA support that only affected 64 bit versions. I don't know that this bug affected anything other than SATA (and have no idea whether it affected LVM and/or SCSI), but I'd do a bit of research on that and see if CentOS offers any newer/patched kernels and/or bug reports just in case it had wider ramifications.

From memory, the bug was fixed around 2.6.10, and it wouldn't let you install 64 bit/SATA at all, so may not be relevant in any way to your circumstance, but I'd still want to spend a few minutes checking into it.
 
Old 10-12-2007, 07:19 AM   #8
cygnus-x1
Member
 
Registered: Jul 2006
Location: Crisfield
Distribution: Slackware, CentOS, RedHat
Posts: 159

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by salasi View Post
Back then there was a bug in SATA support that only affected 64 bit versions. I don't know that this bug affected anything other than SATA (and have no idea whether it affected LVM and/or SCSI), but I'd do a bit of research on that and see if CentOS offers any newer/patched kernels and/or bug reports just in case it had wider ramifications.

From memory, the bug was fixed around 2.6.10, and it wouldn't let you install 64 bit/SATA at all, so may not be relevant in any way to your circumstance, but I'd still want to spend a few minutes checking into it.
Thank you for that tidbit. I will take the time to investigate. One of our options was to wipe and reinstall CentOS 5.0. I would rather just upgrade if that path is available to me.
 
Old 10-12-2007, 01:07 PM   #9
cygnus-x1
Member
 
Registered: Jul 2006
Location: Crisfield
Distribution: Slackware, CentOS, RedHat
Posts: 159

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by syg00 View Post
Mmmm - if swap filling was the issue you'd probably hear from the oom_killer. I suspect you have other problems.
Centos should have systat installed - see if you have that running; sar will give you all the info you'll need, and a good history. Running vmstat in background is another option in addition to the commands above.
Never used any of these commands before. I don't seem to have systat but I do have sar and vmstat. What would sar tell me? Can I look back in time with it? vmstat seems to give up to the minute info, if the kernel panics how would it help then?

thanks

Doug
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Filesystem-Related Kernel Panic Arg0nyX Linux - Kernel 2 07-04-2007 03:28 PM
LILO-related Kernel Panic on Boot sarboras Slackware - Installation 7 04-28-2004 04:28 PM
kernel panic, init related, found cause, but cant fix...help :) sirpelidor Linux - Software 7 12-24-2003 07:28 PM
Kernel Panic re Ext3? OldNewbie Linux - Newbie 1 09-12-2003 11:26 AM
SMP Kernel Panic with ext3 on Redhat 9 Gord_McF Linux - Newbie 7 07-31-2003 07:02 AM


All times are GMT -5. The time now is 11:19 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration