Need advice on tracking down Kernel Panic
 

Hey gang, here is what I got from the boot message (This comes after
the grub menu, it's not a complete list, but you get the idea.) of my
FC4 machine, 2.6.11:

-------------
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Loading jbd.ko module
Loading ext3.ko module
Creating root device
Mounting root filesystem
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Switching to new root
unmounting old /proc
unmounting old /sys
Kernel panic - not syncing: Attempted to kill init!
[<c01211c8>] panic+0x45/0x1e8
[<c01223f1>] profile_task_exit+0x31/0x45
[<c0124217>] do_exit+0x25a/0x369
[<c016c3ea>] do_execve+0x175/0x23c
[<c012437b>] do_group_exit+0x29/0x90
[<c0104025>] syscall_call+0x7/0xb
-------------

Can anyone give me some pointers on tracking down the cause of this
kernel panic? Obviously all grub stages are passing and it appears
initrd is doing it's thing.. but I don't know where to go from here.
ANY and ALL hints/tips/tricks are welcome!

TIA,
ponga

Are that any errors reported prior to the output you posted? Kernel panic errors are almost always caused be corruption in the file referenced in the boot= part of the GRUB kernel line, and those errors (if they're reported at all) are usually several lines prior to the kernel panic message. (A common problem is caused by an "absolute" path for the root file system (i.e., root=/dev/hda1/) when the partitioning or physical drive numbering is changed.)

On the other hand, if you're running a really old, unsupported, distribution like FC4, your hardware may also be ancient. So you might have a bad memory chip, and need to run memtest (which can take a day or so) to identify the problem.

I suggest you download a live CD to verify your hardware and run fsck on your root file system. I've found the System Rescue CD a useful tool to have available.

PTrenholme, thanks for the reply.

I dont have a boot= arg and I should not need one. My menu.lst is listed below.

Moreover, this system has previeously booted fine in the recent past. I've also run a live CD (Ubunut 8.04) and all the hardware checks out, and memtest86+ reports no memory errors. The drive and partition mount just fine under the live CD and fsck reports no errors on them.

What's more, this is all we can get from the boot messages on account that the screen goes by pretty quick after the grub menu. As you can see, there are no real errors being reported like "can't mount such-and-such"... just a straight up kernel panic. Nothing preceeds it, or at least nothing that we can see.

Here is my menu.lst, note that the the label has been verified with e2label.

Thanks, I'm trying reall hard to track down where to look in finding a cause for this. Any help is appreciated,
ponga

edit the kernel line during boot

root=/dev/sda1

amani, I tried that. Still no love. I just can't figure out the cause of the kernel panic, I don't think it's because the fs isn't mounted. But I just don't know... <sigh>
-ponga

Oops! Sorry, :cry: I should have said root=, nor boot=. (I was, obviously, thinking about boot problems.)

Anyhow, the LABEL=/1 looks somewhat strange. Did you really label your / file system as /1? Not that there's any reason that the label need so match the name. (In fact, on this Ubuntu 9.04 system, the root (and sole) partition for the test release is labeled "Jaunty" (with no slash at all) although the root line is root=UUID=. . . .)

But my point remains: Can you run fsck on that partition? It's easy from a Live CD, and easy if you can boot in rescue mode, and easy if you can actually boot, but not if the root file system can't be mounted.

Update
 

I'm a little further down the boot process, but still it's maddening.

I decided to get rid of initrd and go straight to the kernel, which has ext3 support built in. This get me quite a bit further in the boot process, but still ends in kernel panic like so:

It says is can't open root device "sda1" even though I have "root=/dev/sda1" listed in menu.lst. Also disturbing is it says it will list the available partitions, then lists none. This smells like a scsi/sata module or the sort is not getting loaded, which could be, initrd used to be doing that. But initrd is screwed up too. So... I'm just not sure what to do.

PTrenholme, yes, I've run fsck on this partition with no errors or bad blocks. And ya, that disk label is weird, I certainly did not set it like that. But e2label confirms that it's correct, go figure. I've tried /dev/sda2 too with the same result.

Recommendations? This is an FC4 system for a reason; we are doing some development on it, it's required by the customer, don't as me why they want an "ancient" distro... I just work here :/

thanks,
ponga

Perhaps FC4 is so ancient that it's not using the newer combined SATA and PATA driver, and your drive is a PATA one that FC4 would only recognize as /dev/hda1? IIRC, Fedora only went to the combined library with FC7, and, by then, FC4 was, I believe, no longer supported, so that library may have never been backported to FC4.

Try a root=/dev/hda1 to see if that works . . .

PTrenholme,
That would be a pretty good idea, and still worth a try, except that using the partition label should have made this irrelevant. Even if not, I'm looking at the dmesg from the last good bootup, and it's mounting /dev/sda1, not hda. So, :(
Oh well.. I'm quickly running out of options... thanks though!
-ponga

Do you, by any chance, have copies of the old FC4 installation disks or DVD? If so, you could try using them to boot "linux rescue" to see if the installation could be found (by a "native" FC4 kernel) and, if it could be found, you might be able to (re)build the kernel and initial RAM disk images.

Did you check the hardware? Sometimes a loose (or dirty) drive cable connector can cause really strange problems. And, as I suggested above, a bad memory block could also cause the boot problem. In fact, from the consistency of the problem, a bad bit in the area used by the kernel's disk read routine might, possibly, be the cause of your problem. (Although the BIOS POST memory check, unless you've disabled it, should catch a "stuck" bit. Being retired, with no time pressures, I always enable a complete POST on every reboot, but people in other circumstances may feel that it's more important to "start working.")

<edit>
Another thought. On how many different computers have you attempted to boot the drive? Did it fail in the same way on all of them? If so, you problem is not likely to be related to the computer, although a HD problem wouldn't be ruled out.
</edit>

Yes, I still have the FC4 media and it boots just fine into rescue mode on this machine. Like I said I've ran memtest86+ with no errors found in mem. I would try this drive on another machine, in fact I might do that tonight. I was thinking about "reinstalling" the kernel and initrd from the CD.. but don't know how. I will be working this issue some more tonight... keep you posted and thanks for all the suggestions. I'd rather find a solution rather than just "reinstalling."
THANKS,
ponga

Not solved, but case closed
 

Well, I did not solve the problem, but upgrading to FC5 resolved the issue.
I'm still not sure if the problem was in initrd or the kernel (or grub, even) and we'll never know.
But were are booted up now in FC5; the upgrade went off without a hitch, so case closed.

Thanks for everyones help.

--ponga