mount: could not find filesystem '/dev/root'

jemenake · 11-11-2009, 06:46 PM

Here's the backstory: I've got a 1And1 server which was pre-installed with Fedora Core 6, and it's running kernel 2.6.16. We've been experiencing random server hangs (doesn't ssh, doesn't ping, serial console is unresponsive). So, I want to try a new kernel, but only once have I ever been able to build a new kernel for it and have it work (and I don't remember what I did on the one that kinda worked).

The problem I usually run into is the kernel will boot, detect the NVidia SATA controller fine, detect the MD mirrored drives fine, and then I'll get a "mount: could not find filesystem '/dev/root'" and then the kernel panics and restarts the machine.

I have copy/pasted the output from a successful 2.6.16 boot and from the unsuccessful 2.6.31 boot and I've begun comparing them, line-by-line. Except for slight changes in the working of some of the status/debugging messages, the detection of the MD devices is identical. It finds /dev/md1, /dev/md5, /dev/md6, and /dev/md7, so I don't think it's an issue of the kernel not seeing the SATA controller.

The root filesystem is on /dev/md1, and it's ext3. The MD, ext2, and ext3 drivers are compiled into the kernel (not as modules), so even a problem with the initrd shouldn't be a deal-killer at this point.

From the unsuccessful boot:

Code:

...
[    4.883569] md: considering sdb1 ...
[    4.890724] md:  adding sdb1 ...
[    4.897189] md:  adding sda1 ...
[    4.903674] md: created md1
[    4.909268] md: bind<sda1>
[    4.914791] md: bind<sdb1>
[    4.920230] md: running: <sdb1><sda1>
[    4.992837] raid1: raid set md1 active with 2 out of 2 mirrors
[    5.004583] md1: detected capacity change from 0 to 1011548160
[    5.016306] md: ... autorun DONE.
Trying to resume from /dev/sda2
Unable to access resume device (/dev/sda2)
Creating root device.
Mounting root filesystem.
mount: could not find filesystem '/dev/root'
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory

Meanwhile, the successful-booting 2.6.16 kernel finds the ext3 filesystem just fine:

Code:

...
md: considering sdb1 ...
md:  adding sdb1 ...
md:  adding sda1 ...
md: created md1
md: bind<sda1>
md: bind<sdb1>
md: running: <sdb1><sda1>
raid1: raid set md1 active with 2 out of 2 mirrors
md: ... autorun DONE.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 256k freed
INIT: version 2.86 booting
                Welcome to Fedora Core

And, just for kicks, /etc/fstab looks like:

Code:

/dev/md1        /               ext3    defaults        1 1
/dev/sda2       none            swap    sw              
/dev/sdb2       none            swap    sw              
/dev/md5    /usr        xfs     defaults        0 2
/dev/md7        /var    xfs     defaults,usrquota       0       2
/dev/md6        /home           xfs     defaults,usrquota       0 2
devpts          /dev/pts        devpts  gid=5,mode=620  0 0
none            /proc           proc    defaults        0 0
none            /tmp    tmpfs   defaults        0 0

Anybody have any clue what I'm missing here?

syg00 · 11-11-2009, 08:50 PM

If you're doing your own kernel, have you tried running without the initrd ?. What does your boot stanza look like ?.

exvor · 11-11-2009, 10:02 PM

I am thinking your new kernel is missing either the raid stuff or you don't have ext3 file system compiled into the kernel. If you don't want it compiled in then you going to have to include it in your initrd image. Personally I see no point in using initrd images.

jemenake · 11-12-2009, 04:33 AM

Quote:

Originally Posted by exvor

I am thinking your new kernel is missing either the raid stuff or you don't have ext3 file system compiled into the kernel. If you don't want it compiled in then you going to have to include it in your initrd image. Personally I see no point in using initrd images.

Oddly, I have deliberately set the config to have all RAID and ext2/3 compiled in yet, without initrd, I get a different set of errors. Now, I don't get any of the "considering", "adding", "created", "bind" messages from the MD modules, even though the kernel reports registering them.

Code:

...
[    1.918017] md: linear personality registered for level -1
[    1.928980] md: raid0 personality registered for level 0
[    1.939590] md: raid1 personality registered for level 1
[    1.950199] md: raid10 personality registered for level 10
[    1.961154] md: raid6 personality registered for level 6
[    1.971762] md: raid5 personality registered for level 5
[    1.982372] md: raid4 personality registered for level 4
[    1.997431] cpuidle: using governor ladder
[    2.005614] cpuidle: using governor menu
[    2.013467] No iBFT detected.
[    2.019791] TCP cubic registered
[    2.026285] NET: Registered protocol family 10
[    2.035692] lo: Disabled Privacy Extensions
[    2.044421] Mobile IPv6
[    2.049316] NET: Registered protocol family 17
[    2.058292] registered taskstats version 1
[    2.066652] rtc_cmos 00:03: setting system clock to 2009-11-12 09:16:21 UTC (1258017381)
[    2.082887] md: Waiting for all devices to be available before autodetect
[    2.096440] md: If you don't use raid, use raid=noautodetect
[    2.107865] md: Autodetecting RAID arrays.
[    2.116052] md: Scanned 0 and added 0 devices.
[    2.124929] md: autorun ...
[    2.130514] md: ... autorun DONE.
[    2.137323] EXT3-fs: unable to read superblock
[    2.146259] EXT2-fs: unable to read superblock
[    2.155144] List of all partitions:
[    2.162116] No filesystem could mount root, tried:  ext3 ext2
[    2.173686] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,1)

I'm just getting more and more confused....

exvor · 11-12-2009, 01:26 PM

Quote:

[ 2.107865] md: Autodetecting RAID arrays.
[ 2.116052] md: Scanned 0 and added 0 devices.

This is odd. It is loading all of the raid stuff but it seams its not reassembling your raid array. If these were usb drives or something I would say its because the devices are not getting time to load up before the raid detection starts, but I don't think this is your issue.
Did you use raidtools or mdadm to setup your raid?

mostlyharmless · 11-12-2009, 01:34 PM

Your initrd is new as well, I presume? Or do you only have one version? I'm pretty sure you have to make a new initrd for each kernel.

jemenake · 11-12-2009, 02:26 PM

Quote:

Originally Posted by mostlyharmless

Your initrd is new as well, I presume? Or do you only have one version? I'm pretty sure you have to make a new initrd for each kernel.

As per the suggestions of the first replies, I took the initrd out of the boot stanza altogether. That's why I said that it's odd that, even though I've indicated that the raid and ext2/3 drivers should be compiled into the kernel, the ability of the kernel to recognize the raid partitions changes if I remove the initrd from the boot stanza.

mostlyharmless · 11-12-2009, 02:33 PM

Yes, well, I read that. But your initrd might be doing more than loading drivers, are you successful with the old kernel and without the initrd?

jemenake · 11-12-2009, 02:34 PM

Quote:

Originally Posted by exvor

This is odd. It is loading all of the raid stuff but it seams its not reassembling your raid array. If these were usb drives or something I would say its because the devices are not getting time to load up before the raid detection starts, but I don't think this is your issue.
Did you use raidtools or mdadm to setup your raid?

They came pre-made from 1And1. When you sign up for a root-server, they just give you one configured like this. I think it's two 160GB SATA's with something like 4 partitions on each, paired to make 4 MD's. So, /dev/sda1 and /dev/sdb1 make up /dev/md1... etc.

What I find *also* strange is that, even when the MD driver failed to set up the md devices, shouldn't the kernel be able to see the *individual* partitions? For example, shouldn't I be able to rip out the whole /dev/sdb drive and have the kernel still be able to use /dev/sda1 instead of /dev/md1? (Maybe it will, and I just need to change the "root=" part of the boot stanza to be root=/dev/sda1 instead of root=/devb/md1).

jemenake · 11-12-2009, 02:41 PM

Quote:

Originally Posted by mostlyharmless

Yes, well, I read that. But your initrd might be doing more than loading drivers, are you successful with the old kernel and without the initrd?

Old kernel doesn't use initrd.

jemenake · 11-12-2009, 05:42 PM

Here's a related question:

Is there a reliable way to "upgrade" or "update" a .config file from a previous kernel version? I *have* the config file that was used for the working 2.6.16 kernel. However, in the past, I've tried just using a previous .config in a newer kernel and the build would fail. I discovered that, at the least, I'd have to run either "make config" or "make menuconfig" to get any new/changed .config directives into the file... but I still recall having trouble even then.

At present, I'm on the verge of just diff'ing the old config with the current one to make a to-do list of stuff I need to change with menuconfig. Hopefully, there's a slicker way these days.

exvor · 11-13-2009, 11:39 PM

@ jemenake I thought it was make oldconfig or something like that to merge the old one into the new one.

goossen · 02-04-2011, 01:24 PM

I know this is old but I took me a lot of research and headache to find the solution:

Enable CONFIG_SYSFS_DEPRECATED_V2 in the kernel source configuration.
If you run “make menuconfig”, browse to:
General Setup —> and enable (Y) the following:

Code:

enable deprecated sysfs features to support old userspace too (...)

and

Code:

enabled deprecated sysfs features by default

—OR—

Edit your .config file with a text editor and add/modify entry:

Code:

CONFIG_SYSFS_DEPRECATED_V2=y

Mods: I'm replying this thread so other people searching in Google (like me) will find it.