F18 kernel 3.8.2 upgrade hosed boot - dracut rescue mode only
 

I did the usual yum upgrade this morning. Upon reboot, I got as far as dracut's repair prompt. The journal ends with the same message that drops me into the dracut prompt - dracut can't find the root partition.
No system changes, previous upgrades worked, I can still boot into 3.8.1 using the rest of the kernel upgrade packages. Multiple attempts erasing/re-installing the kernel package led to the conclusion that something changed in the kernel install script. My x86_64 system remains the same as when I installed F17, first partition is boot with grub, second partition is a logical volume with seperate / and /home volumes. Dracut can no longer find my root logical volume, the one with everything except /boot and /home, so there must be some assumption that /usr is somehow accessible from the grub boot partition.
The 'Fedora (3.8.1-201.fc18.x86_64)' and 'Fedora (3.8.2-206.fc18.x86_64)' sections are identical; insmod lvm is still in the prologue.
Is there a way to get dracut to find the lvm group-volume containing the root partition?
Thank you for your help!


[The annoying 20-second delay getting a login prompt on a second tty is set aside for another day.]

Is this a normal "update" of the kernel
or
a "upgrade" from fedora 17 to 18

if upgrade , then what did you use
" preupgrade" or " fedup"

Thank you for your help.

F18, x86_64.

`yum -y upgrade` == normal update. I choose the 'upgrade' option to turn on the -obsoletes switch. It's usually the first chore of the day, login as root to a tty console [default telinit 3], invoke the update, if no changes, mount any additional drives/partitions I'll need, then log in to tty2 (after an annoying 20-sec wait) as user and launch XFCE4, then see what's on my todo list.

The list of updated packages for 11Mar13 was:
libwacom-data-0.7-3.fc18.noarch
kernel-tools-libs-3.8.2-206.fc18.x86_64
kernel-tools-3.8.2-206.fc18.x86_64
libwacom-0.7-3.fc18.x86_64
kernel-doc-3.8.2-206.fc18.noarch
kernel-devel-3.8.2-206.fc18.x86_64
systemtap-sdt-devel-2.1-2.fc18.x86_64
1:valgrind-3.8.1-9.fc18.x86_64
kernel-headers-3.8.2-206.fc18.x86_64
kernel-3.8.2-206.fc18.x86_64

Upon reboot, the default kernel-3.8.1 entry was highlighted and the kernel-3.8.2 entry was above it. I usually boot into the new kernel and erase the previous one. My guess is that whatever script creates the initrd after the file installs, /boot links, and grub update, neglected to include the root partition. There were no tools to view and mount any virtual volumes in the rescue mode, which was more focused on displaying the journal. /sbin/new-kernel-package awks for the root device in /etc/fstab, where it usually finds /dev/mapper/vg_[`cat /etc/hostname | cut -d . -f 1`]-lv_root / ext4 ... It then passes --args="root=$rootdevice... to grubby. I thought dracut was the varmint, since that is the prompt where the boot process ceases, unable to find the root device, but after poking around the POSTIN script, /sbin/new-kernel-package may have changed. It has been finding the correct root partition for over a year. The new-kernel file was updated 1/4/2013, but the kernel upgrade to 3.8.1 last month went without a hitch. One of these days, I'll decompress and diff the two initrd files.

Have you tried (from your 3.8.1 boot) running grub2-mkconfig -o /etc/grub2.cfg to create a new grub boot menu, ignoring the RH grubby tool's sometimes problematic intervention? I always do that after any kernel update. :)

so you uninstalled / disabled , i am guessing, "plymouth" gui boot
then use a text only boot up
manually mount drives , instead of having udev do it on boot
using udev rules or fstab ?

then manually launch xfce ( guessing xdm is set and xfce is set as default for "startx" )

A lot of extra work

outputs something like
/devsda1 /boot -- *
dev/sda2 / -- LVM



you can try , as PTrenholme posted , rebuilding grub.conf in grub2

that is NOT recommended !!!
fedora keeps 3 -- three ( might have moved to 5 ) old kernels as an emergency back up
-- and you WILL need them ( a 99% guaranty ) during the 13 month lifetime fedora
-- at least once .

My work is obviously different than other folks. I manually mount partitions because at any time I could have any of a dozen sata drives plugged in, depending on what I'm working on.
I don't want a gui boot. Root sometimes has other stuff to do in the console. Udev does not use the mount options I want.
My user login is also a gui. I have console chores to do there. Startx contains the "--nolisten tcp" server arg, don't know when some update is going to clobber that. Neither udev nor systemd can set up my HPC cluster via NFS shares when I need to do parallel programming.

After over a decade of habit, I invoked rpm -Uvh on the new kernel, now I can't get in and do anything. Grub2.cfg works just fine with both the old and new kernel menu entries identical (soft links in /boot helps). I erase the previous kernel if the current kernel boots and passes my test scripts. I usually roll my own, considering the gazillion modules. I don't want or need pcmcia, serial or parallel ports, samba, and all the other stuff that gets pulled in. Congenital indolence helps me to ignore all that.

I wrote my first program in FORTAN circa 1966, kept it on a roll of yellow tape. In the late eighties I taunted the Mac users by telling them about how they trained a gorilla at the Yerkes Primate Lab to communicate by touching symbols on a screen. But I digress.

--
Henry Pfeil
Principal Support Systems Analyst [retired]
University of Arizona

so is this a manual download from the fedora-update repo
-- current fedora 18 update
"kernel-3.8.2-206.fc18.x86_64.rpm "
bypassing the "yum" package manager and the normal update operations


habits are hard to break but yum triggers the needed scripts to configure the system for a new kernel and gets any needed prerequisites

i would recommend using the std fedora 18 update repo
that will pull all the needed packages that are NEEDED by this update

for manuall rpm install

-- you did blacklist the "kernel", "kernel-devel', and "kernel-headers" along with every Xorg package from being installed by the auto updates and yum ?

you did read the rpm prerequisites needed for"kernel-3.8.2-206.fc18.x86_64.rpm" and dl them also for manual install

For no internet connection - on an isolated box

find all and every needed prerequisite for the downloaded rpm and install them first
-- this dose turn a 5 min. job into a 2 day job


if you do not know fedora updates the kernel and/or xorg about once or twice a week

No.The issue presented itself only after a routine yum update that included the kernel. As usual, yum tags both kernel and kernel-dev as installs, the rest that day were updates.
I resorted to the rpm command only after uninstalling and installing the kernel package using yum. Please read the first message. I know how often updates are pushed to fc18, I check every day. This instance was the first time I've ever seen the dracut journal prompt. When I examined the files from that prompt, the only fs module was btfs, no ext4. Lvm commands all yield not found (`lvm pgscan` `lvm lvscan` etc.). The problems is systemd-udevd. Udev does not pick up the root logical volume. The boot command uses UUID to find the root partition. Suddenly, what worked before stopped working. I've made no changes to the system. Oddly enough, I found a drive with F15 on it. Grub2 booted it without incident. /dev/mapper was populated. The only thing on this broken version that is in /dev/mapper is control. /sys shows seven loop devices, nothing else. Using a Linux From Scratch drive, /dev has a directory named vg_<hostname> that contains vg_root, vg_home, vg_swap, which all appear in /dev/mapper. There's about 40 devices in /dev/block. With systemd-udevd, none of those devices show up.

Problem traced to IBK (Idiot Behind Keyboard). I deleted all of the custom files in /etc/udev/rules.d, /etc/modprobe.d, and /lib/modprobe.d. Et Voilà! Also had to rebuild the rpmdb after yum and rpm got lost in the twilight zone upon invocation, granting me access to the keyboard only after I said the secret word: `ctl-z; kill -9 %1` Thank you everyone who tried to help me. You have mastered the art of patience when dealing with a complete idiot.

If it's just the "I want to set my options" problem you have with udev, have you considered the use of /etc/fstab to set the options you want when you boot your system? The new nofail option makes it simple to have as many "idle" entries you want in fstab that will only take effect if the device is plugged in, or, if you have removable devices, the noauto option can keep a fstab entry from becoming "operational" before you want to use it.

I can empathize with your desire to stick with what has worked for you. (I, too, wrote my first program on punched paper tape.) But there's some joy to be found in using new ways. Sometimes they're even better.

For example, the systemd replacement for init is really a vast improvement, but it took me a long time to find the systemadm tool that makes it understandable. (I was, in fact, writing a similar tool in gawk when I found it.)

Yet again, dracut can't load lvm when root is on a logical volume.
 

--
Obquote: "De gustibus non est disputandum" -(some Roman guy whose ancient works I had to recite in Latin class)

Personally, I don't turn on the computer, then sit there tapping the Enter key waiting for a prompt. I still don't know which parallel algorithm launches everything (openMP, MPI?). The kernel process table gets swamped, then each entry gets a time-slice in turn until they exit? Maybe the IO_WAIT states make it seem like things are zipping right along. Add 20 seconds to get a second console tty....

I have not come to praise systemd. I'm re-opening this thread because it happened again with today's kernel upgrade. Should the occasion arise, this is how I swatted that bug:

dracut --kver 3.8.5-201.fc18.x86_64 --force --show-modules --lvmconf --add lvm --add-drivers sata_nv initramfs-3.8.5-201.fc18.x86_64.img

You don't need --show-modules unless you want to make sure the right modules get tossed into the initramfs. Probably can get away with just "lvm" since that's the root. It's a mystery to me, lvm and the sata drivers are compiled into the kernel. You'd think those would get copied into the tmp/modules, but it just stops after loading btrfs and 7 loops. On the bright side, I didn't cross the palm of Red Hat with gold for the privilege of fixing their code.

If you want to see what's in a dracut generated initrd, the lsinitrd command can be quite useful.

Did you check /etc/dracut.conf, /etc/dracut.conf.d/*, and /usr/lib/dracut/dracut.conf.d/* to see if you have, perhaps, excluded the lvm from the default dracut processing?

Why on earth would I do that?! The same configuration has worked for over a year of kernel upgrades. Someone changed something that hosed what once worked effortlessly. If some goof changed something that required me to go edit some configuration files that used to work flawlessly, courtesy demands giving prior notice, or at least an RFC.

As a general rule, when rpm replaces a configuration file, the old configuration is saved with a .rmpsave appended to its name. (And if, instead, the update script suggests a new configuration, the new suggestion is installed with a .rpmnew appended to the name.

For example:I agree that a "notice" in the log files might be easier to see, but I suspect that the developers think everyone looks at the changelog in the rpm file(s) before they install the change. After all, they always do that.

OK, I'll play once more. /etc/dracut.conf was created February 5, 2013. It has both mdadmconf and lvmconf set to yes. I didn't install it. As for rpmnew/rpmsave, been there; done that. It's a cron.weekly job. At the top of the page, you may notice that it was the kernel upgrade to 3.8.2 that presented the hosed initramfs. On the date of the last time dracut.conf and dracut.conf.d/01-dist.conf,
2013FEB05 - the update installed kernel-3.7.5-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.4-204.fc18.x86_64.
2013FEB09 - the update installed kernel-3.7.6-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.5-201.fc18.x86_64.
2013FEB16 - the update installed kernel-3.7.7-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.6-201.fc18.x86_64.
2013FEB18 - the update installed kernel-3.7.8-202.fc18.x86_64; upon successful boot, I removed kernel-3.7.7-201.fc18.x86_64
2013FEB21 - the update installed kernel-3.7.9-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.8-202.fc18.x86_64.
2013FEB29 - the update installed kernel-3.7.9-205.fc18.x86_64; upon successful boot, I removed kernel-3.7.9-201.fc18.x86_64.
2013MAR03 - the update installed kernel-3.8.1-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.9-205.fc18.x86_64.
2013MAR11 - the update installed kernel-3.8.2-206.fc18.x86_64; no boota dis, stuck in dracut with only btrfs loaded, no ext4, no lvm, no mdadmconf, no joy.
I finished the day using a distribution that has not presented any boot errors since 1994.
2013MAR12 - using kernel-3.8.1-201 that I did not remove yet, since 3.8.2 didn't boot, I wrote the first entry of this thread. I didn't have time to do a complete post-mortem, I mistakenly blamed anything I could think of that would produce such errant behaviour.
Note Bene: the same dracut confs that had been working through seven prior kernel upgrades remained in situ, as did everything else in /etc. Nothing of consequence was installed or upgraded between Mar03 and Mar11, some python and perl modules, yum, mesa, fftw, lapack, etc.
I cannot afford to devote any more non-billable hours to this matter.