LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Fedora
User Name
Password
Fedora This forum is for the discussion of the Fedora Project.

Notices

Reply
 
Search this Thread
Old 03-12-2013, 04:29 PM   #1
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Rep: Reputation: Disabled
F18 kernel 3.8.2 upgrade hosed boot - dracut rescue mode only


I did the usual yum upgrade this morning. Upon reboot, I got as far as dracut's repair prompt. The journal ends with the same message that drops me into the dracut prompt - dracut can't find the root partition.
No system changes, previous upgrades worked, I can still boot into 3.8.1 using the rest of the kernel upgrade packages. Multiple attempts erasing/re-installing the kernel package led to the conclusion that something changed in the kernel install script. My x86_64 system remains the same as when I installed F17, first partition is boot with grub, second partition is a logical volume with seperate / and /home volumes. Dracut can no longer find my root logical volume, the one with everything except /boot and /home, so there must be some assumption that /usr is somehow accessible from the grub boot partition.
The 'Fedora (3.8.1-201.fc18.x86_64)' and 'Fedora (3.8.2-206.fc18.x86_64)' sections are identical; insmod lvm is still in the prologue.
Is there a way to get dracut to find the lvm group-volume containing the root partition?
Thank you for your help!


[The annoying 20-second delay getting a login prompt on a second tty is set aside for another day.]
 
Old 03-12-2013, 07:55 PM   #2
John VV
Guru
 
Registered: Aug 2005
Posts: 13,057

Rep: Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741
Is this a normal "update" of the kernel
or
a "upgrade" from fedora 17 to 18

if upgrade , then what did you use
" preupgrade" or " fedup"
 
Old 03-13-2013, 11:23 AM   #3
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Thank you for your help.

F18, x86_64.

`yum -y upgrade` == normal update. I choose the 'upgrade' option to turn on the -obsoletes switch. It's usually the first chore of the day, login as root to a tty console [default telinit 3], invoke the update, if no changes, mount any additional drives/partitions I'll need, then log in to tty2 (after an annoying 20-sec wait) as user and launch XFCE4, then see what's on my todo list.

The list of updated packages for 11Mar13 was:
libwacom-data-0.7-3.fc18.noarch
kernel-tools-libs-3.8.2-206.fc18.x86_64
kernel-tools-3.8.2-206.fc18.x86_64
libwacom-0.7-3.fc18.x86_64
kernel-doc-3.8.2-206.fc18.noarch
kernel-devel-3.8.2-206.fc18.x86_64
systemtap-sdt-devel-2.1-2.fc18.x86_64
1:valgrind-3.8.1-9.fc18.x86_64
kernel-headers-3.8.2-206.fc18.x86_64
kernel-3.8.2-206.fc18.x86_64

Upon reboot, the default kernel-3.8.1 entry was highlighted and the kernel-3.8.2 entry was above it. I usually boot into the new kernel and erase the previous one. My guess is that whatever script creates the initrd after the file installs, /boot links, and grub update, neglected to include the root partition. There were no tools to view and mount any virtual volumes in the rescue mode, which was more focused on displaying the journal. /sbin/new-kernel-package awks for the root device in /etc/fstab, where it usually finds /dev/mapper/vg_[`cat /etc/hostname | cut -d . -f 1`]-lv_root / ext4 ... It then passes --args="root=$rootdevice... to grubby. I thought dracut was the varmint, since that is the prompt where the boot process ceases, unable to find the root device, but after poking around the POSTIN script, /sbin/new-kernel-package may have changed. It has been finding the correct root partition for over a year. The new-kernel file was updated 1/4/2013, but the kernel upgrade to 3.8.1 last month went without a hitch. One of these days, I'll decompress and diff the two initrd files.
 
Old 03-13-2013, 12:54 PM   #4
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,151

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
Have you tried (from your 3.8.1 boot) running grub2-mkconfig -o /etc/grub2.cfg to create a new grub boot menu, ignoring the RH grubby tool's sometimes problematic intervention? I always do that after any kernel update.

Last edited by PTrenholme; 03-13-2013 at 12:56 PM.
 
Old 03-13-2013, 01:20 PM   #5
John VV
Guru
 
Registered: Aug 2005
Posts: 13,057

Rep: Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741
so you uninstalled / disabled , i am guessing, "plymouth" gui boot
then use a text only boot up
manually mount drives , instead of having udev do it on boot
using udev rules or fstab ?

then manually launch xfce ( guessing xdm is set and xfce is set as default for "startx" )

A lot of extra work

Code:
fdisk -l
outputs something like
/devsda1 /boot -- *
dev/sda2 / -- LVM



you can try , as PTrenholme posted , rebuilding grub.conf in grub2

Quote:
the default kernel-3.8.1 entry was highlighted and the kernel-3.8.2 entry was above it. I usually boot into the new kernel and erase the previous one.
that is NOT recommended !!!
fedora keeps 3 -- three ( might have moved to 5 ) old kernels as an emergency back up
-- and you WILL need them ( a 99% guaranty ) during the 13 month lifetime fedora
-- at least once .
 
Old 03-14-2013, 01:06 PM   #6
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
My work is obviously different than other folks. I manually mount partitions because at any time I could have any of a dozen sata drives plugged in, depending on what I'm working on.
I don't want a gui boot. Root sometimes has other stuff to do in the console. Udev does not use the mount options I want.
My user login is also a gui. I have console chores to do there. Startx contains the "--nolisten tcp" server arg, don't know when some update is going to clobber that. Neither udev nor systemd can set up my HPC cluster via NFS shares when I need to do parallel programming.

After over a decade of habit, I invoked rpm -Uvh on the new kernel, now I can't get in and do anything. Grub2.cfg works just fine with both the old and new kernel menu entries identical (soft links in /boot helps). I erase the previous kernel if the current kernel boots and passes my test scripts. I usually roll my own, considering the gazillion modules. I don't want or need pcmcia, serial or parallel ports, samba, and all the other stuff that gets pulled in. Congenital indolence helps me to ignore all that.

I wrote my first program in FORTAN circa 1966, kept it on a roll of yellow tape. In the late eighties I taunted the Mac users by telling them about how they trained a gorilla at the Yerkes Primate Lab to communicate by touching symbols on a screen. But I digress.

--
Henry Pfeil
Principal Support Systems Analyst [retired]
University of Arizona
 
Old 03-14-2013, 01:50 PM   #7
John VV
Guru
 
Registered: Aug 2005
Posts: 13,057

Rep: Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741Reputation: 1741
Quote:
I invoked rpm -Uvh on the new kernel,
so is this a manual download from the fedora-update repo
-- current fedora 18 update
"kernel-3.8.2-206.fc18.x86_64.rpm "
bypassing the "yum" package manager and the normal update operations


habits are hard to break but yum triggers the needed scripts to configure the system for a new kernel and gets any needed prerequisites

i would recommend using the std fedora 18 update repo
Code:
"su -
yum update kernel kernel-devel
that will pull all the needed packages that are NEEDED by this update

for manuall rpm install

-- you did blacklist the "kernel", "kernel-devel', and "kernel-headers" along with every Xorg package from being installed by the auto updates and yum ?

you did read the rpm prerequisites needed for"kernel-3.8.2-206.fc18.x86_64.rpm" and dl them also for manual install

For no internet connection - on an isolated box

find all and every needed prerequisite for the downloaded rpm and install them first
-- this dose turn a 5 min. job into a 2 day job


if you do not know fedora updates the kernel and/or xorg about once or twice a week

Last edited by John VV; 03-14-2013 at 01:53 PM.
 
Old 03-15-2013, 11:59 AM   #8
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Quote:
...so is this a manual download from the fedora-update repo
No.The issue presented itself only after a routine yum update that included the kernel. As usual, yum tags both kernel and kernel-dev as installs, the rest that day were updates.
I resorted to the rpm command only after uninstalling and installing the kernel package using yum. Please read the first message. I know how often updates are pushed to fc18, I check every day. This instance was the first time I've ever seen the dracut journal prompt. When I examined the files from that prompt, the only fs module was btfs, no ext4. Lvm commands all yield not found (`lvm pgscan` `lvm lvscan` etc.). The problems is systemd-udevd. Udev does not pick up the root logical volume. The boot command uses UUID to find the root partition. Suddenly, what worked before stopped working. I've made no changes to the system. Oddly enough, I found a drive with F15 on it. Grub2 booted it without incident. /dev/mapper was populated. The only thing on this broken version that is in /dev/mapper is control. /sys shows seven loop devices, nothing else. Using a Linux From Scratch drive, /dev has a directory named vg_<hostname> that contains vg_root, vg_home, vg_swap, which all appear in /dev/mapper. There's about 40 devices in /dev/block. With systemd-udevd, none of those devices show up.
 
Old 03-23-2013, 10:58 AM   #9
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Problem traced to IBK (Idiot Behind Keyboard). I deleted all of the custom files in /etc/udev/rules.d, /etc/modprobe.d, and /lib/modprobe.d. Et Voilą! Also had to rebuild the rpmdb after yum and rpm got lost in the twilight zone upon invocation, granting me access to the keyboard only after I said the secret word: `ctl-z; kill -9 %1` Thank you everyone who tried to help me. You have mastered the art of patience when dealing with a complete idiot.
 
Old 03-25-2013, 10:31 AM   #10
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,151

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
Quote:
Originally Posted by hpfeil View Post
My work is obviously different than other folks. I manually mount partitions because at any time I could have any of a dozen sata drives plugged in, depending on what I'm working on.
I don't want a gui boot. Root sometimes has other stuff to do in the console. Udev does not use the mount options I want.
...
If it's just the "I want to set my options" problem you have with udev, have you considered the use of /etc/fstab to set the options you want when you boot your system? The new nofail option makes it simple to have as many "idle" entries you want in fstab that will only take effect if the device is plugged in, or, if you have removable devices, the noauto option can keep a fstab entry from becoming "operational" before you want to use it.

I can empathize with your desire to stick with what has worked for you. (I, too, wrote my first program on punched paper tape.) But there's some joy to be found in using new ways. Sometimes they're even better.

For example, the systemd replacement for init is really a vast improvement, but it took me a long time to find the systemadm tool that makes it understandable. (I was, in fact, writing a similar tool in gawk when I found it.)
 
Old 04-03-2013, 03:04 PM   #11
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Lightbulb Yet again, dracut can't load lvm when root is on a logical volume.

Quote:
For example, the systemd replacement for init is really a vast improvement...
--
Obquote: "De gustibus non est disputandum" -(some Roman guy whose ancient works I had to recite in Latin class)

Personally, I don't turn on the computer, then sit there tapping the Enter key waiting for a prompt. I still don't know which parallel algorithm launches everything (openMP, MPI?). The kernel process table gets swamped, then each entry gets a time-slice in turn until they exit? Maybe the IO_WAIT states make it seem like things are zipping right along. Add 20 seconds to get a second console tty....

I have not come to praise systemd. I'm re-opening this thread because it happened again with today's kernel upgrade. Should the occasion arise, this is how I swatted that bug:
dracut --kver 3.8.5-201.fc18.x86_64 --force --show-modules --lvmconf --add lvm --add-drivers sata_nv initramfs-3.8.5-201.fc18.x86_64.img
You don't need --show-modules unless you want to make sure the right modules get tossed into the initramfs. Probably can get away with just "lvm" since that's the root. It's a mystery to me, lvm and the sata drivers are compiled into the kernel. You'd think those would get copied into the tmp/modules, but it just stops after loading btrfs and 7 loops. On the bright side, I didn't cross the palm of Red Hat with gold for the privilege of fixing their code.
 
Old 04-05-2013, 11:33 PM   #12
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,151

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
If you want to see what's in a dracut generated initrd, the lsinitrd command can be quite useful.

Did you check /etc/dracut.conf, /etc/dracut.conf.d/*, and /usr/lib/dracut/dracut.conf.d/* to see if you have, perhaps, excluded the lvm from the default dracut processing?
 
Old 04-07-2013, 10:36 AM   #13
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Why on earth would I do that?! The same configuration has worked for over a year of kernel upgrades. Someone changed something that hosed what once worked effortlessly. If some goof changed something that required me to go edit some configuration files that used to work flawlessly, courtesy demands giving prior notice, or at least an RFC.
 
Old 04-07-2013, 12:41 PM   #14
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,151

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
Quote:
Originally Posted by hpfeil View Post
... courtesy demands giving prior notice ...
As a general rule, when rpm replaces a configuration file, the old configuration is saved with a .rmpsave appended to its name. (And if, instead, the update script suggests a new configuration, the new suggestion is installed with a .rpmnew appended to the name.

For example:
Code:
$ sudo find /etc -type f -name *.rpm*
/etc/pam.d/system-auth.rpmnew
/etc/pam.d/password-auth.rpmnew
/etc/pam.d/postlogin.rpmnew
/etc/dnf/dnf.conf.rpmnew
/etc/os-release.rpmsave
/etc/pki/tls/certs/ca-bundle.crt.rpmsave
/etc/pki/tls/certs/ca-bundle.trust.crt.rpmsave
/etc/pki/java/cacerts.rpmsave
/etc/kde/kdm/kdmrc.rpmnew
/etc/hp/hplip.conf.rpmnew
/etc/yum.repos.d/fedora-rawhide.repo.rpmsave
/etc/fedora-release.rpmsave
I agree that a "notice" in the log files might be easier to see, but I suspect that the developers think everyone looks at the changelog in the rpm file(s) before they install the change. After all, they always do that.
 
Old 04-08-2013, 01:16 PM   #15
hpfeil
Member
 
Registered: Nov 2010
Location: Tucson, Arizona US
Distribution: Slackware Current, custom kernel, amd64, Beyond LinuxFromScratch
Posts: 130
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
OK, I'll play once more. /etc/dracut.conf was created February 5, 2013. It has both mdadmconf and lvmconf set to yes. I didn't install it. As for rpmnew/rpmsave, been there; done that. It's a cron.weekly job. At the top of the page, you may notice that it was the kernel upgrade to 3.8.2 that presented the hosed initramfs. On the date of the last time dracut.conf and dracut.conf.d/01-dist.conf,
2013FEB05 - the update installed kernel-3.7.5-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.4-204.fc18.x86_64.
2013FEB09 - the update installed kernel-3.7.6-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.5-201.fc18.x86_64.
2013FEB16 - the update installed kernel-3.7.7-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.6-201.fc18.x86_64.
2013FEB18 - the update installed kernel-3.7.8-202.fc18.x86_64; upon successful boot, I removed kernel-3.7.7-201.fc18.x86_64
2013FEB21 - the update installed kernel-3.7.9-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.8-202.fc18.x86_64.
2013FEB29 - the update installed kernel-3.7.9-205.fc18.x86_64; upon successful boot, I removed kernel-3.7.9-201.fc18.x86_64.
2013MAR03 - the update installed kernel-3.8.1-201.fc18.x86_64; upon successful boot, I removed kernel-3.7.9-205.fc18.x86_64.
2013MAR11 - the update installed kernel-3.8.2-206.fc18.x86_64; no boota dis, stuck in dracut with only btrfs loaded, no ext4, no lvm, no mdadmconf, no joy.
I finished the day using a distribution that has not presented any boot errors since 1994.
2013MAR12 - using kernel-3.8.1-201 that I did not remove yet, since 3.8.2 didn't boot, I wrote the first entry of this thread. I didn't have time to do a complete post-mortem, I mistakenly blamed anything I could think of that would produce such errant behaviour.
Note Bene: the same dracut confs that had been working through seven prior kernel upgrades remained in situ, as did everything else in /etc. Nothing of consequence was installed or upgraded between Mar03 and Mar11, some python and perl modules, yum, mesa, fftw, lapack, etc.
I cannot afford to devote any more non-billable hours to this matter.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Possible to boot into linux rescue mode without installation media? veeruk101 Linux - Newbie 3 07-18-2011 11:23 PM
Boot Error + GRUB + Rescue Mode najeebsyed2 Linux - Software 9 11-01-2007 09:08 AM
Debian rescue mode: Kernel panic manudath Linux - Software 0 02-16-2005 12:25 PM
How to boot into rescue mode with 2.4 (alt1) lewisdw Mandriva 0 08-08-2004 12:58 PM
Boot only to Rescue Mode: Filesystem Problems JZL240I-U Linux - General 9 07-12-2004 04:28 AM


All times are GMT -5. The time now is 04:00 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration