I recently installed Debian on a 2nd system, and I was in the process of getting it configured to my liking when it failed on me.

I don't know what happened, because as far as I can tell, I did nothing serious that could've caused such a serious problem. The last thing I did was install the msttcorefonts to try to get the Macromedia Flash plugin working. Unfortunately, it seemed to have the opposite effect, breaking the plugin completely, so I decided to try rebooting. But now it fails to come back up.

The system makes it through the hardware detection phase smoothly, but when it goes to load the filesystem it can't do it. It shows the following (and I'm having to type it by hand, so it's a bit abbreviated):

From there it drops me into a busybox shell prompt.

I've already tried booting into Knoppix and running fsck on the drive, so I know it's not a hardware problem. My best guess is that something is keeping udev from starting, and at the same time the static /dev is also not working. And I have no idea how to fix it either. I've never had to work on Linux at such a low level before. Any suggestions?

All the "/root" stuff in the output looks weird. What's your /etc/fstab look like?

I can't find any fstab at all when I'm in the emergency shell. The /etc directory only has 3 entries. I suppose it's still just a temporary ram drive at that point. But from Knoppix, I can access the original, and there seems to be nothing strange about it. I haven't even edited the default entries yet, except to reorganize them a little and add some nfs entries.

Here's the important lines for my main drive. Pretty standard:
I have no idea where the '/root' stuff is coming from either, but it doesn't seem to be from here. I tried checking out the "/scripts/init-bottom" stuff, which I found under /hda1 /etc/mkinitramfs, but they're just empty directories under Knoppix.

Am I going to have to give up and reinstall again? I'd really hate to do that. Even though it isn't that difficult to do at this point, it would be a pain to have to redo a lot of the changes I've already made.

Given it's a fresh install, why not start from scratch? I know you might get it working from the single user prompt, but it'll probably take you more time then a re-install and success isn't guaranteed. Just at thought...

Regards, Dutch Master

<edit: I see you posted a reply just before mine. I still think re-installing and then customising your system would be the fastest way forward now>

What does grub's config look like? (/boot/grub/menu.lst)

I haven't changed anything in my grub config either. I hadn't yet gotten around to compiling a new kernel, so I only have the one stock kernel loaded. The system has gone through several reboots already with no problems, so whatever happened has to be something I changed recently.

@ Dutch Master: Yes, I will of course redo it if necessary, but I'm hoping it won't come to that yet. But besides the time and effort involved, I'm also curious and a bit concerned as to just what happened. I can't think of anything I did that could've caused it. The only system-level changes I remember doing are adding some nfs mounts to fstab and some aliases to /etc/profiles. I've also added and removed some programs, and run update a couple of times, but those don't seem to have caused any trouble. I don't believe I removed anything system-critical.

It's not a crisis situation in any case, so I'm willing to keep at it for a few days. I don't like the idea of just giving up and never learning what happened.

Update: I found a page that describes something almost like my problem.

http://lists.debian.org/debian-kerne.../msg00173.html

It isn't exactly like mine though. His says "mounting /dev/hda2 on /root failed: no such device", for example, whereas mine doesn't. But it's close, and it started just about the same way mine did. I don't remember now if I ran update before booting, but it's likely.

I wanted to try his workaround, but I'm not sure if I'm doing it right. I tried "mount -t reiserfs /dev/hda1 /root" at the busybox prompt, but all I got back was a "device or resource busy" message. Am I doing that right, or is it something I should be doing at the grub login? Or should I be trying something different?

I sure wish I had another kernel installed to fall back on now. I'm wondering now, is there any way to boot my system from a boot floppy or something? I don't have one handy, but I could make one from my other box if someone showed me what to do.

I feel that if I could actually get the thing to boot up, I could run another update, install another kernel, or perhaps roll back something and get it working again. But I don't know how, or if it's possible, to do such things from the outside. How about chroot? would it be possible to chroot into it from Knoppix and install stuff that way? Or would that be impossible since the kernel wouldn't be running?

You may be running into the problem I had last week. For some reason, udev doesn't upgrade when you do an "apt-get upgrade". So, you need to do it manually. I wonder how many people have encountered this and gave up out of frustration or wiped their drives, thinking it was a hardware problem?

You need to get yourself a knoppix CD and open a terminal window in it to do the following:

[HTML]
su -
mount -t ext3 -o rw /dev/hda1 /mnt
chroot /mnt
apt-get update; apt-get upgrade; apt-get install udev
[/HTML]

So it looks like I was right that chroot could get me in there.

Well, I'm trying to do what you suggest, but I can't seem to get it to work. First of all, my /usr is on a different partition, so I also ran "mount -o rw /dev/hda5 /mnt/hda1/usr" before chrooting into /mnt/hda1. That got the programs working so I could run apt-get update, but I kept getting a lot of http error failures for some reason when I did so. I finally got enough package info downloaded to run apt-get upgrade though. But when I try to run it, all I get is the following:
At first I thought it might be something to do with the fact that I mounted /usr from the "outside", so I undid it and tried to mount it from within the chroot, but it refuses to do so. I guess it was trying to use the non-functional hda1 versions instead.

Anyway, looking closer, it seems to be trying to find a working /dev/null, but can't because the kernel for it isn't actually running. I've tried several things such as linking the hda1/dev/null to the knoppix one, but I can't seem to do anything that works.

So what can I do about it?

Rrrr.

Well, I somehow got it to work by simply removing (renaming) /dev/null. The upgrade went smoothly, but udev said that it was already at the newest version. I also installed a newer kernel version just to be safe. But when I rebooted...nothing. It still fails with either kernel. I'm starting to think I might just have to give up after all.

Got it!!!

I downloaded the testing version of udev (0.091-2) from packages.debian.org and installed it via chroot, and that did it. The system booted right up.

Obviously there's a bad bug in udev version 0.092-1. I recommend everyone stay away from this one.

Thanks for all your help. The chroot experience especially will come in handy.

I think it's a problem with updating it - because when I mounted my boot drive, chrooted to it and then did an apt-get upgrade udev, it worked. However, I had JUST done an apt-get upgrade -- so everything should have been the latest version.

Actually, I haven't checked but it may be that udev isn't even installed in the first place. Either way, it definitely sucks. I reinstalled once with a fresh drive before I started looking further into the problem and found that it was a problem with udev.

Hopefully they'll fix that soon. For now, it's good to know you can just chroot from busybox (I've never even seen busybox come up before so I wasn't even sure what it was - figured it must have been something like usermode in Solaris).

Glad to hear things are back and going again.

Woah, sorry to confuse you, but I didn't do it through busybox. This was all done through Knoppix. I'd never even seen Busybox either, but I can only assume that it's a default emergency shell that's just right for accessing the bootup ramdisk. I didn't get the impression that it was powerful enough to do any major work, at least not in that configuration. I couldn't get anything but the most basic of programs to work, for example.

And at least in my case, I don't think it was just an upgrade issue. As I mentioned, it still failed to boot even after I confirmed that udev was installed and the newest version. It was only after I downgraded that I got it working again.

It could be that whatever bug there is may only cause problems on certain combinations of setups though, so even though it worked for you, it could still fail for me.

BTW, I like udev. In spite of the growing pain problems like this, I can already see just how much potential it has. I'm convinced that it's going to put the device management of other OS's to shame, especially the Redmond one. Once I get settled down again, I'm going to go in and play with some of the options, and see what I can get it to do. There are a couple of devices I have that are a bit of a pain to work with, and I think udev will be able to automate most of it, for example.