Root filesystem stuck in "read only" mode.
 

Hi All,

I have Fedora Core 2 installed and was updating some packages when it seems my root disk was corrupted. Error messages indicated that installation was unable to continue because the filesystem was "read only" (it wasn't until then).

I wasn't able to shut down normally, so I hit the ol' reset button. GRUB came up as usual, but things went downhill from there. A multitude of error messages flashed by, most stating that this, that, or the other file could not be started due to "read only filesystem". It finally stopped with a message indicating the X server could not start due to an internal error, and presented me with a text prompt for login.

I logged in as root, then ran "fsck /dev/hda1". A dozen or so problems popped up with a request to fix. I answered "y" to all of them, and the filesystem was ultimately declared "clean".

Now I rebooted again, but am still having the "read only filesystem" errors. Fsck always shows that "/" is clean, but I can't write any files. I was successful with executing "mount -o remount,rw /dev/hda1 /" and then could write files, but when I reboot, it always reverts back to "read only". I'm still unable to run the gui.

Is there some way of getting the "read/write" mode to stick? Would this fix my gui not starting? I'd really hate to have to wipe the disk and start over.

I'm a recent Linux convert, so please be gentle.

Thanks in advance.

You could put the remount command in /etc/rc.d/boot.local (or wherever Fedora puts the boot.local file).

In the same directory, you may also want to look at the “boot” script, which contains the start-up instructions for the boot process, from starting the init process (after the kernel's loaded) through to starting up system services. It's this file that responsible for fscking the hard disks.

In general, the root filesystem will be mounted read-only when your computer starts up, and remounted read/write after it's been checked. The fact that the remount (in the “boot” script) has fallen out somewhere, or is failing, might be a little concerning. Or it could just have been messed up by the packages being installed, especially if it corrupted the disk.

You'll certainly need a read/write /var filesystem to start X-Windows, so this may well be related to your GUI not starting.

There is no boot.local file in the /etc/rc.d directory. I'm not sure what script is being run after the kernel loads.

Is there a way of doing an interactive startup in Linux, the way you can in Windows, where each line of the startup script asks to be executed or not? That might help nail down where the failures are.

Thanks.

Not so sure on FC, but on my Mandrake system it says "Press I for interactive startup"
maybe try hitting I during the bootup
*note* thats a capitol i

After the kernel loads, it loads up the init process, which starts the first boot script. You can find out what this is by reading the /etc/inittab file. Somewhere near the top is a line like this: The “si” means “system init”, but the important thing is that this is the first line with nothing between the first two colons (::); in this case, /etc/init.d/rcS gets loaded first in all cases.

You can trace it through from here by reading the /etc/init.d/rcS script (which is just a bash script) and seeing what other scripts it calls.

This depends on the setup of the system. You don't normally need anything like this because Linux logs everything it does during boot, usually in /var/log/boot or just /var/log/messages (also run dmesg to see kernel messages).

I looked at those files and ran dmesg, but they only show the last successful boot. I figure that's because the unsuccessful boots left the filesystem read-only, so log files could not be written.

I also checked the inittab file. The system initialization line is si::sysinit:/etc/rc.d/rc.sysinit. I looked at that file and only saw one line: touch /var/lock/subsys/local. In the same directory there is also a "rc" file and a bunch of directories named "rc0.d" ... "rc5.d" which contain links to scripts. I suspect that depending on the runlevel, the corresponding directory of scripts is run at boot time. I have yet to see anything that indicates fsck is run at any time!

It would still be nice if there was a mode for the init process to identify each script before it is run and confirm each script's execution. Then I could see what scripts are executed and chose which ones to run. I remember reading something about "confirmation boot mode" or something similar, but can't remember where. Could it possibly be a kernel argument?

Thanks for your help so far. This has been a real learning experience.

There may be an interactive mode. SuSE doesn't have one that I know of, and I don't use Fedora, so I'm afraid I can't help you there.

But I can help you to get some log-files.

There are two boot-loaders commonly used on a Linux system, grub and lilo. Both of these have the option to let you boot into a read-only filesystem, which is normally enabled because you can then check the filesystem on boot before remounting it read/write.

You can turn this off with the bootloader, but you can boot the filesystem read-write with by passing “read-write” as a command-line parameter. You should then at least get a log of whatever's going wrong.

There may be other options in /etc/fstab with nothing between the colons, but I don't get why rc.sysinit would be virtually empty. Is there a line saying “bootwait” at all?

OK, I'm using GRUB, so i change the kernel arguments to read "rw root=LABEL=/ rhgb quiet" (ro was changed to rw) and then booted. While booting, I see the line that shows the kernel argument "rw", but a few lines later, after INIT starts, a line is printed that says "touch: cannot touch '/var/lock/subsys/local': Read-only file system". Various other lines indicate the inability to write or open files. Once I log in, attempting to write a file causes a "read-only filesystem" error.

There is no "bootwait" command in the rc.sysinit file -- there is only one command line (touch /var/lock/subsys/local) and some comments. The comments indicate that this script will be executed *after* all the other init scripts, but it looks like it is the *first* script to be executed by INIT (from the error message I quoted above).

At this point, it might be more time effective for me to wipe the disk and re-install. I don't have much data to save, so all I'll have to do is remember which packages I had installed. As a last resort, I might try moving this thread to the Fedora forum to see if there is any insight there.

Thanks for your help so far! I seem to be gaining a fair bit of knowledge in the Linux boot process.

The “bootwait” line would be in /etc/inittab, not rc.sysinit.

You should also be able to just type the command-line options in at the boot prompt, but re-running grub should have the same effect.