|
Filesystems always mounted as read-only on boot after recent system freeze
I am not sure that this is a Fedora problem but my googles haven't given me a direction to pursue or where I should be looking to figure out the problem so I'm asking those more knowledgeable for help.
I have a dual Opteron machine with a software raided pair of drives running the Fedora x86_64 builds. This machine occasionally freezes which I've guessed to be a problem from the ATI drivers. The problem: I try to boot and all kinds of errors are thrown during startup because nothing can write to the drives and I end up at a semi-normal prompt(hostname setting gets screwed up during the boot so I end up with a user@(none) prompt) in a read-only filesystem. Navigation and read-only actions all seem to work fine. Things I'm guessing could be relevant that were changed before the post-freeze problem started: 1) My machine was a Fedora Core 3 w/updates OS and any of a bunch of kernels up to the most recent available on the 25th for that distrib. I did a: Quote:
3) did a yum update which updated 20 lines worth of files in /var/log/yum.log Everything normal. Eventually had a system freeze and did a power-off reboot. 4) I allowed fsck to look at filesystem as I always do after a freeze since there are always inconsistencies. 5) Next time I checked in on the machine I had a login prompt and didn't notice anything was wrong until I attempted to invoke X and it threw gobs of errors because it couldn't write lock files and whatnot. 5) I reboot, end up in the same spot. Try mounting a partition, in this case /boot and it works fine and I can write to a test file in it. 6) I looked the last entries in /var/log/messages and don't see anything that appears likely. This and other logs seem to all stop at the point filesystem started being booted as read only. They all look normal to me. 7) I wonder if grub is doing something and I remove the 'ro' portions from a test case kernel version (reg and smp) boot arguments. No luck so I'm hoping for ideas. What do I need to do to track down how to fix this? |
Check your /etc/fstab to make sure that it is intact, correct. Also If you could post your Boot output using /var/log/messages, dmesg, /var/log/syslog.
This might help explain why it's failing to mount disks, or why it's booting wierd. ALSO, it might be helpful to see your menu.1st / grub.conf to make usre you're not running the setup with ro |
log files
I'm not sure as to how I can capture the boot output or anything else that is being output while the system loads as the logs all stopped being updated/written to when this problem started due presumably to their wanting to write to a system that has already been mounted as read only. The last entries in the boot, messages logs date to the 24th, before the problem. I haven't included them until I hear you are still interested in seeing them.
I've copied the dmesg output, my fstab which seems ok on a quick glance, and syslog below. fstab first: hmm, this was last touched by something on the day of doom, the 25th Quote:
Quote:
Quote:
Quote:
|
Can you mount the disks after you boot up?
If not, what errors do you get? Can you give me an idea of what the errors are? also, pressing the 'pause' button on the keyboard sometimes will stop kernel output, <enter> should restart it. |
I know essentially nothing about mounting and unmounting. I've done it manually a few times but its just rote repetition of something I've seen shown somewhere without any deep understanding of what I am doing.
After a boot, I've been able to: mount /boot and get into the boot partition in a r/w manner. That is how I was able to generate the dmesg results. If I try to: umount -v / I get '/dev/md0 unmounted' but I actually have no problems navigating to it and using it still. If I then immediately try to remount it: mount -v / I get 'mount: /dev/md0 already mounted or / busy' 'mount: according to mtab, /dev/md0 is already mounted on /' Which makes sense since umount isn't able to write to mtab. I haven't managed to get the boot to pause yet through hitting the pause/break button. |
I can view part of the boot buffer upt to just prior to the "Welcome to Fedora Core" message using shift+pg up b
---------------- md: ... autorun DONE Creating root device Mounting root filesystem kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Switching to new root INIT: version 2.85 booting ----------------- Now the 1st error I can see in the portion of the buffer I can reach... ----------------- /etc/rc.d/rc.sysinit: line 30: /etc/init.d/functions: No such file or directory Welcome to Fedora Core Press 'I' to enter interactive startup. ------------------ Pressing 'I' is ignored as are pause, scroll lock though pause does work prior to grub getting going. There are further errors... ------------------ /sbin/start_udev: line 35: /etc/init.d/funcitons: No such file or directory /etc/rc.d/rc.sysinit: line 135: strstr: command not found ## GOBS OF SYSINIT COMMAND NOT FOUND ERRORS ## ------------------- This is followed by lots of hardware being found and initialized properly. and then a desire to see if I want to foce a file sysstem integrity check [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/md0 ## AND OTHER PARTITIONS ## and a bunch of clean-up stuff in /var/lock throwing errors such as this example: rm: cannot remove '/var/lock/subsys/acpid' : Read-only file system And then I get a login prompt which will get me logged on and able to view the filesystem though not modify it. |
after logging in, mount /boot
them mount /dev/hda1 (or your root partition) /boot/grub or create a directory in /boot and mount it there see if you can mount it readonly, and if not, why |
More info about the symptoms of this problem in response to mrGenixus' last post.
I did the following: # mount /boot # cd /boot # mkdir test_mnt # mount /dev/md0 test_mnt This then mounts my root filesystem in read-only mode under test_mnt I can also do this to set my root filesystem to read-write once logged on # mount -o remount / This will get me write access to the root filesystem. However this is lost upon reboot. Where could this be getting set improperly? |
Please try to get a complete list of errors from boot
|
With regard to getting more messages from the boot process someone will have to come up with a way for me to log these messages. This system boots very rapidly, it scrolls info far too quickly for human eyes to track. Normally that is a plus! Logging is not happening at all as far as I can tell. I can not pause the boot process via any technique to be able to look at std out. I think I've posted the methods I used in attempting to pause the boot in an earlier email and have hammered every key I could in just hopes. The last point where the boot process will respond to keyboard input is grub.
Is this a new safety feature of some sort reminiscent of safe-mode resulting from the inconsistent journal with fsck.ext3 just not making the changes it should or releasing the drive as clean? I vaguely recollect some mention that fsck might have the power to get drives with problems (though it doesn't actually find any) to boot in read-only mode. Does this ring any bells? This system always thinks its crashing even though reboots are normal and working fine because the system isn't writing its shutdown info. I would think it would be able to write once I've remounted the root filesystem in writable mode but perhaps there are services that aren't getting started that should during the messed up boot with the errors mentioned in my previous message listing those errors which were visible to me. |
wothout a complete list of erros, we cannot correc them. PLease reinstall Fedorea Core 3 to correct the problems you are having
|
I appreciate the help I've received. People have to be brave to try to tackle the fairly tough probs. However, I'm not quite ready to throw in the towel and reinstall after hunting down all the important data on the machine and scp'g it to somewhere else temporarily though the time cost is getting close to about break even now.
I have made some progress this morning. I found this page: http://openskills.info/infobox.php?ID=228 I then began walking through the boot steps and quickly focused on the first boot error I HAD been able to recover: (copied from an earlier post) /etc/rc.d/rc.sysinit: line 30: /etc/init.d/functions: No such file or directory I hunted down 'functions' which appears to be residing at /etc/rc.d/init.d/functions and created a symlink to it in /etc (ln -s /etc/rc.d/init.d /etc/init.d) I certainly didn't make changes to rc.sysinit nor functions I am now wondering if my system freeze occurred actually during the last yum upgrade after switching yum to think it was the Fedora 4 test 1 release. Perhaps these files have been changed in FC4? Doing this has gotten me a lot further into the boot process including getting the root filesystem to mount in a write mode. I now make it to a normal and working login prompt. However attempting a startx is deadly leading to completely blank screens and an unresponsive keyboard. While there are still problems to solve, now that I am getting most services started properly and have logging working, I should be able to attack them. Thanks again for the help. |
In case anyone ends up on this thread due to the title, the read-only filesystem problem was almost certainly due to the rc.sysinit script which appears to intentionally do so for the sake of fsck program. This supposedly makes your system more stable following a crash. :) My version was having problems running the script because of some wrong location pointers in the script. Fixing these paths DID NOT give me a healthy machine but it did get me a bit further into the normal boot sequence and gave me a writeable root filesystem. If I had been able to get rpm to be able to set locks properly, I might have managed to get YUM to upgrade it into a healthy state. But I couldn't figure out why it claimed locks couldn't be set since it WAS actually writing them in the /var/lib/rpm directory.
I ended up deciding that it wasn't worth any more time and have gone back to doing a fresh install of Fedora 3 stable after dumping the important data off of the machine. |
Thanks for letting us know..
|
read-only root fs
Hi ,
I have the similar problem. It is a FC3. I have /etc/init.d/functions file. The 'hdparam /dev/hda' tells this is not a read-only disk. The related /etc/fstab line has "default" option. So it means rw. /etc/mtab file has only two line and it isn't the truth, but /proc/mounts looks like ok. So this two file isn't the same. /var/log/dmesg is old one, but of course I can "mount /dev/hda1 -o remount,rw" than "dmesg >dmesg.log". Unfortunately, there is nothing. No info in /var/log/message file about the error. But, when the sytem boots up, It writes error to the screen only. :( It is something like recovery needed on root partition. After check (no error found) tries to remount it automatically, but there is error : I can stop it by [scroll lock]: Remounting root filesystem in read-write mode: mount: LABEL=/ duplicate - not mounted Setting up Logical Volume Management: 3 logical volumes in volume group... mount: LABEL=/ duplicate - not mounted ... Mounting local file systems: mount: none already mounted or /dev/pts busy mount: none already mounted or /sys busy ... rm: cannot remove '/tmp/.X11-unix': Read-only file system has anyone some idea ? Thanx a lot, Barna. |
| All times are GMT -5. The time now is 03:30 PM. |