I have been running an FC4 server for a while, when I noticed the / (root) partition becoming full. I ssh-ed the server, deleted a few /var/log files, bringing it down to 98%. I then rebooted and since then the system will boot but will refuse to mount anything.

(cat /etc/fstab produces an error message notifying of not being able to run fstab-sync.)

During boot, the only error message I have been able to spot is this:

Also, in "quiet" boot, I get at some point:
After which there are lots of failure to access all kind of system resource due to filesystem being read-only.

The bottom line is that, despite being able to boot the system, I am unable to fix anything because nothing is mounted and I cannot even check what went wrong, why this is happening, and what to write were in order to fix this.

Any help, tip, insight, idea, instructions would be greatly appreciated.

Thanks.

fedora? on a server?? why?? and an obsolete version at that.... why??? That distro is so old, just disable SELinux (edit /etc/selinux/config) and move on.

I am asking the same question, too. I inherited it. I intend to upgrade it to the latest CentOS, but until I have the resources to do so safely, I need to revive this system, for day-to-day operation.

Thank you for this tip. I will try this shortly and report back how it went.

OK, I just tried to edit /etc/selinux/config (from "permissive" to "disabled") but the system is in such bad situation, it won't let me to write the file back because is everything (that managed to be mounted) is mounted as a read-only filesystem.

I could use a SysRescueCD to mount that /etc writable, but given that SELinux was set to "permissive" (i.e. warnings only), is that really the solution to the problem?

hmm, ok I guess i wasn't thinking wide enough...

So you can't cat fstab? what about under the rescue CD?

Correct. It is empty, except for a single comment line saying that "This file is edited by fstab-sync - see 'man fstab-sync' for details"

Under the rescue CD it comes up empty (and so does /etc/mtab).

I managed to use SysRescueCD to completely disable SELinux (via /etc/selinux/config) but the problem persists.

It seems that the core problem is being unable to mount the / filesystem writeable, and as a result /etc/fstab and/or /etc/mtab cannot be generated by the system, and then there is a chain-reaction in which everything is mounted read-only.

The question now is what causes this?

If I can fix the root cause, the system will proceed normally on itself.

But what is that root cause?

What makes a system decide to mount read-only?

How did that /etc/fstab (or /etc/mtab) disappear?

well I'd look at having a stab at writing the fstab file again manually, shouldn't be too hard to get the system usable again. Not had much experience with fstab-sync, long gone now... I'd probably look to at least make a backup copy of your new fstab in case it does get pwned, maybe uninstall / delete the fstab-sync binary if that's too involved.

That did the trick. (fortunately, I had a backup copy of the working fstab lying around and all I had to do is copy it over)

Now the big mystery is why, and how, this happened.

I will probably never know because this incident convinced me that I should perform the upgrade sooner rather than later. The problem is that the guy who passed this system on to me, installed zillion services that make that server tick (wordpress, sitebar, cups, backup and other daemons), it would take me forever to re-build that functionality on a newly installed server.

well unless someone managed to nuke it manually, a file system corruption could be to blame. anything in lost+found?