remote server mounts /usr as read-only. How to fix without physical access?

thatto · 10-21-2009, 04:27 PM

All,
I'll be brief:
* /usr mounts as read only.
* The filesystem is Reiserfs
* fsck.reiserfs --fix-fixable reports that there is a bad superblock on the filesystem

Quote:

Will check consistency of the filesystem on /usr
and will fix what can be fixed without --rebuild-tree
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
bread: Cannot read the block (2): (Is a directory).
reiserfs_open: bread failed reading block 2
bread: Cannot read the block (16): (Is a directory).
reiserfs_open: bread failed reading block 16

reiserfs_open: the reiserfs superblock cannot be found on /usr.
Failed to open the filesystem.

If the partition table has not been changed, and the partition is
valid and it really contains a reiserfs partition, then the
superblock is corrupted and you need to run this utility with
--rebuild-sb.

* Obviously I cannot unmount /usr without losing my connection.
* Single user mode is out because I do not have physical access to the machine.

As I see it, to fix this problem, I have to do one of three things.
a. wait until I have access to the server.
b. move sshd so to a different partition (and reboot)
c. ??? (what does the wisdom of the crowd say?)

I am using openSuSE 11.0 Kernel 2.6.25.20-0.5

Thanks for reading this far!

jhwilliams · 10-21-2009, 05:40 PM

Quote:

Originally Posted by thatto

* Obviously I cannot unmount /usr without losing my connection.

Well, you wouldn't lose your connection necessarily, it would just become a fairly boring connection. ;-) (sshd is memory resident if you're connected, so it doesn't matter where /usr goes.*) You'll still have /bin and /sbin, of course. Which is okay:

/sbin/umount, /sbin/mount, /sbin/fsck will all be available and should have all you need to fsck.

Code:

# First, kill everything using /usr:
lsof | grep /usr
kill <pids>
# Unmount /usr:
umount /usr
# fsck the problem:
fsck.resierfs <usr partition>
# remount, and enjoy us(e)r binaries again:
mount -a
# test a usr bin:
/usr/bin/awk -Wversion

* As a related example of this: If you hit rm -rf /* while on an ssh connection, your connection will continue, but as soon as you lose it, you'll need to reinstall the operating system via physical access.