fs errors root mount RO, how to mount RW without rebooting?

DaveQB · 05-08-2008, 08:59 PM

Not sure which forum this belongs in. A few choices for it, so I put it here. I hope its the best place.

I have a pair of 4.3GB SCSI disk as the root FS in my server. I sometimes get errors or one of them dropping out of the RAID 1 they are in. I have ran physical manufacture checks on them and they have passed.

So right now, after the most recent case of re-adding one of the disks, I have found the fs to be mounted read-only. Although mount says its rw.
I have a nice 140 day uptime, so dont want to have to reboot to re-mount this FS.

I managed to run fsck on it while it was mounted and it fixed a few errors.

See notes below:

Code:

root@server:~# mount -o remount,rw /
mount: block device /dev/mapper/mainsystem-slash is write-protected, mounting read-only

Code:

root@server:~# fsck /dev/mapper/mainsystem-slash
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
slash: clean, 149162/453376 files, 714018/905216 blocks

Code:

root@server:~# mount -l | grep slash
/dev/mapper/mainsystem-slash on / type ext3 (rw,acl) [slash]

eggixyz · 05-08-2008, 10:16 PM

Depending on how your raid1 is setup, you should be able to pull out one disk at a time from the mirror and fix it. I wouldn't recommend running fsck on / if it's mounted. It's caused headaches for me in the past.

If you need help with the pulling the disks (not physically) out of the mirror and all that, can you post some spec's on what mirroring system/software you're using? Assuming lvm, but maybe not.

Best wishes,

Mike

DaveQB · 05-08-2008, 11:55 PM

Thanks eggixyz
Lots of good info there. I didn't think fsck on a mount fs was a good idea, but sorta desperate haha.

Makes perfect sense to pull one volumne out at a time and run fsck on it.

What I have is software RAID 1 and then LVM on top of that.

Code:

root@server:~# cat /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=272a1b0a:34bd2a9b:fbef58a1:944cf217
   devices=/dev/sda1,/dev/sdb1
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=2bbb54f8:fd4e80f5:27771627:d5ca1b00
   devices=/dev/sda2,/dev/sdb2
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=dc70c7e1:9fffd982:b26f79a7:64b0b8b9
   devices=/dev/hda1,/dev/hdc1
MAILADDR root
root@server:~# mdadm -D /dev/md1
/dev/md1:
        Version : 00.90.03
  Creation Time : Sat Feb 24 22:52:32 2007
     Raid Level : raid1
     Array Size : 4144704 (3.95 GiB 4.24 GB)
    Device Size : 4144704 (3.95 GiB 4.24 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Fri May  9 14:48:44 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 2bbb54f8:fd4e80f5:27771627:d5ca1b00
         Events : 0.565562

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
root@server:~#

Code:

root@server:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               80gig
  PV Size               74.53 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              19079
  Free PE               0
  Allocated PE          19079
  PV UUID               xTD2S4-zZJj-pt2Y-2Id2-yaPs-UZOu-w8D4uA

  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               mainsystem
  PV Size               3.95 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              1011
  Free PE               0
  Allocated PE          1011
  PV UUID               XzZKIz-z3O9-7W3g-mrOV-JIKQ-5PGI-FS7xv1

Code:

root@server:~# vgdisplay mainsystem
  --- Volume group ---
  VG Name               mainsystem
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.95 GB
  PE Size               4.00 MB
  Total PE              1011
  Alloc PE / Size       1011 / 3.95 GB
  Free  PE / Size       0 / 0
  VG UUID               sBRsL9-tsyR-m4aS-kAXQ-xaf1-dVkE-gSB3uC

Code:

root@server:~# lvdisplay mainsystem
  --- Logical volume ---
  LV Name                /dev/mainsystem/slash
  VG Name                mainsystem
  LV UUID                5QxhDm-J32C-QZZ5-0jl2-dDoL-1MgI-ERV7N8
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                3.45 GB
  Current LE             884
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/mainsystem/temp
  VG Name                mainsystem
  LV UUID                2JwPWb-AAlj-uGf7-stZi-AL4z-urWw-UMddJ0
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                508.00 MB
  Current LE             127
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:1

Code:

root@server:~# mount | grep [ts][ml][pa] | grep mapper
/dev/mapper/mainsystem-slash on / type ext3 (rw,acl)
/dev/mapper/mainsystem-temp on /tmp type ext3 (rw,noexec,nosuid,nodev,acl)

Curious, /tmp is writable!

I'll pull out /dev/sda2 and try fsck on it and then /dev/sdb2 and do the same. How do I then tell the system "the raid is ok, mount it as rw" ?

DaveQB · 05-08-2008, 11:58 PM

AHHH!!

Code:

root@server:~# mdadm /dev/md1 -f /dev/sda2
mdadm: set /dev/sda2 faulty in /dev/md1
root@server:~# mdadm /dev/md1 -r /dev/sda2
mdadm: hot removed /dev/sda2
root@server:~# fsck /dev/sda2
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sda2

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

So I need to fsck the raid while its together ? There must be a way to check a partition at a time?? append -b 8193 ?

eggixyz · 05-09-2008, 12:16 AM

Hey There,

Thanks for the info. More than I needed, but I never complain about that

Once you're done fixing both disks, you can just run

mount -o remount,rw /

If this doesn't work try using the n option, also, so the mount doesn't try writing to the mnttab. Sometimes the write to mnttab fails and can make the mount fail even though it should be all right.

mount -no remount,rw /

or

mount -n -o remount,rw /

Since you have / listed in /etc/fstab you shouldn't have to define the device in your mount command.

Best wishes

, Mike

DaveQB · 05-11-2008, 07:36 AM

Yeah mounting doesn't work as the FS has issues and is read-only. I need to work out a way to fix this FS without rebooting. Might have to bite the bullet and re-boot

eggixyz · 05-11-2008, 03:36 PM

Yeah,

If mount with the "remount" option doesn't work, there's no other way to change it from ro to rw. I've done it before on /var and /usr, but it may be that the OS won't allow it for / if it's the root filesystem

Best wishes,

Mike