-   Linux - General (
-   -   fs errors root mount RO, how to mount RW without rebooting? (

DaveQB 05-08-2008 09:59 PM

fs errors root mount RO, how to mount RW without rebooting?
Not sure which forum this belongs in. A few choices for it, so I put it here. I hope its the best place.

I have a pair of 4.3GB SCSI disk as the root FS in my server. I sometimes get errors or one of them dropping out of the RAID 1 they are in. I have ran physical manufacture checks on them and they have passed.

So right now, after the most recent case of re-adding one of the disks, I have found the fs to be mounted read-only. Although mount says its rw.
I have a nice 140 day uptime, so dont want to have to reboot to re-mount this FS.

I managed to run fsck on it while it was mounted and it fixed a few errors.

See notes below:


root@server:~# mount -o remount,rw /
mount: block device /dev/mapper/mainsystem-slash is write-protected, mounting read-only


root@server:~# fsck /dev/mapper/mainsystem-slash
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
slash: clean, 149162/453376 files, 714018/905216 blocks


root@server:~# mount -l | grep slash
/dev/mapper/mainsystem-slash on / type ext3 (rw,acl) [slash]

eggixyz 05-08-2008 11:16 PM

Depending on how your raid1 is setup, you should be able to pull out one disk at a time from the mirror and fix it. I wouldn't recommend running fsck on / if it's mounted. It's caused headaches for me in the past.

If you need help with the pulling the disks (not physically) out of the mirror and all that, can you post some spec's on what mirroring system/software you're using? Assuming lvm, but maybe not.

Best wishes,


DaveQB 05-09-2008 12:55 AM

Thanks eggixyz
Lots of good info there. I didn't think fsck on a mount fs was a good idea, but sorta desperate haha.

Makes perfect sense to pull one volumne out at a time and run fsck on it.

What I have is software RAID 1 and then LVM on top of that.


root@server:~# cat /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=272a1b0a:34bd2a9b:fbef58a1:944cf217
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=2bbb54f8:fd4e80f5:27771627:d5ca1b00
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=dc70c7e1:9fffd982:b26f79a7:64b0b8b9
root@server:~# mdadm -D /dev/md1
        Version : 00.90.03
  Creation Time : Sat Feb 24 22:52:32 2007
    Raid Level : raid1
    Array Size : 4144704 (3.95 GiB 4.24 GB)
    Device Size : 4144704 (3.95 GiB 4.24 GB)
  Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Fri May  9 14:48:44 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

          UUID : 2bbb54f8:fd4e80f5:27771627:d5ca1b00
        Events : 0.565562

    Number  Major  Minor  RaidDevice State
      0      8        2        0      active sync  /dev/sda2
      1      8      18        1      active sync  /dev/sdb2


root@server:~# pvdisplay
  --- Physical volume ---
  PV Name              /dev/md2
  VG Name              80gig
  PV Size              74.53 GB / not usable 0
  Allocatable          yes (but full)
  PE Size (KByte)      4096
  Total PE              19079
  Free PE              0
  Allocated PE          19079
  PV UUID              xTD2S4-zZJj-pt2Y-2Id2-yaPs-UZOu-w8D4uA

  --- Physical volume ---
  PV Name              /dev/md1
  VG Name              mainsystem
  PV Size              3.95 GB / not usable 0
  Allocatable          yes (but full)
  PE Size (KByte)      4096
  Total PE              1011
  Free PE              0
  Allocated PE          1011
  PV UUID              XzZKIz-z3O9-7W3g-mrOV-JIKQ-5PGI-FS7xv1


root@server:~# vgdisplay mainsystem
  --- Volume group ---
  VG Name              mainsystem
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access            read/write
  VG Status            resizable
  MAX LV                0
  Cur LV                2
  Open LV              2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size              3.95 GB
  PE Size              4.00 MB
  Total PE              1011
  Alloc PE / Size      1011 / 3.95 GB
  Free  PE / Size      0 / 0
  VG UUID              sBRsL9-tsyR-m4aS-kAXQ-xaf1-dVkE-gSB3uC


root@server:~# lvdisplay mainsystem
  --- Logical volume ---
  LV Name                /dev/mainsystem/slash
  VG Name                mainsystem
  LV UUID                5QxhDm-J32C-QZZ5-0jl2-dDoL-1MgI-ERV7N8
  LV Write Access        read/write
  LV Status              available
  # open                1
  LV Size                3.45 GB
  Current LE            884
  Segments              1
  Allocation            inherit
  Read ahead sectors    0
  Block device          253:0

  --- Logical volume ---
  LV Name                /dev/mainsystem/temp
  VG Name                mainsystem
  LV UUID                2JwPWb-AAlj-uGf7-stZi-AL4z-urWw-UMddJ0
  LV Write Access        read/write
  LV Status              available
  # open                1
  LV Size                508.00 MB
  Current LE            127
  Segments              1
  Allocation            inherit
  Read ahead sectors    0
  Block device          253:1


root@server:~# mount | grep [ts][ml][pa] | grep mapper
/dev/mapper/mainsystem-slash on / type ext3 (rw,acl)
/dev/mapper/mainsystem-temp on /tmp type ext3 (rw,noexec,nosuid,nodev,acl)

Curious, /tmp is writable!

I'll pull out /dev/sda2 and try fsck on it and then /dev/sdb2 and do the same. How do I then tell the system "the raid is ok, mount it as rw" ?

DaveQB 05-09-2008 12:58 AM



root@server:~# mdadm /dev/md1 -f /dev/sda2
mdadm: set /dev/sda2 faulty in /dev/md1
root@server:~# mdadm /dev/md1 -r /dev/sda2
mdadm: hot removed /dev/sda2
root@server:~# fsck /dev/sda2
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sda2

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

So I need to fsck the raid while its together ? There must be a way to check a partition at a time?? append -b 8193 ?

eggixyz 05-09-2008 01:16 AM

Hey There,

Thanks for the info. More than I needed, but I never complain about that :)

Once you're done fixing both disks, you can just run

mount -o remount,rw /

If this doesn't work try using the n option, also, so the mount doesn't try writing to the mnttab. Sometimes the write to mnttab fails and can make the mount fail even though it should be all right.

mount -no remount,rw /


mount -n -o remount,rw /

Since you have / listed in /etc/fstab you shouldn't have to define the device in your mount command.

Best wishes :)

, Mike

DaveQB 05-11-2008 08:36 AM

Yeah mounting doesn't work as the FS has issues and is read-only. I need to work out a way to fix this FS without rebooting. Might have to bite the bullet and re-boot :(

eggixyz 05-11-2008 04:36 PM


If mount with the "remount" option doesn't work, there's no other way to change it from ro to rw. I've done it before on /var and /usr, but it may be that the OS won't allow it for / if it's the root filesystem

Best wishes,


All times are GMT -5. The time now is 06:03 AM.