LinuxQuestions.org - fs errors root mount RO, how to mount RW without rebooting?

- Linux - General (https://www.linuxquestions.org/questions/linux-general-1/)

- - fs errors root mount RO, how to mount RW without rebooting? (https://www.linuxquestions.org/questions/linux-general-1/fs-errors-root-mount-ro-how-to-mount-rw-without-rebooting-640938/)

fs errors root mount RO, how to mount RW without rebooting?

Not sure which forum this belongs in. A few choices for it, so I put it here. I hope its the best place.

I have a pair of 4.3GB SCSI disk as the root FS in my server. I sometimes get errors or one of them dropping out of the RAID 1 they are in. I have ran physical manufacture checks on them and they have passed.

So right now, after the most recent case of re-adding one of the disks, I have found the fs to be mounted read-only. Although mount says its rw.
I have a nice 140 day uptime, so dont want to have to reboot to re-mount this FS.

I managed to run fsck on it while it was mounted and it fixed a few errors.

See notes below:

Code:

root@server:~# mount -o remount,rw /

mount: block device /dev/mapper/mainsystem-slash is write-protected, mounting read-only

Code:

root@server:~# fsck /dev/mapper/mainsystem-slash

fsck 1.40-WIP (14-Nov-2006)

e2fsck 1.40-WIP (14-Nov-2006)

slash: clean, 149162/453376 files, 714018/905216 blocks

Code:

root@server:~# mount -l | grep slash

/dev/mapper/mainsystem-slash on / type ext3 (rw,acl) [slash]

Depending on how your raid1 is setup, you should be able to pull out one disk at a time from the mirror and fix it. I wouldn't recommend running fsck on / if it's mounted. It's caused headaches for me in the past.

If you need help with the pulling the disks (not physically) out of the mirror and all that, can you post some spec's on what mirroring system/software you're using? Assuming lvm, but maybe not.

Best wishes,

Mike

Thanks eggixyz
Lots of good info there. I didn't think fsck on a mount fs was a good idea, but sorta desperate haha.

Makes perfect sense to pull one volumne out at a time and run fsck on it.

What I have is software RAID 1 and then LVM on top of that.

Code:

root@server:~# cat /etc/mdadm/mdadm.conf

DEVICE partitions

ARRAY /dev/md0 level=raid1 num-devices=2 UUID=272a1b0a:34bd2a9b:fbef58a1:944cf217

  devices=/dev/sda1,/dev/sdb1

ARRAY /dev/md1 level=raid1 num-devices=2 UUID=2bbb54f8:fd4e80f5:27771627:d5ca1b00

  devices=/dev/sda2,/dev/sdb2

ARRAY /dev/md2 level=raid1 num-devices=2 UUID=dc70c7e1:9fffd982:b26f79a7:64b0b8b9

  devices=/dev/hda1,/dev/hdc1

MAILADDR root

root@server:~# mdadm -D /dev/md1

/dev/md1:

        Version : 00.90.03

  Creation Time : Sat Feb 24 22:52:32 2007

    Raid Level : raid1

    Array Size : 4144704 (3.95 GiB 4.24 GB)

    Device Size : 4144704 (3.95 GiB 4.24 GB)

  Raid Devices : 2

  Total Devices : 2

Preferred Minor : 1

    Persistence : Superblock is persistent



    Update Time : Fri May  9 14:48:44 2008

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 0



          UUID : 2bbb54f8:fd4e80f5:27771627:d5ca1b00

        Events : 0.565562



    Number  Major  Minor  RaidDevice State

      0      8        2        0      active sync  /dev/sda2

      1      8      18        1      active sync  /dev/sdb2

root@server:~#

Code:

root@server:~# pvdisplay

  --- Physical volume ---

  PV Name              /dev/md2

  VG Name              80gig

  PV Size              74.53 GB / not usable 0

  Allocatable          yes (but full)

  PE Size (KByte)      4096

  Total PE              19079

  Free PE              0

  Allocated PE          19079

  PV UUID              xTD2S4-zZJj-pt2Y-2Id2-yaPs-UZOu-w8D4uA



  --- Physical volume ---

  PV Name              /dev/md1

  VG Name              mainsystem

  PV Size              3.95 GB / not usable 0

  Allocatable          yes (but full)

  PE Size (KByte)      4096

  Total PE              1011

  Free PE              0

  Allocated PE          1011

  PV UUID              XzZKIz-z3O9-7W3g-mrOV-JIKQ-5PGI-FS7xv1

Code:

root@server:~# vgdisplay mainsystem

  --- Volume group ---

  VG Name              mainsystem

  System ID

  Format                lvm2

  Metadata Areas        1

  Metadata Sequence No  5

  VG Access            read/write

  VG Status            resizable

  MAX LV                0

  Cur LV                2

  Open LV              2

  Max PV                0

  Cur PV                1

  Act PV                1

  VG Size              3.95 GB

  PE Size              4.00 MB

  Total PE              1011

  Alloc PE / Size      1011 / 3.95 GB

  Free  PE / Size      0 / 0

  VG UUID              sBRsL9-tsyR-m4aS-kAXQ-xaf1-dVkE-gSB3uC

Code:

root@server:~# lvdisplay mainsystem

  --- Logical volume ---

  LV Name                /dev/mainsystem/slash

  VG Name                mainsystem

  LV UUID                5QxhDm-J32C-QZZ5-0jl2-dDoL-1MgI-ERV7N8

  LV Write Access        read/write

  LV Status              available

  # open                1

  LV Size                3.45 GB

  Current LE            884

  Segments              1

  Allocation            inherit

  Read ahead sectors    0

  Block device          253:0



  --- Logical volume ---

  LV Name                /dev/mainsystem/temp

  VG Name                mainsystem

  LV UUID                2JwPWb-AAlj-uGf7-stZi-AL4z-urWw-UMddJ0

  LV Write Access        read/write

  LV Status              available

  # open                1

  LV Size                508.00 MB

  Current LE            127

  Segments              1

  Allocation            inherit

  Read ahead sectors    0

  Block device          253:1

Code:

root@server:~# mount | grep [ts][ml][pa] | grep mapper

/dev/mapper/mainsystem-slash on / type ext3 (rw,acl)

/dev/mapper/mainsystem-temp on /tmp type ext3 (rw,noexec,nosuid,nodev,acl)

Curious, /tmp is writable!

I'll pull out /dev/sda2 and try fsck on it and then /dev/sdb2 and do the same. How do I then tell the system "the raid is ok, mount it as rw" ?

AHHH!!

Code:

root@server:~# mdadm /dev/md1 -f /dev/sda2

mdadm: set /dev/sda2 faulty in /dev/md1

root@server:~# mdadm /dev/md1 -r /dev/sda2

mdadm: hot removed /dev/sda2

root@server:~# fsck /dev/sda2

fsck 1.40-WIP (14-Nov-2006)

e2fsck 1.40-WIP (14-Nov-2006)

Couldn't find ext2 superblock, trying backup blocks...

fsck.ext2: Bad magic number in super-block while trying to open /dev/sda2



The superblock could not be read or does not describe a correct ext2

filesystem.  If the device is valid and it really contains an ext2

filesystem (and not swap or ufs or something else), then the superblock

is corrupt, and you might try running e2fsck with an alternate superblock:

    e2fsck -b 8193 <device>

So I need to fsck the raid while its together ? There must be a way to check a partition at a time?? append -b 8193 ?

Hey There,

Thanks for the info. More than I needed, but I never complain about that :)

Once you're done fixing both disks, you can just run

mount -o remount,rw /

If this doesn't work try using the n option, also, so the mount doesn't try writing to the mnttab. Sometimes the write to mnttab fails and can make the mount fail even though it should be all right.

mount -no remount,rw /

or

mount -n -o remount,rw /

Since you have / listed in /etc/fstab you shouldn't have to define the device in your mount command.

Best wishes :)

, Mike

Yeah mounting doesn't work as the FS has issues and is read-only. I need to work out a way to fix this FS without rebooting. Might have to bite the bullet and re-boot :(

Yeah,

If mount with the "remount" option doesn't work, there's no other way to change it from ro to rw. I've done it before on /var and /usr, but it may be that the OS won't allow it for / if it's the root filesystem

Best wishes,

Mike