software raid - read only filesystem
 

I am using centos 5.1 for setting up raid 1 software raid using 2 seagate 500g hdd.

after running for several months, the filesystem becomes readonly. all system services such as httpd fails. Even creating a new file is impossible.

Below is the result of some commands. pls advise.

[root@localhost ~]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Wed Dec 24 23:14:24 2008
Raid Level : raid1
Array Size : 471997632 (450.13 GiB 483.33 GB)
Used Dev Size : 471997632 (450.13 GiB 483.33 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Thu Mar 19 11:57:40 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0

UUID : b7974505:46381e1b:6f332600:70eecbe8
Events : 0.2138762

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 0 0 1 removed

2 8 17 - faulty spare /dev/sdb1

[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[2](F) sda2[0]
16386176 blocks [2/1] [U_]

md0 : active raid1 sdb1[2](F) sda1[0]
471997632 blocks [2/1] [U_]

unused devices: <none>

Reading /proc/mdstat it says that you have 2 raid devices, md0, md1. Each has a partition from sda and sdb. sdb has died (specifically sdb2), see also last line of

mdadm --detail /dev/md0.
.
.
2 8 17 - faulty spare /dev/sdb1


You'll get similar output from mdadm --detail /dev/md1.

Get a backup ASAP and either replace sdb, or run fdisk sdb (good luck).

I realised that sdb is faulty but sda is ok. the whole system is nearly unusable as the filesystem is readonly.

eg. i type
root> touch a.txt
touch: cannot touch `a': Read-only file system

it is a software raid. if sda is ok, the system should be normal. otherwise, sw raid is not stable enough for production use.

pls kindly advise.

Some information on recovering linux soft raid can be found at http://wpkg.org/Software_RAID_in_Linux

Your filesystem maybe marked read only, until you do an fsck. Boot to single user mode. The way to boot into single user mode, will depend on your Linux distribution.

After booting into single user mode, do a file system check on the partition. Unmount the partition if, its mounted.

umount /mnt/yourmountpoint

Then do a filesystem check

fsck /dev/sda1
fsck /dev/sdb1

If you're paranoid, you can remove the drive from the computer and put it in another computer to do the fsck.