HDD failure in a raid1

chiendarret · 03-03-2009, 12:44 PM

One of the two disk in a raid one (debian amd64 lenny, ext3, software raid) failed and was removed by the system. Although common problem, it is unclear to me (user, no system expert available here) if the failed disk has to be replaced with a new one or it could be forced into place, tried, reformatted, and what else should be done. If to be replaced, it would be the second time in a less than 2000 hours service. I was not aware that WD Raptor is so short living (there is a good UPS and the disks are mounted on Zahlman coolers with a very string fan directly on them.

Unfortunately I have a badly planned huge raid.

cat /proc/mdstat

Personalities : [raid1]
md6 : active raid1 sdb8[1]
102341952 blocks [2/1] [_U]

md5 : active raid1 sdb7[1]
1951744 blocks [2/1] [_U]

md4 : active raid1 sdb6[1]
2931712 blocks [2/1] [_U]

md3 : active raid1 sdb5[1]
14651136 blocks [2/1] [_U]

md1 : active (auto-read-only) raid1 sdb2[1]
6835584 blocks [2/1] [_U]

md0 : active raid1 sdb1[1]
2931712 blocks [2/1] [_U]

md2 : active raid1 sdb3[1]
14651200 blocks [2/1] [_U]

unused devices: <none>
========================================================================

mdadm --detail /dev/md0

/dev/md0:
Version : 00.90
Creation Time : Wed Sep 13 16:50:11 2006
Raid Level : raid1
Array Size : 2931712 (2.80 GiB 3.00 GB)
Used Dev Size : 2931712 (2.80 GiB 3.00 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Tue Mar 3 19:05:21 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : 5afe70f3:fa1dcb8c:5cc73e84:1c5db3e0
Events : 0.912

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 17 1 active sync /dev/sdb1
==================================================================
Similar situation with all other raids
==================================================================

smartctl --all /dev/sda (the disk removed):

smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Raptor family
Device Model: WDC WD1500ADFD-00NLR1
Serial Number: WD-WMAP41185084
Firmware Version: 20.07P20
User Capacity: 150,039,945,216 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 published, ANSI INCITS 397-2005
Local Time is: Tue Mar 3 19:41:15 2009 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (4783) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 72) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x103f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 168 166 021 Pre-fail Always - 4625
4 Start_Stop_Count 0x0032 099 099 040 Old_age Always - 1031
5 Reallocated_Sector_Ct 0x0033 189 189 140 Pre-fail Always - 168
7 Seek_Error_Rate 0x000a 200 200 051 Old_age Always - 0
9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 7259
10 Spin_Retry_Count 0x0012 100 100 051 Old_age Always - 0
11 Calibration_Retry_Count 0x0012 100 100 051 Old_age Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1031
194 Temperature_Celsius 0x0022 123 106 000 Old_age Always - 24
196 Reallocated_Event_Count 0x0032 180 180 000 Old_age Always - 20
197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0012 200 200 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x000a 200 253 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 051 Old_age Offline - 0

SMART Error Log Version: 1
ATA Error Count: 12 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 12 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:49.600 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:49.600 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA

Error 11 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:46.400 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:46.400 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA

Error 10 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:42.950 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:42.950 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA

Error 9 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:39.800 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:39.800 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA

Error 8 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:36.600 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:36.600 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:36.600 READ DMA
c8 00 08 3f 9d 9b 4c 00 00:52:36.600 READ DMA
c8 00 08 37 9d 9b 4c 00 00:52:36.600 READ DMA

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
============================================================

sdb reported no errors
============================================================

I would appreciate suggestions
thanks

francesco pietra