[SOLVED] Help recovering raid5 array please

RottenMutt · 04-04-2015, 05:28 PM

... HELP Please ...
In ClearOS (router/firewall distribution) shortly after installing smart monitor (checks smart status of drives) i lost my raid array, march 23 exactly. i think smart tools knocked the array offline. i've attempted to re-add the two missing drives sd[hi]1 but get "not large enough to join array".
the only think i can think of at this point is to --assemble --force /dev/md0 /dev/sd[abcdehikl]1, sdh and sdi are the drives missing.

Code:

mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
   Raid Devices : 10
  Total Devices : 9
    Persistence : Superblock is persistent

    Update Time : Sat Apr  4 14:55:18 2015
          State : active, FAILED, Not Started
 Active Devices : 8
Working Devices : 9
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : orion.localdomain:0
           UUID : 2002d641:a6021121:103a0ed7:81d48464
         Events : 145962

    Number   Major   Minor   RaidDevice State
      10       8       65        0      active sync   /dev/sde1
      11       8       17        1      active sync   /dev/sdb1
       2       8       49        2      active sync   /dev/sdd1
       3       8       33        3      active sync   /dev/sdc1
       4       8        1        4      active sync   /dev/sda1
       6       8       81        5      active sync   /dev/sdf1
       9       8      177        6      active sync   /dev/sdl1
       8       8      161        7      active sync   /dev/sdk1
      16       0        0       16      removed
      18       0        0       18      removed

      12       8       97        -      spare   /dev/sdg1


mdadm --examine /dev/sd[a-z]1 | egrep 'Event|/dev/sd'
/dev/sda1:
         Events : 145962
/dev/sdb1:
         Events : 145962
/dev/sdc1:
         Events : 145962
/dev/sdd1:
         Events : 145962
/dev/sde1:
         Events : 145962
/dev/sdf1:
         Events : 145962
/dev/sdg1:
         Events : 145946
/dev/sdh1:
         Events : 145947
/dev/sdi1:
         Events : 145947
/dev/sdk1:
         Events : 145962
/dev/sdl1:
         Events : 145962

mdadm --examine /dev/sd[a-z]1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 352b3be8:074585e2:9887a950:511dc0b1

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : 65378398 - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907023874 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 128 sectors
   Super Offset : 8 sectors
   Unused Space : before=48 sectors, after=2 sectors
          State : clean
    Device UUID : f81bb836:f5e1511e:6d1a3560:6f385277

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : c49eb2db - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 5a54237b:a8333001:697b3e52:b736c442

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : a963a85a - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : f6b9e98c:4e650e14:7b3a8d59:bb5d2ab1

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : 43bd25b2 - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907023874 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 128 sectors
   Super Offset : 8 sectors
   Unused Space : before=48 sectors, after=2 sectors
          State : clean
    Device UUID : 2da43fbc:43fbfa27:5408cfb5:5ca30640

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : 721dad32 - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : d31e4d85:8d325917:e2d3242a:f09b8a09

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : 68632f6d - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907023874 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 128 sectors
   Super Offset : 8 sectors
   Unused Space : before=48 sectors, after=2 sectors
          State : clean
    Device UUID : 0387b7f6:fd56ca8b:42d08295:714c0b9d

    Update Time : Sun Mar 22 13:18:54 2015
       Checksum : 4d1622aa - correct
         Events : 145946

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907023874 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 128 sectors
   Super Offset : 8 sectors
   Unused Space : before=48 sectors, after=2 sectors
          State : clean
    Device UUID : c8b2a372:293296c3:5d5f5263:2b661a66

    Update Time : Sun Mar 22 14:32:40 2015
       Checksum : 97ace3bb - correct
         Events : 145947

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 9
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907023874 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 128 sectors
   Super Offset : 8 sectors
   Unused Space : before=48 sectors, after=2 sectors
          State : clean
    Device UUID : 783cec58:981ca542:93d9fccb:88f21a21

    Update Time : Sun Mar 22 14:32:40 2015
       Checksum : 20af5e6e - correct
         Events : 145947

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 8
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sdj1.
/dev/sdk1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907023874 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 128 sectors
   Super Offset : 8 sectors
   Unused Space : before=48 sectors, after=2 sectors
          State : clean
    Device UUID : 4e191b1d:d2bc76f1:0ad9675c:057adf9c

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : 9fe68b3f - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 7
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 2002d641:a6021121:103a0ed7:81d48464
           Name : orion.localdomain:0
  Creation Time : Sun Mar 20 14:23:38 2011
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3907023874 (1863.01 GiB 2000.40 GB)
     Array Size : 17581607424 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 128 sectors
   Super Offset : 8 sectors
   Unused Space : before=48 sectors, after=2 sectors
          State : clean
    Device UUID : ec2344bb:dd405bd2:9c62e643:79c256bd

    Update Time : Sat Apr  4 14:55:18 2015
       Checksum : 26e9ebf0 - correct
         Events : 145962

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AAAAAAAA.. ('A' == active, '.' == missing, 'R' == replacing)

what should i try???

syg00 · 04-04-2015, 09:52 PM

What do the actual disks look like ?

Code:

parted /dev/sdh "print free"
smartctl -A /dev/sdh
parted /dev/sdi "print free"
smartctl -A /dev/sdi

RottenMutt · 04-05-2015, 04:19 PM

Code:

parted /dev/sdh "print free"
Model: ATA WDC WD20EZRX-00D (scsi)
Disk /dev/sdh: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      32.3kB  2000GB  2000GB  primary               raid
        2000GB  2000GB  2613kB           Free Space


smartctl -A /dev/sdh
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-504.8.1.v6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   174   173   021    Pre-fail  Always       -       4283
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       38
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       5391
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       38
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       25
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2163
194 Temperature_Celsius     0x0022   119   111   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

Code:

parted /dev/sdi "print free"
Model: ATA WDC WD20EZRX-00D (scsi)
Disk /dev/sdi: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      32.3kB  2000GB  2000GB  primary               raid
        2000GB  2000GB  2613kB           Free Space


smartctl -A /dev/sdi
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-504.8.1.v6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   177   176   021    Pre-fail  Always       -       4133
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       38
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       5397
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       38
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       25
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2219
194 Temperature_Celsius     0x0022   116   108   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

i did try
mdadm --stop /dev/md0
mdadm --create /dev/md0 --level=5 --raid-devices=10 --chunk=512 --ame=orion.localdomain:0 /dev/sd[abcdefhikl]1 --assume-clean
partitions on /dev/md0 will not mount or are not there

Code:

mdadm --examine /dev/sd[a-z]1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=3248 sectors
          State : clean
    Device UUID : 0c49f213:698ef716:30759256:188ee71c

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c9df4fc3 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=130 sectors
          State : clean
    Device UUID : 5a269a15:57b22583:a3c3a6c1:046f12a3

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 28f47433 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=3248 sectors
          State : clean
    Device UUID : 8b7751a5:f356f1bb:2af02dc6:7f331acb

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 1e066732 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=3248 sectors
          State : clean
    Device UUID : 6ac5f4ee:c8b5fe72:962ec160:ce2da35f

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 4dd34ca1 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=130 sectors
          State : clean
    Device UUID : 292aca5d:b37c8872:7ef1cdd4:e7288541

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 12212a1f - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=3248 sectors
          State : clean
    Device UUID : b258e615:207afe11:ce31564b:98335fc7

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6615ad44 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=130 sectors
          State : clean
    Device UUID : 4672ef20:040775b6:56e6fbf3:f042917e

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 756d0b74 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=130 sectors
          State : clean
    Device UUID : 2713777e:c8fc2ce2:0fb21798:9d7e17a5

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c94ea97b - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=130 sectors
          State : clean
    Device UUID : 7dff22c6:822ffef2:eb20de7c:9d923712

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 73b24b68 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 7
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sdj1.
/dev/sdk1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=130 sectors
          State : clean
    Device UUID : 6d35dcac:0578ae4d:12465702:141ffa0b

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 34577b79 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 8
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6db73c01:02e7e8a6:4f14c675:ba33ac3b
           Name : orion.localdomain:0
  Creation Time : Sat Apr  4 20:20:52 2015
     Raid Level : raid5
   Raid Devices : 10

 Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=130 sectors
          State : clean
    Device UUID : 35fceaca:0ce1f9fe:37a39223:2eba9f13

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr  5 05:27:15 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 2c92a389 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 9
   Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

S.Haran · 04-06-2015, 03:07 PM

Well from a quick look I can see at least one problem here. You used the wrong drive order when you did the mdadm --create. Partition sda1 is initially identified as "Device Role : Active device 4" and after the create " Device Role : Active device 0"

I work on RAID recovery cases like this all the time and it is very important when doing a mdadm --create you must be 100% sure of the correct mdadm RAID parameters and drive order.

And ideally you would have made images of the drives before attempting recovery.

Having said that I think recovery may still be possible.

RottenMutt · 04-06-2015, 04:16 PM

Arrg! I will try again, this time with a different order. I thought mdadm would know.

Under mdadm -detail, isn't the minor # the order? Otherwise 10 factorial is a huge #!!!

This appears to be the correct order to me sd[abcdefhikl] with either h & i in either order of the minor # is correct. But if the raid number is correct then sd[ebdcaflkih] is correct.

This forum needs a separate raid section!!!

RottenMutt · 04-06-2015, 06:32 PM

I tried /dev/sd[ebdcaflkih] but no partitions. should i try creating with h or i failed?

edit: when running mdadm -detail i found it processes the above list in alphabetical order, it needs to be:

mdadm --create /dev/md0 --level=5 --raid-devices=10 --chunk=512 --name=orion.localdomain:0 /dev/sde1 /dev/sdb1 /dev/sdd1 /dev/sdc1 /dev/sda1 /dev/sdf1 /dev/sdl1 /dev/sdk1 /dev/sdi1 /dev/sdh1 --assume-clean

still no file system:P

Code:

mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Apr  6 21:24:35 2015
     Raid Level : raid5
     Array Size : 17580427776 (16766.00 GiB 18002.36 GB)
  Used Dev Size : 1953380864 (1862.89 GiB 2000.26 GB)
   Raid Devices : 10
  Total Devices : 10
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Apr  6 21:24:35 2015
          State : active
 Active Devices : 10
Working Devices : 10
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : orion.localdomain:0
           UUID : 2661e06d:c95c9948:25f770d6:0a59d5b8
         Events : 0

    Number   Major   Minor   RaidDevice State
       0       8       65        0      active sync   /dev/sde1
       1       8       17        1      active sync   /dev/sdb1
       2       8       49        2      active sync   /dev/sdd1
       3       8       33        3      active sync   /dev/sdc1
       4       8        1        4      active sync   /dev/sda1
       5       8       81        5      active sync   /dev/sdf1
       6       8      177        6      active sync   /dev/sdl1
       7       8      161        7      active sync   /dev/sdk1
       8       8      129        8      active sync   /dev/sdi1
       9       8      113        9      active sync   /dev/sdh1

RottenMutt · 04-07-2015, 08:09 AM

S.Haran · 04-07-2015, 01:51 PM

If you have the RAID configured correctly you should see at least a trace of a filesystem even if the event counts differ slightly. Since you don't I suspect misconfiguration.

One common test I run is to use Photorec to carve out jpg files. If a large jpg renders fine then you know you have the correct drive order, parity, and chunk size.

You can also look in /var/log/messages for any clues or evidence of drive failure. Also check the drives with a hex editor.

Sometimes these recoveries can be a bit tricky especially a RAID5 with 10 member disks. For example if you need to guess at the drive order for 10 drives there are over 3 million possible drive combinations. I use scripts to automate things but it is still a challenge.

RottenMutt · 04-14-2015, 11:53 PM

my original array had different "Data Offsets" for different drives, is there a way i can specify the old offset when creating per drive???

See my first post, the array had different data offsets, and when recreated in the third post it has the same offsets.

RottenMutt · 04-17-2015, 12:13 AM

How do i get this to work

$ sudo mdadm --create /dev/md0 --level=5 --chunk=512 --name=orion.localdomain:0 --assume-clean --raid-devices=10 --data-offset=variable /dev/sdc1:2048s /dev/sdd1:2048s /dev/sdf1:2048s /dev/sde1:2048s /dev/sdh1:128s /dev/sdg1:128s /dev/sdl1:128s /dev/sdk1:128s /dev/sdj1:128s /dev/sdi1:128s

mdadm: cannot open /dev/sdc1:2048s: No such file or directory

im using the latest fedora live usb distribution
uname -a
Linux localhost 3.17.4-301.fc21.x86_64 #1 SMP Thu Nov 27 19:09:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Code:

$ man mdadm
....
       --data-offset=....

              When creating an array, --data-offset can be specified as
              variable.  In the case each member device is expected to have
              a offset appended to the name, separated by a colon.  This
              makes it possible to recreate exactly an array which has
              varying data offsets (as can happen when different versions of
              mdadm are used to add different devices).

edit: i needed to add --data-offset=variable

RottenMutt · 04-18-2015, 02:07 PM

i got the array to reassemble (did complain in dmesg that the array was missaligned) and am offloading data to usb drives now.

when i'm done getting my files, i plan on stopping the array and deleting the partitions and reconstructing the array in raid 6. this time i will save mdadm data and drive serial numbers.

mdadm --examine /dev/sd[a-z]1
hdparm -I /dev/sd[a-z] | grep -E "Number|/dev"

syg00 · 04-18-2015, 08:42 PM

Sorry nobody has been able to help - this could get real ugly if we gave you bad advice, so better not to.
Thanks for continuing to keep us updated on your progress.