I am running a software raid using md with gentoo. I have 4 500 gig hard rives (I added one later about 6 months after setting up the raid). Its always been ok, except recently it has been acting "weird". Let me elaborate. On bootup mdadm automatically detects the array I suppose by looking in mdadm.conf. During the boot sequence it adds the drives and then mounts the array. The problem is that recently it has still been detecting the drives, but when it finishes booting the array isn't there. Even worse I get mailed a status message that says a device went missing. But then when I run mdadm --auto-detect, the raid gets detected and goes right up (I don't think in a degraded state, I checked at least once). Now 2 days ago I started hearing those dreaded clicking noises your hdd makes when its about to die. And about a week ago I was unable to transfer a 3 gig file onto the array, although at the time I thought it was the drive that the data was coming off of. Additionally Yesterday I finally connected the dots when I couldn't transfer data from my boot drive to the raid. I then tried to copy a DVD to the drive and that stopped (floundered and just locked). None of the failed transfers have given me errors, they just stop. So I am wondering, is this the behavior of a raid with a device that is going bad? I had a hardrive die before, but it just would just stop showing up. Secondly how do I tell which one it is. I already ran smartctl -t long on all the drives and they came back clean. Should I just remove one at a time and badblock it or is there a better way?

just curious - what's your hardware?

I have a Core 2 Quad 6600, with an Abit P35 motherboard. Also the raid drives are all on the intel sata controller:

# lspci
00:00.0 Host bridge: Intel Corporation DRAM Controller (rev 02)
00:01.0 PCI bridge: Intel Corporation PCI Express Root Port (rev 02)
00:1a.0 USB Controller: Intel Corporation USB UHCI Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation USB UHCI Controller #5 (rev 02)
00:1a.2 USB Controller: Intel Corporation USB UHCI Controller #6 (rev 02)
00:1a.7 USB Controller: Intel Corporation USB2 EHCI Controller #2 (rev 02)
00:1b.0 Audio device: Intel Corporation HD Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation PCI Express Port 1 (rev 02)
00:1c.4 PCI bridge: Intel Corporation PCI Express Port 5 (rev 02)
00:1d.0 USB Controller: Intel Corporation USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation USB UHCI Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation LPC Interface Controller (rev 02)
00:1f.2 IDE interface: Intel Corporation 4 port SATA IDE Controller (rev 02)
00:1f.3 SMBus: Intel Corporation SMBus Controller (rev 02)
00:1f.5 IDE interface: Intel Corporation 2 port SATA IDE Controller (rev 02)
01:00.0 VGA compatible controller: nVidia Corporation NV44 [GeForce 6200 LE] (rev a1)
03:00.0 SATA controller: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02)
03:00.1 IDE interface: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10)
04:01.0 Ethernet controller: Realtek
Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10)
04:03.0 Multimedia video controller: Internext Compression Inc iTVC16 (CX23416) MPEG-2 Encoder (rev 01)
04:04.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
04:05.0 Ethernet controller: Atheros Communications, Inc. AR5212 802.11abg NIC (rev 01)
#

So also, here is the bootup string and then what is going on after boot:
md: Autodetecting RAID arrays.
md: autorun ...
md: considering sdd1 ...
md: adding sdd1 ...
md: adding sdc1 ...
md: adding sdb1 ...
md: adding sda1 ...
md: created md0
md: bind<sda1>
md: bind<sdb1>
md: bind<sdc1>
md: bind<sdd1>
md: running: <sdd1><sdc1><sdb1><sda1>
md: personality for level 5 is not loaded!
md: do_md_run() returned -22
md: md0 stopped.
md: unbind<sdd1>
md: export_rdev(sdd1)
md: unbind<sdc1>
md: export_rdev(sdc1)
md: unbind<sdb1>
md: export_rdev(sdb1)
md: unbind<sda1>
md: export_rdev(sda1)
md: ... autorun DONE.
ReiserFS: sde2: found reiserfs format "3.6" with standard journal
ReiserFS: sde2: using ordered data mode
ReiserFS: sde2: journal params: device sde2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: sde2: checking transaction log (sde2)
ReiserFS: sde2: Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) readonly.
Freeing unused kernel memory: 220k freed
udev: renamed network interface eth0 to eth2
ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 22 (level, low) -> IRQ 22
PCI: Setting latency timer of device 0000:00:1b.0 to 64
ivtv: Start initialization, version 1.0.0
ivtv0: Initializing card #0
ivtv0: Autodetected Hauppauge card (cx23416 based)
ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 22 (level, low) -> IRQ 22
ivtv0: Loaded v4l-cx2341x-enc.fw firmware (376836 bytes)
ivtv0: Encoder revision: 0x02060039
tveeprom 0-0050: Hauppauge model 26552, rev G168, serial# 8880817
tveeprom 0-0050: tuner model is LG TAPE H001F MK3 (idx 68, type 47)
tveeprom 0-0050: TV standards NTSC(M) (eeprom 0x08)
tveeprom 0-0050: audio processor is CX25843 (idx 37)
tveeprom 0-0050: decoder processor is CX25843 (idx 30)
tveeprom 0-0050: has radio, has no IR receiver, has no IR transmitter
ivtv0: Autodetected Hauppauge WinTV PVR-150
tuner 0-0043: chip found @ 0x86 (ivtv i2c driver #0)
tda9887 0-0043: tda988[5/6/7] found @ 0x43 (tuner)
tuner 0-0061: chip found @ 0xc2 (ivtv i2c driver #0)
cx25840 0-0044: cx25843-23 found @ 0x88 (ivtv i2c driver #0)
cx25840 0-0044: loaded v4l-cx25840.fw firmware (16382 bytes)
wm8775 0-001b: chip found @ 0x36 (ivtv i2c driver #0)
tuner 0-0061: type set to 47 (LG NTSC (TAPE series))
ivtv0: Registered device video0 for encoder MPEG (4 MB)
ivtv0: Registered device video32 for encoder YUV (2 MB)
ivtv0: Registered device vbi0 for encoder VBI (1 MB)
ivtv0: Registered device video24 for encoder PCM audio (1 MB)
ivtv0: Registered device radio0 for encoder radio
ivtv0: Initialized card #0: Hauppauge WinTV PVR-150
JMB363: IDE controller at PCI slot 0000:03:00.1
ivtv: End initialization
PCI: Enabling device 0000:03:00.1 (0000 -> 0001)
ACPI: PCI Interrupt 0000:03:00.1[B] -> GSI 17 (level, low) -> IRQ 17
JMB363: chipset revision 2
JMB363: 100% native mode on irq 17
PCI: Setting latency timer of device 0000:03:00.1 to 64
ide0: BM-DMA at 0xcb00-0xcb07, BIOS settings: hdaio, hdbio
ide1: BM-DMA at 0xcb08-0xcb0f, BIOS settings: hdcio, hddio
Probing IDE interface ide0...
hda: LITE-ON DVD SHD-16P1S, ATAPI CD/DVD-ROM drive
hdb: MAXTOR STM3500630A, ATA DISK drive
hda: selected mode 0x42
hdb: selected mode 0x45
ide0 at 0xcf00-0xcf07,0xce02 on irq 17
hda: ATAPI 48X DVD-ROM drive, 1725kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
hdb: max request size: 512KiB
hdb: 976773168 sectors (500107 MB) w/16384KiB Cache, CHS=60801/255/63, UDMA(100)
hdb: cache flushes supported
hdb: hdb1
Probing IDE interface ide1...
r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 23 (level, low) -> IRQ 23
eth0: RTL8169sc/8110sc at 0xffffc20001166000, 00:50:8d:b6:11:af, XID 18000000 IRQ 23
r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
ACPI: PCI Interrupt 0000:04:01.0[A] -> GSI 22 (level, low) -> IRQ 22
eth1: RTL8169sc/8110sc at 0xffffc20001168000, 00:50:8d:b6:11:b0, XID 18000000 IRQ 22
nvidia: module license 'NVIDIA' taints kernel.
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:01:00.0 to 64
NVRM: loading NVIDIA Linux x86_64 Kernel Module 96.43.01 Wed Sep 5 18:44:48 PDT 2007
ath_hal: 0.9.18.0 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
wlan: 0.8.4.2 (0.9.3.3)
ath_pci: 0.9.4.5 (0.9.3.3)
ACPI: PCI Interrupt 0000:04:05.0[A] -> GSI 20 (level, low) -> IRQ 20
ath_rate_sample: 1.2 (0.9.3.3)
wifi0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
wifi0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: turboG rates: 6Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: H/W encryption support: WEP AES AES_CCM TKIP
wifi0: mac 7.9 phy 4.5 radio 5.6
wifi0: Use hw queue 1 for WME_AC_BE traffic
wifi0: Use hw queue 0 for WME_AC_BK traffic
wifi0: Use hw queue 2 for WME_AC_VI traffic
wifi0: Use hw queue 3 for WME_AC_VO traffic
wifi0: Use hw queue 8 for CAB traffic
wifi0: Use hw queue 9 for beacons
wifi0: Atheros 5212: mem=0xfdce0000, irq=20
nf_conntrack_ftp: Unknown parameter `port'
vboxdrv: Trying to deactivate the NMI watchdog permanently...
vboxdrv: Successfully done.
vboxdrv: Successfully loaded version 1.5.4 (interface 0x00050002).
md: md0 stopped.
SGI XFS with large block/inode numbers, no debug enabled
XFS: SB read failed
r8169: eth0: link down
r8169: eth1: link down
eth2: link up, 100Mbps, full-duplex, lpa 0xC5E1
Adding 1048568k swap on /swapfile. Priority:-1 extents:701 across:177661880k

So md0 gets detected then at the end of the boot cycle I get that
md: md0 stopped

But then:
# mdadm --auto-detect

#
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda1[0] sdb1[3] sdc1[2] sdd1[1]
1465150464 blocks level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

(I forgot to get the before of /proc/mdstat, it said that no raids were detected or something like that)
mdadm --detail still doesn't give me the correct information

So I have a theory about the possible cause of my lock ups. I don't know why mdadm isn't starting, but I think I might know why the transfers are dying. I started looking through the logs and low and behold:

Feb 13 02:39:27 oldcub wpa_cli invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0
Feb 13 02:39:27 oldcub
Feb 13 02:39:27 oldcub Call Trace:
Feb 13 02:39:27 oldcub [<ffffffff80263322>] out_of_memory+0x1a2/0x210
Feb 13 02:39:27 oldcub [<ffffffff80264e6d>] __alloc_pages+0x2dd/0x330
Feb 13 02:39:27 oldcub [<ffffffff80266ae5>] __do_page_cache_readahead+0x165/0x300
Feb 13 02:39:27 oldcub [<ffffffff8050acd5>] __wait_on_bit_lock+0x65/0x80
Feb 13 02:39:27 oldcub [<ffffffff8025ff3f>] __lock_page+0x5f/0x70
Feb 13 02:39:27 oldcub [<ffffffff8024c420>] wake_bit_function+0x0/0x30
Feb 13 02:39:27 oldcub [<ffffffff80262468>] filemap_fault+0x358/0x440
Feb 13 02:39:27 oldcub [<ffffffff8026c45a>] __do_fault+0x6a/0x450
Feb 13 02:39:27 oldcub [<ffffffff8026e1d0>] handle_mm_fault+0x1a0/0x860
Feb 13 02:39:27 oldcub [<ffffffff80223aae>] do_page_fault+0x1ae/0x910
Feb 13 02:39:27 oldcub [<ffffffff8049167e>] sys_recvfrom+0xfe/0x1a0
Feb 13 02:39:27 oldcub [<ffffffff80498fe0>] datagram_poll+0x0/0x110
Feb 13 02:39:27 oldcub [<ffffffff8050c45d>] error_exit+0x0/0x84
Feb 13 02:39:27 oldcub
Feb 13 02:39:27 oldcub Mem-info:
Feb 13 02:39:27 oldcub DMA per-cpu:
Feb 13 02:39:27 oldcub CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 13 02:39:27 oldcub CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 13 02:39:27 oldcub CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 13 02:39:27 oldcub CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 13 02:39:27 oldcub DMA32 per-cpu:
Feb 13 02:39:27 oldcub CPU 0: Hot: hi: 186, btch: 31 usd: 62 Cold: hi: 62, btch: 15 usd: 21
Feb 13 02:39:27 oldcub CPU 1: Hot: hi: 186, btch: 31 usd: 97 Cold: hi: 62, btch: 15 usd: 49
Feb 13 02:39:27 oldcub CPU 2: Hot: hi: 186, btch: 31 usd: 99 Cold: hi: 62, btch: 15 usd: 5
Feb 13 02:39:27 oldcub CPU 3: Hot: hi: 186, btch: 31 usd: 83 Cold: hi: 62, btch: 15 usd: 52
Feb 13 02:39:27 oldcub Active:230491 inactive:243 dirty:0 writeback:0 unstable:0
Feb 13 02:39:27 oldcub free:1978 slab:14213 mapped:45 pagetables:1562 bounce:0
Feb 13 02:39:27 oldcub DMA free:3988kB min:40kB low:48kB high:60kB active:7216kB inactive:0kB present:10736kB pages_scanned:11768 all_unreclaimable? yes
Feb 13 02:39:27 oldcub lowmem_reserve[]: 0 993 993 993
Feb 13 02:39:27 oldcub DMA32 free:3924kB min:4008kB low:5008kB high:6012kB active:914748kB inactive:972kB present:1016944kB pages_scanned:1680387 all_unreclaimable? yes
Feb 13 02:39:27 oldcub lowmem_reserve[]: 0 0 0 0
Feb 13 02:39:27 oldcub DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3988kB
Feb 13 02:39:27 oldcub DMA32: 16*4kB 12*8kB 3*16kB 2*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3984kB
Feb 13 02:39:27 oldcub Swap cache: add 0, delete 0, find 0/0, race 0+0
Feb 13 02:39:27 oldcub Free swap = 0kB
Feb 13 02:39:27 oldcub Total swap = 0kB
Feb 13 02:39:27 oldcub Free swap: 0kB

I have no swap, mostly because I am an idiot, but also because I thought, "why would a linux box with a whole gig of ram need swap !!!11one". I do have a swapfile I turn on every once in a while. So that looks pretty bad, but it only matters if mdadm or something related dies like....

# zgrep 'oom-killer' * | grep xfs
messages.1.gz:Feb 13 02:39:32 oldcub xfs_db invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0
messages.1.gz:Feb 13 02:43:05 oldcub xfs_db invoked oom-killer: gfp_mask=0xa80d2, order=0, oomkilladj=0
# zgrep 'oom-killer' * | grep mdadm
messages.24.gz:Jan 13 13:03:45 oldcub mdadm invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0
messages.24.gz:Jan 13 13:03:45 oldcub mdadm invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0

So I am testing now with a second gig of ram and swap on. We'll see what happens.

Alright so I figured out the other half. I need to stop posting here, because I inevitably figure it out about 24 hours after I do. The problem appears to be something with boot order in gentoo. Its not a problem per se, it just wasn't loading the md module early enough. It used to, but for some reason it stopped. I started building md into the kernel and now it works. On the other hand, maybe there is a deadlock in md somehow? I am running multicore and 64 bit so maybe there is a bug?

Hmm nevermind, it is happening right now. Everything *seems* fine except that a copy to the drive has just frozen. Any ideas?

found this:
http://www.mail-archive.com/linux-ra.../msg10310.html