30 second io waits with raid arrays on Centos 4.3 i86_64

kheldar · 06-29-2006, 12:24 PM

Hi,

I've just upgraded my motherboard/cpus from a dual athlon to a dual opteron, and Fedora3 to Centos4.3 64 bit, and I'm getting the most bizare IO waits.

Basically, the whole IO on the machines stops for 30 seconds at a time. CPU only functions aren't affected, but anything that wants the disk stops. Also, it seems that there is nothing thrashing the cpu, or interrupt storms or just plain slow raid, as gkrellm reports that the CPUs are doing next to nothing during this time - just running the x and gkrellm stuff.

My lspci is as follows:
00:00.0 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.1 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.2 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.3 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.4 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.7 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI bridge [K8T800/K8T890 South]
00:06.0 RAID bus controller: Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller (rev 02)
00:0b.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5705 Gigabit Ethernet (rev 03)
00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800/K8T890 South]
00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: ATI Technologies Inc RV350 AS [Radeon 9550]
01:00.1 Display controller: ATI Technologies Inc RV350 ?? [Radeon 9550] (Secondary)

uname -a:
Linux monolith 2.6.9-34.0.1.ELsmp #1 SMP Wed May 24 05:28:30 CDT 2006 x86_64 x86_64 x86_64 GNU/Linux

The filesystem is:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
4128448 285804 3632932 8% /
/dev/md2 124323 22805 95099 20% /boot
none 1028868 0 1028868 0% /dev/shm
/dev/mapper/VolGroup00-LogVol01
10321208 6326608 3470312 65% /home
/dev/mapper/VolGroup00-LogVol06
4128448 2289652 1629084 59% /opt
/dev/mapper/VolGroup00-LogVol04
6321720 49496 5951152 1% /tmp
/dev/mapper/VolGroup00-LogVol02
41284928 37178056 2009720 95% /u01
/dev/md1 474931328 304256880 151374304 67% /u02
/dev/mapper/VolGroup00-LogVol03
10321208 3982444 5814476 41% /usr
/dev/mapper/VolGroup00-LogVol05
10321208 9192684 604236 94% /var

I've done some iostats during before, during and after the wait period:

Linux 2.6.9-34.0.1.ELsmp (monolith) 29/06/06

avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 1.20 2.40 96.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 1.80 0.00 2.81 0.00 38.48 0.00 19.24 13.71 0.00 0.21 0.21 0.06
hdb 0.60 1.80 64.33 2.81 4083.37 38.48 2041.68 19.24 61.40 0.06 0.84 0.81 5.47
hde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 64.93 3.61 4083.37 28.86 2041.68 14.43 60.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 64.93 3.01 4083.37 24.05 2041.68 12.02 60.46 0.06 0.82 0.81 5.49
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.60 0.00 4.81 0.00 2.40 8.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Linux 2.6.9-34.0.1.ELsmp (monolith) 29/06/06

avg-cpu: %user %nice %sys %iowait %idle
0.46 0.02 0.89 17.28 81.35

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 3.70 3.28 17.74 2.11 1008.58 43.15 504.29 21.57 52.97 0.91 44.49 11.41 22.66
hdb 3.79 3.30 19.21 2.13 1137.05 43.49 568.53 21.74 55.32 0.95 43.44 9.61 20.51
hde 2.23 0.01 0.10 0.03 18.65 0.33 9.33 0.16 148.76 0.00 14.53 8.79 0.11
hdf 2.15 0.02 0.10 0.04 17.97 0.45 8.98 0.22 136.68 0.00 18.06 9.97 0.13
hdg 0.14 0.00 0.01 0.00 0.59 0.00 0.30 0.00 48.71 0.00 406.65 406.65 0.50
md0 0.00 0.00 37.99 4.92 2105.66 39.35 1052.83 19.68 49.99 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.96 0.06 72.83 0.50 36.42 0.25 71.96 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.11 0.00 0.54 0.01 0.27 0.00 5.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 1.76 1.08 50.89 8.65 25.45 4.32 20.96 0.14 50.14 37.77 10.73
dm-1 0.00 0.00 28.38 0.88 1770.82 7.03 885.41 3.51 60.76 0.59 19.09 7.16 20.95
dm-2 0.00 0.00 0.10 0.00 0.83 0.01 0.42 0.01 7.96 0.00 9.07 1.80 0.02
dm-3 0.00 0.00 0.11 0.23 0.87 1.87 0.44 0.94 7.99 0.13 379.53 274.06 9.43
dm-4 0.00 0.00 0.66 0.07 5.31 0.56 2.66 0.28 7.99 0.02 31.87 28.97 2.13
dm-5 0.00 0.00 4.61 2.31 214.10 18.46 107.05 9.23 33.64 0.94 134.40 19.29 13.33
dm-6 0.00 0.00 2.12 0.35 60.71 2.77 30.36 1.39 25.73 0.25 100.77 40.00 9.87

avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.00 100.00 0.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 0.00 100.02
hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 0.00 100.02
hde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.20 0.00 1.60 0.00 0.80 8.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.20 0.00 1.60 0.00 0.80 8.00 0.00 0.00 2.00 0.04
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 0.00 100.02
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 100.02
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.20 79.28 20.52

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 3.21 7.83 3.41 477.11 51.41 238.55 25.70 47.00 1.65 1084.79 71.34 80.22
hdb 0.00 3.21 0.00 3.41 0.00 49.80 0.00 24.90 14.59 1.54 3311.59 231.82 79.14
hde 0.00 0.00 0.20 0.20 1.61 1.61 0.80 0.80 8.00 0.00 0.50 0.50 0.02
hdf 0.00 0.00 0.20 0.20 1.61 1.61 0.80 0.80 8.00 0.00 0.50 0.50 0.02
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 7.63 5.02 477.11 40.16 238.55 20.08 40.89 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.20 0.00 1.61 0.00 0.80 8.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 1.00 0.00 8.03 0.00 4.02 8.00 0.81 804.40 787.20 79.04
dm-1 0.00 0.00 7.63 2.21 477.11 17.67 238.55 8.84 50.29 1.60 1154.27 81.49 80.18
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 1.81 0.00 14.46 0.00 7.23 8.00 0.80 2064.67 437.67 79.10
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.10 0.00 1.30 3.20 95.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 70.58 1.39 42.74 1.79 2953.48 25.45 1476.74 12.72 66.89 0.11 2.57 1.05 4.69
hdb 70.58 1.39 6.16 1.79 613.92 25.45 306.96 12.72 80.40 0.09 11.57 2.23 1.77
hde 75.75 0.00 1.19 0.40 615.51 3.18 307.75 1.59 389.00 0.04 25.50 18.50 2.94
hdf 75.55 0.00 1.39 0.40 615.51 3.18 307.75 1.59 345.78 0.05 27.89 21.56 3.86
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 36.98 1.79 2342.74 14.31 1171.37 7.16 60.80 0.00 0.00 0.00 0.00
md1 0.00 0.00 10.93 0.60 2446.12 4.77 1223.06 2.39 212.55 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 36.98 1.79 2342.74 14.31 1171.37 7.16 60.80 0.04 1.08 1.07 4.16
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Any suggestions as to what I could try, or what could be causing it would be great.

Note: hda and hdb are connected to the ATA software raid controller (it's software raid features aren't used - I'm just using it as an additional controller). hde and hdf are connected to the motherboard ata controller.

The motherboard has a SATA raid controller built in which I am not using as I have no SATA drives.

I have subsequently switched off the SATA controller in the BIOS, but that hasn't helped.

I've also tested moving large files from the /dev/md0 (raid1) to a usbdisk, and that has worked fine. From /dev/md1 (raid5) the problem occurred, but only once in a fairly large file transfer.