LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware
User Name
Password
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?

Notices

Reply
 
Search this Thread
Old 06-29-2006, 01:24 PM   #1
kheldar
Member
 
Registered: Jul 2004
Location: Nottingham, England
Distribution: Fedora 17, Centos 6.3 x86_64
Posts: 35

Rep: Reputation: 2
30 second io waits with raid arrays on Centos 4.3 i86_64


Hi,

I've just upgraded my motherboard/cpus from a dual athlon to a dual opteron, and Fedora3 to Centos4.3 64 bit, and I'm getting the most bizare IO waits.

Basically, the whole IO on the machines stops for 30 seconds at a time. CPU only functions aren't affected, but anything that wants the disk stops. Also, it seems that there is nothing thrashing the cpu, or interrupt storms or just plain slow raid, as gkrellm reports that the CPUs are doing next to nothing during this time - just running the x and gkrellm stuff.

My lspci is as follows:
00:00.0 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.1 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.2 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.3 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.4 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:00.7 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge
00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI bridge [K8T800/K8T890 South]
00:06.0 RAID bus controller: Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller (rev 02)
00:0b.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5705 Gigabit Ethernet (rev 03)
00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800/K8T890 South]
00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: ATI Technologies Inc RV350 AS [Radeon 9550]
01:00.1 Display controller: ATI Technologies Inc RV350 ?? [Radeon 9550] (Secondary)


uname -a:
Linux monolith 2.6.9-34.0.1.ELsmp #1 SMP Wed May 24 05:28:30 CDT 2006 x86_64 x86_64 x86_64 GNU/Linux

The filesystem is:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
4128448 285804 3632932 8% /
/dev/md2 124323 22805 95099 20% /boot
none 1028868 0 1028868 0% /dev/shm
/dev/mapper/VolGroup00-LogVol01
10321208 6326608 3470312 65% /home
/dev/mapper/VolGroup00-LogVol06
4128448 2289652 1629084 59% /opt
/dev/mapper/VolGroup00-LogVol04
6321720 49496 5951152 1% /tmp
/dev/mapper/VolGroup00-LogVol02
41284928 37178056 2009720 95% /u01
/dev/md1 474931328 304256880 151374304 67% /u02
/dev/mapper/VolGroup00-LogVol03
10321208 3982444 5814476 41% /usr
/dev/mapper/VolGroup00-LogVol05
10321208 9192684 604236 94% /var


I've done some iostats during before, during and after the wait period:

Linux 2.6.9-34.0.1.ELsmp (monolith) 29/06/06

avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 1.20 2.40 96.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 1.80 0.00 2.81 0.00 38.48 0.00 19.24 13.71 0.00 0.21 0.21 0.06
hdb 0.60 1.80 64.33 2.81 4083.37 38.48 2041.68 19.24 61.40 0.06 0.84 0.81 5.47
hde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 64.93 3.61 4083.37 28.86 2041.68 14.43 60.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 64.93 3.01 4083.37 24.05 2041.68 12.02 60.46 0.06 0.82 0.81 5.49
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.60 0.00 4.81 0.00 2.40 8.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Linux 2.6.9-34.0.1.ELsmp (monolith) 29/06/06

avg-cpu: %user %nice %sys %iowait %idle
0.46 0.02 0.89 17.28 81.35

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 3.70 3.28 17.74 2.11 1008.58 43.15 504.29 21.57 52.97 0.91 44.49 11.41 22.66
hdb 3.79 3.30 19.21 2.13 1137.05 43.49 568.53 21.74 55.32 0.95 43.44 9.61 20.51
hde 2.23 0.01 0.10 0.03 18.65 0.33 9.33 0.16 148.76 0.00 14.53 8.79 0.11
hdf 2.15 0.02 0.10 0.04 17.97 0.45 8.98 0.22 136.68 0.00 18.06 9.97 0.13
hdg 0.14 0.00 0.01 0.00 0.59 0.00 0.30 0.00 48.71 0.00 406.65 406.65 0.50
md0 0.00 0.00 37.99 4.92 2105.66 39.35 1052.83 19.68 49.99 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.96 0.06 72.83 0.50 36.42 0.25 71.96 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.11 0.00 0.54 0.01 0.27 0.00 5.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 1.76 1.08 50.89 8.65 25.45 4.32 20.96 0.14 50.14 37.77 10.73
dm-1 0.00 0.00 28.38 0.88 1770.82 7.03 885.41 3.51 60.76 0.59 19.09 7.16 20.95
dm-2 0.00 0.00 0.10 0.00 0.83 0.01 0.42 0.01 7.96 0.00 9.07 1.80 0.02
dm-3 0.00 0.00 0.11 0.23 0.87 1.87 0.44 0.94 7.99 0.13 379.53 274.06 9.43
dm-4 0.00 0.00 0.66 0.07 5.31 0.56 2.66 0.28 7.99 0.02 31.87 28.97 2.13
dm-5 0.00 0.00 4.61 2.31 214.10 18.46 107.05 9.23 33.64 0.94 134.40 19.29 13.33
dm-6 0.00 0.00 2.12 0.35 60.71 2.77 30.36 1.39 25.73 0.25 100.77 40.00 9.87

avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.00 100.00 0.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 0.00 100.02
hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 0.00 100.02
hde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.20 0.00 1.60 0.00 0.80 8.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.20 0.00 1.60 0.00 0.80 8.00 0.00 0.00 2.00 0.04
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 0.00 100.02
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 100.02
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.20 79.28 20.52

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 3.21 7.83 3.41 477.11 51.41 238.55 25.70 47.00 1.65 1084.79 71.34 80.22
hdb 0.00 3.21 0.00 3.41 0.00 49.80 0.00 24.90 14.59 1.54 3311.59 231.82 79.14
hde 0.00 0.00 0.20 0.20 1.61 1.61 0.80 0.80 8.00 0.00 0.50 0.50 0.02
hdf 0.00 0.00 0.20 0.20 1.61 1.61 0.80 0.80 8.00 0.00 0.50 0.50 0.02
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 7.63 5.02 477.11 40.16 238.55 20.08 40.89 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.20 0.00 1.61 0.00 0.80 8.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 1.00 0.00 8.03 0.00 4.02 8.00 0.81 804.40 787.20 79.04
dm-1 0.00 0.00 7.63 2.21 477.11 17.67 238.55 8.84 50.29 1.60 1154.27 81.49 80.18
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 1.81 0.00 14.46 0.00 7.23 8.00 0.80 2064.67 437.67 79.10
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.10 0.00 1.30 3.20 95.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hda 70.58 1.39 42.74 1.79 2953.48 25.45 1476.74 12.72 66.89 0.11 2.57 1.05 4.69
hdb 70.58 1.39 6.16 1.79 613.92 25.45 306.96 12.72 80.40 0.09 11.57 2.23 1.77
hde 75.75 0.00 1.19 0.40 615.51 3.18 307.75 1.59 389.00 0.04 25.50 18.50 2.94
hdf 75.55 0.00 1.39 0.40 615.51 3.18 307.75 1.59 345.78 0.05 27.89 21.56 3.86
hdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 36.98 1.79 2342.74 14.31 1171.37 7.16 60.80 0.00 0.00 0.00 0.00
md1 0.00 0.00 10.93 0.60 2446.12 4.77 1223.06 2.39 212.55 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 36.98 1.79 2342.74 14.31 1171.37 7.16 60.80 0.04 1.08 1.07 4.16
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00


Any suggestions as to what I could try, or what could be causing it would be great.

Note: hda and hdb are connected to the ATA software raid controller (it's software raid features aren't used - I'm just using it as an additional controller). hde and hdf are connected to the motherboard ata controller.

The motherboard has a SATA raid controller built in which I am not using as I have no SATA drives.

I have subsequently switched off the SATA controller in the BIOS, but that hasn't helped.

I've also tested moving large files from the /dev/md0 (raid1) to a usbdisk, and that has worked fine. From /dev/md1 (raid5) the problem occurred, but only once in a fairly large file transfer.

Last edited by kheldar; 06-30-2006 at 03:15 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
reiserfs slight occasional corruption on software RAID-5 arrays Cairan Linux - Software 3 07-11-2006 05:11 PM
Transferring two RAID arrays Dee-ehn Linux - Software 2 09-01-2004 01:12 AM
WARNING: Some disks in your RAID arrays seem to have failed! patrickkenlock Debian 4 04-26-2004 03:19 AM
Lilo problems useing Promise raid card with no arrays deadlyicon Red Hat 1 11-12-2003 04:54 AM
Raid Arrays and IDE Probes Tarball_Phreak Linux - Hardware 0 07-15-2003 05:42 PM


All times are GMT -5. The time now is 01:08 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration