Slow disk access

Grabo · 10-06-2013, 02:29 PM

Hi!

I have a server with 11 disks.

Some of the disks are slow, no matters if it is on raidcard or mobo controller.

The disk are:

3xWD5003AZEX (2 works ok, 1 is slow r/w, all of them are on raidcard)
3xST2000DM001 (all of them working fine, mobo controller)
1xWD3200AVVS (as slow as the 'bad' one WD5003AZEX, mobo controller)
1xWD2500AAJS (works fine, mobo controller)
1xWD1002FAEX (works fine, mobo controller)
1xSV300S37A (works fine,raid controller)

Other components:
MOBO: Asus P5Q Deluxe / ASROCK 960GMU3S3
CPU: Core2Quad Q9550 / Phenom2 X4 960T
RAM: 2x2GB 1066MHz / 1x8GB 1333MHz
PSU: 550W XFX / 750W TT (just for testing)
RAIDCARD: HP Smart Array P400
System: Ubuntu 12.04.03 LTS / Debian 7

free:

Code:

root@server:/home/user1# free
             total       used       free     shared    buffers     cached
Mem:       4060768    3886892     173876          0      22980    2068720
-/+ buffers/cache:    1795192    2265576
Swap:      8383484     105192    8278292

uname:

Code:

root@server:/home/user1# uname -a
Linux s3rver 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux

hdparm of the 'bad' WD5003AZEX:

Code:

root@server:/home/user1# hdparm -Tt /dev/cciss/c0d3

/dev/cciss/c0d3:
 Timing cached reads:   9794 MB in  2.00 seconds = 4901.81 MB/sec
 Timing buffered disk reads: 140 MB in  3.02 seconds =  46.38 MB/sec

hdparm of the WD3200AVVS:

Code:

root@server:/home/user1# hdparm -Tt /dev/sde

/dev/sde:
 Timing cached reads:     2 MB in 28.99 seconds =  70.66 kB/sec
 Timing buffered disk reads:   4 MB in 12.52 seconds = 327.09 kB/sec

dd of the 'bad' WD5003AZEX (interrupted):

Code:

root@server:/mnt/raid/wd3# dd if=/dev/zero of=ddfile bs=8k
^C407917+0 beolvasott rekord
407917+0 kiírt rekord
3341656064 bájt (3,3 GB) másolva, 911,528 mp, 3,7 MB/s

All of the hardwares were replaced (the sata cables too), only the HDD-s are the same.
The two slow disk were tested at home, and they made a good result under Windows.

I'm out of ideas..
Thanks for your help, and sorry for my bad english.

Gabor

JJJCR · 10-06-2013, 11:04 PM

have you tried checking the RPM for those hard drives?

what's the raid configuration? RAID 5?

Grabo · 10-07-2013, 07:38 AM

Quote:

Originally Posted by JJJCR

have you tried checking the RPM for those hard drives?

what's the raid configuration? RAID 5?

Hi!
Thanks for your reply.

All of the disks are 7200RPM disks, and there isn't any raid configuration, just bunch of disks.

JJJCR · 10-08-2013, 08:03 PM

Hi Grabo, have you tried to check whether there are linux hard drive updates for the components of your system?

Linux Hard drive update for the Mobo / Sata Controller / RAID update

Linux Hard drive update for the HDD itself from the manufacturer website

Linux OS update for the hard drive

Grabo · 10-09-2013, 10:02 AM

Quote:

Originally Posted by JJJCR

Hi Grabo, have you tried to check whether there are linux hard drive updates for the components of your system?

Linux Hard drive update for the Mobo / Sata Controller / RAID update

Linux Hard drive update for the HDD itself from the manufacturer website

Linux OS update for the hard drive

Hi,

The hard drive firmwares are'nt public, but all of the WD Blacks (WD5003AZEX) have the same firmware, so it can't be a problem. The mobo's bios is the freshest,and the raid controller too.

chrism01 · 10-09-2013, 07:37 PM

Might be worth using this http://linux.die.net/man/8/smartd to run some checks/tests.

JJJCR · 10-09-2013, 07:38 PM

Quote:

The hard drive firmwares are'nt public

not the firmware, like for IBM they provide an hdd driver for Linux, Windows. Check with the server manufacturer, whether they have such updates for your OS.

Grabo · 10-10-2013, 08:41 AM

Quote:

Originally Posted by JJJCR

not the firmware, like for IBM they provide an hdd driver for Linux, Windows. Check with the server manufacturer, whether they have such updates for your OS.

The disks are slow on the raidcard too, so I think there is a configuration problem or kernel problem somewhere.

The smart condition looks good.

Code:

root@server:/# smartctl -H /dev/sde
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-4-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Code:

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       412         -
# 2  Extended offline    Completed without error       00%         1         -

JJJCR · 10-11-2013, 05:53 AM

Quote:

so I think there is a configuration problem or kernel problem somewhere.

Yes, it could be but what was the hard drive that was configured on the server before this issue started?

Does it has any issues before? or this is a scratch installation?

Grabo · 10-11-2013, 06:32 AM

Quote:

Originally Posted by JJJCR

Yes, it could be but what was the hard drive that was configured on the server before this issue started?

Does it has any issues before? or this is a scratch installation?

It is a scratch install, at the time the server runs Debian 7.
Today I will try the disk speed under Windows, to rule out the hardware failure.

EDIT:

Now, 8 disks are connected to the raidcard of 10, the disks are still slow. The test, under Windows was not implemented yet.

Grabo · 10-28-2013, 12:39 PM

Hi!

If i try running dd one of the slow disks, in the dmesg appears the following:

[ 806.743885] Buffer I/O error on device sdb1, logical block 151037
[ 806.744051] Buffer I/O error on device sdb1, logical block 151038
[ 806.744168] Buffer I/O error on device sdb1, logical block 151039
[ 806.744316] EXT4-fs warning (device sdb1): ext4_end_bio:250: I/O error writing to inode 12 (offset 467140608 size 524288 starting block 151296)
[ 806.744330] sd 7:0:0:0: [sdb] Unhandled error code
[ 806.744331] sd 7:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 806.744334] sd 7:0:0:0: [sdb] CDB: Write(10): 2a 00 00 12 78 00 00 04 00 00
[ 806.744339] end_request: I/O error, dev sdb, sector 1210368

The disk tests seems ok, there is'nt any error (long test).

smeezekitty · 10-28-2013, 02:47 PM

That looks like hardware issues.

Grabo · 10-28-2013, 02:49 PM

The hdd is fine, i have tested it at my home computer, under Windows,and the speed was right.

smeezekitty · 10-28-2013, 03:04 PM

Have you tried a different SATA cable?

Grabo · 10-28-2013, 03:06 PM

Yes, i tried. Now I think maybe the raidcard is the source of the problems..
I will try out soon.
I also tried a CentOS install to WD3200AVVS but it was slow.

Can it be the OS?