3Ware 7450, poor performance

aimed · 01-26-2009, 02:54 AM

I recently purchased couple of 3Ware 7450-4 RAID card and I'm not so thrilled about the performance of the cards. Currently I configured a 3 HD (á 160Gb) RAID-5 (sda) and in the fourth channel I've one 160Gb HD for testing purposes (sdb). Here's the results of hdparm:

Code:

[root@localhost root]# hdparm -tT /dev/sda
/dev/sda:
 Timing buffer-cache reads:   672 MB in  2.00 seconds = 336.00 MB/sec
 Timing buffered disk reads:   50 MB in  3.01 seconds =  16.61 MB/sec
[root@localhost root]# hdparm -tT /dev/sdb

/dev/sdb:
 Timing buffer-cache reads:   696 MB in  2.00 seconds = 348.00 MB/sec
 Timing buffered disk reads:  182 MB in  3.01 seconds =  60.47 MB/sec
[root@localhost root]#

Output of lspci -vvv follows:

Code:

00:09.0 RAID bus controller: 3ware Inc 7xxx/8xxx-series PATA/SATA-RAID (rev 01)
        Subsystem: 3ware Inc 7xxx/8xxx-series PATA/SATA-RAID
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (2250ns min), cache line size 08
        Interrupt: pin A routed to IRQ 16
        Region 0: I/O ports at dc00 [size=16]
        Region 1: Memory at da800000 (32-bit, non-prefetchable) [size=16]
        Region 2: Memory at da000000 (32-bit, non-prefetchable) [size=8M]
        Expansion ROM at <unassigned> [disabled] [size=64K]
        Capabilities: [40] Power Management version 1
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

dmesg:

Code:

SCSI subsystem driver Revision: 1.00
3ware Storage Controller device driver for Linux v1.02.00.037.
scsi0 : Found a 3ware Storage Controller at 0xdc00, IRQ: 16, P-chip: 1.3
scsi0 : 3ware Storage Controller
blk: queue c2dd9218, I/O limit 4095Mb (mask 0xffffffff)
  Vendor: 3ware     Model: Logical Disk 0    Rev: 1.0
  Type:   Direct-Access                      ANSI SCSI revision: 00
blk: queue c2dd9018, I/O limit 4095Mb (mask 0xffffffff)
  Vendor: 3ware     Model: Logical Disk 1    Rev: 1.0
  Type:   Direct-Access                      ANSI SCSI revision: 00
blk: queue e7991e18, I/O limit 4095Mb (mask 0xffffffff)
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0
SCSI device sda: 625159424 512-byte hdwr sectors (320082 MB)
Partition check:
 sda: sda1 sda2
SCSI device sdb: 312581808 512-byte hdwr sectors (160042 MB)
 sdb: sdb1

With four disk RAID-5 configuration the speed is about 20MB/sec. Should the card reach atleast 40MB/sec.

And one other thing is that when copying let's say an ISO image to the server via scp the load average maxes at server rises a lot:

Code:

 10:39:57  up  3:05,  2 users,  load average: 1.58, 0.57, 0.20
41 processes: 38 sleeping, 3 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total   34.4%    0.0%   16.4%   4.9%     2.8%   13.2%   28.3%
           cpu00   41.4%    0.0%   15.4%   9.6%     4.4%   26.4%    2.8%
           cpu01   27.4%    0.0%   17.4%   0.2%     1.2%    0.0%   53.8%
Mem:   637940k av,  624592k used,   13348k free,       0k shrd,    7428k buff
                     27432k actv,  435164k in_d,    8964k in_c
Swap:  979956k av,       0k used,  979956k free                  580432k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
 1392 root      25   0  2668 2668  1776 R    41.7  0.4   1:17   0 sshd
 1394 root      25   0   992  992   808 R     8.1  0.1   0:14   1 scp
   20 root      15   0     0    0     0 SW    0.5  0.0   0:00   0 kjournald
    7 root      15   0     0    0     0 SW    0.4  0.0   0:00   1 kswapd
 1402 root      15   0  1072 1072   884 R     0.1  0.1   0:00   1 top
    1 root      15   0   500  500   440 S     0.0  0.0   0:04   0 init

I've tested several OS's and versions (ubuntu from 6.4 to 8.10, centos 3 and 5, freebsd6) and the performance (or lack of it) is not OS related. The computer in which the card is now installed has dual P3 1000MHz CPU's and roughly 1GB RAM. I tested also with other computer (800Duron) which has a motherboard officially supported by the 3Ware but it didn't change anything. I've the same problem with both of the RAID cards so it's not card related either (unless both of those are broken :|)

What can I do to enhance the performance and lower the CPU usage?

aimed · 01-27-2009, 08:02 AM

Did some tests with Bonnie
sda:

Code:

[root@localhost tmp]# bonnie++ -u root -s 1024 -r 512 -n 5
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.94       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.locald 1G    97  99 13757  17  8429  10   790  84 27036  17 309.8   3
Latency             84238us   25376ms    2714ms   65687us     593ms    1444ms
Version  1.94       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  5  2799  73 +++++ +++ +++++ +++  3291  83 +++++ +++  7887  58
Latency             35777us      83us     627us    1002us      22us     647us
1.93c,1.94,localhost.localdomain,1,1233056162,1G,,97,99,13757,17,8429,10,790,84,27036,17,309.8,3,5,,,,,2799,73,+++++,+++,+++++,+++,3291,83,+++++,+++,7887,58,84238us,25376ms,2714ms,65687us,593ms,1444ms,35777us,83us,627us,1002us,22us,647us

sdb:

Code:

[root@localhost home]# bonnie++ -u root -s 1024 -r 512 -n 5
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.94       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.locald 1G    96  99 44658  54 20135  26   813  84 41561  26 243.5   2
Latency             87029us     895ms     874ms     170ms   69755us     566ms
Version  1.94       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  5  3168  81 +++++ +++ +++++ +++  3738  94 +++++ +++ +++++ +++
Latency             45440us      63us     101us   29804us      42us     725us
1.93c,1.94,localhost.localdomain,1,1233058953,1G,,96,99,44658,54,20135,26,813,84,41561,26,243.5,2,5,,,,,3168,81,+++++,+++,+++++,+++,3738,94,+++++,+++,+++++,+++,87029us,895ms,874ms,170ms,69755us,566ms,45440us,63us,101us,29804us,42us,725us

farslayer · 01-27-2009, 08:28 AM

Is NCQ enabled on your drives ? (you did buy Drives that support NCQ right ?)

How can I improve performance using 3ware controllers with the Linux 2.6 kernel?

I also saw a post where the guy improved his throughput to the array an additional 15-20% by switching from ext3 to xfs file system. ymmv.

I am not getting the performance I expect, what can I do ?

aimed · 01-27-2009, 01:30 PM

Thank you for your answer and links. I go through those and report back.

The card I have has standard IDE channels (PATA) so NCQ is nothing I need to worry about I suppose.

farslayer · 01-27-2009, 02:20 PM

Ahh this sounded like a new setup, so I incorrectly assumed it was SATA with new drives.. my bad.

Let us know what you find out, hope that info helps.

Electro · 01-28-2009, 12:13 AM

What is the block or stripe size that you are using for your array?

What is the brand of the hard drives that you are using?

aimed · 01-28-2009, 01:08 AM

Stripe size is 64kB which can't be changed at the time when creating the array in the 3ware BIOS.

Code:

Unit 0 (Controller ID 0)
Status 	OK
Capacity   	447.15 GB
Type   	RAID 5
Stripe   	64kB
Subunits   	4
Subunit 0
Status   	OK
Type   	DISK
Port   	0
	
Subunit 1
Status   	OK
Type   	DISK
Port   	1
	
Subunit 2
Status   	OK
Type   	DISK
Port   	2
	
Subunit 3
Status   	OK
Type   	DISK
Port   	3

Code:

Drive Information (Controller ID 0)
Port 	Model 	Capacity 	Serial # 	Firmware 	Unit 	Status
0 	WDC WD1600AAJB-00WRA0 	149.05 GB 	WD-WCAS2C137588 	58.01H58 	0 	OK
1 	ST3160812A 	149.05 GB 	4LS3MS6Y 	3.AAJ 	0 	OK
2 	WDC WD1600AAJB-00WRA0 	149.05 GB 	WD-WCAS2C262152 	58.01H58 	0 	OK
3 	ST3160023A 	149.05 GB 	5JS3W4HJ 	8.01 	0 	OK

Also the write cache is enabled.

aimed · 01-28-2009, 06:00 AM

All right! Tweaking indeed helped a bit. Here's what I got:

With blockdev --setra 16384 /dev/sda

Code:

$ sudo hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   394 MB in  2.00 seconds = 197.03 MB/sec
 Timing buffered disk reads:   90 MB in  3.00 seconds =  29.99 MB/sec

Code:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
filer            1G 14061  98 32569  38 21408  26 16246  98 88235  56 621.9   2

The other settings didn't seem to affect that much.

The 29.99MB/sec is not that much I expected to have but it's something I can live with.

Now with the other issue. The CPU load. Is it normal that the load average is around 0.5 - 0.6 in situation where I'm copying four ISO images to the server simultaneously and copying one image from the server? Since the RAID card is hardware based shouldn't it do all the work and leave the CPU alone?

Electro · 01-31-2009, 04:18 PM

A hardware RAID controller is designed to off load the processor when doing RAID tasks. It will not help during reading and writing data. The main processor is handling the file system.

I suggest for better throughput is test both the Western Digital and Seagate hard drives separately. Set pairs of hard drives with the same brand in RAID level 0 and then test the raw throughput using hdparm. Do the same for other pair of hard drives. This separates which one is causing a conflict or a performance penalty. You should get close to 100 megabytes per second.

Bonnie++ is OK if you can read it correctly. I suggest iozone. It mainly tests everything as a whole.

aimed · 02-06-2009, 04:34 PM

It was indeed one of the disks that caused the poor performance (ST3160023A I think it was). Earlier I tested the configuration with 1+0 and it gave near 70Mb/sec so I thought all the disk are OK. Silly from me for not to do test separately for each disk time ago :|

Thanks to you all who have answered to my posts.

Cheers!