SCSI/software RAID performance inadequate

stefan_nicolau · 07-19-2006, 09:48 PM

Hi,
I recently bought a used Compaq Proliant (specs below) and I've been surprised with the low performance of its SCSI drives.

I compared the performances of three machines:
1: Compaq Proliant ML370: dual p3/800, 896MB RAM, 2x9.1GB SCSI 10K RPM in a RAID-1 array (software), debian Etch, circa 2000
2: IBM: p3/800, 256MB RAM, 10GB IDE (5400 RPM?), debian Etch, circa 2000
3: custom: p4/2.6, 1GB RAM, 80GB IDE 7200 RPM, debian Sid, 2004

benchmarks are (in order)
1- bonnie++ (all filesystems are ext3)
2- dd
3- hdparm

note:

1's performance is much lower than I expected for a RAID1 scsi 10K machine
1 beats 3 in random seeks
in 1 and 3, dd is faster (!) than hdparm
3 hugely beats 1 in most tests
1's scsi drives have the same hdparm speed as 2's standard, slow, IDE
In most tests, CPU is not maxed, so that's not the bottleneck
hdparm and dd are not nearly twice as fast for the RAID array as for the individual drives. This is probably the most striking thing.

What I'm most interested in is boosting 1's output performance. I don't understand why, with a raid array of two much faster disks and an extra CPU, its performance is often not even double that of 2.

I've looked at the performances in http://www.linuxquestions.org/questi...ad.php?t=75181 and 2 and 3 seem to fit in well with what others get with similar setups.

I performed all tests several times and got consistent results.

Code:

------------------------ 1 ------------------------

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
1                2G 12989  97 27557  32 13027  15 12728  90 23434  13 399.9   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   625  99 +++++ +++ 32322 100   639  98 +++++ +++  2124  96

dd if=/dev/md0 of=/dev/null bs=4K count=2M
1977968+0 records in
1977968+0 records out
8101756928 bytes (8.1 GB) copied, 248.317 seconds, 32.6 MB/s

dd if=/dev/sda of=/dev/null bs=4K count=2M
2097152+0 records in
2097152+0 records out
8589934592 bytes (8.6 GB) copied, 264.077 seconds, 32.5 MB/s

/dev/sda:
 Timing cached reads:   872 MB in  2.00 seconds = 434.98 MB/sec
 Timing buffered disk reads:   74 MB in  3.02 seconds =  24.49 MB/sec
(sdb the same)

/dev/md0:
 Timing cached reads:   876 MB in  2.01 seconds = 436.32 MB/sec
 Timing buffered disk reads:   84 MB in  3.01 seconds =  27.87 MB/sec

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[0] sdb1[1]
      7911872 blocks [2/2] [UU]

lsscsi -lv
[0:0:4:0]    tape    ARCHIVE  Python 04106-XXX 7270  /dev/st0
  state=running queue_depth=2 scsi_level=3 type=1 device_blocked=0 timeout=900
  dir: /sys/bus/scsi/devices/0:0:4:0  [/sys/devices/pci0000:00/0000:00:01.0/host0/target0:0:4/0:0:4:0]
[1:0:0:0]    disk    COMPAQ   BD0096349A       3B12  /dev/sda
  state=running queue_depth=16 scsi_level=3 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/1:0:0:0  [/sys/devices/pci0000:00/0000:00:01.1/host1/target1:0:0/1:0:0:0]
[1:0:1:0]    disk    COMPAQ   BD0096349A       3B12  /dev/sdb
  state=running queue_depth=16 scsi_level=3 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/1:0:1:0  [/sys/devices/pci0000:00/0000:00:01.1/host1/target1:0:1/1:0:1:0]

sginfo -a /dev/sda (edited)
INQUIRY response (cmd: 0x12)
----------------------------
Device Type                        0
Vendor:                    COMPAQ  
Product:                   BD0096349A      
Revision level:            3B12

Serial Number '3BV16RW60000V1220D21'

Caching mode page (0x8)
-----------------------
Initiator Control                  0
ABPF                               0
CAP                                0
DISC                               0
SIZE                               0
Write Cache Enabled                0
MF                                 0
Read Cache Disabled                0
Demand Read Retention Priority     0
Demand Write Retention Priority    0
Disable Pre-fetch Transfer Length  65535
Minimum Pre-fetch                  0
Maximum Pre-fetch                  65535
Maximum Pre-fetch Ceiling          65535
FSW                                1
LBCSS                              0
DRA                                0
NV_DIS                             0
Number of Cache Segments           64
Cache Segment size                 0
Non-Cache Segment size             0

Control mode page (0xa)
-----------------------
TST                                0
TMF_ONLY                           0
D_SENSE                            0
GLTSD                              0
RLEC                               0
Queue Algorithm Modifier           1
QErr                               0
DQue [obsolete]                    0
TAS                                0
RAC                                0
UA_INTLCK_CTRL                     0
SWP                                0
RAERP [obs.]                       0
UAAERP [obs.]                      0
EAERP [obs.]                       0
ATO                                0
TAS                                0
AUTOLOAD MODE                      0
Ready AER Holdoff Period [obs.]    0
Busy Timeout Period                0
Extended self-test completion time 419

Code:

------------------------ 2 ------------------------

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
2              496M 13449  96 15763  13  7321   7  9492  64 16676   9 122.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   631  99 +++++ +++ +++++ +++   596  98 +++++ +++  2077  93

dd if=/dev/hda of=/dev/null bs=4K count=2M
2097152+0 records in
2097152+0 records out
8589934592 bytes (8.6 GB) copied, 366.365 seconds, 23.4 MB/s

/dev/hda:
 Timing cached reads:   952 MB in  2.00 seconds = 475.60 MB/sec
 Timing buffered disk reads:   74 MB in  3.00 seconds =  24.63 MB/sec

Code:

------------------------ 3 ------------------------

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
3                2G 33328  92 46015  24 19707   7 28781  65 48270   5 131.2   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2788  97 +++++ +++ +++++ +++  2883  99 +++++ +++  6718  99


dd if=/dev/hda of=/dev/null bs=4K count=2M
2097152+0 records in
2097152+0 records out
8589934592 bytes (8.6 GB) copied, 146.344 seconds, 58.7 MB/s

/dev/hda:
 Timing cached reads:   2624 MB in  2.00 seconds = 1313.13 MB/sec
 Timing buffered disk reads:  170 MB in  3.01 seconds =  56.57 MB/sec