extremly slow random read on new raid5 array

Tomasu · 05-30-2009, 01:03 AM

I just setup a new md raid5 array with 4 Seagate 7200.12 1TB disks. The sequential read and write is pretty amazing to me, but the random read is going very slowly. iostat says its going at about 500KB/s vs the 300MB/s+ and 170MB/s+ for sequential read and write respectively.

I've seen people report they get random reads in the double digit MB range, and I would have at least liked to see 50MB/s or something along those lines.

The raid array is setup with a 64KB chunk, I'm using ext4 (the mkfs and fsck times for the full array with ext3 would have killed me, and I'm starting to dislike the occasional corruption and slowness I get with XFS).

What am I missing? There's got to be something I misconfigured some place.

Tomasu · 05-30-2009, 02:47 PM

My iozone test finished, so I'll post the results:

Code:

natasha:/mnt/test-data# time iozone -a -s 8G
        Iozone: Performance Test of File I/O
                Version $Revision: 3.308 $  
                Compiled for 64 bit mode.   
                Build: linux                
 
        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
 
        Run began: Fri May 29 21:16:37 2009
 
        Auto Mode
        File size set to 8388608 KB
        Command line used: iozone -a -s 8G
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         8388608       4  174685  213251   355843   365144     586    1284   12223  2280563    16758   218247   263474  358397   366568
         8388608       8  223099  240634   355358   365280    1168    2397   18998  2642433    18052   210898   233876  350771   365105
         8388608      16  226324  229919   365406   360961    2315    4390   26815  2810571     8341   218916   251165  361090   357974
         8388608      32  218344  255547   359488   365617    4532    7680   28945  3173752     9059   211130   247218  351760   366038
         8388608      64  214133  216581   360049   357493    8666   12576   35039  3067238    11112   215186   258125  352486   364836
         8388608     128  215983  263689   361029   355805   14172   22884   44144  3134393    17952   220013   262199  359932   358492
         8388608     256  219437  221301   361070   365407   24231   27027   55767  3249251    31228   219924   250765  355612   368312
         8388608     512  224317  249989   362780   359418   44359   44989   70847  3114559    75262   218529   256738  352011   359514
         8388608    1024  220018  245323   363428   357883   77843   71840  102576  2949858    89856   222584   246536  357558   358643
         8388608    2048  223986  257198   362738   364082  127198  100024  143284  2867324   140889   217681   248418  353124   364953
         8388608    4096  223963  240834   360078   360076  197791  121379  203363  2333949   211259   218647   259376  352413   358489
         8388608    8192  217639  235952   356011   356751  246708  143259  262451  2173888   264425   214914   234224  356112   367021
         8388608   16384  217533  231808   352947   367823  301731  161226  296074  2091368   297689   221477   243934  364667   365455
 
iozone test complete.
 
real    967m19.809s
user    1m28.250s
sys     54m3.239s

Out of the tests, only about the last 4 are adequate. I'd really like to see some better times from the smaller record sizes as well.