RAID, stride, and stripe-size, oh my!

digdogger · 11-17-2010, 06:30 PM

Hello. I've been Google searching for ways to improve the performance of our webhosting server, as disk I/O latency and throughput are currently huge bottlenecks in the performance of our servers. I've found:

http://storageadvisors.adaptec.com/2...t-stripe-size/
http://wiki.centos.org/HowTos/Disk_Optimization
http://osdir.com/ml/linux.file-syste.../msg00045.html
http://old.nabble.com/Calculating-st...d10979268.html
http://uclibc.org/~aldot/mkfs_stride.html

But reading through all this just leaves me with more questions than answers.

1. According to http://wiki.centos.org/HowTos/Disk_Optimization : "Sadly, the 'stripe-width=' extended-option has disappeared from man mkfs.ext3 as of CentOS 5.3" ... anybody have any idea why this is? Perhaps the "stripe-width" option is not valid for ext3, and it was a bug to include it in the first place, so that is why it was removed? It appears to be available again in mkfs.ext4 .

2. What is the effect of using tune2fs (or tune4fs) on an existing filesystem to change the values of stride and stripe-size ? Does it just effect files that are newly created or modified after the changes are made, so that existing files on the filesystem are not optimized?

3. How do you query the current values of stride and stripe-size on an existing filesystem? dumpe2fs -h doesn't appear to have the info, nor does dumpe4fs -h or tune4fs -l .

4. Does using LVM have any effect on the effectiveness of the stripe-width or stride options?

Thanks.

Electro · 11-19-2010, 08:49 PM

It seems you have set up a RAID-0 or a RAID-5. This is fine for database and file servers, but not the right one for web hosting. Web hosting requires latency, but not throughput. You should setup a RAID-1 or RAID-10. I have not used a RAID, but from experience 16 kilobyte is best for hard drives although do a test on your own. Setting the stripe and the ext spec stride is a trial and benching is only way to get the best performance.

The utility tune2fs only will make changes on newly creating files.

LVM2 or EVMS will not provide you any performance. It gives you higher capacity.

If you think you have a bottleneck for your server, it is best to look at the software first. If using MySQL, is the queries too complex? If so any server will be slow and it does not matter how fast the server is. Also if the server opens up too many connections, the bandwidth will choke and it will seem it is behind a phone modem. There is the kernel you can optimize by using the utility sysctl. Not using GUI will help a lot. If the scripts allow it, setup caching. Caching can help with repetitive tasks.

From reading some sites, "dumpe2fs -h" does in fact provide information. You should pipe it through grep.

digdogger · 11-26-2010, 05:49 PM

Thanks for your reply, Electro. We were previously using raid 6, but when hosting around 1000 ish sites you really start to notice the IO wait lag. Using the 'sar' command provided by the 'sysstat' RPM package, we're seeing 10 minute %iowait shooting up to 30% or 40% at times. So we're switching to raid 10. According to dbench and bonnie++, the throughput is higher and latency is much lower using raid 10 on our hardware (IBM x3650).

When we setup a new machine with raid 10, I took the time to do dbench tests with and without stride and stripe-width settings set. The system is hardware raid 10, 6 disks, 256 KB stripe size, and a block size of 4096 . According to http://uclibc.org/~aldot/mkfs_stride.html , we should be using stride=64 and stripe-width=192 . I ran dbench 10 times in a row and saved the output into text files (with and without stide/stripe-width set, for a total of 20 times). The exact command was: dbench 10 -t 60 -c /usr/local/share/client.txt . I then created a Perl script to find the minimum, average, and maximum of the throughput and latency of all of the 'execute' lines in each of the dbench runs. The results were:

Code:

RAID10 and noatime turned on:

                     Minimum              Average              Maximum
         Throughput: 100.26               174.163508474576     222.91     (MB/sec)
            Latency: 46.232               231.828379661017     1176.877   (ms)

Code:

RAID10 and noatime turned on and stride / stripe-width set:

                     Minimum              Average              Maximum
         Throughput: 77.13                158.045508474576     194.74     (MB/sec)
            Latency: 39.927               251.216361016949     1432.605   (ms)

So it appears to have made very little difference. In fact, it appears to have made things a bit worse. Which is why I wonder if stride / stripe-width does anything at all in ext3? I was asking about LVM as I was wondering if using LVM has any effect on the stride / stripe-width calculations, since LVM creates various data structures that will consume disk space, and perhaps create some sort of 'offset' that needs to be taken into account when doing the stride / stripe-width calculations.

And you're saying "From reading some sites, "dumpe2fs -h" does in fact provide information. You should pipe it through grep." Do you happen to have any links to those sites handy? I have run "dumpe2fs -h", "dumpe4fs -h", and "tune4fs -l" with "grep -i stride" and "grep -i stripe" and didn't get any results.