high NFS load--filesystems, tuning, etc

Mirlyn · 04-13-2009, 05:03 PM

I have a NFS server (dual Xenon X3220, 8gb ram, 3ware SATA controller in RAID5 on an intel board) serving a gigabit network of approx 30 I/O-intensive clients (email servers) and am seeing rising loads. The array is ext3, using NFS in Debian Etch and vanilla 2.6.27.

Currently, I have 32 instances set to start up. Loads are consistently in the 3-5 range. I am starting to see the delay at the client side mounts (hanging, crashing, etc).

NFS options used in fstab on the clients are: rw,tcp,bg,soft

On the NFS server:
# cat /proc/net/rpc/nfsd
rc 2 66748797 1172589670
fh 333 0 0 0 0
io 2375901293 1279693650
th 32 3404 676579.148 101531.952 17663.252 5604.840 5468.832 1360.520 469.652 79.680 26.320 23.864
ra 64 11100893 0 0 0 0 0 0 0 0 0 7618937
net 1239368221 3 1239321998 42485
rpc 1239347096 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 147 949441929 17250670 82983997 96346844 0 18719731 17463189 11296091 7592 0 0 13736344 1023 2935776 4058078 6223757 10020851 86147 25236 0 8597582
proc4 2 0 0
proc4ops 40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

A few questions:

Should I be using something other than what's included with 2.6 and the distro?

What other filesystems should I consider? I've read reiser is better for smaller, more numerous files. Is ext3 slowing me down?

I've played with the suggestions in the Optimizing NFS Performance docs but no real results have been seen.

Thanks!

jschiwal · 04-13-2009, 05:47 PM

This article compared the ext2, ext3, xfs, jfs and reiser file systems.
http://linuxgazette.net/102/piszcz.html

Would using UDP instead of TCP increase the throughput or would retransmissions at a higher level be actually slower. TCP is stateful so that could be the cause of the hangs you are seeing.

The nfsd manpage suggests starting 8 clients (nproc argument) as a starting point and checking traffic throughput with nfsstat.

Quote:

Originally Posted by nfsd manpage

nproc specify the number of NFS server threads. By default, just one thread is started. However, for optimum performance several threads should be used. The actual figure depends on the number of and the work load created by the NFS clients, but a useful starting point is 8 threads. Effects of modifying that number can be checked using the nfsstat(8) program.

My understanding is that ext3 and reiserfs are not speed demons but are more robust. XFS may be the best for serving large files. The iostat and nfsstat programs may help in determining whether the problem is with the filesystem used or with nfs or with network problems.