NFS performance help
 

This is repost from the networking forum where I got no responses. I think it was the wrong audience after reading the sticky posts, so here it is again, hopefully in front of the right audience.

I have a pretty busy NFS server. There are a LOT of small I/O requests that come across. It is a cluster built on RHEL5 where I distribute each NFS export as a service across the cluster so it is in essence, an active/active cluster. Each export is ext3. Here is one of the mount points being shared out:
/dev/mapper/vg_data01a-lv_data01a on /data01a type ext3 (rw,noatime,nodiratime,data=writeback,commit=30)

I believe these to be optimum settings for performance, someone correct me if I am wrong.

I am starting 256 nfsd processes on each node. Here are the client mount settings:
omadvnfs01-nfs-h:/data01h /data01h nfs defaults,rsize=32768,wsize=32768 0 0

Here are some tests that drew me to NFS performance:

WRITING TO ISCSI DISK
[root@omadvnfs01b data01h]# time dd if=/dev/zero of=foo bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.547895 seconds, 490 MB/s

real 0m0.578s
user 0m0.004s
sys 0m0.540s
READING FROM ISCSI DISK
[root@omadvnfs01b data01h]# time dd if=foo of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.142407 seconds, 1.9 GB/s

real 0m0.144s
user 0m0.008s
sys 0m0.135s


WRITING TO NFS EXPORT
[root@omadvdss01a data01h]# time dd if=/dev/zero of=foo bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 9.48628 seconds, 28.3 MB/s

real 0m14.394s
user 0m0.004s
sys 0m0.408s

READING FROM NFS EXPORT
[root@omadvdss01a data01h]# time dd if=foo of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 4.03699 seconds, 66.5 MB/s

real 0m4.039s
user 0m0.007s
sys 0m0.302s


You can see the huge difference from direct I/O to the disk vs. nfs. I would like to close this gap a bit somehow. Any ideas?

Those speeds would indicate a speed of read: 532Mbits/sec write: 226Mbits/sec. I would assume this is a Gigabit link? 532Mbits/sec over a gigabit link would not be an unreasonable max throughput. What is the problem here?

I was looking at the difference between iscsi and nfs performance. This is a backup application so it mostly writes that I am concerned with. Given I saw over 1 Gb/s speeds on iscsi tells me I was hitting some cache (missed this before I originally posted.

However, I don't think 226 Mb/s is good for writing to an NFS volume. I don't think 532 Mb/s reading is good either. I would expect somewhere near 700 for reading and 500 for writing.

I would try to narrow down the issue. Have you tested the throughput you are able to achieve over your link? Using perhaps iperf or similar? Could even be the switch that is the bottleneck.
Alex

Ahhh, layer 2? Brilliant! Shame on me for skipping ahead.

Here's the most I could push through under default iperf settings:
[root@omadvdss01b ~]# iperf -c omadvnfs01a
------------------------------------------------------------
Client connecting to omadvnfs01a, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.199.1.101 port 35086 connected with 10.199.1.110 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 790 MBytes 662 Mbits/sec

The system is under light load right now when I was running this test.

Not sure what to think. I don't think that's that good.

I would be very surprised to see anything much over 600Mbits/sec. Also, try iperf -u on both ends to get udp mode, which should give you closer to the theoretical max throughput.

Also, you should set rsize=32768,wsize=32768 to something closer to 16k if that is the block sizes you are going to test. Whether this will be similar to your real nfs traffic or not depends on whether you are sending a lot of small requests or large requests. Tune as appropriate. Honestly though, I am very surprised that you are getting speeds as good as you are. If anything, a little rsize and wsize may go a long way.

Alex