LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 12-19-2008, 12:31 PM   #1
td3201
Member
 
Registered: Jan 2002
Location: Omaha, NE US
Distribution: Red Hat/CentOS
Posts: 224

Rep: Reputation: 30
Question NFS performance help


This is repost from the networking forum where I got no responses. I think it was the wrong audience after reading the sticky posts, so here it is again, hopefully in front of the right audience.

I have a pretty busy NFS server. There are a LOT of small I/O requests that come across. It is a cluster built on RHEL5 where I distribute each NFS export as a service across the cluster so it is in essence, an active/active cluster. Each export is ext3. Here is one of the mount points being shared out:
/dev/mapper/vg_data01a-lv_data01a on /data01a type ext3 (rw,noatime,nodiratime,data=writeback,commit=30)

I believe these to be optimum settings for performance, someone correct me if I am wrong.

I am starting 256 nfsd processes on each node. Here are the client mount settings:
omadvnfs01-nfs-h:/data01h /data01h nfs defaults,rsize=32768,wsize=32768 0 0

Here are some tests that drew me to NFS performance:

WRITING TO ISCSI DISK
[root@omadvnfs01b data01h]# time dd if=/dev/zero of=foo bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.547895 seconds, 490 MB/s

real 0m0.578s
user 0m0.004s
sys 0m0.540s
READING FROM ISCSI DISK
[root@omadvnfs01b data01h]# time dd if=foo of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.142407 seconds, 1.9 GB/s

real 0m0.144s
user 0m0.008s
sys 0m0.135s


WRITING TO NFS EXPORT
[root@omadvdss01a data01h]# time dd if=/dev/zero of=foo bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 9.48628 seconds, 28.3 MB/s

real 0m14.394s
user 0m0.004s
sys 0m0.408s

READING FROM NFS EXPORT
[root@omadvdss01a data01h]# time dd if=foo of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 4.03699 seconds, 66.5 MB/s

real 0m4.039s
user 0m0.007s
sys 0m0.302s


You can see the huge difference from direct I/O to the disk vs. nfs. I would like to close this gap a bit somehow. Any ideas?
 
Old 12-24-2008, 12:51 PM   #2
alexhwest
Member
 
Registered: Dec 2008
Location: Cleveland, OH
Distribution: Ubuntu
Posts: 30

Rep: Reputation: 15
Those speeds would indicate a speed of read: 532Mbits/sec write: 226Mbits/sec. I would assume this is a Gigabit link? 532Mbits/sec over a gigabit link would not be an unreasonable max throughput. What is the problem here?
 
Old 12-24-2008, 04:17 PM   #3
td3201
Member
 
Registered: Jan 2002
Location: Omaha, NE US
Distribution: Red Hat/CentOS
Posts: 224

Original Poster
Rep: Reputation: 30
I was looking at the difference between iscsi and nfs performance. This is a backup application so it mostly writes that I am concerned with. Given I saw over 1 Gb/s speeds on iscsi tells me I was hitting some cache (missed this before I originally posted.

However, I don't think 226 Mb/s is good for writing to an NFS volume. I don't think 532 Mb/s reading is good either. I would expect somewhere near 700 for reading and 500 for writing.
 
Old 12-26-2008, 11:54 AM   #4
alexhwest
Member
 
Registered: Dec 2008
Location: Cleveland, OH
Distribution: Ubuntu
Posts: 30

Rep: Reputation: 15
I would try to narrow down the issue. Have you tested the throughput you are able to achieve over your link? Using perhaps iperf or similar? Could even be the switch that is the bottleneck.
Alex
 
Old 12-26-2008, 03:29 PM   #5
td3201
Member
 
Registered: Jan 2002
Location: Omaha, NE US
Distribution: Red Hat/CentOS
Posts: 224

Original Poster
Rep: Reputation: 30
Ahhh, layer 2? Brilliant! Shame on me for skipping ahead.

Here's the most I could push through under default iperf settings:
[root@omadvdss01b ~]# iperf -c omadvnfs01a
------------------------------------------------------------
Client connecting to omadvnfs01a, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.199.1.101 port 35086 connected with 10.199.1.110 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 790 MBytes 662 Mbits/sec

The system is under light load right now when I was running this test.

Not sure what to think. I don't think that's that good.
 
Old 12-29-2008, 04:03 PM   #6
alexhwest
Member
 
Registered: Dec 2008
Location: Cleveland, OH
Distribution: Ubuntu
Posts: 30

Rep: Reputation: 15
I would be very surprised to see anything much over 600Mbits/sec. Also, try iperf -u on both ends to get udp mode, which should give you closer to the theoretical max throughput.
 
Old 12-29-2008, 04:07 PM   #7
alexhwest
Member
 
Registered: Dec 2008
Location: Cleveland, OH
Distribution: Ubuntu
Posts: 30

Rep: Reputation: 15
Also, you should set rsize=32768,wsize=32768 to something closer to 16k if that is the block sizes you are going to test. Whether this will be similar to your real nfs traffic or not depends on whether you are sending a lot of small requests or large requests. Tune as appropriate. Honestly though, I am very surprised that you are getting speeds as good as you are. If anything, a little rsize and wsize may go a long way.

Alex
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
NFS Performance -- any ideas? td3201 Linux - Networking 1 12-26-2008 12:10 PM
NFS Performance markham82 Linux - General 4 10-06-2004 01:39 PM
Cross Platform NFS performance marist89 Linux - Networking 0 03-31-2004 10:55 AM
NFS Performance Tuning joeclark Linux - Networking 2 03-16-2004 09:08 PM
Can you rate your nfs performance with this command for me? BrianK Linux - Networking 4 02-03-2004 02:24 PM


All times are GMT -5. The time now is 09:31 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration