NFS response is very slow
 

Hi Everyone,

We are using Dell Power Edge R510 server as a central storage in Linux network. I have configured RAID6 on 10 near-line SAS hard disks. Using NFS i have exported one directory and mounted it on other machines.

Recently we are facing one problem, when anyone read/write big file (2-3GB)from NFS drive,others feel very slow access to that drive. Sometime we can't open simple text file till complete read/write process.

I have remounted NFS drive but still problem as it is.

Can anyone help me to solve this problem.

You don't provide many details, such as what version of NFS you're using, your network configuration, how many users, what kind of systems the users are using, and what version/distro of Linux. There are several NFS tuning guides you can try first:
http://www.tldp.org/HOWTO/NFS-HOWTO/performance.html
http://www.slashroot.in/how-do-linux...d-optimization

Thanks for taking interest in this issue.

I will go though link which you sent in your previos quote.

Please find the details of NFS
NFS version:
nfs-utils-1.0.9-44.el5
system-config-nfs-1.3.23-1.el5
nfs4-acl-tools-0.3.3-1.el5
nfs-utils-lib-1.0.8-7.6.el5
Network structure:
It is NIS/NFS network.We have added central storage (NFS) for easy to access our projects files from any machines where nfs drive mounted. We have 15 compute nodes and each node have 12 cores processors.our 30 users running his simulation on that compute nodes and stored data on nfs.We have connected all compute nodes and nfs through layer 3 giga bit switch

Operating system of server where nfs configured: RHEL 5.5 64 bit


Umesh

First off...RHEL 5.5 is VERY OLD, and it's a commercial, PAY FOR distro. Have you contacted Red Hat about this?? The latest version is 7.x, and contains many improvements over what you have now. And is your system patched and current to RHEL's repositories? Are you PAYING FOR RHEL???

And you haven't mentioned what the users workstations are running, which makes a difference too. Going through basic NFS performance tuning should be done first, though.

We have not renewed subscription of RHEL, hence we can't take a support from red hat team.

All are branded workstations of HP & Dell.
Model numbers are:
HP Proliant DL 160
Dell PowerEdge R610

but please let me know is it os problem?

I have checked network performance on web console of manageable switch ,it shows normal network load.

I have checked disks iops using "dstate" utility which comes with operating system,I observed that when we read/write above 2-3 GB file on nfs, cpu "wait" time goes up.

Please find the report below:
----total-cpu-usage---- -----time----- -dsk/total----dsk/sda-----dsk/sdb--
usr sys idl wai hiq siq| date/time | read writ: read writ: read writ
0 0 100 0 0 0|28-02 16:41:20| 466k 95k: 33k 6656B: 200k 41k
0 0 100 0 0 0|28-02 16:42:20|4779B 30k:2389B 15k: 0 0
0 0 100 0 0 0|28-02 16:43:20|9284B 13k:1297B 6690B:3345B 0
0 1 94 4 0 0|28-02 16:44:21| 161M 21M:1502B 3959B: 81M 10M
0 1 88 10 0 0|28-02 16:45:21| 191M 42M:3618B 12k: 95M 21M
0 0 91 8 0 0|28-02 16:46:21| 43M 44M:6895B 4915B: 21M 22M
0 0 92 8 0 0|28-02 16:47:21| 0 49M: 0 1092B: 0 24M
0 0 88 12 0 0|28-02 16:48:22|4779B 47M:2389B 4847B: 0 24M
0 0 92 8 0 0|28-02 16:48:26| 0 47M: 0 0 : 0 23M

Disk type: near line SAS with 7200rpm
Qty: 10
Raid type: Raid 6

Regards,
Umesh

Then stop where you are and RENEW IT. There is no point in spending such a large amount of money on hardware and software (as you have described), to not be supported. And AGAIN, I will point out that RHEL 5.5 is OLD, and may only be receiving security updates (NOT patches/software updates) at this point, and you'll only get THOSE if you pay. Load CentOS if you don't plan on paying for RHEL.
Again, no idea, since you haven't said anything about how NFS is configured, or if you tried ANY of the tips in the NFS tuning guides. And again, you still haven't said WHAT OPERATING SYSTEM THE CLIENTS ARE RUNNING. The models and hardware make no difference.
And AGAIN, have you read through the performance guides you were given before???? There are MANY things to tune and monitor for NFS in those guides...did you bother to try ANY of them yet????

From @TBone http://www.tldp.org/HOWTO/NFS-HOWTO/performance.html
Actually, the quadratic nature of the time taken illustrates almost exactly my performance stats between a modern Debian kernel when writing to a cheapo backup device, using rsync over nfs.
No measurements, but it takes roughly 100 times as long to transfer a 10GB file as a 1GB file.

I blame my cheapo backup device (an ebay purchase). :newbie:

Thanks JeremyBoden for your interest.

Actually updated kernel version is available on that server where we created NFS drive:
Kernel version: 2.6.18-194.el5xen

Ok...and you STILL have not answered any questions about the clients, or acknowledged the NFS tuning instructions. Unless you actually TRY something, performance isn't going to increase. You have said nothing about what operating system the clients are running, what NFS client they're running, or said if you did ANYTHING to the NFS server, after reading the NFS performance tuning guides.

Read the guides, and follow them. If you're still having problems, then you need to come back and PROVIDE DETAILS and ANSWER QUESTIONS, or there is nothing anyone, anywhere can do to help you.