LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-26-2015, 01:50 AM   #1
umesh.pawar
LQ Newbie
 
Registered: Feb 2015
Location: India
Posts: 6

Rep: Reputation: Disabled
NFS response is very slow


Hi Everyone,

We are using Dell Power Edge R510 server as a central storage in Linux network. I have configured RAID6 on 10 near-line SAS hard disks. Using NFS i have exported one directory and mounted it on other machines.

Recently we are facing one problem, when anyone read/write big file (2-3GB)from NFS drive,others feel very slow access to that drive. Sometime we can't open simple text file till complete read/write process.

I have remounted NFS drive but still problem as it is.

Can anyone help me to solve this problem.

Last edited by umesh.pawar; 02-26-2015 at 02:16 AM.
 
Old 02-26-2015, 10:06 AM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,334

Rep: Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891
Quote:
Originally Posted by umesh.pawar View Post
Hi Everyone,
We are using Dell Power Edge R510 server as a central storage in Linux network. I have configured RAID6 on 10 near-line SAS hard disks. Using NFS i have exported one directory and mounted it on other machines.

Recently we are facing one problem, when anyone read/write big file (2-3GB)from NFS drive,others feel very slow access to that drive. Sometime we can't open simple text file till complete read/write process. I have remounted NFS drive but still problem as it is.
You don't provide many details, such as what version of NFS you're using, your network configuration, how many users, what kind of systems the users are using, and what version/distro of Linux. There are several NFS tuning guides you can try first:
http://www.tldp.org/HOWTO/NFS-HOWTO/performance.html
http://www.slashroot.in/how-do-linux...d-optimization
 
Old 02-27-2015, 01:01 AM   #3
umesh.pawar
LQ Newbie
 
Registered: Feb 2015
Location: India
Posts: 6

Original Poster
Rep: Reputation: Disabled
Thanks for taking interest in this issue.

I will go though link which you sent in your previos quote.

Please find the details of NFS
NFS version:
nfs-utils-1.0.9-44.el5
system-config-nfs-1.3.23-1.el5
nfs4-acl-tools-0.3.3-1.el5
nfs-utils-lib-1.0.8-7.6.el5
Network structure:
It is NIS/NFS network.We have added central storage (NFS) for easy to access our projects files from any machines where nfs drive mounted. We have 15 compute nodes and each node have 12 cores processors.our 30 users running his simulation on that compute nodes and stored data on nfs.We have connected all compute nodes and nfs through layer 3 giga bit switch

Operating system of server where nfs configured: RHEL 5.5 64 bit


Umesh
 
Old 02-27-2015, 10:17 AM   #4
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,334

Rep: Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891
Quote:
Originally Posted by umesh.pawar View Post
Thanks for taking interest in this issue.
I will go though link which you sent in your previos quote.

Please find the details of NFS
NFS version:
nfs-utils-1.0.9-44.el5
system-config-nfs-1.3.23-1.el5
nfs4-acl-tools-0.3.3-1.el5
nfs-utils-lib-1.0.8-7.6.el5
Network structure:
It is NIS/NFS network.We have added central storage (NFS) for easy to access our projects files from any machines where nfs drive mounted. We have 15 compute nodes and each node have 12 cores processors.our 30 users running his simulation on that compute nodes and stored data on nfs.We have connected all compute nodes and nfs through layer 3 giga bit switch

Operating system of server where nfs configured: RHEL 5.5 64 bit
First off...RHEL 5.5 is VERY OLD, and it's a commercial, PAY FOR distro. Have you contacted Red Hat about this?? The latest version is 7.x, and contains many improvements over what you have now. And is your system patched and current to RHEL's repositories? Are you PAYING FOR RHEL???

And you haven't mentioned what the users workstations are running, which makes a difference too. Going through basic NFS performance tuning should be done first, though.
 
Old 02-28-2015, 06:41 AM   #5
umesh.pawar
LQ Newbie
 
Registered: Feb 2015
Location: India
Posts: 6

Original Poster
Rep: Reputation: Disabled
We have not renewed subscription of RHEL, hence we can't take a support from red hat team.

All are branded workstations of HP & Dell.
Model numbers are:
HP Proliant DL 160
Dell PowerEdge R610

but please let me know is it os problem?

I have checked network performance on web console of manageable switch ,it shows normal network load.

I have checked disks iops using "dstate" utility which comes with operating system,I observed that when we read/write above 2-3 GB file on nfs, cpu "wait" time goes up.

Please find the report below:
----total-cpu-usage---- -----time----- -dsk/total----dsk/sda-----dsk/sdb--
usr sys idl wai hiq siq| date/time | read writ: read writ: read writ
0 0 100 0 0 0|28-02 16:41:20| 466k 95k: 33k 6656B: 200k 41k
0 0 100 0 0 0|28-02 16:42:20|4779B 30k:2389B 15k: 0 0
0 0 100 0 0 0|28-02 16:43:20|9284B 13k:1297B 6690B:3345B 0
0 1 94 4 0 0|28-02 16:44:21| 161M 21M:1502B 3959B: 81M 10M
0 1 88 10 0 0|28-02 16:45:21| 191M 42M:3618B 12k: 95M 21M
0 0 91 8 0 0|28-02 16:46:21| 43M 44M:6895B 4915B: 21M 22M
0 0 92 8 0 0|28-02 16:47:21| 0 49M: 0 1092B: 0 24M
0 0 88 12 0 0|28-02 16:48:22|4779B 47M:2389B 4847B: 0 24M
0 0 92 8 0 0|28-02 16:48:26| 0 47M: 0 0 : 0 23M

Disk type: near line SAS with 7200rpm
Qty: 10
Raid type: Raid 6

Regards,
Umesh
 
Old 02-28-2015, 10:15 AM   #6
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,334

Rep: Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891
Quote:
Originally Posted by umesh.pawar View Post
We have not renewed subscription of RHEL, hence we can't take a support from red hat team.
Then stop where you are and RENEW IT. There is no point in spending such a large amount of money on hardware and software (as you have described), to not be supported. And AGAIN, I will point out that RHEL 5.5 is OLD, and may only be receiving security updates (NOT patches/software updates) at this point, and you'll only get THOSE if you pay. Load CentOS if you don't plan on paying for RHEL.
Quote:
All are branded workstations of HP & Dell.
Model numbers are:
HP Proliant DL 160
Dell PowerEdge R610

but please let me know is it os problem?
Again, no idea, since you haven't said anything about how NFS is configured, or if you tried ANY of the tips in the NFS tuning guides. And again, you still haven't said WHAT OPERATING SYSTEM THE CLIENTS ARE RUNNING. The models and hardware make no difference.
Quote:
I have checked network performance on web console of manageable switch ,it shows normal network load. I have checked disks iops using "dstate" utility which comes with operating system,I observed that when we read/write above 2-3 GB file on nfs, cpu "wait" time goes up.

Disk type: near line SAS with 7200rpm
And AGAIN, have you read through the performance guides you were given before???? There are MANY things to tune and monitor for NFS in those guides...did you bother to try ANY of them yet????
 
Old 02-28-2015, 12:58 PM   #7
JeremyBoden
Senior Member
 
Registered: Nov 2011
Distribution: Debian
Posts: 1,025

Rep: Reputation: 203Reputation: 203Reputation: 203
Quote:
Originally Posted by umesh.pawar View Post
Hi Everyone,

We are using Dell Power Edge R510 server as a central storage in Linux network. I have configured RAID6 on 10 near-line SAS hard disks. Using NFS i have exported one directory and mounted it on other machines.

Recently we are facing one problem, when anyone read/write big file (2-3GB)from NFS drive,others feel very slow access to that drive. Sometime we can't open simple text file till complete read/write process.
From @TBone http://www.tldp.org/HOWTO/NFS-HOWTO/performance.html
Quote:
7.5. When I transfer really big files, NFS takes over all the CPU cycles on the server and it screeches to a halt.

This is a problem with the fsync() function in 2.2 kernels that causes all sync-to-disk requests to be cumulative, resulting in a write time that is quadratic in the file size. If you can, upgrading to a 2.4 kernel should solve the problem. Also, exporting with the no_wdelay option forces the program to use o_sync() instead, which may prove faster.
Actually, the quadratic nature of the time taken illustrates almost exactly my performance stats between a modern Debian kernel when writing to a cheapo backup device, using rsync over nfs.
No measurements, but it takes roughly 100 times as long to transfer a 10GB file as a 1GB file.

I blame my cheapo backup device (an ebay purchase).
 
Old 03-02-2015, 06:03 AM   #8
umesh.pawar
LQ Newbie
 
Registered: Feb 2015
Location: India
Posts: 6

Original Poster
Rep: Reputation: Disabled
Thanks JeremyBoden for your interest.

Actually updated kernel version is available on that server where we created NFS drive:
Kernel version: 2.6.18-194.el5xen

Last edited by umesh.pawar; 03-02-2015 at 06:08 AM.
 
Old 03-02-2015, 10:18 AM   #9
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,334

Rep: Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891Reputation: 3891
Quote:
Originally Posted by umesh.pawar View Post
Thanks JeremyBoden for your interest.

Actually updated kernel version is available on that server where we created NFS drive: Kernel version: 2.6.18-194.el5xen
Ok...and you STILL have not answered any questions about the clients, or acknowledged the NFS tuning instructions. Unless you actually TRY something, performance isn't going to increase. You have said nothing about what operating system the clients are running, what NFS client they're running, or said if you did ANYTHING to the NFS server, after reading the NFS performance tuning guides.

Read the guides, and follow them. If you're still having problems, then you need to come back and PROVIDE DETAILS and ANSWER QUESTIONS, or there is nothing anyone, anywhere can do to help you.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
NFS Behavior in different distros - NFS becomes slow in Ubuntu client dwsideri Linux - Networking 2 10-11-2012 09:55 AM
Nfs client no response from server mikusita Linux - Networking 4 07-08-2011 02:46 PM
X server slow response when connected to slow Wifi bluesmanu Linux - Software 0 04-09-2008 07:56 AM
Slow response robeadam Linux - General 6 09-13-2007 10:31 AM
Slow response as ..... umok Linux - Newbie 4 07-11-2003 01:54 PM


All times are GMT -5. The time now is 11:15 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration