Problem with NFS mount during high load

holst · 09-15-2007, 09:12 AM

I get a kernel message and a failed rsync when I copy files from nfs share A to nfs share B using rsync. The message is:

nfs: too small RPC reply size (0 bytes)

When I change to using UDP I get a server timeout instead.

Any ideas how I can fix this?

Thanks for your help,
Henrik Holst, Sweden.

soroccoheaven · 09-16-2007, 04:15 AM

use the nfsstat command to look at nfs transactions, client and server statistics, network statistics. The "-o net" option will show you the number of dropped packets in relation to the total number of transactions.

check rsize and wsize, should be less than MTU.

holst · 09-16-2007, 06:30 AM

Quote:

Originally Posted by soroccoheaven

use the nfsstat command to look at nfs transactions, client and server statistics, network statistics. The "-o net" option will show you the number of dropped packets in relation to the total number of transactions.

check rsize and wsize, should be less than MTU.

You mean to mount with rsize=1400 if my MTU is 1500? Isn't that something that will give very poor performance? I tried it just now and the performance went down from 25-30 mb/s to 15-20 mb/s.

I will see in a few hours if it now is stable or not. I would actually like to have both performance and stability (of course!).

/Henrik Holst

holst · 09-16-2007, 08:11 AM

Quote:

Originally Posted by holst

You mean to mount with rsize=1400 if my MTU is 1500? Isn't that something that will give very poor performance? I tried it just now and the performance went down from 25-30 mb/s to 15-20 mb/s.

I will see in a few hours if it now is stable or not. I would actually like to have both performance and stability (of course!).

/Henrik Holst

The errors are continuing to show up. Rsync stops and dmesg on the client which is copying from NFS server A to NFS server B shows:

Code:

nfs: server fs-6000.domain.org not responding, timed out
nfs: server fs-6000.domain.org not responding, timed out

Nothing shows up in the system log on the NFS servers.

When I look at the two NFS servers I see that they are not 100% loaded all the time (they are idle almost 100% a few seconds now and then.)

Also, the network utilization graph is not so funny

Code:

################################################################################
RX    MiB
    23.41 ...***................******........................**..****
    19.51 **.*****.........***.************..........**..***..********
    15.61 ********.........***.************..........*****************
    11.71 ********.........*****************.........*****************
     7.80 ********.........*****************.........*****************
     3.90 ********:*:*:.::******************:***::*:****************** [-0.46%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s
TX    MiB
    31.37 .*....*..........*.*............................*...........
    26.14 .**...**...*.....***................**..*...*.*****.........
    20.91 ***...**...*.....****.......***.**.***..*...*.*****.........
    15.68 ***...**.*.*...*.****.......**********..*..********.........
    10.46 ***..*******..********.....**************..********.........
     5.23 *************:***************************::***************** [-0.46%]
          1   5   10   15   20   25   30   35   40   45   50   55   60 s
################################################################################

soroccoheaven · 09-16-2007, 08:15 AM

pls. see this ..might be helpful..NFS Performance