slow nfs "uploads" on current

rogan · 02-03-2018, 06:11 AM

I think I have tried every trick in the book, but I just
can't get the speed to within more than a tenth of what I get
when uploading to a 14.2 server.
Ruled out:
hardware, kernel (tried 4.4.88 on current), normal mounting options like varying values for rsize and wsize (I'm NOT mounting async on either 14.2 or current).

Clarification:
Large file copy has about the same performance.
However copying e.g. a kernel source tree ~60000 files ~700Mb
takes 13 min from a current system to a 14.2 system
and more than 2 hours the other way which makes a nfs file server
built on current almost unusable.

Does anybody else see this regression?
Does anybody have a clue what might have happened?

Thanks in advance

DarkVision · 02-04-2018, 01:16 AM

Quote:

Originally Posted by rogan

Does anybody else see this regression?
Does anybody have a clue what might have happened?

I have no (more) 14.2 NFS-server but i can second that uploading 700Mb/kernel-tree to a -current NFS-server takes ages. Tested from a -current VirtualBox to my -current Desktop. About 6-8h here. The issue seem to be those many single files, not the 700Mb itself. Transfer a 1.1Gb file from Desktop to vbox takes about 10seconds... from vbox to desktop also 10-15seconds.

P.S. I'm using 4.9.79 as desktop kernel and 4.14.16 inside vbox.

rogan · 02-04-2018, 05:43 AM

Quote:

Originally Posted by DarkVision

uploading 700Mb/kernel-tree to a -current NFS-server takes ages. Tested from a -current VirtualBox to my -current Desktop. The issue seem to be those many single files

I get the feeling something has introduced enormous overhead for file
transfers. I can literally count the files one by one in mc when
copying even very small ones.
I suspected retpoline fixes but those are supposed to be kernel only
right? Anyway I tried different kernels and a number of different
machines on current, but no luck so far.

kjhambrick · 02-04-2018, 06:26 AM

Quote:

Originally Posted by rogan

I get the feeling something has introduced enormous overhead for file
transfers. I can literally count the files one by one in mc when
copying even very small ones.
I suspected retpoline fixes but those are supposed to be kernel only
right? Anyway I tried different kernels and a number of different
machines on current, but no luck so far.

rogan --

I wonder if you're not onto something there ?

My -current box is sorely out-of-date ...

Do you see the same speed regression when you copy the same set of files via rsync or scp or ftp ?

P.S. Maybe something useful here: https://wiki.archlinux.org/index.php...ormance_issues ?

-- kjh

keefaz · 02-04-2018, 06:54 AM

No errors, warnings... in logs? (/var/log/messages, /var/log/syslog, /var/log/nfsd/*)
I don't run current, but it seems there is new nfs configuration file in /etc/default (set default options), maybe worth a look?

Arch wiki nfs document looks informative

rogan · 02-04-2018, 08:50 AM

Quote:

Originally Posted by kjhambrick

Do you see the same speed regression when you copy the same set of files via rsync or scp or ftp
-- kjh

I cannot say for sure. I'm going to test that asap.
keefaz: Not a single warning anywhere in any of the logs I checked.
That would have been too easy i guess ... Slacking is real hard sometimes

rogan · 02-04-2018, 10:11 AM

uploading via scp is fast as hell
it took maybe 30 seconds for the whole linux-4.15.1 file tree

sftp _and_ ftp is about 100 times faster than current nfs on small files.
I think about as fast as my raid1 spinning hard drives can deliver

kjhambrick · 02-06-2018, 06:25 AM

Quote:

Originally Posted by rogan

uploading via scp is fast as hell
it took maybe 30 seconds for the whole linux-4.15.1 file tree

sftp _and_ ftp is about 100 times faster than current nfs on small files.
I think about as fast as my raid1 spinning hard drives can deliver

rogan --

I set aside Slackware64 14.2 and installed the latest Slackware64 current on an older Laptop on Sunday Morning.

I copied over my configs but I've not had time to set up nfs for testing yet.

Kinda like you, I found that rsync-over-ssh simply screams when I copied my local Slackware Current and SBo repos from my Slackware64 14.2 Laptop to the new Slackware64 current Laptop.

Before I mess around with nfs testing, did you figure out any issues / workarounds with your system or nfs configs ?

Thanks,

-- kjh

p.s. I thought I posted this earlier but it's not here ... must have pressed [Back] button while previewing. If you see a dupe, sorry ... I'll delete it ...

rogan · 02-06-2018, 10:09 AM

I have not found anything in particular. The tests I did suggested that no matter the
mounting options I could not avoid this strange overhead/file when copying to the server.
All tests were done on the ext4 file system mounted defaults,nodev,nosuid (as usual).
Since I am in the process of replacing all file systems with btrfs (other reason)
I'm going to do some more testing on exported btrfs volumes.

rogan · 02-06-2018, 01:07 PM

The sync option which have been the default since nfs-utils 1.0.0 (14.2 have 1.3.3)
seems to have a tremendous effect on current. If i export from the server (rw,async) I can
copy a kernel file tree to it in less than 3 minutes (without it in no less than 2 hours).
Somehow, it seems, sync have increased it's cost by 40 times at least.

kjhambrick · 02-06-2018, 04:48 PM

rogan --

Yes, that's what I see too.

The following seems to give the best throughput for me when copying the 4.4.115 Kernel Tree from my Slackware64 14.2 to Slackware64 current:

Code:

# grep -vH '^#' /etc/exports

/etc/exports:  /opt/export   192.168.0.0/24(rw,async,subtree_check,no_root_squash)

But can one live with the async option ?

My testing log is below ( ! note the rsync time compared to everything else ! )

-- kjh

Code:

# make a pair of mount points for testing.
#
# mkdir -p /mnt/nfs-sam /mnt/smb-sam
#
# how much data are we playing with ( from /usr/src/ on the 14.2 client ) ?
#
# du -sm linux-4.4.115.kjh
# 723    linux-4.4.115.kjh
#
# removed sam:/opt/export/linux-4.4.115.kjh/
#
# ---------------------------------------------------------
# rsync 
# ---------------------------------------------------------
#
# time rsync -e 'ssh -p 22 -lroot' -aH linux-4.4.115.kjh sam:/opt/export
 
real    0m53.357s
user    0m12.766s
sys     0m3.198s
#
# ---------------------------------------------------------
# scp 
# ---------------------------------------------------------
#
# removed sam:/opt/export/linux-4.4.115.kjh/
#
# time scp -pqr -P22 linux-4.4.115.kjh sam:/opt/export/

real    3m7.011s
user    0m19.755s
sys     0m26.682s
#
# ---------------------------------------------------------
# copy to cifs mounted share
# ---------------------------------------------------------
#
# removed sam:/opt/export/linux-4.4.115.kjh/
#
# mount.cifs sam:/exports /mnt/smb-sam -o 'rw,credentials=/root/.smbcred',ip=192.168.0.7'
#
# time cp -R linux-4.4.115.kjh /mnt/smb-sam/

real    3m35.261s
user    0m1.996s
sys     0m23.648s
#
# ---------------------------------------------------------
# copy to nfs mounted share -- no special options
# ---------------------------------------------------------
#
# removed sam:/opt/export/linux-4.4.115.kjh/
# umount /mnt/smb-sam
# mount.nfs sam:/opt/export /mnt/nfs-sam -o 'rw'
#
# time cp -R linux-4.4.115.kjh /mnt/nfs-sam/
# 
# date
# Tue Feb  6 14:24:05 CST 2018
# Tue Feb  6 14:36:06 CST 2018
# Tue Feb  6 14:50:27 CST 2018
#
# gave up after 40 minutes and only 111 MB of 723 MB transferred.

real    40m6.012s
user    0m1.299s
sys     0m12.774s
#
# ---------------------------------------------------------
# copy to nfs mounted share -- force version 4 on the Client
# ---------------------------------------------------------
#
# removed sam:/opt/export/linux-4.4.115.kjh/
# umount /mnt/nfs-sam
# mount.nfs sam:/opt/export /mnt/nfs-sam -o 'rw,vers=4'
#
# time cp -R linux-4.4.115.kjh /mnt/nfs-sam/
#
# date
# Tue Feb  6 15:11:01 CST 2018
# Tue Feb  6 15:17:21 CST 2018
# Tue Feb  6 15:22:52 CST 2018
#
# gave up after 13 minutes and only 79 MB of 723 MB transferred.

real    13m7.323s
user    0m0.516s
sys     0m5.273s
#
# ---------------------------------------------------------
# copy to nfs mounted share -- add no_wdelay to server
# ---------------------------------------------------------
#
# try this( man 5 nfs ):
#
# ssh -p22 sam grep -v '^#' /etc/exports

  /etc/exports:  /opt/export   192.168.0.0/24(rw,no_wdelay,subtree_check,no_root_squash)
#
# umount /mnt/nfs-sam
# mount.nfs sam:/opt/export /mnt/nfs-sam -o 'rw,vers=4'
#
# /etc/rc.d/rc.nfs restart
# time cp -R linux-4.4.115.kjh /mnt/nfs-sam/
#
# killed the copy after ~10 minutes ( only 58 MB of 723 MB copied )
# date
# Tue Feb  6 16:15:53 CST 2018
# Tue Feb  6 16:19:57 CST 2018
# Tue Feb  6 16:26:02 CST 2018

real    10m24.712s
user    0m0.361s
sys     0m4.066s
#
# ---------------------------------------------------------
# the following works if you can live with the async option
# ---------------------------------------------------------
#
# removed sam:/opt/export/linux-4.4.115.kjh/
# modified sam:/etc/exports:
# ssh -p22 sam grep -v '^#' /etc/exports

  /opt/export   192.168.0.6/32(rw,async,no_subtree_check,no_root_squash)
#
# /etc/rc.d/rc.nfs restart
#
# umount /mnt/nfs-sam
# mount.nfs sam:/opt/export /mnt/nfs-sam -o 'rw,vers=4,async'
# time cp -R linux-4.4.115.kjh /mnt/nfs-sam/
#
real    4m16.140s
user    0m2.058s
sys     0m25.714s
#
# ssh -p 22 sam du -sm /opt/export/linux-4.4.115.kjh
# 722     linux-4.4.115.kjh
#
# final answer ...
#
# ssh -p22 sam grep -v '^#' /etc/exports

  /etc/exports:  /opt/export   192.168.0.0/24(rw,async,subtree_check,no_root_squash)
#
# /etc/rc.d/rc.nfs restart
#
# time cp -R linux-4.4.115.kjh /mnt/nfs-sam/
#
# date
# Tue Feb  6 16:03:20 CST 2018

real    4m42.483s
user    0m2.005s
sys     0m26.672s

keefaz · 02-06-2018, 07:21 PM

Interesting data. Kernel source directory contains a large number of small files, it would have been cool to test copying just one large file as well.
But thanks anyway (I am surprised with the speed difference of rsync vs scp)

rogan · 02-07-2018, 01:09 AM

Preliminary tests seem to suggest that the file system
on the exported volume matters, but not enough to be a 'solution'.

I'm going to run some more tests to see which file
system has the heaviest penalty for the sync op.

Maybe we can learn something from it...

kjhambrick: If the server goes goes down during a write
or if a client A expects to find a file written by a client B
when it thinks the write is done, bad things happen...
I'd think sync is necessary in all 'serious' circumstances.

keefaz: rsync uses all kinds of trickery to speed up copying
nothing I have seen even comes close. It's a brilliant piece of software.
The sync cost is per file, and since the bandwidth is still there,
there's no particular slow down for really large single files.
However imagine using a 'modern' web browser on a nfs mounted user home,
only able to write one file/second, probably not a nice experience.

kjhambrick · 02-07-2018, 08:41 AM

Quote:

Originally Posted by keefaz

Interesting data. Kernel source directory contains a large number of small files, it would have been cool to test copying just one large file as well.
But thanks anyway (I am surprised with the speed difference of rsync vs scp)

keefaz --

See below.

More tests coming after I build multilib versions gcc and glibc for today's updates on my 14.2 Laptop.

Googling the 'grace period' and 'stable storage' errors I see in /var/log/syslog, the 'fix' might be to build and install nfs-utils-2.3.1 on Slackware 14.2 ???

Code:

Feb  7 07:13:15 samsung kernel: [ 8677.550991] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Feb  7 07:15:18 samsung kernel: [ 8800.735383] NFSD: Unable to end grace period: -110
Feb  7 07:15:51 samsung kernel: [ 8833.503364] NFSD: Unable to create client record on stable storage: -110

-- kjh

Code:

# ----------------------------------------------------------------------
# WORKS:  one huge file ; server-side defaults ; client-side async
# ----------------------------------------------------------------------
#
# cli:  tar -cvf /opt/tmp/linux-4.4.115.kjh.tar linux-4.4.115.kjh
# cli:  ls -la /opt/tmp/linux-4.4.115.kjh.tar
  -rw-r--r-- 1 root root 656046080 Feb  7 06:17 /opt/tmp/linux-4.4.115.kjh.tar
#
# cli:  umount /mnt/nfs-sam
# srv:  rm -rf /opt/export/linux-4.4.115.kjh/
# srv:  grep -v '^#' /etc/exports
  /opt/export   192.168.0.0/24(rw,subtree_check,no_root_squash)
# srv:  /etc/rc.d/rc.nfs restart
# cli:  mount.nfs srv:/opt/export /mnt/nfs-sam -o 'rw,vers=4,async'
# srv:  while [ 1 = 1 ] ; do echo "" ; date ; ls -la  linux-4.4.115.kjh ; sleep 30 ; done
# cli:  time cp -pR /opt/tmp/linux-4.4.115.kjh.tar /mnt/nfs-sam/
# ----------------------------------------------------------------------

Wed Feb  7 06:26:58 CST 2018
/bin/ls: cannot access 'linux-4.4.115.kjh.tar': No such file or directory

Wed Feb  7 06:27:28 CST 2018
---------- 1 root root 0 Dec 28  1979 linux-4.4.115.kjh.tar

<<snip>> ---------------------------------------------------------------

Wed Feb  7 06:28:28 CST 2018
-rw------- 1 root root 625999872 Feb  7 06:28 linux-4.4.115.kjh.tar

Wed Feb  7 06:28:58 CST 2018
-rw-r--r-- 1 root root 656046080 Feb  7 06:17 linux-4.4.115.kjh.tar

real    1m28.382s
user    0m0.000s
sys     0m0.348s

# ----------------------------------------------------------------------
# WORKS:  one huge file ; server and client side defaults (sleep 10)
# ----------------------------------------------------------------------
#
# cli:  umount /mnt/nfs-sam
# srv:  rm linux-4.4.115.kjh.tar
# srv:  grep -v '^#' /etc/exports
  /opt/export   192.168.0.0/24(rw,subtree_check,no_root_squash)
# srv:  /etc/rc.d/rc.nfs restart
# cli:  mount.nfs srv:/opt/export /mnt/nfs-sam -o 'rw,vers=4'
# srv:  while [ 1 = 1 ] ; do echo "" ; date ; ls -la  linux-4.4.115.kjh ; sleep 10 ; done
# cli:  time cp -pR /opt/tmp/linux-4.4.115.kjh.tar /mnt/nfs-sam/
# ----------------------------------------------------------------------

Wed Feb  7 06:33:38 CST 2018
---------- 1 root root 0 Jan  2  1980 linux-4.4.115.kjh.tar

Wed Feb  7 06:33:48 CST 2018
---------- 1 root root 0 Jan  2  1980 linux-4.4.115.kjh.tar

Wed Feb  7 06:33:58 CST 2018
---------- 1 root root 0 Jan  2  1980 linux-4.4.115.kjh.tar

Wed Feb  7 06:34:08 CST 2018
-rw------- 1 root root 7340032 Feb  7 06:34 linux-4.4.115.kjh.tar

<<snip>> ---------------------------------------------------------------

Wed Feb  7 06:34:58 CST 2018
-rw------- 1 root root 595591168 Feb  7 06:34 linux-4.4.115.kjh.tar

Wed Feb  7 06:35:08 CST 2018
-rw-r--r-- 1 root root 656046080 Feb  7 06:17 linux-4.4.115.kjh.tar

real    1m26.770s
user    0m0.000s
sys     0m0.348s

# ----------------------------------------------------------------------
# clean up
# ----------------------------------------------------------------------
#
# cli:  umount /mnt/nfs-sam
# cli:  rm /opt/tmp/linux-4.4.115.kjh.tar
# srv:  rm /opt/export/linux-4.4.115.kjh.tar

rogan · 02-07-2018, 02:02 PM

Ok...
I've done some testing also. Since sync has become so damn expensive on 'current',
maybe the file system on the exported directory has something to do with it?

Test:
The server: AMD 8320 8G ram Slackware 'current' (who runs Intel these days)...
The client: AMD 9590 32G ram Slackware-14.2
Both operating systems are up to date withing a few days. Nic is gigabit on a gigabit lan.
Copy from a regular spinning hard drive dedicated for this use, to a nfs
exported directory from 'current' on another spinning hard drive dedicated for this test.
Sync/async is declared in /etc/exports. Mounts were done with default options (none given).
time cp -r /mnt/hd/linux-4.13.1/Documentation /mnt/tmp/rogan/tmp/ on client.
There were some cache effects but completely insignificant in comparison.
These are user times averaged over three runs and rounded:

BTRFS:
sync: 3 minutes
async: 10 seconds

EXT4:
sync: 8 minutes
async: 10 seconds

ext2:
sync: 1 minute 40 seconds
async: 10 seconds

XFS:
sync: 5 minutes
async: 10 seconds

JFS:
sync: 25 seconds
async: 20 seconds

If I recall correctly jfs has 'delayed writes'. It certainly works for nfs.
If one wants, one can mention something about death by creeping featuritis
but I would never do that