Speeding up SCP

szboardstretcher · 04-01-2011, 09:27 AM

For anyone that needs it, Ill summarize my findings with regards to speeding up SCP.

If you search around on the web, you will find that SCP's speed is limited to SSH's default window size. There are patches provided to patch up SSH and increase the window size, improving the speed by up to 500% depending.

However, that requires you to compile SSH and install it on both endpoints, which you might not be able to do. So, built into the SSH suite is a little known flag called -c or "cypher" which allows you to pick a different cypher. The default is '3des' which is secure -- but taxing.

Here are my quick tests with a 1GB file over a 100mbit connection.

Code:

scp output.dat 192.168.111.11:/tmp/               2%   27MB   5.2MB/s   03:01 ETA
scp -c blowfish output.dat 192.168.111.11:/tmp/   6%  100MB   6.8MB/s   02:52 ETA
scp -c arcfour output.dat 192.168.111.11:/tmp/   24%  235MB   7.2MB/s   01:42 ETA

As you can see, specifying the -c arcfour cypher on the scp line, adds 2MegaBytes per second throughput.

Good luck.

jefro · 04-01-2011, 04:14 PM

Did you also test various compression levels? Guess in some cases you could choose none if not you are not worried about content only connection.

arcfour
blowfish-cbc
cast128-cbc
aes128-cbc
rijndael128-cbc
3des-cbc

Is there any other options?

szboardstretcher · 04-13-2011, 01:33 PM

Quote:

Originally Posted by jefro

Did you also test various compression levels? Guess in some cases you could choose none if not you are not worried about content only connection.

arcfour
blowfish-cbc
cast128-cbc
aes128-cbc
rijndael128-cbc
3des-cbc

Is there any other options?

I listed the fastest 3 out of those. I don't see any other options in the SSL man page.

marcojrfurtado · 07-06-2012, 08:12 AM

You are seeing these results because your CPU is the bottleneck. At moderns machines, the disk or the network becomes the bottleneck, so you’ll get comparable performance out of all ciphers. I benchmarked my system through this little script here:

Code:

for cipher in aes128-ctr aes192-ctr aes256-ctr arcfour256 arcfour128 aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc arcfour ; do
        echo "$cipher"
        for try in 1 2 ; do
                scp  -c "$cipher" Tomcat.tar.bz2 user3@m21:~/
        done
done

Just look at my results, as you can see the network won't transfer more than 11.3MB/s.

aes128-ctr
Tomcat.tar.bz2 100% 508MB 11.0MB/s 00:46
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
aes192-ctr
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
aes256-ctr
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
arcfour256
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
arcfour128
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
aes128-cbc
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
3des-cbc
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.0MB/s 00:46
blowfish-cbc
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.0MB/s 00:46
cast128-cbc
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.0MB/s 00:46
aes192-cbc
Tomcat.tar.bz2 100% 508MB 11.0MB/s 00:46
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
aes256-cbc
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
arcfour
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45
Tomcat.tar.bz2 100% 508MB 11.3MB/s 00:45

browny_amiga · 10-19-2012, 08:37 PM

I cannot agree with you there:
The CPU is definitely not the limiting factor. My CPU (dual core 2.6 ghz) is not really sweating, my disk is underwhelmed and my network could theoratically transfer 70 mb/s (1 Gbit) and yet ssh and scp is very very slow.
From my years and years of transfering files over it, I found that the protocol must have some built in bottle neck that no fast CPU or anything can fix. But that makes sense, ssh (secure SHELL) was never ever done for transfering bulk amounts of data, so it is not surprising that it sucks at it.

I wish they would fix it, because for easy fast access to files, satisfying these conditions:
1. use existing users
2. do access secure and encrypted
3. use something that available and installed on any Linux system on the planet
4. is fast and easy to configure and use.

there is only ssh currently.
NFS fails in point 2 and 4.

nbritton · 11-07-2014, 09:59 PM

Below is a script I made that you can use to benchmark each cipher. I use /dev/zero, /dev/null, and localhost to eliminate potential i/o bottlenecks. The results should report realistic maximum transfer throughput. There is also the "openssl speed" benchmark, but I find that doesn't produce accurate data transfer throughput results.

The following bash one liner was tested, and works, on Linux, Mac OS X, and Solaris:

for i in 3des-cbc aes128-cbc aes128-ctr aes128-gcm@openssh.com aes192-cbc aes192-ctr aes256-cbc aes256-ctr aes256-gcm@openssh.com arcfour arcfour128 arcfour256 blowfish-cbc cast128-cbc chacha20-poly1305@openssh.com rijndael-cbc@lysator.liu.se; do dd if=/dev/zero bs=1000000 count=1000 2> /dev/null | ssh -c $i localhost "(time -p cat) > /dev/null" 2>&1 | grep real | awk '{print "'$i': "1000 / $2" MB/s" }'; done

The arcfour (RC4) ciphers are usually the fastest on systems that do not have encryption acceleration hardware. When available, arcfour256 or arcfour128 should be use rather then plain arcfour, this is because arcfour128 and arcfour256 include a discard step, meaning the first 1536 bytes produced by the cipher are dropped (see rfc 4345). I firmly believe that RC4 is still a secure cipher and I regularly use it when I need faster transfer rates.

Ciphers can be invoked as follows:

scp -c arcfour128 ...
scp -o Cipher=arcfour128 ...

sftp -c arcfour128 ...
sftp -o Cipher=arcfour128 ...

rsync -e "ssh -c arcfour128" ...
rsync --rsh="ssh -c arcfour128" ...

Results:

With AES-NI and CLMUL instructions, aes128-gcm@openssh.com is the fastest cipher on OpenSSH Intel systems:

Intel Core i7-2635QM (MacBookPro8,2) / Fedora 21:
3des-cbc: 18.34 MB/s
aes128-cbc: 167.54 MB/s
aes128-ctr: 230.26 MB/s
aes128-gcm@openssh.com: 328.37 MB/s
aes192-cbc: 158.13 MB/s
aes192-ctr: 228.28 MB/s
aes256-cbc: 148.89 MB/s
aes256-ctr: 222.80 MB/s
aes256-gcm@openssh.com: 317.49 MB/s
arcfour: 178.52 MB/s
arcfour128: 177.52 MB/s
arcfour256: 178.50 MB/s
blowfish-cbc: 66.75 MB/s
cast128-cbc: 58.68 MB/s
chacha20-poly1305@openssh.com: 131.70 MB/s
rijndael-cbc@lysator.liu.se: 149.04 MB/s
Mac OS X cipher performance is abysmal compared to Linux, take a look at the results:

Intel Core i7-2635QM (MacBookPro8,2) / Mac OS X 10.10:
3des-cbc: 14.98 MB/s
aes128-cbc: 88.08 MB/s
aes128-ctr: 87.18 MB/s
aes192-cbc: 85.18 MB/s
aes192-ctr: 84.10 MB/s
aes256-cbc: 82.51 MB/s
aes256-ctr: 81.48 MB/s
arcfour: 133.93 MB/s
arcfour128: 133.99 MB/s
arcfour256: 133.93 MB/s
blowfish-cbc: 39.96 MB/s
cast128-cbc: 39.06 MB/s
rijndael-cbc@lysator.liu.se: 82.24 MB/s
Lastly, here is a Synology DS1010+ that I converted to Zentyal:
Intel Atom D510 / Ubuntu 12.04:
3des-cbc: 8.12 MB/s
aes128-cbc: 15.28 MB/s
aes128-ctr: 11.45 MB/s
aes192-cbc: 13.43 MB/s
aes192-ctr: 10.04 MB/s
aes256-cbc: 11.55 MB/s
aes256-ctr: 8.89 MB/s
arcfour: 41.63 MB/s
arcfour128: 42.65 MB/s
arcfour256: 40.30 MB/s
blowfish-cbc: 28.65 MB/s
cast128-cbc: 16.77 MB/s
rijndael-cbc@lysator.liu.se: 11.34 MB/s

nbritton · 11-12-2014, 11:58 AM

Quote:

Originally Posted by nbritton

The following bash one liner was tested, and works, on Linux, Mac OS X, and Solaris:

for i in 3des-cbc aes128-cbc aes128-ctr aes128-gcm@openssh.com aes192-cbc aes192-ctr aes256-cbc aes256-ctr aes256-gcm@openssh.com arcfour arcfour128 arcfour256 blowfish-cbc cast128-cbc chacha20-poly1305@openssh.com rijndael-cbc@lysator.liu.se; do dd if=/dev/zero bs=1000000 count=1000 2> /dev/null | ssh -c $i localhost "(time -p cat) > /dev/null" 2>&1 | grep real | awk '{print "'$i': "1000 / $2" MB/s" }'; done

Amazon AWS EC2 t2.micro (throttled) / Ubuntu 14.04:

3des-cbc 1.13 MB/s
aes128-cbc 13.10 MB/s
aes128-ctr 16.98 MB/s
aes128-gcm@openssh.com 25.14 MB/s
aes192-cbc 12.38 MB/s
aes192-ctr 16.18 MB/s
aes256-cbc 11.51 MB/s
aes256-ctr 15.52 MB/s
aes256-gcm@openssh.com 22.87 MB/s
arcfour 11.69 MB/s
arcfour128 11.79 MB/s
arcfour256 11.96 MB/s
blowfish-cbc 4.51 MB/s
cast128-cbc 4.01 MB/s
chacha20-poly1305@openssh.com 9.23 MB/s
rijndael-cbc@lysator.liu.se 11.35 MB/s

Amazon AWS EC2 t2.micro (unthrottled) / Ubuntu 14.04:

3des-cbc 11.22 MB/s
aes128-cbc 117.97 MB/s
aes128-ctr 141.21 MB/s
aes128-gcm@openssh.com 206.39 MB/s
aes192-cbc 113.52 MB/s
aes192-ctr 140.77 MB/s
aes256-cbc 108.98 MB/s
aes256-ctr 138.07 MB/s
aes256-gcm@openssh.com 198.00 MB/s
arcfour 110.61 MB/s
arcfour128 109.68 MB/s
arcfour256 109.98 MB/s
blowfish-cbc 42.09 MB/s
cast128-cbc 38.60 MB/s
chacha20-poly1305@openssh.com 89.23 MB/s
rijndael-cbc@lysator.liu.se 110.81 MB/s