Slow Rsync SSH Over Internet
 

Hi all,

I'm a relative newbie to linux. I have an Ubuntu server set up at my house for file storage. Every night, I run a rsync script using Cron to backup my data to a 2nd hard drive (in case the first one dies).

I also have a Ubuntu server set up on a laptop at my parents house. I then repeat the same rsync backup except to a hard drive connected to that laptop. Note that I can successfully access the laptop from my home Ubuntu server through ssh (I've opened the ports etc.).

The local rsync backup happens very fast, however, the remote backup is very slow... in the range of 100 - 200 kB/s. It works, as the files eventually get transferred... it is just very slow. For me to backup say 100 MB of data would take eons.

I've tested upload and download speeds using speed test and they were very fast >1MB/s however backing up using rsync through SSH is painfully slow.

Any suggestions?

Hmm, looks like a tricky one. Sorry for asking the obvious, but...

• when you say you ran speed tests for upload and download, did you run them on both machines?

• Are you using the exact same options for both rsync calls (local and remote)

You actually might want to post your rsync commands for people to be able to see what options you are using.

Other than these simple things, I guess your best bet would be to try and isolate possible causes. E.g. when you set up a web server at your house and put a large file there, can you wget it from the laptop at your parents' house with a better transfer rate? Cause if you can't then you can rule out everything related to ssh / rsync and will probably just have to live with it. A standard speed test is not representative of the link between your house and your parent's house, as there might be a network node between the two that acts as a bottleneck. You might not suffer from this bottleneck during the speed tests...

Btw this leads me to another question: How far apart from each other are these two houses?

Thanks for the response. Our house are about a 5 minute drive away in an urban center so certainly not long distance. In fact we use the same internet provider so I would hope that speed between us would be fast.

I'll run some additional tests. I did do a speed test on both systems and got > 1MB/s upload and download speeds. I'll try the webserver method to wget a file.

Thanks
AB

I don't know if this is related, but...

Is it 1MB in the speed test or much higher?

Speed tests generally measure by megabit rather then megabyte
Rsync will report it as megabyte (like most other programs)

I believe there's 8 bits to a byte so..

1megabit will equal 125 kilobytes

Thanks again,

Speed test for download at my parents was about 36 Mbit/s = 4.5 MB/s
Speed test for upload at my house was about 9.7 Mbit/s = 1.2 MB/s

I then did a wget of a file from my home webserver to my parents computer. Average transfer speed was 1.29 MB/s (which corresponds to above).

I've tried multiple rsync options but basically a sample one looks like this:

rsync -avh -e ssh --info=progress2 "local_file" "username@remote_ip:remote_folder"

Using this command transferred a 200 MB file at an average of 186 kB/s which is 8 times slower than it should be.

Thanks again!

Just did another test. This time I issued the rsync command from the remote computer to basically fetch the 200MB file from my local server.

Transfer speed was about the same (201 kB/s).

Very strange. Is there anything specific about the rsync command that would slow this down (the wget command was very fast downloading the file).

AB

This link might help
Could be the encryption itself is slowing it down.
http://serverfault.com/questions/377...-rsync-so-slow

Funny enough, I was reading this link earlier wondering if it could the encryption.

I've issued a different rsync command using the following options:

My speed is now about 457 kB/s for a 50 MB file. 2-3 times faster :-)

I wonder if I can improve even more????

AB

SSH interferes with some apps (not rsync?) by injecting itself INSIDE packet instead of being a wrapper (example: it has been known to damage X11 packets, fixed?)

SSH upon DEFAULT (debian/ubuntu) install takes a while to "generate a key" but that DOES NOT MEAN it is setup (up to sqeeze anyhows).

if you use "tcpdump" to examine packets they ARE NOT ENcRYPTED, and wont be until you read a book of SSH configuration and spend and hour doing it.

$ rsync --rsh=rsh

# why: to turn of ssh

i suggest using "private network" (vpn) if you really need security

but some sites do require scftp (ssh hacked ftp) so you really can't just "delete ssh" for trade of using an alternate method. note i would not bet this is a mistake ... they work together to enforce use of certain people's wares uploads.

I'm a relative newbie to linux. I have an Ubuntu server set up at my house for file storage. Every night, I run a rsync script using Cron to backup my data to a 2nd hard drive (in case the first one dies).

I also have a Ubuntu server set up on a laptop at my parents house. I then repeat the same rsync backup except to a hard drive connected to that laptop. Note that I can successfully access the laptop from my home Ubuntu server through ssh (I've opened the ports etc.).

--------------------
hold right there that's your problem :)

invest $50 in a terabyte class drive and run your backups to it in your own home (across your local lan / usb wire)

or just get a used/refurbished laptop for the cause

you'll thank yourself later i promise

your not transfering your parents files to your computer are you? if you are that's illegal dont do it

Hi Debguy... thanks for the responses.

I have a backup solution working in my own house (rsync speed no issue there). So if my primary storage device fails, I'll have a backup at my house.

The backup I'm trying to achieve in this case is a 2nd backup that would exist at my parents house. That will give me offsite backup should my house every get damaged (fire/flood/etc.). So I'm not transferring files from my parents to my house.... it is the opposite. Files from my house to my parents.

I could probably buy some cloud storage but of course that would cost money and put my data with a third party. I'm convinced I can do this with my own hardware for relatively little expense.

I have also tried using VPN (software VPN) but found the speed was slow with that as well... which lead me to backup through SSH. I'm trying to find the bottleneck. Not sure what it might be since I can download a file from my web server (also located at my house) very fast (see previous post above).

By what logic is this true? Backing up files for someone is NEVER illegal, if they give you permission. Unless they're pirating software, there is ZERO that's illegal about making a copy of something that you own. Please, don't post things like this.

What made you arrive at this conclusion?