LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 11-24-2015, 01:34 PM   #1
ablundon
LQ Newbie
 
Registered: Dec 2014
Location: Paradise, NL, Canada
Distribution: Ubuntu
Posts: 11

Rep: Reputation: Disabled
Slow Rsync SSH Over Internet


Hi all,

I'm a relative newbie to linux. I have an Ubuntu server set up at my house for file storage. Every night, I run a rsync script using Cron to backup my data to a 2nd hard drive (in case the first one dies).

I also have a Ubuntu server set up on a laptop at my parents house. I then repeat the same rsync backup except to a hard drive connected to that laptop. Note that I can successfully access the laptop from my home Ubuntu server through ssh (I've opened the ports etc.).

The local rsync backup happens very fast, however, the remote backup is very slow... in the range of 100 - 200 kB/s. It works, as the files eventually get transferred... it is just very slow. For me to backup say 100 MB of data would take eons.

I've tested upload and download speeds using speed test and they were very fast >1MB/s however backing up using rsync through SSH is painfully slow.

Any suggestions?
 
Old 11-24-2015, 02:21 PM   #2
joe_2000
Senior Member
 
Registered: Jul 2012
Location: Aachen, Germany
Distribution: Void, Debian
Posts: 1,016

Rep: Reputation: 307Reputation: 307Reputation: 307Reputation: 307
Hmm, looks like a tricky one. Sorry for asking the obvious, but...
  • when you say you ran speed tests for upload and download, did you run them on both machines?
  • Are you using the exact same options for both rsync calls (local and remote)
You actually might want to post your rsync commands for people to be able to see what options you are using.

Other than these simple things, I guess your best bet would be to try and isolate possible causes. E.g. when you set up a web server at your house and put a large file there, can you wget it from the laptop at your parents' house with a better transfer rate? Cause if you can't then you can rule out everything related to ssh / rsync and will probably just have to live with it. A standard speed test is not representative of the link between your house and your parent's house, as there might be a network node between the two that acts as a bottleneck. You might not suffer from this bottleneck during the speed tests...

Btw this leads me to another question: How far apart from each other are these two houses?
 
Old 11-24-2015, 02:32 PM   #3
ablundon
LQ Newbie
 
Registered: Dec 2014
Location: Paradise, NL, Canada
Distribution: Ubuntu
Posts: 11

Original Poster
Rep: Reputation: Disabled
Thanks for the response. Our house are about a 5 minute drive away in an urban center so certainly not long distance. In fact we use the same internet provider so I would hope that speed between us would be fast.

I'll run some additional tests. I did do a speed test on both systems and got > 1MB/s upload and download speeds. I'll try the webserver method to wget a file.

Thanks
AB
 
Old 11-24-2015, 03:26 PM   #4
Sefyir
Member
 
Registered: Mar 2015
Distribution: Linux Mint
Posts: 632

Rep: Reputation: 315Reputation: 315Reputation: 315Reputation: 315
I don't know if this is related, but...

Quote:
I did do a speed test on both systems and got > 1MB/s upload and download speeds.
the remote backup is very slow... in the range of 100 - 200 kB/s.
Is it 1MB in the speed test or much higher?

Speed tests generally measure by megabit rather then megabyte
Rsync will report it as megabyte (like most other programs)

I believe there's 8 bits to a byte so..

1megabit will equal 125 kilobytes
 
Old 11-24-2015, 08:50 PM   #5
ablundon
LQ Newbie
 
Registered: Dec 2014
Location: Paradise, NL, Canada
Distribution: Ubuntu
Posts: 11

Original Poster
Rep: Reputation: Disabled
Thanks again,

Speed test for download at my parents was about 36 Mbit/s = 4.5 MB/s
Speed test for upload at my house was about 9.7 Mbit/s = 1.2 MB/s

I then did a wget of a file from my home webserver to my parents computer. Average transfer speed was 1.29 MB/s (which corresponds to above).

I've tried multiple rsync options but basically a sample one looks like this:

rsync -avh -e ssh --info=progress2 "local_file" "username@remote_ip:remote_folder"

Using this command transferred a 200 MB file at an average of 186 kB/s which is 8 times slower than it should be.

Thanks again!

Last edited by ablundon; 11-24-2015 at 10:12 PM.
 
Old 11-25-2015, 07:37 AM   #6
ablundon
LQ Newbie
 
Registered: Dec 2014
Location: Paradise, NL, Canada
Distribution: Ubuntu
Posts: 11

Original Poster
Rep: Reputation: Disabled
Just did another test. This time I issued the rsync command from the remote computer to basically fetch the 200MB file from my local server.

Transfer speed was about the same (201 kB/s).

Very strange. Is there anything specific about the rsync command that would slow this down (the wget command was very fast downloading the file).

AB
 
Old 11-25-2015, 11:44 AM   #7
Sefyir
Member
 
Registered: Mar 2015
Distribution: Linux Mint
Posts: 632

Rep: Reputation: 315Reputation: 315Reputation: 315Reputation: 315
This link might help
Could be the encryption itself is slowing it down.
http://serverfault.com/questions/377...-rsync-so-slow
 
Old 11-25-2015, 01:00 PM   #8
ablundon
LQ Newbie
 
Registered: Dec 2014
Location: Paradise, NL, Canada
Distribution: Ubuntu
Posts: 11

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Sefyir View Post
This link might help
Could be the encryption itself is slowing it down.
http://serverfault.com/questions/377...-rsync-so-slow
Funny enough, I was reading this link earlier wondering if it could the encryption.

I've issued a different rsync command using the following options:

Code:
rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" "Source" "username@IP:Target"
My speed is now about 457 kB/s for a 50 MB file. 2-3 times faster :-)

I wonder if I can improve even more????

AB
 
Old 11-27-2015, 09:10 AM   #9
debguy
Member
 
Registered: Oct 2014
Location: U.S.A.
Distribution: mixed, mostly debian slackare today
Posts: 207

Rep: Reputation: 19
SSH interferes with some apps (not rsync?) by injecting itself INSIDE packet instead of being a wrapper (example: it has been known to damage X11 packets, fixed?)

SSH upon DEFAULT (debian/ubuntu) install takes a while to "generate a key" but that DOES NOT MEAN it is setup (up to sqeeze anyhows).

if you use "tcpdump" to examine packets they ARE NOT ENcRYPTED, and wont be until you read a book of SSH configuration and spend and hour doing it.
 
Old 11-27-2015, 09:13 AM   #10
debguy
Member
 
Registered: Oct 2014
Location: U.S.A.
Distribution: mixed, mostly debian slackare today
Posts: 207

Rep: Reputation: 19
$ rsync --rsh=rsh

# why: to turn of ssh

i suggest using "private network" (vpn) if you really need security

but some sites do require scftp (ssh hacked ftp) so you really can't just "delete ssh" for trade of using an alternate method. note i would not bet this is a mistake ... they work together to enforce use of certain people's wares uploads.
 
Old 11-27-2015, 09:15 AM   #11
debguy
Member
 
Registered: Oct 2014
Location: U.S.A.
Distribution: mixed, mostly debian slackare today
Posts: 207

Rep: Reputation: 19
I'm a relative newbie to linux. I have an Ubuntu server set up at my house for file storage. Every night, I run a rsync script using Cron to backup my data to a 2nd hard drive (in case the first one dies).

I also have a Ubuntu server set up on a laptop at my parents house. I then repeat the same rsync backup except to a hard drive connected to that laptop. Note that I can successfully access the laptop from my home Ubuntu server through ssh (I've opened the ports etc.).

--------------------
hold right there that's your problem

invest $50 in a terabyte class drive and run your backups to it in your own home (across your local lan / usb wire)

or just get a used/refurbished laptop for the cause

you'll thank yourself later i promise
 
Old 11-27-2015, 09:16 AM   #12
debguy
Member
 
Registered: Oct 2014
Location: U.S.A.
Distribution: mixed, mostly debian slackare today
Posts: 207

Rep: Reputation: 19
your not transfering your parents files to your computer are you? if you are that's illegal dont do it
 
Old 11-27-2015, 09:47 AM   #13
ablundon
LQ Newbie
 
Registered: Dec 2014
Location: Paradise, NL, Canada
Distribution: Ubuntu
Posts: 11

Original Poster
Rep: Reputation: Disabled
Hi Debguy... thanks for the responses.

I have a backup solution working in my own house (rsync speed no issue there). So if my primary storage device fails, I'll have a backup at my house.

The backup I'm trying to achieve in this case is a 2nd backup that would exist at my parents house. That will give me offsite backup should my house every get damaged (fire/flood/etc.). So I'm not transferring files from my parents to my house.... it is the opposite. Files from my house to my parents.

I could probably buy some cloud storage but of course that would cost money and put my data with a third party. I'm convinced I can do this with my own hardware for relatively little expense.

I have also tried using VPN (software VPN) but found the speed was slow with that as well... which lead me to backup through SSH. I'm trying to find the bottleneck. Not sure what it might be since I can download a file from my web server (also located at my house) very fast (see previous post above).
 
Old 11-27-2015, 10:28 AM   #14
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 23,659

Rep: Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915Reputation: 6915
Quote:
Originally Posted by debguy View Post
your not transfering your parents files to your computer are you? if you are that's illegal dont do it
By what logic is this true? Backing up files for someone is NEVER illegal, if they give you permission. Unless they're pirating software, there is ZERO that's illegal about making a copy of something that you own. Please, don't post things like this.
 
2 members found this post helpful.
Old 12-01-2015, 03:15 PM   #15
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 917Reputation: 917Reputation: 917Reputation: 917Reputation: 917Reputation: 917Reputation: 917Reputation: 917
Quote:
Originally Posted by debguy View Post
your not transfering your parents files to your computer are you? if you are that's illegal dont do it
What made you arrive at this conclusion?
 
  


Reply

Tags
backup, rsync+ssh, slow, ubuntu


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync is slow jayadhanesh Linux - Newbie 1 01-24-2014 04:49 AM
rsync over ssh VS rsync.d RISKS tripialos Linux - Security 4 02-20-2013 06:22 PM
[SOLVED] rsync fails in cron - ssh key prob for rsync? jonathansfl Linux - Server 6 12-09-2010 09:48 AM
Rsync server vs rsync over ssh humbletech99 Linux - Networking 1 10-18-2006 12:10 PM
New Fedora Internet connection slow slow slow matrim Fedora 9 07-29-2005 01:39 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:55 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration