LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-21-2019, 01:30 PM   #1
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 289

Rep: Reputation: 8
RSYNC over SSH slow speed


I use such command to backup QNAP NAS to mounted on-demand (only for backup needs) USBv3 HDD connected to PC inside local LAN. The connection between QNAP and PC is done over 10 GB Ethernet. So network is not bottleneck.

This is related part of backup script:

Code:
/opt/bin/rsync -qratx --delete --exclude-from=/share/homes/media/Pobrane/pomin_piotr.txt --force --progress -e '/opt/bin/ssh -x -q -i /share/homes/media/.ssh/id_media' /share/homes/media/Gosia_ISO/* root@$serwer:$backup_path/Gosia_ISO
The $backup_path/Gosia_ISO dir consist only 6 files but large:

Code:
[root@Piotr ~]# ls -la /home/maciek/back/RSYNC/Gosia_ISO/
razem 12308364
drwxrwxr-x  2 toruser toruser       4096 gru 20 16:47 ./
drwxrwxrwx 23 root    root          4096 gru 21 12:50 ../
-rw-rw-r--  1 toruser toruser 6287774842 gru 20 05:52 gosia_sda5.img.gz
-rw-r--r--  1 toruser toruser         81 gru 20 05:52 gosia_sda5.img.gz.md5
-rw-rw-r--  1 toruser toruser   72909190 gru 20 06:05 gosia_sda7.img.gz
-rw-r--r--  1 toruser toruser         81 gru 20 06:05 gosia_sda7.img.gz.md5
-rw-rw-r--  1 toruser toruser 6243042497 gru 20 06:16 gosia_sda8.img.gz
-rw-r--r--  1 toruser toruser         81 gru 20 06:17 gosia_sda8.img.gz.md5
If I run above command, the sync speed is appx 5 M/s (as iotop stands). But when I first delete content of $backup_path/Gosia_ISO dir and then run rsync command, the throughput is appx 70 M/s.

Why the speed is so different in both cases? What to do to be the same without former deleting of $backup_path/Gosia_ISO dir content?
 
Old 12-21-2019, 02:11 PM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,661

Rep: Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133
rsync is designed to work on slow networks, where checking the content and transfer only the differences is much more efficient. I think in your case transferring the whole file is much faster than finding the differences.
 
Old 12-22-2019, 03:41 AM   #3
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 289

Original Poster
Rep: Reputation: 8
As far as I know, rsync (in default config) looks for files needs to be synced by comparing names, size and time of modification. And there is no inspection of files content. Of course it takes a lot of time if You have a huge amount of files. But after then (in my case), if newer version of file is avaliable, it is transmitted to destination with name beginning with dot and random string at the end of the name of this new file. When new file is fully transmitted, it replace older one.
Moreover. If I connect USB HDD to local USB port in NAS, rsync always runs with maximal speed.
And look that amount of files in my case is very low. 6 or 8 files in each directory.
So still I don know why deleting files before rsync speed-up throughput dramatically.
 
Old 12-22-2019, 03:59 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,661

Rep: Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133
here you can read how rsync really works: https://rsync.samba.org/how-rsync-works.html (and there you can find also some links to additional details) [believe me, really interesing].
Quote:
If a file is not to be skipped, any existing version on the receiving side becomes the "basis file" for the transfer, and is used as a data source that will help to eliminate matching data from having to be sent by the sender. To effect this remote matching of data, block checksums are created for the basis file and sent to the sender immediately following the file's index number.
I don't really understand your situation "If I connect USB HDD to local USB port in NAS, rsync always runs with maximal speed.", so I guess only: if the server and client are the same host (=nas) rsync will do [more or less] a simple copy.

anyway, you can check the load of your system and you will see if rsync is really processing those files. From the other hand if a simple copy is definitely faster, use that. Or tell --whole-file to rsync (or just for testing).

Last edited by pan64; 12-22-2019 at 04:06 AM.
 
Old 12-22-2019, 04:05 AM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,260
Blog Entries: 3

Rep: Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597Reputation: 2597
The rsync manual page has this:

Quote:
-I, --ignore-times
Normally rsync will skip any files that are already the same
size and have the same modification timestamp. This option
turns off this "quick check" behavior, causing all files to be
updated.
So that indicates that it does initially only check size + date to guess about which files can be skipped for checksumming.

However, if I recall correctly the checksumming can be done safely even with something as old as MD4, but for non-technical reasons a slower hashing algorithm was chosen. You can set it back to MD4 using the --checksum-choice option, and it should remain as accurate Or you could go with --size-only but at the expense of accuracy. The read times being slow will be dependent on how your file system is mounted and what kind of device is underneath.
 
Old 12-22-2019, 04:14 AM   #6
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 289

Original Poster
Rep: Reputation: 8
Quote:
I don't really understand your situation "If I connect USB HDD to local USB port in NAS, rsync always runs with maximal speed.", so I guess only: if the server and client are the same host (=nas) rsync will do [more or less] a simple copy.
Yes, in this test, the server and client are the same host (=nas).

But now another test. Filesystem located on USB HDD and connected to PC, I mounted via NFS to dir in NAS. And try to rsync files de-facto locally. Always throughput is full, does not matter dir containing large ISO files is empty or not.
 
Old 12-22-2019, 04:53 AM   #7
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,661

Rep: Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133Reputation: 5133
so that was again a local copy.
 
Old 12-22-2019, 06:43 AM   #8
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 289

Original Poster
Rep: Reputation: 8
Yes. Its like local copy. But this solution has advantages what I wanted. Off-line backup on HDD connected to other LAN connected computer, mounted on-demand and high throughput both with empty and not empty destination dir. Thats all. From my point of view - solved.

Last edited by mackowiakp; 12-22-2019 at 09:59 AM.
 
  


Reply

Tags
back-up, rsync+ssh, throughput


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow Rsync SSH Over Internet ablundon Linux - Software 18 03-17-2016 07:33 AM
rsync over ssh VS rsync.d RISKS tripialos Linux - Security 4 02-20-2013 07:22 PM
Rsync running on IPSEC VPN timeout or getting very slow speed Moaahk Linux - Hardware 0 06-11-2008 10:46 PM
Rsync server vs rsync over ssh humbletech99 Linux - Networking 1 10-18-2006 01:10 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:02 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration