LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Closed Thread
  Search this Thread
Old 08-05-2016, 01:22 AM   #1
gbell12
LQ Newbie
 
Registered: Aug 2015
Posts: 11

Rep: Reputation: Disabled
rsync over sshfs reallllly slow


Using rsync 3.1.1 (or HEAD) and sshfs (2.5) under ubuntu is really slow over the internet.

Rsync via ssh or scp without sshfs involved is fine.

An strace shows rsync doing lots of lstats and chmods, and not bothering to transmit most of the time when those are happening.

Reducing all of those remote checks with things like --whole-file and --no-owner improves the situation, but it still spends most of its time querying the remote file system, and little time transmitting.

The reason I'm not using rsync's ssh transport is that I'm using encfs to create a virtual local file system so that I don't have to have double the disk space to build a local copy of everything for an encrypted remote mirror.

Has anybody got a reasonable setup with rsync and sshfs working quickly?
 
Old 08-05-2016, 06:17 AM   #2
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS,Manjaro
Posts: 5,658

Rep: Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708Reputation: 2708
Well, sshfs is an emulated filesystem over sftp. Rsync is normally run as rsync protocol (which MUST do a lit of checksum and metadata checking) generally over ssh (originally over rsh). Sshfs was designed for convenience when better solutions are not available, it is not known for global efficiency. Running rsync in local mode strips out the ssh piece, but you are still layering a lot of communication protocol into a tower of inelegance.

I would recommend running rsync between native storage locations to avoid redundant layering. Run that way it seems fast and elegant, though performance may not scale elegantly. If you must involve the sshfs storage, there may be a tool more suited to your goal.

Detail on that goal, and what you are REALLY trying to accomplish, MIGHT help some of us make some targeted recommendations. (Of course, it also might not, but we can only try.)
 
Old 08-05-2016, 06:31 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,131

Rep: Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121
This is a duplicate post - more detail in this thread. Reporting this for closure.

My initial response was also "too many moving pieces" - especially with fuse involved (at least) twice. I used sshfs years ago, and IIRC I dropped it because of performance.
 
  


Closed Thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Off-site mirroring scheme using rsync, sshfs and encfs too slow to be usable gbell12 Linux - Server 6 08-10-2016 03:31 AM
rsync is slow jayadhanesh Linux - Newbie 1 01-24-2014 04:49 AM
RSYNC is running slow perhaps bcyork Linux - Software 5 08-18-2011 03:07 AM
Rsync is dead slow. Why? junglepunk Linux - Server 2 08-13-2010 09:07 AM
RSync Performance slow.... geeksedition Linux - Software 1 04-25-2008 06:34 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 09:12 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration