LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 07-31-2016, 08:04 PM   #1
gbell12
LQ Newbie
 
Registered: Aug 2015
Posts: 11

Rep: Reputation: Disabled
Off-site mirroring scheme using rsync, sshfs and encfs too slow to be usable


I have a scheme using rsync, encfs and sshfs that mirrors my data off-site.

I like it because I don't mirror junk (because I can use rsync's --exclude), and it doesn't require a local copy to be made (because of the encfs mount point).

So the data flow looks like:

rsync of my data -> encfs mount point -> sshfs mount point

It's really really slow, though. Unusably so.

Turns out simply rsync'ing to the sshfs mount point is slow.

cp is fine (floods the connection).

This seems like something easy to search Google for and linuxquestions.org, and indeed I have. I've tried a few recommended sshfs options with no change.

It's hard to debug since rsync does different things at different times, but mostly the symptom is:
  1. My CPU is mostly idle
  2. The connection is mostly idle
  3. Each <100KB file that's being processed takes 10+ seconds

If I strace -f -p the head rsync process (there are three spawned for some reason), I see it sitting on a select:

Code:
write(1, "gbell/.PhpStorm2016.1/system/in"..., 145) = 145
read(4, "\0\275\0\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32768) = 32768
select(7, [6], [5], [6], {60, 0})       = 1 (in [6], left {59, 999994})
read(6, "\0\0\0\0\0\1\0\240\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\240\0\0\0\0\0"..., 114) = 114
select(6, [], [5], [], {60, 0}
That last select is the one taking the longest. Handle 5 is

Code:
rsync   15129 root    5u  unix 0xe0eb3980      0t0  5162991 socket
At this point I'm stuck.

This is with rsync-HEAD-20160630-1605GMT as well as the one that's current with Ubuntu 14.04.1 LTS (v3.1.0).

Here's my rsync command line.

Code:
rsync -rv --size-only --delay-updates --no-group --no-owner --no-perms --no-times --whole-file --exclude 'cache/' --exclude 'caches/' /data /mnt/remote/unencrypted
A simpler command line does the same thing (but is even slower since it's more interactive with the remote site due to trying to preserve permissions, times, etc.)

The --delay-updates I threw in there as an experiment, and it makes the initial flurry of lstats go very fast, but then it slows to the 2 seconds per file pace.

Targeting other machines over the internet is the same.

Targeting a machine on my LAN is fine (floods the connection). This is probably the least logical result.

Rsync using ssh transport to the same machine is fine (floods the connection).

What do I try or look at next?

Last edited by gbell12; 07-31-2016 at 08:30 PM. Reason: change 'backup' to 'mirror'
 
Old 07-31-2016, 08:49 PM   #2
gbell12
LQ Newbie
 
Registered: Aug 2015
Posts: 11

Original Poster
Rep: Reputation: Disabled
Update: I mistakenly thought strace -f <parent pid> would trace all three rsync processes simultaneously. If I do strace all three, I see the last one is waiting on a select, as documented above, but the other two are doing 100s of seemingly useless operations like this:

Code:
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage.values", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage.values.at", 0xbf89a944) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage.values.at", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage.values.s", 0xbf89a944) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage.values.s", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage_i", 0xbf89a944) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage_i", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage_i.len", 0xbf89a944) = -1 ENOENT (No such file or directory)
And this:

Code:
lstat64(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.~tmp~", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
lstat64(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction$1.class.xhqKFI", {st_mode=S_IFREG|0600, st_size=2035, ...}) = 0
chmod(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction$1.class.xhqKFI", 0640) = 0
rename(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction$1.class.xhqKFI", ".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.~tmp~/ShiftLeftVisualAction$1.class") = 0
open(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/ShiftLeftVisualAction.class", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction.class.YtPv6X", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 1
fchmod(1, 0600)                         = 0
100's of lines of those, for each file rsync reports it's working on.

So am I looking at a question for the rsync devs?

Last edited by gbell12; 07-31-2016 at 09:10 PM. Reason: clarify
 
Old 08-05-2016, 08:20 AM   #3
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,380

Rep: Reputation: 404Reputation: 404Reputation: 404Reputation: 404Reputation: 404
Some years ago, I struggled with the same problems. I wanted to send a backup to a remote server with rsync, and I was worried about the remote server. It could get stolen or something, and I didn't want them to be able to read all the backup files. Like you, I had problems because it was terribly slow. I never really solved the problem, but I found 2 different workarounds.

1) Don't use sshfs. Instead just use encfs on the remote server, and use rsync normally (over ssh). Something like this:

echo "mysecretpassword" | ssh $remoteserver "encfs --stdinpass /home/.backup /home/backup"
rsync -a /data $remoteserver:/home/backup
ssh $remoteserver fusermount -u /home/backup

This is a lot faster. The downside is that you have to trust root on the remote server. Root can read all the files while the backup is being done, sniff the password, or do other nasty things. But in my situation, I was more worried someone would actually steal the server by breaking into the house. When the server is turned off, it's automaticly unmounted. So then all the files are encrypted, and the password to decrypt is not stored anywhere.

2) If you have lots of diskspace, you can use rsync locally first to an filesystem with encfs. Then unmount the enfs filesystem, and rsync the encrypted files. This is a safer solution, but it takes double the disk space on the local machine.
 
Old 08-05-2016, 08:40 AM   #4
gbell12
LQ Newbie
 
Registered: Aug 2015
Posts: 11

Original Poster
Rep: Reputation: Disabled
Thanks @Guttorm. The issue is that I need the files to land on the remote server already encrypted, because I don't trust root or other users.

Last edited by gbell12; 08-05-2016 at 05:44 PM.
 
Old 08-09-2016, 12:53 AM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,895

Rep: Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613
Quote:
cp is fine (floods the connection).
so the cxn looks like a local disk (or did you mean scp ??)

If you have the diskspace locally, you could create an encrypted copy using gnupg, then just cp or scp or even ftp it ...
 
Old 08-09-2016, 05:04 PM   #6
gbell12
LQ Newbie
 
Registered: Aug 2015
Posts: 11

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by chrism01 View Post
so the cxn looks like a local disk (or did you mean scp ??)
Connection looks like a local disk because of sshfs. So I can 'cp' to it.

Quote:
Originally Posted by chrism01 View Post
If you have the diskspace locally, you could create an encrypted copy using gnupg, then just cp or scp or even ftp it ...
I do not have the space locally.
 
Old 08-10-2016, 03:31 AM   #7
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,895

Rep: Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613Reputation: 2613
I'd say you definitely need to talk to the remote admins.

What it sounds like is that anything thing that looks like network socket cxn is bandwidth limited at one end or the other (or both).
(It may be case that even for an rsync that appears to a 'local' copy job, it still uses network sockets)
cp itself doesn't use an explicit network socket (although its a remote mnt 'underneath'). I'm not quite sure how cp really works for a remote mnt here.
 
  


Reply

Tags
performance, rsync, sshfs


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Encrypted offsite backup with EncFS, Amazon S3, and s3cmd LXer Syndicated Linux News 1 07-31-2016 07:39 PM
sshfs to mount VPS hosted site on Linux Mint ramsforums Linux - Newbie 2 07-29-2016 03:44 AM
[SOLVED] Using rsnapshot with sshfs to backup lpallard Linux - Software 3 08-29-2012 10:41 AM
encfs over sshfs buggy - workaround? Guttorm Linux - Security 1 10-01-2009 10:38 AM
Using RSync to backup a secondary off-site backup server pezdspencer Linux - Software 4 06-29-2007 03:40 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 08:39 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration