Off-site mirroring scheme using rsync, sshfs and encfs too slow to be usable

gbell12 · 07-31-2016, 08:04 PM

I have a scheme using rsync, encfs and sshfs that mirrors my data off-site.

I like it because I don't mirror junk (because I can use rsync's --exclude), and it doesn't require a local copy to be made (because of the encfs mount point).

So the data flow looks like:

rsync of my data -> encfs mount point -> sshfs mount point

It's really really slow, though. Unusably so.

Turns out simply rsync'ing to the sshfs mount point is slow.

cp is fine (floods the connection).

This seems like something easy to search Google for and linuxquestions.org, and indeed I have. I've tried a few recommended sshfs options with no change.

It's hard to debug since rsync does different things at different times, but mostly the symptom is:

My CPU is mostly idle
The connection is mostly idle
Each <100KB file that's being processed takes 10+ seconds

If I strace -f -p the head rsync process (there are three spawned for some reason), I see it sitting on a select:

Code:

write(1, "gbell/.PhpStorm2016.1/system/in"..., 145) = 145
read(4, "\0\275\0\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32768) = 32768
select(7, [6], [5], [6], {60, 0})       = 1 (in [6], left {59, 999994})
read(6, "\0\0\0\0\0\1\0\240\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\240\0\0\0\0\0"..., 114) = 114
select(6, [], [5], [], {60, 0}

That last select is the one taking the longest. Handle 5 is

Code:

rsync   15129 root    5u  unix 0xe0eb3980      0t0  5162991 socket

At this point I'm stuck.

This is with rsync-HEAD-20160630-1605GMT as well as the one that's current with Ubuntu 14.04.1 LTS (v3.1.0).

Here's my rsync command line.

Code:

rsync -rv --size-only --delay-updates --no-group --no-owner --no-perms --no-times --whole-file --exclude 'cache/' --exclude 'caches/' /data /mnt/remote/unencrypted

A simpler command line does the same thing (but is even slower since it's more interactive with the remote site due to trying to preserve permissions, times, etc.)

The --delay-updates I threw in there as an experiment, and it makes the initial flurry of lstats go very fast, but then it slows to the 2 seconds per file pace.

Targeting other machines over the internet is the same.

Targeting a machine on my LAN is fine (floods the connection). This is probably the least logical result.

Rsync using ssh transport to the same machine is fine (floods the connection).

What do I try or look at next?

gbell12 · 07-31-2016, 08:49 PM

Update: I mistakenly thought strace -f <parent pid> would trace all three rsync processes simultaneously. If I do strace all three, I see the last one is waiting on a select, as documented above, but the other two are doing 100s of seemingly useless operations like this:

Code:

lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage.values", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage.values.at", 0xbf89a944) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage.values.at", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage.values.s", 0xbf89a944) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage.values.s", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage_i", 0xbf89a944) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/.~tmp~/php.variable.shortName.storage_i", 0xbf89a8e4) = -1 ENOENT (No such file or directory)
lstat64(".PhpStorm2016.1/system/index/stubs/php.variable.shortname/php.variable.shortName.storage_i.len", 0xbf89a944) = -1 ENOENT (No such file or directory)

And this:

Code:

lstat64(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.~tmp~", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
lstat64(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction$1.class.xhqKFI", {st_mode=S_IFREG|0600, st_size=2035, ...}) = 0
chmod(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction$1.class.xhqKFI", 0640) = 0
rename(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction$1.class.xhqKFI", ".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.~tmp~/ShiftLeftVisualAction$1.class") = 0
open(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/ShiftLeftVisualAction.class", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open(".PhpStorm2016.1/config/plugins/IdeaVim/classes/com/maddyhome/idea/vim/action/change/shift/.ShiftLeftVisualAction.class.YtPv6X", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 1
fchmod(1, 0600)                         = 0

100's of lines of those, for each file rsync reports it's working on.

So am I looking at a question for the rsync devs?

Guttorm · 08-05-2016, 08:20 AM

Some years ago, I struggled with the same problems. I wanted to send a backup to a remote server with rsync, and I was worried about the remote server. It could get stolen or something, and I didn't want them to be able to read all the backup files. Like you, I had problems because it was terribly slow. I never really solved the problem, but I found 2 different workarounds.

1) Don't use sshfs. Instead just use encfs on the remote server, and use rsync normally (over ssh). Something like this:

echo "mysecretpassword" | ssh $remoteserver "encfs --stdinpass /home/.backup /home/backup"
rsync -a /data $remoteserver:/home/backup
ssh $remoteserver fusermount -u /home/backup

This is a lot faster. The downside is that you have to trust root on the remote server. Root can read all the files while the backup is being done, sniff the password, or do other nasty things. But in my situation, I was more worried someone would actually steal the server by breaking into the house. When the server is turned off, it's automaticly unmounted. So then all the files are encrypted, and the password to decrypt is not stored anywhere.

2) If you have lots of diskspace, you can use rsync locally first to an filesystem with encfs. Then unmount the enfs filesystem, and rsync the encrypted files. This is a safer solution, but it takes double the disk space on the local machine.

gbell12 · 08-05-2016, 08:40 AM

Thanks @Guttorm. The issue is that I need the files to land on the remote server already encrypted, because I don't trust root or other users.

chrism01 · 08-09-2016, 12:53 AM

Quote:

cp is fine (floods the connection).

so the cxn looks like a local disk (or did you mean scp ??)

If you have the diskspace locally, you could create an encrypted copy using gnupg, then just cp or scp or even ftp it ...

gbell12 · 08-09-2016, 05:04 PM

Quote:

Originally Posted by chrism01

so the cxn looks like a local disk (or did you mean scp ??)

Connection looks like a local disk because of sshfs. So I can 'cp' to it.

Quote:

Originally Posted by chrism01

If you have the diskspace locally, you could create an encrypted copy using gnupg, then just cp or scp or even ftp it ...

I do not have the space locally.

chrism01 · 08-10-2016, 03:31 AM

I'd say you definitely need to talk to the remote admins.

What it sounds like is that anything thing that looks like network socket cxn is bandwidth limited at one end or the other (or both).
(It may be case that even for an rsync that appears to a 'local' copy job, it still uses network sockets)
cp itself doesn't use an explicit network socket (although its a remote mnt 'underneath'). I'm not quite sure how cp really works for a remote mnt here.