[SOLVED] Batching Copying Files Across Network

NobleOne · 07-09-2017, 11:43 AM

I've read a lot of posts on multiple sites about this, but not one post that seems to combine BOTH the ability to BATCH copy files over a network (i.e. scp) with a secure solution.

The goal is to routinely backup files from Linux box A to Linux box B via cron job. So obviously, entering a password isn't an option, and using a password-less login is of course not secure. So how best to get this done?

Turbocapitalist · 07-09-2017, 12:13 PM

Usually rsync is a good way to go for transferring multiple files. The sftp program works, too, but cannot easily resume if interrupted. Both work over SSH so that means you can use keys.

Code:

rsync -a -v -H -e "ssh -i ~/.ssh/somekey.rsa" \
        /source/dir/ user@remote.example.com:/dest/dir/

Usually if you do not use a passphrase on your key you should lock it down substantially with a forced command on the server. Otherwise, with a little bit of configuration, you can use an agent to hold the key for that connection.

NobleOne · 07-09-2017, 12:28 PM

Thanks. Since your post, I have been reading up on rsync and rsync_t and my eyes are glazing over. Seems pretty complicated.

Turbocapitalist · 07-09-2017, 12:33 PM

You can go quite deep, but that is not really necessary. With password authentication, via SSH, you would run it like this:

Code:

rsync -a -v -H \
        /source/dir/ user@remote.example.com:/dest/dir/

And to use a key see, the previous example in post #2.

Can you give more details about the transfer you have in mind and how you have used rsync so far. Also, how have you been using keys for regular SSH connections?

NobleOne · 07-09-2017, 12:36 PM

Thanks! I'll give it a try.

szboardstretcher · 07-09-2017, 12:38 PM

Passwordless SSH keys are not insecure. Its the standard way of unattended copying to remote hosts.

Set up your passwordless SSH keys under your preferred user: http://www.linuxproblem.org/art_9.html

That will probably save your key as .ssh/id_rsa.pub unless you specify otherwise.

Open up your cron as that user:

Code:

su - user1
crontab -e

add your rsync command to the cron and save (for example every 30 minutes):

Code:

*/30 * * * * rsync -varh -e "ssh -i ~/.ssh/id_rsa.pub" /local/source/dir/ remoteuser@remotemachine:/remote/dest/dir/

Keep in mind that the user you are setting SSH keys up for has to have permissions to the local files you are trying to copy.

NobleOne · 07-09-2017, 12:41 PM

Super, sz. I think I can handle this. I love this forum!

Turbocapitalist · 07-09-2017, 12:45 PM

Quote:

Originally Posted by szboardstretcher

That will probably save your key as .ssh/id_rsa.pub unless you specify otherwise.

I'd recommend specifying a unique filename for the key pair so that later when you have more than one it will be easier to keep track of them. Likewise embedding a comment in the public key helps, too, once you have several public keys together. The -f and -C options do that. See "man sshkey-gen"

NobleOne · 07-09-2017, 12:47 PM

Got it, thanks again!