How to RSYNC files between two servers?

michaelk · 05-21-2015, 02:29 PM

FYI it is possible to scp between two remote hosts but you need to use keys. I can't remember the version. From the man page

Quote:

-3 Copies between two remote hosts are transferred through the local
host. Without this option the data is copied directly between
the two remote hosts. Note that this option disables the
progress meter.

suicidaleggroll · 05-21-2015, 02:36 PM

For #2 see #4

Without the trailing slash with -a or -r, rsync will copy the directory and all of its contents. With the trailing slash and with -a or -r, rsync will only copy the contents of the directory.

So if you had:

Code:

$ ls /datafolder
dir1 dir2 dir3 file1 file2 file3

This would be the result:

Code:

$ rsync -a /datafolder /destination
$ ls /destination
datafolder
$ ls /destination/datafolder
dir1 dir2 dir3 file1 file2 file3

or

Code:

$ rsync -a /datafolder/ /destination
$ ls /destination
dir1 dir2 dir3 file1 file2 file3

Just put the commands in a script and call the script from cron. You'll want to add a lot of error checking/handling though.

Beryllos · 05-21-2015, 03:03 PM

Quote:

Originally Posted by tearsforhari

Help. I really need to rsync from computer1 to computer2, using computer3. I am trying to sync a directory in two computers. I tried the following command on computer3:
>rsync -rav -e ssh user1@75.35.111.205:/home user2@75.35.208.199:/volume1/backups/
The source and destination cannot both be remote.
rsync error: syntax or usage error (code 1) at main.c(1166) [Receiver=3.0.9]

Any suggestions?

There is another way. Set up NFS so that Computer 3 can access the files on Computer 1:
* Computer 1 exports /home to Computer 3. Read-only access is sufficient.
* Computer 3 mounts it.
* Then Computer 3 can rsync from the mount point to Computer 2.

Beryllos · 05-21-2015, 03:15 PM

I just read suicidaleggroll's suggestion to use sshfs. That looks similar to my suggestion to use NFS, except I guess sshfs is more secure and it may be easier to set up.

tearsforhari · 05-21-2015, 03:42 PM

Thanks for the trailing / lesson. I have been a little unclear on that.

Code:

You'll want to add a lot of error checking/handling though.

How do you do that? Would it be something like this in a script.sh file:

sshfs user1@host1:/datafolder /some/temporary/location /dev/null 2>&1
rsync -av /some/temporary/location user2@host2:/mirrordatafolder /dev/null 2>&1
umount /some/temporary/location /dev/null 2>&1

tearsforhari · 05-21-2015, 03:45 PM

michaelk: Yes, SCP -3 user1@host1 user2@host2 copies just fine from computer3. But it isn't what I am looking for, because I just want to add new contents from computer1 to computer2--not overwrite each time.

tearsforhari · 05-21-2015, 04:17 PM

The amount of data that I am mirroring everyday is large 200GB to 1 TB. I'm unclear whether SSHFS actually writes (copies) the contents of the source directory to the created temporary directory, or rather instantaneously links the source directory to temporary mount directory that can be read by a destination directory without really copying anything?

willc86 · 05-21-2015, 04:27 PM

great post; however, what about rsync and restoring from a backup. I am currently having problems restoring my entire root directory

rsync -avze 'ssh' willc86@venus:/home/backups/root /*

keeps saying If arg is a remote file/dir, prefix it with a colon (

.
rsync error: syntax or usage error (code 1) at main.c(1362) [Receiver=3.1.0]

to better explain it, I am logged on saturn. I need the back up thats in venus to be rsync back to saturn since I messed up in saturn. I already backed up saturn to venus, but now i need that file back and its giving me that error.

so [saturn] <-----rsync the files--- [venus]

suicidaleggroll · 05-21-2015, 05:39 PM

Quote:

Originally Posted by tearsforhari

The amount of data that I am mirroring everyday is large 200GB to 1 TB. I'm unclear whether SSHFS actually writes (copies) the contents of the source directory to the created temporary directory, or rather instantaneously links the source directory to temporary mount directory that can be read by a destination directory without really copying anything?

sshfs works like nfs, it's just a remote mount, it doesn't copy anything.

suicidaleggroll · 05-21-2015, 05:41 PM

Quote:

Originally Posted by willc86

great post; however, what about rsync and restoring from a backup. I am currently having problems restoring my entire root directory

rsync -avze 'ssh' willc86@venus:/home/backups/root /*

keeps saying If arg is a remote file/dir, prefix it with a colon (

.
rsync error: syntax or usage error (code 1) at main.c(1362) [Receiver=3.1.0]

to better explain it, I am logged on saturn. I need the back up thats in venus to be rsync back to saturn since I messed up in saturn. I already backed up saturn to venus, but now i need that file back and its giving me that error.

so [saturn] <-----rsync the files--- [venus]

It's because your destination is a wildcard that's matching god knows how many locations. What rsync sees is not

Code:

rsync -avze 'ssh' user@ip:/home/backups/root /*

Because the globbing pattern is expanded by the shell before the call to rsync. What rsync sees is

Code:

rsync -avze 'ssh' user@ip:/home/backups/root /alt /bin /boot /dev /etc /home /lib /lib64 /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /usr /var

You need to stick the trailing slash on /home/backups/root/, since you're probably trying to copy the contents of that directory, rather than the directory itself, right? And the destination should just be "/".

Also, this really has nothing to do with the original question, and hijacking is not very polite.

suicidaleggroll · 05-21-2015, 05:44 PM

Quote:

Originally Posted by tearsforhari

Thanks for the trailing / lesson. I have been a little unclear on that.

Code:

You'll want to add a lot of error checking/handling though.

How do you do that? Would it be something like this in a script.sh file:

sshfs user1@host1:/datafolder /some/temporary/location /dev/null 2>&1
rsync -av /some/temporary/location user2@host2:/mirrordatafolder /dev/null 2>&1
umount /some/temporary/location /dev/null 2>&1

That's error suppression, which is the opposite of error checking (not to mention the syntax is wrong, you're missing the >).

It would be more like

Code:

sshfs user1@host1:/datafolder /some/temporary/location
errstat=$?
if [[ $errstat -ne 0 ]]; then
   echo sshfs remote mount failed with error code $errstat, stopping
   exit 1
fi

rsync -av /home/temporary/location user2@host2:/mirrordatafolder
errstat=$?
if [[ $errstat -ne 0 ]]; then
   echo rsync failed with error code $errstat
fi

umount /some/temporary/location

tearsforhari · 05-22-2015, 11:48 AM

Brilliant! It works!

I tried to modify the script to mail error reports, if you could take a glance:

Also, in my limited script writing, I have used "if [ "$?" = "1" ]; then" rather than "if [[ $errstat -ne 0 ]]; then". Would either work?

Is the script missing an "exit 1" in the second IF statement?

Code:

sshfs user1@host1:/datafolder /some/temporary/location > /tmp/SSHFSerrors.log
errstat=$?
if [[ $errstat -ne 0 ]]; then
  echo sshfs remote mount failed with error code $errstat, stopping
  mail -s "SSHFS Backup issue" user1@google.com < /tmp/SSHFSerrors.log
   exit 1
fi

rsync -av /home/temporary/location user2@host2:/mirrordatafolder > /tmp/RSYNCerrors.log
errstat=$?
if [[ $errstat -ne 0 ]]; then
   echo rsync failed with error code $errstat
   mail -s "SSHFS Backup issue" user1@google.com < /tmp/RSYNCerrors.log
fi

umount /some/temporary/location

suicidaleggroll · 05-22-2015, 11:57 AM

Quote:

Originally Posted by tearsforhari

I tried to modify the script to mail error reports, if you could take a glance:

You appear to be trying to send /tmp/SSHFSerrors.log as the message, what is in that file?

Quote:

Originally Posted by tearsforhari

Also, in my limited script writing, I have used "if [ "$?" = "1" ]; then" rather than "if [[ $errstat -ne 0 ]]; then". Would either work?

Three things:
1) Double brackets "[[ ]]" is recommended over single brackets "[ ]"

2) You do not want to check if the exit status is 1, you want to check if it's anything other than 0. Different failure modes will throw different error codes. You want to catch any of them, not just the error that happens to give you a code of 1

3) Using $? would work in the if statement, but $? gets replaced after every command, so you wouldn't be able to add it to the output, email, etc. Storing it in a variable lets you do whatever you want with it, without having to worry about it being replaced after every command.

Quote:

Originally Posted by tearsforhari

Is the script missing an "exit 1" in the second IF statement?

There are many ways of handling errors. If you did put an exit 1 (or exit 2, to give it a unique exit status), you would need to duplicate the umount inside that if statement to keep it from lingering. Or umount it before the error test. It doesn't really matter, you just want to make sure you still run the umount even if the rsync fails.

tearsforhari · 05-22-2015, 01:46 PM

Sorry, I corrected the code above to write an output log after each command. I just want a log file after each command and to mail if any errors.

Much obliged for the help!

tearsforhari · 05-29-2015, 09:33 AM

Suicidaleggroll, anyone?

I wrote the above script called "myscript.sh" and put it in a directory with path /home/scripts/myscript.sh. It works just fine. I then set several similar cronjobs, which also work just fine. But I nevertheless get an error mailed as root:

Code:

/home/scripts/myscript1.sh: 13: /home/scripts/myscript1.sh: [[: not found
/home/scripts/myscript2.sh: 23: /home/scripts/myscript2.sh: [[: not found
/home/scripts/myscript3.sh: 33: /home/scripts/myscript3.sh: [[: not found

I am wondering if this has something to do with needing to add /home/scripts to the PATH of the root user. But I don't understand why it is mailing this error when it actually worked?