LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-21-2015, 03:29 PM   #31
michaelk
Moderator
 
Registered: Aug 2002
Posts: 16,412

Rep: Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935Reputation: 1935

FYI it is possible to scp between two remote hosts but you need to use keys. I can't remember the version. From the man page

Quote:
-3 Copies between two remote hosts are transferred through the local
host. Without this option the data is copied directly between
the two remote hosts. Note that this option disables the
progress meter.
 
Old 05-21-2015, 03:36 PM   #32
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,556

Rep: Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116
For #2 see #4

Without the trailing slash with -a or -r, rsync will copy the directory and all of its contents. With the trailing slash and with -a or -r, rsync will only copy the contents of the directory.

So if you had:
Code:
$ ls /datafolder
dir1 dir2 dir3 file1 file2 file3
This would be the result:
Code:
$ rsync -a /datafolder /destination
$ ls /destination
datafolder
$ ls /destination/datafolder
dir1 dir2 dir3 file1 file2 file3
or
Code:
$ rsync -a /datafolder/ /destination
$ ls /destination
dir1 dir2 dir3 file1 file2 file3


Just put the commands in a script and call the script from cron. You'll want to add a lot of error checking/handling though.
 
Old 05-21-2015, 04:03 PM   #33
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 362

Rep: Reputation: 157Reputation: 157
Quote:
Originally Posted by tearsforhari View Post
Help. I really need to rsync from computer1 to computer2, using computer3. I am trying to sync a directory in two computers. I tried the following command on computer3:
>rsync -rav -e ssh user1@75.35.111.205:/home user2@75.35.208.199:/volume1/backups/
The source and destination cannot both be remote.
rsync error: syntax or usage error (code 1) at main.c(1166) [Receiver=3.0.9]

Any suggestions?
There is another way. Set up NFS so that Computer 3 can access the files on Computer 1:
* Computer 1 exports /home to Computer 3. Read-only access is sufficient.
* Computer 3 mounts it.
* Then Computer 3 can rsync from the mount point to Computer 2.

Last edited by Beryllos; 05-21-2015 at 04:05 PM.
 
Old 05-21-2015, 04:15 PM   #34
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 362

Rep: Reputation: 157Reputation: 157
I just read suicidaleggroll's suggestion to use sshfs. That looks similar to my suggestion to use NFS, except I guess sshfs is more secure and it may be easier to set up.
 
Old 05-21-2015, 04:42 PM   #35
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Thanks for the trailing / lesson. I have been a little unclear on that.

Code:
You'll want to add a lot of error checking/handling though.
How do you do that? Would it be something like this in a script.sh file:

sshfs user1@host1:/datafolder /some/temporary/location /dev/null 2>&1
rsync -av /some/temporary/location user2@host2:/mirrordatafolder /dev/null 2>&1
umount /some/temporary/location /dev/null 2>&1

Last edited by tearsforhari; 05-21-2015 at 04:48 PM.
 
Old 05-21-2015, 04:45 PM   #36
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
michaelk: Yes, SCP -3 user1@host1 user2@host2 copies just fine from computer3. But it isn't what I am looking for, because I just want to add new contents from computer1 to computer2--not overwrite each time.
 
Old 05-21-2015, 05:17 PM   #37
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
The amount of data that I am mirroring everyday is large 200GB to 1 TB. I'm unclear whether SSHFS actually writes (copies) the contents of the source directory to the created temporary directory, or rather instantaneously links the source directory to temporary mount directory that can be read by a destination directory without really copying anything?
 
Old 05-21-2015, 05:27 PM   #38
willc86
Member
 
Registered: Dec 2014
Posts: 55

Rep: Reputation: Disabled
great post; however, what about rsync and restoring from a backup. I am currently having problems restoring my entire root directory

rsync -avze 'ssh' willc86@venus:/home/backups/root /*

keeps saying If arg is a remote file/dir, prefix it with a colon (.
rsync error: syntax or usage error (code 1) at main.c(1362) [Receiver=3.1.0]

to better explain it, I am logged on saturn. I need the back up thats in venus to be rsync back to saturn since I messed up in saturn. I already backed up saturn to venus, but now i need that file back and its giving me that error.

so [saturn] <-----rsync the files--- [venus]
 
Old 05-21-2015, 06:39 PM   #39
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,556

Rep: Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116
Quote:
Originally Posted by tearsforhari View Post
The amount of data that I am mirroring everyday is large 200GB to 1 TB. I'm unclear whether SSHFS actually writes (copies) the contents of the source directory to the created temporary directory, or rather instantaneously links the source directory to temporary mount directory that can be read by a destination directory without really copying anything?
sshfs works like nfs, it's just a remote mount, it doesn't copy anything.
 
Old 05-21-2015, 06:41 PM   #40
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,556

Rep: Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116
Quote:
Originally Posted by willc86 View Post
great post; however, what about rsync and restoring from a backup. I am currently having problems restoring my entire root directory

rsync -avze 'ssh' willc86@venus:/home/backups/root /*

keeps saying If arg is a remote file/dir, prefix it with a colon (.
rsync error: syntax or usage error (code 1) at main.c(1362) [Receiver=3.1.0]

to better explain it, I am logged on saturn. I need the back up thats in venus to be rsync back to saturn since I messed up in saturn. I already backed up saturn to venus, but now i need that file back and its giving me that error.

so [saturn] <-----rsync the files--- [venus]
It's because your destination is a wildcard that's matching god knows how many locations. What rsync sees is not
Code:
rsync -avze 'ssh' user@ip:/home/backups/root /*
Because the globbing pattern is expanded by the shell before the call to rsync. What rsync sees is
Code:
rsync -avze 'ssh' user@ip:/home/backups/root /alt /bin /boot /dev /etc /home /lib /lib64 /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /usr /var
You need to stick the trailing slash on /home/backups/root/, since you're probably trying to copy the contents of that directory, rather than the directory itself, right? And the destination should just be "/".

Also, this really has nothing to do with the original question, and hijacking is not very polite.
 
Old 05-21-2015, 06:44 PM   #41
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,556

Rep: Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116
Quote:
Originally Posted by tearsforhari View Post
Thanks for the trailing / lesson. I have been a little unclear on that.

Code:
You'll want to add a lot of error checking/handling though.
How do you do that? Would it be something like this in a script.sh file:

sshfs user1@host1:/datafolder /some/temporary/location /dev/null 2>&1
rsync -av /some/temporary/location user2@host2:/mirrordatafolder /dev/null 2>&1
umount /some/temporary/location /dev/null 2>&1
That's error suppression, which is the opposite of error checking (not to mention the syntax is wrong, you're missing the >).

It would be more like
Code:
sshfs user1@host1:/datafolder /some/temporary/location
errstat=$?
if [[ $errstat -ne 0 ]]; then
   echo sshfs remote mount failed with error code $errstat, stopping
   exit 1
fi

rsync -av /home/temporary/location user2@host2:/mirrordatafolder
errstat=$?
if [[ $errstat -ne 0 ]]; then
   echo rsync failed with error code $errstat
fi

umount /some/temporary/location
 
Old 05-22-2015, 12:48 PM   #42
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Brilliant! It works!

I tried to modify the script to mail error reports, if you could take a glance:

Also, in my limited script writing, I have used "if [ "$?" = "1" ]; then" rather than "if [[ $errstat -ne 0 ]]; then". Would either work?

Is the script missing an "exit 1" in the second IF statement?



Code:
sshfs user1@host1:/datafolder /some/temporary/location > /tmp/SSHFSerrors.log
errstat=$?
if [[ $errstat -ne 0 ]]; then
  echo sshfs remote mount failed with error code $errstat, stopping
  mail -s "SSHFS Backup issue" user1@google.com < /tmp/SSHFSerrors.log
   exit 1
fi

rsync -av /home/temporary/location user2@host2:/mirrordatafolder > /tmp/RSYNCerrors.log
errstat=$?
if [[ $errstat -ne 0 ]]; then
   echo rsync failed with error code $errstat
   mail -s "SSHFS Backup issue" user1@google.com < /tmp/RSYNCerrors.log
fi

umount /some/temporary/location

Last edited by tearsforhari; 05-22-2015 at 02:40 PM.
 
Old 05-22-2015, 12:57 PM   #43
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,556

Rep: Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116Reputation: 2116
Quote:
Originally Posted by tearsforhari View Post
I tried to modify the script to mail error reports, if you could take a glance:
You appear to be trying to send /tmp/SSHFSerrors.log as the message, what is in that file?

Quote:
Originally Posted by tearsforhari View Post
Also, in my limited script writing, I have used "if [ "$?" = "1" ]; then" rather than "if [[ $errstat -ne 0 ]]; then". Would either work?
Three things:
1) Double brackets "[[ ]]" is recommended over single brackets "[ ]"

2) You do not want to check if the exit status is 1, you want to check if it's anything other than 0. Different failure modes will throw different error codes. You want to catch any of them, not just the error that happens to give you a code of 1

3) Using $? would work in the if statement, but $? gets replaced after every command, so you wouldn't be able to add it to the output, email, etc. Storing it in a variable lets you do whatever you want with it, without having to worry about it being replaced after every command.

Quote:
Originally Posted by tearsforhari View Post
Is the script missing an "exit 1" in the second IF statement?
There are many ways of handling errors. If you did put an exit 1 (or exit 2, to give it a unique exit status), you would need to duplicate the umount inside that if statement to keep it from lingering. Or umount it before the error test. It doesn't really matter, you just want to make sure you still run the umount even if the rsync fails.
 
Old 05-22-2015, 02:46 PM   #44
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Sorry, I corrected the code above to write an output log after each command. I just want a log file after each command and to mail if any errors.

Much obliged for the help!
 
Old 05-29-2015, 10:33 AM   #45
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Suicidaleggroll, anyone?

I wrote the above script called "myscript.sh" and put it in a directory with path /home/scripts/myscript.sh. It works just fine. I then set several similar cronjobs, which also work just fine. But I nevertheless get an error mailed as root:
Code:
/home/scripts/myscript1.sh: 13: /home/scripts/myscript1.sh: [[: not found
/home/scripts/myscript2.sh: 23: /home/scripts/myscript2.sh: [[: not found
/home/scripts/myscript3.sh: 33: /home/scripts/myscript3.sh: [[: not found
I am wondering if this has something to do with needing to add /home/scripts to the PATH of the root user. But I don't understand why it is mailing this error when it actually worked?

Last edited by tearsforhari; 05-29-2015 at 10:35 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] rsync between two different servers astalavista2000 Linux - Server 7 12-13-2010 10:10 PM
how to rsync files on two servers that have two different file systems mrotsliah Linux - General 2 08-11-2010 04:29 PM
rsync can not rsync files with include filter... xiutuo Linux - Server 2 07-23-2010 03:10 AM
which is the better way to rsync files between web servers? cooljai Linux - Server 8 05-22-2010 12:32 AM
Could I run rsync to download files from a server without rsync daemon? Richard.Yang Linux - Software 1 09-18-2009 05:08 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 12:48 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration