LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   How to RSYNC files between two servers? (https://www.linuxquestions.org/questions/linux-newbie-8/how-to-rsync-files-between-two-servers-4175542424/)

tearsforhari 05-12-2015 08:35 PM

How to RSYNC files between two servers?
 
Hello--can someone tell me how to RSYNC bteween two servers with IP addresses, username and passwords while removing the files at the senders end? Do I need to login as root on both servers? I appreciate your input.

Ztcoracat 05-12-2015 09:24 PM

Hi:

These links might help.
http://www.tomsitpro.com/articles/li...t,2-777-4.html
http://www.tecmint.com/sync-two-apac...s-using-rsync/
http://www.sysadminshare.com/2012/05...tween-two.html

I don't know if you should be logged into both servers as root or not:-

evo2 05-12-2015 09:41 PM

Hi,

you need to learn to use rm, rsync and (indirectly ssh).

Eg to rysnc from server1 to server2 (with IP address 192.168.1.234), get a shell on server1, then run rsync. Eg.

Code:

rsync -auv -e ssh somedir user2@192.168.1.234:
Then once you are completely sure that you copied the "somedir" correctly you can remove it and its contents with 'rm -r'.

Whether or not you need root depends on the permissions of the files in question.

Evo2.

tearsforhari 05-12-2015 11:17 PM

Yes, thank you. It is slowing coming back to me now. But in your code, I believe that there are some missing parts. FYI: I am trying to write a crontab to do this automatically everyday. So, I am wondering about the following modifiers:

1. Rsync has a --remove-system-file modifier that I believe will remove files from Server 1, no?

2. I am wondering about how ssh will know the password of user2 in your example. Reading one of the links, I believe the ssh-keygen command enables this, no?

How would the code combine to include both these command above?

evo2 05-12-2015 11:29 PM

Hi,

your original post never mentioned anything about automating this. If you want this to be automated (eg with cron) you should probably use public key authentication (with the key unprotected by a passphrase).
You can use ssh-keygen and ssh-copy-id to set it up.

I'm not familiar with the --remove-system-file option and it is not documented in the rsync man page (on my Debian 8.0/ system).

Evo2.

tearsforhari 05-13-2015 12:26 AM

Sorry, that should be --remove-source-files for Rsync. Where would I put that in your code? I am little unclear on whether that would retain older files on the destination folder (Server 2) that were previously deleted on the source folder (Server 1)?

In the links provided, the ssh-genkey is implemented separate from the Rsync code. But I remember including that in the code itself. I am setting up several crontabs to rsync between different external computers (Server 1 and Server 2, Server 1 and Server 3). I am a little worried about or unclear setting up multiple ssh-keys. I believe that the ssh-key could be implemented in the code. Is that correct?

I have never heard of ssh-copy-id. How is that different than ssh-genkey?

chrism01 05-13-2015 05:01 AM

ssh-keygen & ssh-copy-id http://www.cyberciti.biz/faq/install...y-remote-host/

evo2 05-13-2015 06:48 PM

Hi,
Quote:

Originally Posted by tearsforhari (Post 5361704)
Sorry, that should be --remove-source-files for Rsync. Where would I put that in your code?

It is just an commandline option to rsync, so put it with the other options. Are you experienced in using the commandline? If not I strongly suggest you spend some time on the basics before trying to implement this. Eg learn and use cd, ls, pwd, mkdir, cp, mv, rm, cat, less, etc. Then learn about basic shell scripts. There are plenty of tutorials online.

Quote:

Originally Posted by tearsforhari (Post 5361704)
I am little unclear on whether that would retain older files on the destination folder rver 2) that were previously deleted on the source folder (Server 1)?

Is the documentation unclear? What does it say?


Quote:

Originally Posted by tearsforhari (Post 5361704)
In the links provided, the ssh-genkey is implemented separate from the Rsync code. But I remember including that in the code itself. I am setting up several crontabs to rsync between different external computers (Server 1 and Server 2, Server 1 and Server 3). I am a little worried about or unclear setting up multiple ssh-keys. I believe that the ssh-key could be implemented in the code. Is that correct?

I'm assuming that when you write "ssh-genkey" and "ssh-key" you actually mean "ssh-keygen". But what do you mean by "implemented in the code"? Each client should have its own private key with the corresponding public key installed on the server (here client means the machine on which rsync is run, and server is the other machine).

Quote:

Originally Posted by tearsforhari (Post 5361704)
I have never heard of ssh-copy-id. How is that different than ssh-genkey?

It is a helpful little script for installing public keys. It exists primarily so that people can avoid common mistakes when installing keys (eg wrong location, wrong permissions, overwriting other keys, etc). Look it up.

Evo2.

Beryllos 05-13-2015 08:50 PM

Quote:

Originally Posted by tearsforhari (Post 5361704)
Sorry, that should be --remove-source-files for Rsync. Where would I put that in your code? I am little unclear on whether that would retain older files on the destination folder (Server 2) that were previously deleted on the source folder (Server 1)?

It would retain them. To investigate questions like this, try rsync between two folders that you don't need. The folders don't have to be on different servers to find out how most rsync options work.

Example:
Code:

$ mkdir tempdir1
$ touch tempdir1/file1
$ ls tempdir1
file1
$ rsync -a --remove-source-files tempdir1/ tempdir2/
$ ls tempdir1
$ ls tempdir2
file1
$ touch tempdir1/file2
$ rsync -a --remove-source-files tempdir1/ tempdir2/
$ ls tempdir1
$ ls tempdir2
file1  file2
$ touch tempdir1/file3
$ rsync -a --delete tempdir1/ tempdir2/
$ ls tempdir1
file3
$ ls tempdir2
file3
$

So you see that --remove-source-files removes files from the source after copying them. Since you asked, the option which does not retain older files on the destination folder that were previously deleted on the source folder is --delete. It removes files from the destination if they are not present in the source.

tearsforhari 05-13-2015 10:34 PM

Great Example! Yes, indeed, rsync can be used for directories within a server, too. That was very clear Beryllos.

So far, my code has taken this shape:

Code:

>rsync -rave --remove-source-files ssh remoteuser@remotehost:/remote/dir /this/dir/
From the links that people have posted, I can generate a file from ssh-keygen so that the password of the remoteuser on the remotehost can be read. For some reason, I thought that the ssh-keygen was part of the code; but perhaps I'm mistaken. My only question left is that if I put this into a crontab, which directory does the ssh-keygen file need to located to be read? I understand that this file also has to be copied on the other server, too. But where?

Thanks guys!

evo2 05-14-2015 01:15 AM

Hi,
Quote:

Originally Posted by tearsforhari (Post 5362206)
So far, my code has taken this shape:

Code:

>rsync -rave --remove-source-files ssh remoteuser@remotehost:/remote/dir /this/dir/

The "ssh" should be an argument to the -e option. Eg
Code:

>rsync -rav -e ssh --remove-source-files remoteuser@remotehost:/remote/dir /this/dir/
However it won't be needed if the RSYNC_RSH environment variable is set to ssh.
Quote:

Originally Posted by tearsforhari (Post 5362206)
From the links that people have posted, I can generate a file from ssh-keygen so that the password of the remoteuser on the remotehost can be read. For some reason, I thought that the ssh-keygen was part of the code; but perhaps I'm mistaken. My only question left is that if I put this into a crontab, which directory does the ssh-keygen file need to located to be read? I understand that this file also has to be copied on the other server, too. But where?

The command ssh-keygen generates two files:
1. The private key (eg ~/.ssh/id_rsa)
2. The corresponding public key (eg ~/ssh/id_rsa.pub)
The contents of the public key file needs to be put into the ~/.ssh/authorized_keys file (or similar) on the remote server. You can use the ssh-copy-id script to do this.

Evo2.

tearsforhari 05-14-2015 11:33 PM

Thank you, Evo. That was very helpful. I am having one problem and have another question:

1. I am trying to rsync to synology, and the ssh-copy-id cannot write to the disk. Any suggestions?

2. How would I go about setting rsync and sshkey-gen on a third party linux box, where computer 3 accesses files from computer 1 and transfers them to computer 2?

evo2 05-18-2015 03:33 AM

Hi,
Quote:

Originally Posted by tearsforhari (Post 5362630)
1. I am trying to rsync to synology, and the ssh-copy-id cannot write to the disk. Any suggestions?

What is synology? Exactly what error message are you seeing?

Quote:

Originally Posted by tearsforhari (Post 5362630)
2. How would I go about setting rsync and sshkey-gen on a third party linux box, where computer 3 accesses files from computer 1 and transfers them to computer 2?

What do you mean by "setting"? Do you mean installing? If so, then by using the package manager for that system. Why does there need to be a 3rd computer in this scenario? Why can't you copy directly from computer 1 to computer 2?

Evo2.

michaelk 05-18-2015 08:05 AM

I assume that Synology is in reference to a NAS box.

Did you enable ssh/sftp via its control panel? Did you try copying the keys to a regular user (if any were created) or root? The .ssh directory may not exist or if copying to a regular user, home services is not enabled which allows regular users to access their home directories.

tearsforhari 05-19-2015 11:03 AM

Sorry. By Synology, I mean NAS. (I'm a true newbie.)

Here is my problem: when I do the following command, I get an error:
>rsync -av -e ssh --remove-source-files /home/ftp_user/stage/datafolder root@75.36.201.198::backups/system/MachineName/stage
root@75.36.201.198's password:
Password:
sending incremental file list
datafolder/
rsync: failed to set times on "/system/MachineName/stage/datafolder" (in backups): Operation not permitted (1)
datafolder/filename1.zip
datafolder/filename2.zip
rsync: mkstemp "/system/MachineName/stage/datafolder/.filename1.wv1RXC" (in backups) failed: Permission denied (13)
rsync: mkstemp "/system/MachineName/stage/datafolder/.filename2.RfXual" (in backups) failed: Permission denied (13)
sent 1891165115 bytes received 107 bytes 13753928.89 bytes/sec
total size is 1890933885 speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]


However, when I implement the same command using a directory (testfolder containing the same files) that I made by myself, I get no error.

>rsync -av -e ssh --remove-source-files /home/ftp_user/stage/testfolder root@75.36.201.198::backups/system/MachineName/stage

root@'s password:
Permission denied, please try again.
root@75.36.201.198's password:
Password:
sending incremental file list

sent 33 bytes received 9 bytes 1.95 bytes/sec
total size is 0 speedup is 0.00

Now, I thought there are permission issues as owners and groups of the directories and files contained within them. Previously the owner and group were ftp_user. But changing permissions to root for both owner and group did not solve the problem as illustrated above:
>ls -als
4 drwxrwxr-x 2 root root 4096 May 15 17:35 testfolder
4 drwxr-xr-x 2 root root 4096 May 14 15:05 datafolder

So, why is the folder "testfolder" that I made able to rsync, while the "datafolder" that was made by a shell script (that made the zip files) not able to rync?


All times are GMT -5. The time now is 01:02 AM.