LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-12-2015, 09:35 PM   #1
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Rep: Reputation: Disabled
How to RSYNC files between two servers?


Hello--can someone tell me how to RSYNC bteween two servers with IP addresses, username and passwords while removing the files at the senders end? Do I need to login as root on both servers? I appreciate your input.
 
Old 05-12-2015, 10:24 PM   #2
Ztcoracat
LQ Guru
 
Registered: Dec 2011
Distribution: Slackware
Posts: 6,407
Blog Entries: 6

Rep: Reputation: Disabled
Hi:

These links might help.
http://www.tomsitpro.com/articles/li...t,2-777-4.html
http://www.tecmint.com/sync-two-apac...s-using-rsync/
http://www.sysadminshare.com/2012/05...tween-two.html

I don't know if you should be logged into both servers as root or not:-
 
Old 05-12-2015, 10:41 PM   #3
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,753

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
Hi,

you need to learn to use rm, rsync and (indirectly ssh).

Eg to rysnc from server1 to server2 (with IP address 192.168.1.234), get a shell on server1, then run rsync. Eg.

Code:
rsync -auv -e ssh somedir user2@192.168.1.234:
Then once you are completely sure that you copied the "somedir" correctly you can remove it and its contents with 'rm -r'.

Whether or not you need root depends on the permissions of the files in question.

Evo2.
 
Old 05-13-2015, 12:17 AM   #4
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Yes, thank you. It is slowing coming back to me now. But in your code, I believe that there are some missing parts. FYI: I am trying to write a crontab to do this automatically everyday. So, I am wondering about the following modifiers:

1. Rsync has a --remove-system-file modifier that I believe will remove files from Server 1, no?

2. I am wondering about how ssh will know the password of user2 in your example. Reading one of the links, I believe the ssh-keygen command enables this, no?

How would the code combine to include both these command above?
 
Old 05-13-2015, 12:29 AM   #5
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,753

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
Hi,

your original post never mentioned anything about automating this. If you want this to be automated (eg with cron) you should probably use public key authentication (with the key unprotected by a passphrase).
You can use ssh-keygen and ssh-copy-id to set it up.

I'm not familiar with the --remove-system-file option and it is not documented in the rsync man page (on my Debian 8.0/ system).

Evo2.
 
Old 05-13-2015, 01:26 AM   #6
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Sorry, that should be --remove-source-files for Rsync. Where would I put that in your code? I am little unclear on whether that would retain older files on the destination folder (Server 2) that were previously deleted on the source folder (Server 1)?

In the links provided, the ssh-genkey is implemented separate from the Rsync code. But I remember including that in the code itself. I am setting up several crontabs to rsync between different external computers (Server 1 and Server 2, Server 1 and Server 3). I am a little worried about or unclear setting up multiple ssh-keys. I believe that the ssh-key could be implemented in the code. Is that correct?

I have never heard of ssh-copy-id. How is that different than ssh-genkey?
 
Old 05-13-2015, 06:01 AM   #7
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
ssh-keygen & ssh-copy-id http://www.cyberciti.biz/faq/install...y-remote-host/
 
Old 05-13-2015, 07:48 PM   #8
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,753

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
Hi,
Quote:
Originally Posted by tearsforhari View Post
Sorry, that should be --remove-source-files for Rsync. Where would I put that in your code?
It is just an commandline option to rsync, so put it with the other options. Are you experienced in using the commandline? If not I strongly suggest you spend some time on the basics before trying to implement this. Eg learn and use cd, ls, pwd, mkdir, cp, mv, rm, cat, less, etc. Then learn about basic shell scripts. There are plenty of tutorials online.

Quote:
Originally Posted by tearsforhari View Post
I am little unclear on whether that would retain older files on the destination folder rver 2) that were previously deleted on the source folder (Server 1)?
Is the documentation unclear? What does it say?


Quote:
Originally Posted by tearsforhari View Post
In the links provided, the ssh-genkey is implemented separate from the Rsync code. But I remember including that in the code itself. I am setting up several crontabs to rsync between different external computers (Server 1 and Server 2, Server 1 and Server 3). I am a little worried about or unclear setting up multiple ssh-keys. I believe that the ssh-key could be implemented in the code. Is that correct?
I'm assuming that when you write "ssh-genkey" and "ssh-key" you actually mean "ssh-keygen". But what do you mean by "implemented in the code"? Each client should have its own private key with the corresponding public key installed on the server (here client means the machine on which rsync is run, and server is the other machine).

Quote:
Originally Posted by tearsforhari View Post
I have never heard of ssh-copy-id. How is that different than ssh-genkey?
It is a helpful little script for installing public keys. It exists primarily so that people can avoid common mistakes when installing keys (eg wrong location, wrong permissions, overwriting other keys, etc). Look it up.

Evo2.
 
Old 05-13-2015, 09:50 PM   #9
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 304

Rep: Reputation: 121Reputation: 121
Quote:
Originally Posted by tearsforhari View Post
Sorry, that should be --remove-source-files for Rsync. Where would I put that in your code? I am little unclear on whether that would retain older files on the destination folder (Server 2) that were previously deleted on the source folder (Server 1)?
It would retain them. To investigate questions like this, try rsync between two folders that you don't need. The folders don't have to be on different servers to find out how most rsync options work.

Example:
Code:
$ mkdir tempdir1
$ touch tempdir1/file1
$ ls tempdir1
file1
$ rsync -a --remove-source-files tempdir1/ tempdir2/
$ ls tempdir1
$ ls tempdir2
file1
$ touch tempdir1/file2
$ rsync -a --remove-source-files tempdir1/ tempdir2/
$ ls tempdir1
$ ls tempdir2
file1  file2
$ touch tempdir1/file3
$ rsync -a --delete tempdir1/ tempdir2/
$ ls tempdir1
file3
$ ls tempdir2
file3
$
So you see that --remove-source-files removes files from the source after copying them. Since you asked, the option which does not retain older files on the destination folder that were previously deleted on the source folder is --delete. It removes files from the destination if they are not present in the source.
 
Old 05-13-2015, 11:34 PM   #10
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Great Example! Yes, indeed, rsync can be used for directories within a server, too. That was very clear Beryllos.

So far, my code has taken this shape:

Code:
>rsync -rave --remove-source-files ssh remoteuser@remotehost:/remote/dir /this/dir/
From the links that people have posted, I can generate a file from ssh-keygen so that the password of the remoteuser on the remotehost can be read. For some reason, I thought that the ssh-keygen was part of the code; but perhaps I'm mistaken. My only question left is that if I put this into a crontab, which directory does the ssh-keygen file need to located to be read? I understand that this file also has to be copied on the other server, too. But where?

Thanks guys!
 
Old 05-14-2015, 02:15 AM   #11
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,753

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
Hi,
Quote:
Originally Posted by tearsforhari View Post
So far, my code has taken this shape:

Code:
>rsync -rave --remove-source-files ssh remoteuser@remotehost:/remote/dir /this/dir/
The "ssh" should be an argument to the -e option. Eg
Code:
>rsync -rav -e ssh --remove-source-files remoteuser@remotehost:/remote/dir /this/dir/
However it won't be needed if the RSYNC_RSH environment variable is set to ssh.
Quote:
Originally Posted by tearsforhari View Post
From the links that people have posted, I can generate a file from ssh-keygen so that the password of the remoteuser on the remotehost can be read. For some reason, I thought that the ssh-keygen was part of the code; but perhaps I'm mistaken. My only question left is that if I put this into a crontab, which directory does the ssh-keygen file need to located to be read? I understand that this file also has to be copied on the other server, too. But where?
The command ssh-keygen generates two files:
1. The private key (eg ~/.ssh/id_rsa)
2. The corresponding public key (eg ~/ssh/id_rsa.pub)
The contents of the public key file needs to be put into the ~/.ssh/authorized_keys file (or similar) on the remote server. You can use the ssh-copy-id script to do this.

Evo2.
 
Old 05-15-2015, 12:33 AM   #12
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Thank you, Evo. That was very helpful. I am having one problem and have another question:

1. I am trying to rsync to synology, and the ssh-copy-id cannot write to the disk. Any suggestions?

2. How would I go about setting rsync and sshkey-gen on a third party linux box, where computer 3 accesses files from computer 1 and transfers them to computer 2?
 
Old 05-18-2015, 04:33 AM   #13
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,753

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
Hi,
Quote:
Originally Posted by tearsforhari View Post
1. I am trying to rsync to synology, and the ssh-copy-id cannot write to the disk. Any suggestions?
What is synology? Exactly what error message are you seeing?

Quote:
Originally Posted by tearsforhari View Post
2. How would I go about setting rsync and sshkey-gen on a third party linux box, where computer 3 accesses files from computer 1 and transfers them to computer 2?
What do you mean by "setting"? Do you mean installing? If so, then by using the package manager for that system. Why does there need to be a 3rd computer in this scenario? Why can't you copy directly from computer 1 to computer 2?

Evo2.
 
Old 05-18-2015, 09:05 AM   #14
michaelk
Moderator
 
Registered: Aug 2002
Posts: 14,938

Rep: Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520Reputation: 1520
I assume that Synology is in reference to a NAS box.

Did you enable ssh/sftp via its control panel? Did you try copying the keys to a regular user (if any were created) or root? The .ssh directory may not exist or if copying to a regular user, home services is not enabled which allows regular users to access their home directories.
 
Old 05-19-2015, 12:03 PM   #15
tearsforhari
Member
 
Registered: Mar 2015
Posts: 79

Original Poster
Rep: Reputation: Disabled
Sorry. By Synology, I mean NAS. (I'm a true newbie.)

Here is my problem: when I do the following command, I get an error:
>rsync -av -e ssh --remove-source-files /home/ftp_user/stage/datafolder root@75.36.201.198::backups/system/MachineName/stage
root@75.36.201.198's password:
Password:
sending incremental file list
datafolder/
rsync: failed to set times on "/system/MachineName/stage/datafolder" (in backups): Operation not permitted (1)
datafolder/filename1.zip
datafolder/filename2.zip
rsync: mkstemp "/system/MachineName/stage/datafolder/.filename1.wv1RXC" (in backups) failed: Permission denied (13)
rsync: mkstemp "/system/MachineName/stage/datafolder/.filename2.RfXual" (in backups) failed: Permission denied (13)
sent 1891165115 bytes received 107 bytes 13753928.89 bytes/sec
total size is 1890933885 speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]


However, when I implement the same command using a directory (testfolder containing the same files) that I made by myself, I get no error.

>rsync -av -e ssh --remove-source-files /home/ftp_user/stage/testfolder root@75.36.201.198::backups/system/MachineName/stage

root@'s password:
Permission denied, please try again.
root@75.36.201.198's password:
Password:
sending incremental file list

sent 33 bytes received 9 bytes 1.95 bytes/sec
total size is 0 speedup is 0.00

Now, I thought there are permission issues as owners and groups of the directories and files contained within them. Previously the owner and group were ftp_user. But changing permissions to root for both owner and group did not solve the problem as illustrated above:
>ls -als
4 drwxrwxr-x 2 root root 4096 May 15 17:35 testfolder
4 drwxr-xr-x 2 root root 4096 May 14 15:05 datafolder

So, why is the folder "testfolder" that I made able to rsync, while the "datafolder" that was made by a shell script (that made the zip files) not able to rync?

Last edited by tearsforhari; 05-19-2015 at 12:09 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] rsync between two different servers astalavista2000 Linux - Server 7 12-13-2010 10:10 PM
how to rsync files on two servers that have two different file systems mrotsliah Linux - General 2 08-11-2010 04:29 PM
rsync can not rsync files with include filter... xiutuo Linux - Server 2 07-23-2010 03:10 AM
which is the better way to rsync files between web servers? cooljai Linux - Server 8 05-22-2010 12:32 AM
Could I run rsync to download files from a server without rsync daemon? Richard.Yang Linux - Software 1 09-18-2009 05:08 AM


All times are GMT -5. The time now is 03:39 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration