LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-26-2020, 09:22 AM   #1
atomiccomp
Member
 
Registered: Dec 2008
Location: UK
Distribution: Debian/Ubuntu
Posts: 41

Rep: Reputation: 0
Rsync question


After advice I received on this excellent forum I plan to start using rsync rather than SFTP to back up remotely.

My question is if I start the job from a crontab, and the following day the session is still backing up what happens when the crontab runs again?

Will I end up with two different rsync proccesess at different points of the backup running?

I just want to make sure I don't get corrupt data or files missed on a backup.
 
Old 03-26-2020, 09:41 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,873

Rep: Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481
that mainly depends on the configuration. In general there will be two different rsync processes, but they will not conflict with each other (if configured properly). And also you may configure your system to avoid parallel execution too.
 
Old 03-26-2020, 10:10 AM   #3
atomiccomp
Member
 
Registered: Dec 2008
Location: UK
Distribution: Debian/Ubuntu
Posts: 41

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pan64 View Post
that mainly depends on the configuration. In general there will be two different rsync processes, but they will not conflict with each other (if configured properly). And also you may configure your system to avoid parallel execution too.

So I have my own debian server which will connect to about 10 remote file servers and run; rsync -a username@myhost:/home/clientsdata /clientdataonmyserver/clientname

I plan to run it through a crontab with 10 different sessions an hour appart one for each client.

Would this work OK?

Also It would be nice if I could make it so whatever is on my server doesn't get deleted if the remote directory deletes a file so I have a backup if a user accidently deletes a file.
 
Old 03-26-2020, 10:24 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,873

Rep: Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481Reputation: 4481
if you use rsync to copy the files into the same dir then you may have conflicts. You need to be sure only one rsync is working on that dir. By the way you need to know how long does it take at all. Subsequent calls usually will run much faster than the first invocation.
From the other hand there is another question: if a user [accidentally] modifies a file which one do you want to backup? The old or the new one?
 
Old 03-26-2020, 10:29 AM   #5
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,576
Blog Entries: 3

Rep: Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297
Make sure that the directories passed to rsync are identified as such by ending them with slashes so that you don't get the wrong result:

Code:
rsync -a username@myhost:/home/clientsdata/ /clientdataonmyserver/clientname/
If this is scripted, you'll probably need to have the authentication happen using an SSH key:

Code:
rsync -a -e 'ssh -i /home/you/.ssh/some_key_ed25519'\
        username@myhost:/home/clientsdata/ /clientdataonmyserver/clientname/
Then on the server side you can lock down the key so that all it can do is to run rsync with only those preset parameters. That is done by prepending command="..." into the key on the server account's authorized_keys file. The value for "..." would be determined by adding a -v to the SSH client above. But it would look something like this:

Code:
command="/usr/bin/rsync --server --sender -logDtpre.iLsfxC . /home/clientsdata/" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMtBe1tMc1LrccAPUX15cOVwWzfqnRCgXhwrNRrjyLLx user source host myhost
If you are worried about load, you might throw in ionice in there, too.

As for your main question, you could have the cron job call a shell script which in turn calls rsync but first calls flock. See "man 1 flock" about that. Through it, you would ensure that only one instance of the script is active at a time.
 
1 members found this post helpful.
Old 03-26-2020, 11:12 AM   #6
atomiccomp
Member
 
Registered: Dec 2008
Location: UK
Distribution: Debian/Ubuntu
Posts: 41

Original Poster
Rep: Reputation: 0
Thanks for the replies.

I've tested log in via private/public key and it works great.

How can I stop multiple rsyncs running? For example some days there may only be 10 small files to sync (10mb 10 minutes) or another day a user may drop 1 GB in and it takes 2 days to sync.
 
Old 03-26-2020, 11:19 AM   #7
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,576
Blog Entries: 3

Rep: Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297Reputation: 2297
Rather than call rsync directly from cron, have it call a shell script. Have that shell script check in with lock file using flock first.

That would be my own approach, though there are also other ways.
 
1 members found this post helpful.
Old 03-26-2020, 12:24 PM   #8
agillator
Member
 
Registered: Aug 2016
Distribution: Mint 19.1
Posts: 306

Rep: Reputation: Disabled
My recommendation is to NOT use a cron job the first time. The reason: rsync or no it will have to copy EVERYTHING and that can take a long,long time (24 hours or more if you have a LOT of files). Run it from the terminal in the background or from 'screen' and let it go. After it finishes you can start using cron and it shouldn't take long.
 
1 members found this post helpful.
Old 03-26-2020, 01:18 PM   #9
michaelk
Moderator
 
Registered: Aug 2002
Posts: 19,692

Rep: Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215Reputation: 3215
rysnc will only delete files at the destination if you select one of the delete options. Additional clarification as to what you meany by drop files. I assume these are new files not currently in the client's backup directory.

The only problem might be if your clients continuously drop or edit files into the directory while rsync is still running. Granted rsync is smart but I don't think it looks for open files which means a "long" rsync might not copy everything but technically should at the next run.

You have not provided any specific hardware or upload/download speed information but the speed limitation most likely is due to the upload speed of the client versus running multiple rysnc's at the same time on the server but I agree that ionice is something to consider.

Is this on the same LAN or remote clients?

Last edited by michaelk; 03-26-2020 at 01:20 PM.
 
Old 03-26-2020, 02:09 PM   #10
atomiccomp
Member
 
Registered: Dec 2008
Location: UK
Distribution: Debian/Ubuntu
Posts: 41

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by michaelk View Post
rysnc will only delete files at the destination if you select one of the delete options. Additional clarification as to what you meany by drop files. I assume these are new files not currently in the client's backup directory.

The only problem might be if your clients continuously drop or edit files into the directory while rsync is still running. Granted rsync is smart but I don't think it looks for open files which means a "long" rsync might not copy everything but technically should at the next run.

You have not provided any specific hardware or upload/download speed information but the speed limitation most likely is due to the upload speed of the client versus running multiple rysnc's at the same time on the server but I agree that ionice is something to consider.

Is this on the same LAN or remote clients?
They are all remote clients on 80/20 VDSL lines and I have a 80/20 line.

I've run the system using LFTP and SFTP for years and its been largely ok, however I want to be able to not have any credentials out in the wild or opertunities for hackers so I want my server to connect to theirs using a key and pull the data back to my server which has LUKS encryption to protect against theft ect. I also use fail to ban to harden the ssh a bit on mine and all my clients ssh with 1 year IP bans for 3 fails. Speaking of hardware my main server is ancient. Its a dual core Athlon with 2GB ram and it's about 15 years old. It does have new drives and PSU at least.

Last edited by atomiccomp; 03-26-2020 at 02:14 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync with --link-dest: Question about interrupted rsync session bvz Linux - Server 9 09-10-2016 11:31 AM
rsync solaris + ld.so.1: rsync: fatal: libiconv.so.2: open failed: xxx_anuj_xxx Solaris / OpenSolaris 25 02-23-2012 03:23 AM
Could I run rsync to download files from a server without rsync daemon? Richard.Yang Linux - Software 1 09-18-2009 04:08 AM
Rsync server vs rsync over ssh humbletech99 Linux - Networking 1 10-18-2006 12:10 PM
Windows Rsync Upload to Linux Rsync - permissions inspleak Linux - Software 0 10-12-2004 02:49 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 07:23 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration