more complex rsync cron job script

JamesM83 · 05-17-2017, 04:44 PM

Hi,

I am having trouble with setting up a bash script, and thought I would ask here for some help. I have the following setup:

(1) two servers A and B running ubuntu
(2) server A constantly downloads data into directory ~/data/ into a deep subdirectory structure. Each download is done at most after 15 minutes.
(3) Once per day I want to move data, that is older than 15 minutes from ~/data/ to ~/transferdata/ preserving the subdirectory structure
(4) this should then be transferred via rsync to server B:~/data/
(5) if this succeeded, the directory A:~/transferdata/ should be gzipped into A:~/transfers/[currentDate.tar.gz] and if that is succesfull, the directory should be cleared
(6) if all of this is succesful, another script called notifySuccessB.sh should be executed, on any failure in between notifyErrorB.sh should be executed

Could you give me some hints, especially on (3) because I couldn't find a flag for rsync that depended on the modification date, and (6) as I don't really know how to react on errors within a script?

Thanks, James

suicidaleggroll · 05-17-2017, 04:51 PM

For (3), you could use find's -ctime (or -mtime) flag to find the files order than 15 minutes, then use exec to make the necessary directory structure at the destination and copy the file over

Checking for errors in a script is straight forward, but you have to do it yourself, the script won't do it for you. The variable "$?" always contains the exit code of the most recently executed command. 99.99999% of the time an exit code of 0 means it was successful, anything else means something went wrong. You'd have to check the documentation for the program to see what each flag means, but it's generally a safe bet to use 0 for good, !=0 for bad. For example:

Code:

rsync -av $dir1 $dir2
stat=$?
if [[ $stat -ne 0 ]]; then
   echo "rsync failed with exit code $stat"
fi

tar -zcf $(date +%Y%m%d).tgz $dir1
stat=$?
if [[ $stat -ne 0 ]]; then
   echo "tar failed with exit code $stat"
fi

syg00 · 05-17-2017, 08:35 PM

I would be (extremely) concerned with the complexity of this. And resultant likely fragility.
I'm a big believer in snapshot - have been for over a decade. I would snap once a day ignoring the 15 min requirement. No data movement, instantaneous. Rsync, then tar, do what you want, with the snap as source, knowing the data can't change.
When it's done, delete the snap to recover any space allocated in the meantime. LVM offers this functionality.

At the target end worry about the 15 min problem - easier to delete the last 15 min of files than copy/move 23 hours 45 min worth ...
KISS.

JeremyBoden · 05-18-2017, 06:04 AM

The OP's suggested procedure is very fragile for data - how would a restore of data work?

JamesM83 · 05-18-2017, 07:54 AM

Quote:

Originally Posted by suicidaleggroll

For (3), you could use find's -ctime (or -mtime) flag to find the files order than 15 minutes, then use exec to make the necessary directory structure at the destination and copy the file over

How would I make the necessary directory structure?

Unfortunately I am not able to create snapshots on that machine, since I am not allowed to make changes to the LVM.

In which sense is it very fragile?

Regarding restoring data, this is actually not a concern, since as soon as the data is on server B this is of no concern anymore, server A is purely a server that gets the data, and should once a day forward the data, no local backup is really needed (step 5 is only for sanity checks in the beginning)

Turbocapitalist · 05-18-2017, 08:26 AM

Directory structure aside, one option is to have incron monitor the directory for completed files using the IN_CLOSE_WRITE trigger. That could be used to call rsync on the file or the newest files.

JeremyBoden · 05-18-2017, 08:59 AM

Provided that all files are forced to disk on server A and that there are no incomplete transactions on the fifteen minute boundary and also provided that the rsync doesn't fail in mid-flight then there is no problem.

I would suggest you probably need to close any databases before the start of the rsync and re-open them on successful completion.

suicidaleggroll · 05-18-2017, 10:20 AM

Quote:

Originally Posted by JamesM83

How would I make the necessary directory structure?

A combination of dirname and mkdir -p. It may make the most sense to write your own little script to do it and call that from exec, eg:

Code:

find . -mmin +15 -exec mycopy "{}" "$dir2" \;

mycopy:

Code:

if [[ $# -eq 2 ]]; then
   if [[ -f "$1" ]]; then
      path=$(dirname "$1")
      if [[ ! -d "$2/$path" ]]; then
         mkdir -p "$2/$path"
      fi
      cp -a "$1" "$2/$path/"
   fi
fi