script to run rsync process, stop and clean exit.

tonj · 02-10-2014, 02:03 PM

I'm a newbie with linux scripts so please go easy....
I'm using the following script on a centos 6 server to rsync a centos mirror to my own local mirror:

Quote:

#!/bin/bash

while [ 1 ]
do
/usr/bin/rsync -art --delete -vv --progress rsync://mirror.nsc.liu.se/centos-debuginfo/6/x86_64/ /var/www/html/debug.info/6/x86_64
if [ "$?" = "0" ] ; then
echo "rsync completed normally"
exit
else
echo "Rsync failure. Backing off and retrying..."
sleep 20
fi
done

this script works well in that it auto resumes rsyncing when the connection is broken. What I want to do however is make this process run for a set amount of time and then stop the rsync process and clean exit. Is there a way to do this?
I've tried putting this in at various aplaces:

Quote:

killall -SIGTERM rsync
exit 0

but it doesn't work, the script just carries on.
I've also tried the script this way:

Quote:

/usr/bin/rsync -art -vv --progress rsync://mirror.nsc.liu.se/centos-debuginfo/6/x86_64/ /var/www/html/debug.info/6/x86_64 &
sleep 20
killall -SIGTERM rsync
exit 0

and it does stop after 20 secs but it's messy, it doesn't clean exit and it doesn't contain any auto resume. What I ultimately want to do is run this script as a cron job during the night so it starts, runs for a set time and then stops clean and exits. Is there a way to have both these things in one script? Thanks for any help.

Nephew · 02-11-2014, 03:00 AM

Just thinking that maybe their is no clean exit for rsync if it is busy, the only clean exit is to let it finish.

tonj · 02-11-2014, 03:26 AM

but my problem is this:
I have a monthly download limit imposed by my isp, and the volume needed to build up a local mirror eats it away. However downloading between midnight and 8am isn't counted so I use a cron job to download during these hours, but so far it's messy trying to stop one download and start another during the night. I'm getting errors and failures to download.

Habitual · 02-11-2014, 08:46 AM

Quote:

Originally Posted by tonj

I'm getting errors and failures to download.

Better to diagnose the "errors and failures" and once those are corrected, schedule it as a late night cronjob, no?
What errors and failures are you getting?

60.4'ish G of data in that remote directory.
Do you have room for that?

tonj · 02-11-2014, 01:38 PM

the failures are because (using the second script) the internet connection sometimes gets broken and the stop and exit from the script is messy. The first script solves these problems, but the first script doesn't have a time limit in it - which I would like, if I could get some help on how to correctly script it.

Habitual · 02-11-2014, 02:03 PM

http://wiki.centos.org/HowTos/CreateLocalMirror
says "find a mirror near you that supports rsync" and rsync://mirror.nsc.liu.se/CentOS is listed, but http://mirror.nsc.liu.se/centos-debuginfo is not.

I don't think that site allows rsync into that directory.

This however works for me here:

Code:

cd /var/www/html/debug.info/6/x86_64
lftp -c "open http://mirror.nsc.liu.se/centos-debuginfo/6/x86_64/ ; mirror"

and seems to supports a resume-type function
I watched a stat command on one of the existing files and killed the job and restarted it and the file *times weren't changed at all.

tonj · 02-11-2014, 02:10 PM

ok thanks but I'm fairly certain the problem here is in the script I'm using, not in the site I'm downloading from. The stop and clean exit has to be done using a time limit in the script (preferably in the first script I mentioned) not from a human pressing keys on the keyboard.

Habitual · 02-11-2014, 02:42 PM

I finally got it to "run" with this partial output:

Code:

/usr/bin/rsync -art --delete -vv --progress rsync://mirror.nsc.liu.se/centos-debuginfo/6/x86_64/
opening tcp connection to mirror.nsc.liu.se port 873
sending daemon args: --server --sender -vvlogDtpre.iLsf . centos-debuginfo/6/x86_64/ 
NSC public mirror server
Admin contact: mirror-admin@nsc.liu.se

receiving incremental file list
delta-transmission enabled

and a list of 2900 files, but NO files were downloaded.

I'm telling you lftp is the way to go

Habitual · 02-11-2014, 02:55 PM

Now, this is working

Code:

rsync -art --delete -vv --progress rsync://mirror.nsc.liu.se/centos-debuginfo/6/x86_64/ .

notice the period? #ignore this comment as /var/www/html/debug.info/6/x86_64 satisfies.

Once you have a command that doesn't error out, then you can work on the timing/resume issue in your script.

Habitual · 02-11-2014, 03:16 PM

Something like this may work?

Code:

#!/bin/bash
SLEEP=20m

while :   #infinite loop, jump out with break
do
  /usr/bin/rsync -art --delete -vv --progress rsync://mirror.nsc.liu.se/centos-debuginfo/6/x86_64/ /var/www/html/debug.info/6/x86_64 && break
  sleep $SLEEP
done