rsync backup script help

henryyy · 03-02-2013, 06:41 PM

Hey guys,

I wonder if you could give me some pointers on setting up a script to backup my NAS to a online storage account.

I have signed up for a months trial account with http://onlinestoragesolution.com/ and have managed to perform a manual backup with the following commands.

ssh -L 873:localhost:873 username@fm.ols18.com

rsync -avz --progress -b ~/Downloads rsync://username@localhost/username

wich is all well and good but I would like to set it to run as a cron job to backup nightly.

I did try this command
rsync -avz --progress -e ssh username@fm.ols18.com:/username ~/Downloads/

but I was told that my shell was dirty and on reviewing the help on the online storage solution website it seems that it doesn't support rsync over ssl you have to use a tunnel.

Code:

MacBook:~$ rsync -avz --progress -e ssh username@fm.ols18.com:/username ~/Downloads/
username@fm.ols18.com's password: 
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at /SourceCache/rsync/rsync-42/rsync/compat.c(61) [receiver=2.6.9]
MacBook:~$

If anyone has any suggestions that would be much appreciated.

lykwydchykyn · 03-02-2013, 07:19 PM

Why not just put the commands that worked manually into a script and run that in cron?

propofol · 03-02-2013, 11:51 PM

How about:

Code:

rsync -avz --progress -e 'ssh -L 873:localhost:873' username@fm.ols18.com:/username ~/Downloads/

henryyy · 03-03-2013, 04:09 AM

Quote:

Originally Posted by lykwydchykyn

Why not just put the commands that worked manually into a script and run that in cron?

Im not sure that would work as you need to run 'ssh -L 873:localhost:873 username@fm.ols18.com' as root so it asks you for both your root password then your ols password. Once they have both been entered it holds the terminal window open where no more commands can be entered.

Code:

Welcome to Ubuntu 11.10 (GNU/Linux 3.0.0-28-server x86_64)

 * Documentation:  https://help.ubuntu.com/11.10/serverguide/C
New release 'precise' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Sat Mar  2 18:27:35 2013 from XXX-XXX-XXX-XXX.zone9.bethere.co.uk



Connection: XXX.XXX.XXX.XXX 53624 64.64.131.105 22
Client: XXX.XXX.XXX.XXX 53624 22
Terminal: /dev/pts/7

*************************************

You then need to open a second terminal window to run the rsync command.
rsync -avz --progress -b ~/Downloads rsync://username@localhost/username

Code:

MacBook:~$ rsync -avz --progress -b ~/Downloads rsync://username@localhost/username
Password: 
building file list ... 
819 files to consider
Downloads/Movie.avi
     3768320   0%  106.51kB/s    1:54:15

So I have no idea where to start scripting that or even if it is possible.

Quote:

Originally Posted by propofol

How about:

Code:

rsync -avz --progress -e 'ssh -L 873:localhost:873' username@fm.ols18.com:/username ~/Downloads/

Thanks but that didn't work it still tells me I have a dirty shell

Code:

MacBook:~$ sudo rsync -avz --progress -e 'ssh -L 873:localhost:873' username@fm.ols18.com:/username ~/Downloads/
Password:
username@fm.ols18.com's password: 
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at /SourceCache/rsync/rsync-42/rsync/compat.c(61) [receiver=2.6.9]
MacBook:~$

This is the example they show on the ols website
https://fm.ols18.com/Screen%20shot%2...57.56%20PM.png

Thanks again for the help.

lykwydchykyn · 03-03-2013, 04:54 PM

Quote:

Originally Posted by henryyy

Im not sure that would work as you need to run 'ssh -L 873:localhost:873 username@fm.ols18.com' as root so it asks you for both your root password then your ols password. Once they have both been entered it holds the terminal window open where no more commands can be entered.

This can be fixed with 3 things:

- add the -f switch to the command to fork ssh into the background
- Setup passwordless login using ssh keys
- Configure sudo on your machine to let you run this command (and just this command) without a password. I don't know why you need to be root to setup a tunnel, though, unless it's just because you have to use a port < 1024. Does it have to run on 873?

Quote:

You then need to open a second terminal window to run the rsync command.
rsync -avz --progress -b ~/Downloads rsync://username@localhost/username

Seems like you could specify an alternate port here, and avoid the root login requirement when you create the tunnel.

lleb · 03-03-2013, 05:47 PM

yes in order to tunnel via ssh in a reserved port range you must be root.

the rest of your comment is 100% the way to fix the OPs issue.

propofol · 03-03-2013, 08:37 PM

Code:

MacBook:~$ sudo rsync -avz --progress -e 'ssh -L 873:localhost:873' username@fm.ols18.com:/username ~/Downloads/
Password:
username@fm.ols18.com's password: 
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at /SourceCache/rsync/rsync-42/rsync/compat.c(61) [receiver=2.6.9]
MacBook:~$

Now that I'm thinking about it, this (-L 873:localhost:873) does not make sense. The " -e ssh " option does the transfer securely over ssh anyway as I understand it. It seems that "rsync error: protocol incompatibility" happens because of bash output on the fm.ols18.com server side. I assume that they have a ssh forced command thing in place so this will be difficult to fix from your end.

One option is to set up a tunnel which remains active in the background:

Code:

ssh -NTfL 8730:127.0.0.1:873 user@fm.ols18.com

For your crontab entry use this:

Quote:

export RSYNC_PASSWORD=yourpassword; rsync -av --progress ~/mydata rsync://user@localhost:8730/user/

The only downside is that you would have to make sure that the ssh tunnel is active. Note that using port 8730 avoids having to invoke ssh as root.

Regards,
Stefan

henryyy · 03-05-2013, 02:31 PM

Nice one guys,

I have a few things to try now so will post back soon and let you know if I manages to get things working,