LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Rsync and Crontab not playing well together (https://www.linuxquestions.org/questions/linux-newbie-8/rsync-and-crontab-not-playing-well-together-597993/)

phillc 11-08-2007 03:51 AM

Rsync and Crontab not playing well together
 
So, I can't seem to get rsync and crontab to work together.

When I manually run rsync with the following command:

rsync -aHrz --inplace --size-only --bwlimit=50 /www/kmtv/test /mnt/s3

It works fine, and the "test" folder with three files is copied to /mnt/s3

However, I can't get it to work with cron.

I created a rsync.sh script. The script says:

#!/bin/bash

rsync -aHrz --inplace --size-only --bwlimit=50 /www/kmtv /mnt/s3
rsync -aHrz --inplace --size-only --bwlimit=50 /var/lib/mysql/kmtv /mnt/s3
rsync -aHrz --inplace --size-only --bwlimit=50 /var/lib/mysql/scottish /mnt/s3

Saved it in /root/scripts Chmod to 755 so it has the correct permissions. Then added the following to crontab:

00 8 * * * /root/scripts/rsync.sh

Obviously the 8am time was just for testing purposes. But basically, now at 9.42am, nothing has been copied over.

To edit the crontab I'm using crontab -e

Any ideas why I can't get this to work?

laucian 11-08-2007 04:00 AM

there is an actual thread about cron here.
which distro are you using? can that be suse ?

phillc 11-08-2007 04:10 AM

Thanks for the link to the cron discussion. Although maybe your problem seems a little different to mine. It appears you can't get the actual command he wants to execute to work correctly? I can get rsync to work just fine when executing it from the command line. What I can't get is for crontab to execute it.

I'm using Debian.

colucix 11-08-2007 04:47 AM

Maybe your rsync command is not under /usr/bin and cron cannot locate it, since cron has a very limited PATH (different from the PATH you have on the command line). You can try to put the absolute path to rsync, when invoking from the script. Also, have you received some message onto the system mail for root? Usually it is the place where standard output and standard error are sent, if not explicitly redirected somewhere else. Furthermore you can check if the cron job is properly executed adding a simple command to your script, as
Code:

echo $PATH > /root/testfile
and see if /root/testfile is created.

phillc 11-08-2007 06:08 AM

I can confirm that rsync is located under /user/bin

After reading root's mail, I can see lots of errors! But of course I have no idea what they mean. I also don't understand why when I execute rsync manually from the command line it works, but not as part of crontab.

Here's the mail output from the most recent crontab attempt:

I can confirm that rsync is located under /user/bin

After reading root's mail, I can see lots of errors! But of course I have no idea what they mean. I also don't understand why when I execute rsync manually from the command line it works, but not as part of crontab.

Here's the mail output from the most recent crontab attempt:

Quote:

From: root@domainname.com (Cron Daemon)
To: root@domainname.com
Subject: Cron <root@servername> /root/scripts/rsync.sh
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <20071108104507.510584E01B5@servername>
Date: Thu, 8 Nov 2007 10:45:07 +0000 (GMT)

rsync: symlink "/mnt/s3/kmtv/scottish" -> "/home/scottish/" failed: No such fil$
rsync: recv_generator: mkdir "/mnt/s3/kmtv/admin" failed: No such file or direc$
*** Skipping everything below this failed directory ***

<snip - lots more similar type failures for each directory>

rsync: recv_generator: mkdir "/mnt/s3/kmtv/test" failed: No such file or direct$
*** Skipping everything below this failed directory ***
rsync: open "/mnt/s3/kmtv/.htaccess" failed: No such file or directory (2)
rsync: open "/mnt/s3/kmtv/empty.mov" failed: No such file or directory (2)

<snip lots more similar type failures for each file>

rsync: open "/mnt/s3/kmtv/sitemap.xml" failed: No such file or directory (2)
rsync: open "/mnt/s3/kmtv/update_rss.php" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(977) [send$

<snip lots more similar type failures for each file>

rsync: open "/mnt/s3/kmtv/kmtv_user_agents.frm" failed: No such file or directo$
rsync error: some files could not be transferred (code 23) at main.c(977) [send$
rsync: writefd_unbuffered failed to write 227 bytes [sender]: Broken pipe (32)
rsync: write failed on "/mnt/s3/scottish/jos_content.MYD": Input/output error ($
rsync error: error in file IO (code 11) at receiver.c(259) [receiver=2.6.9]
rsync: connection unexpectedly closed (1692 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [genera$
rsync: connection unexpectedly closed (696 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender$
/usr/bin:/bin /root/testfile
An earlier cron job from 8am this morning had slightly different errors. They are shown below:

Quote:

rsync error: some files could not be transferred (code 23) at main.c(977) [send$
rsync: recv_generator: mkdir "/mnt/s3/kmtv" failed: Device or resource busy (16)
*** Skipping everything below this failed directory ***
rsync error: some files could not be transferred (code 23) at main.c(977) [send$
rsync: writefd_unbuffered failed to write 313 bytes [sender]: Broken pipe (32)
rsync: write failed on "/mnt/s3/scottish/jos_content.MYD": Input/output error ($
rsync error: error in file IO (code 11) at receiver.c(259) [receiver=2.6.9]
rsync: connection unexpectedly closed (1692 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [genera$
rsync: connection unexpectedly closed (674 bytes received so far) [sender]
$sender$

phillc 11-08-2007 06:34 AM

As a test I decided to edit my rsync.sh script to run the EXACT SAME rsync job that works manually. That is:

rsync -r --inplace --size-only --bwlimit=50 /www/kmtv/test /mnt/s3

(Actually I also removed the -a -H and -z parameters)

After doing that I can confirm that the directory "test" and the three files contained therein have been created/copied to /mnt/s3

The output of the cron mail for this job is:

Quote:

Delivered-To: root@domainname.com
Received: by servername.com (Postfix, from userid 0)
id 925F14E01B5; Thu, 8 Nov 2007 11:25:02 +0000 (GMT)
From: root@domainname.com (Cron Daemon)
To: root@domainname.com
Subject: Cron <root@servername> /root/scripts/rsync.sh
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <20071108112502.925F14E01B5@servername.com>
Date: Thu, 8 Nov 2007 11:25:02 +0000 (GMT)

/usr/bin:/bin /root/testfile

phillc 11-08-2007 06:47 AM

So I changed my rsync line in the rsync.sh script to the following:

rsync -r --inplace --size-only --bwlimit=50 /www/kmtv /mnt/s3

This is just asking rsync to work at one directory level higher than the /www/kmtv/test from before.

And of course it didn't work. The mail output errors are:

Quote:

rsync: writefd_unbuffered failed to write 21046 bytes [sender]: Broken pipe (32)
skipping non-regular file "kmtv/scottish"
rsync: write failed on "/mnt/s3/kmtv/hazardous.php": Input/output error (5)
rsync: connection unexpectedly closed (146 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender$
/usr/bin:/bin /root/testfile
I have no more ideas.

colucix 11-08-2007 07:25 AM

I don't really understand why it works from command line and not from crontab, but from your test is evident that there was a failure when copying symbolic and/or hard links. The archive mode (option -a) tries to preserve a lot of stuff including symbolic links and the first entry in the error log is just related to the creation of a symbolic link
Quote:

rsync: symlink "/mnt/s3/kmtv/scottish" -> "/home/scottish/" failed: No such file or directory.
To debug you can try to explicitly pass the options related to the archive mode and remove them one at a time, starting from -l. To see which are these options, check rsync --help and you will get something like
Code:

-a, --archive              archive mode; equals -rlptgoD (no -H,-A,-X)
Also, you may consider to use the option --copy-links instead, if it is suitable for your needs.

phillc 11-08-2007 07:37 AM

OK, I have some more information here and I think the problem now lies with my /mnt/s3 rather than rsync or crontab.

I've just run my rsync command to copy to another local directory, in this case /root/rsynctest , and everything worked fine. No errors.

Basically /mnt/s3 is my Amazon S3 storage service. I've installed Jungledisk as per the instructions/tutorial here:

http://el-studio.com/article/jungledisk-linux-backups

So, I think the problem must be with the way I have S3 and Jungledisk setup. I'll head over to their forums for some assistance.

Thanks for your time here.

phillc 11-08-2007 10:22 AM

So I'm back here again.....

I have Jungledisk running correctly. It would seem that some rsync commands work totally perfectly. Others are a miserable failure.

Here's the output of my latest effort:

Quote:

rsync -aHz --inplace --size-only --bwlimit=50 /var/lib/mysql/kmtv /mnt/s3/mysql

rsync: writefd_unbuffered failed to write 1 bytes [sender]: Broken pipe (32)
rsync: write failed on "/mnt/s3/mysql/kmtv/kmtv_content.MYD": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(259) [receiver=2.6.9]
rsync: connection unexpectedly closed (915 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [generator=2.6.9]
rsync: connection unexpectedly closed (190 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9]


All times are GMT -5. The time now is 07:29 AM.