LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 11-08-2007, 04:51 AM   #1
phillc
LQ Newbie
 
Registered: Nov 2007
Posts: 10

Rep: Reputation: 7
Rsync and Crontab not playing well together


So, I can't seem to get rsync and crontab to work together.

When I manually run rsync with the following command:

rsync -aHrz --inplace --size-only --bwlimit=50 /www/kmtv/test /mnt/s3

It works fine, and the "test" folder with three files is copied to /mnt/s3

However, I can't get it to work with cron.

I created a rsync.sh script. The script says:

#!/bin/bash

rsync -aHrz --inplace --size-only --bwlimit=50 /www/kmtv /mnt/s3
rsync -aHrz --inplace --size-only --bwlimit=50 /var/lib/mysql/kmtv /mnt/s3
rsync -aHrz --inplace --size-only --bwlimit=50 /var/lib/mysql/scottish /mnt/s3

Saved it in /root/scripts Chmod to 755 so it has the correct permissions. Then added the following to crontab:

00 8 * * * /root/scripts/rsync.sh

Obviously the 8am time was just for testing purposes. But basically, now at 9.42am, nothing has been copied over.

To edit the crontab I'm using crontab -e

Any ideas why I can't get this to work?
 
Old 11-08-2007, 05:00 AM   #2
laucian
Member
 
Registered: Oct 2005
Distribution: Ubuntu 9.04
Posts: 124
Blog Entries: 2

Rep: Reputation: 15
there is an actual thread about cron here.
which distro are you using? can that be suse ?
 
Old 11-08-2007, 05:10 AM   #3
phillc
LQ Newbie
 
Registered: Nov 2007
Posts: 10

Original Poster
Rep: Reputation: 7
Thanks for the link to the cron discussion. Although maybe your problem seems a little different to mine. It appears you can't get the actual command he wants to execute to work correctly? I can get rsync to work just fine when executing it from the command line. What I can't get is for crontab to execute it.

I'm using Debian.

Last edited by phillc; 11-08-2007 at 05:21 AM.
 
Old 11-08-2007, 05:47 AM   #4
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Maybe your rsync command is not under /usr/bin and cron cannot locate it, since cron has a very limited PATH (different from the PATH you have on the command line). You can try to put the absolute path to rsync, when invoking from the script. Also, have you received some message onto the system mail for root? Usually it is the place where standard output and standard error are sent, if not explicitly redirected somewhere else. Furthermore you can check if the cron job is properly executed adding a simple command to your script, as
Code:
echo $PATH > /root/testfile
and see if /root/testfile is created.
 
Old 11-08-2007, 07:08 AM   #5
phillc
LQ Newbie
 
Registered: Nov 2007
Posts: 10

Original Poster
Rep: Reputation: 7
I can confirm that rsync is located under /user/bin

After reading root's mail, I can see lots of errors! But of course I have no idea what they mean. I also don't understand why when I execute rsync manually from the command line it works, but not as part of crontab.

Here's the mail output from the most recent crontab attempt:

I can confirm that rsync is located under /user/bin

After reading root's mail, I can see lots of errors! But of course I have no idea what they mean. I also don't understand why when I execute rsync manually from the command line it works, but not as part of crontab.

Here's the mail output from the most recent crontab attempt:

Quote:
From: root@domainname.com (Cron Daemon)
To: root@domainname.com
Subject: Cron <root@servername> /root/scripts/rsync.sh
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <20071108104507.510584E01B5@servername>
Date: Thu, 8 Nov 2007 10:45:07 +0000 (GMT)

rsync: symlink "/mnt/s3/kmtv/scottish" -> "/home/scottish/" failed: No such fil$
rsync: recv_generator: mkdir "/mnt/s3/kmtv/admin" failed: No such file or direc$
*** Skipping everything below this failed directory ***

<snip - lots more similar type failures for each directory>

rsync: recv_generator: mkdir "/mnt/s3/kmtv/test" failed: No such file or direct$
*** Skipping everything below this failed directory ***
rsync: open "/mnt/s3/kmtv/.htaccess" failed: No such file or directory (2)
rsync: open "/mnt/s3/kmtv/empty.mov" failed: No such file or directory (2)

<snip lots more similar type failures for each file>

rsync: open "/mnt/s3/kmtv/sitemap.xml" failed: No such file or directory (2)
rsync: open "/mnt/s3/kmtv/update_rss.php" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(977) [send$

<snip lots more similar type failures for each file>

rsync: open "/mnt/s3/kmtv/kmtv_user_agents.frm" failed: No such file or directo$
rsync error: some files could not be transferred (code 23) at main.c(977) [send$
rsync: writefd_unbuffered failed to write 227 bytes [sender]: Broken pipe (32)
rsync: write failed on "/mnt/s3/scottish/jos_content.MYD": Input/output error ($
rsync error: error in file IO (code 11) at receiver.c(259) [receiver=2.6.9]
rsync: connection unexpectedly closed (1692 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [genera$
rsync: connection unexpectedly closed (696 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender$
/usr/bin:/bin /root/testfile
An earlier cron job from 8am this morning had slightly different errors. They are shown below:

Quote:
rsync error: some files could not be transferred (code 23) at main.c(977) [send$
rsync: recv_generator: mkdir "/mnt/s3/kmtv" failed: Device or resource busy (16)
*** Skipping everything below this failed directory ***
rsync error: some files could not be transferred (code 23) at main.c(977) [send$
rsync: writefd_unbuffered failed to write 313 bytes [sender]: Broken pipe (32)
rsync: write failed on "/mnt/s3/scottish/jos_content.MYD": Input/output error ($
rsync error: error in file IO (code 11) at receiver.c(259) [receiver=2.6.9]
rsync: connection unexpectedly closed (1692 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [genera$
rsync: connection unexpectedly closed (674 bytes received so far) [sender]
$sender$

Last edited by phillc; 11-08-2007 at 07:30 AM.
 
Old 11-08-2007, 07:34 AM   #6
phillc
LQ Newbie
 
Registered: Nov 2007
Posts: 10

Original Poster
Rep: Reputation: 7
As a test I decided to edit my rsync.sh script to run the EXACT SAME rsync job that works manually. That is:

rsync -r --inplace --size-only --bwlimit=50 /www/kmtv/test /mnt/s3

(Actually I also removed the -a -H and -z parameters)

After doing that I can confirm that the directory "test" and the three files contained therein have been created/copied to /mnt/s3

The output of the cron mail for this job is:

Quote:
Delivered-To: root@domainname.com
Received: by servername.com (Postfix, from userid 0)
id 925F14E01B5; Thu, 8 Nov 2007 11:25:02 +0000 (GMT)
From: root@domainname.com (Cron Daemon)
To: root@domainname.com
Subject: Cron <root@servername> /root/scripts/rsync.sh
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <20071108112502.925F14E01B5@servername.com>
Date: Thu, 8 Nov 2007 11:25:02 +0000 (GMT)

/usr/bin:/bin /root/testfile

Last edited by phillc; 11-08-2007 at 07:35 AM.
 
Old 11-08-2007, 07:47 AM   #7
phillc
LQ Newbie
 
Registered: Nov 2007
Posts: 10

Original Poster
Rep: Reputation: 7
So I changed my rsync line in the rsync.sh script to the following:

rsync -r --inplace --size-only --bwlimit=50 /www/kmtv /mnt/s3

This is just asking rsync to work at one directory level higher than the /www/kmtv/test from before.

And of course it didn't work. The mail output errors are:

Quote:
rsync: writefd_unbuffered failed to write 21046 bytes [sender]: Broken pipe (32)
skipping non-regular file "kmtv/scottish"
rsync: write failed on "/mnt/s3/kmtv/hazardous.php": Input/output error (5)
rsync: connection unexpectedly closed (146 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender$
/usr/bin:/bin /root/testfile
I have no more ideas.
 
Old 11-08-2007, 08:25 AM   #8
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
I don't really understand why it works from command line and not from crontab, but from your test is evident that there was a failure when copying symbolic and/or hard links. The archive mode (option -a) tries to preserve a lot of stuff including symbolic links and the first entry in the error log is just related to the creation of a symbolic link
Quote:
rsync: symlink "/mnt/s3/kmtv/scottish" -> "/home/scottish/" failed: No such file or directory.
To debug you can try to explicitly pass the options related to the archive mode and remove them one at a time, starting from -l. To see which are these options, check rsync --help and you will get something like
Code:
-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
Also, you may consider to use the option --copy-links instead, if it is suitable for your needs.
 
Old 11-08-2007, 08:37 AM   #9
phillc
LQ Newbie
 
Registered: Nov 2007
Posts: 10

Original Poster
Rep: Reputation: 7
OK, I have some more information here and I think the problem now lies with my /mnt/s3 rather than rsync or crontab.

I've just run my rsync command to copy to another local directory, in this case /root/rsynctest , and everything worked fine. No errors.

Basically /mnt/s3 is my Amazon S3 storage service. I've installed Jungledisk as per the instructions/tutorial here:

http://el-studio.com/article/jungledisk-linux-backups

So, I think the problem must be with the way I have S3 and Jungledisk setup. I'll head over to their forums for some assistance.

Thanks for your time here.
 
Old 11-08-2007, 11:22 AM   #10
phillc
LQ Newbie
 
Registered: Nov 2007
Posts: 10

Original Poster
Rep: Reputation: 7
So I'm back here again.....

I have Jungledisk running correctly. It would seem that some rsync commands work totally perfectly. Others are a miserable failure.

Here's the output of my latest effort:

Quote:
rsync -aHz --inplace --size-only --bwlimit=50 /var/lib/mysql/kmtv /mnt/s3/mysql

rsync: writefd_unbuffered failed to write 1 bytes [sender]: Broken pipe (32)
rsync: write failed on "/mnt/s3/mysql/kmtv/kmtv_content.MYD": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(259) [receiver=2.6.9]
rsync: connection unexpectedly closed (915 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [generator=2.6.9]
rsync: connection unexpectedly closed (190 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9]
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Backup with rsync and crontab gjblackford Linux - General 4 01-04-2007 09:53 AM
rsync in crontab loonykroons Linux - Software 3 11-28-2006 04:08 AM
Rsync server vs rsync over ssh humbletech99 Linux - Networking 1 10-18-2006 01:10 PM
Windows Rsync Upload to Linux Rsync - permissions inspleak Linux - Software 0 10-12-2004 03:49 PM
system-wide crontab in /etc/crontab ner Linux - General 2 11-18-2003 01:35 PM


All times are GMT -5. The time now is 04:39 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration