LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-26-2007, 07:02 AM   #1
magli
LQ Newbie
 
Registered: Nov 2005
Location: South Tirol, Italy
Distribution: gentoo/ubuntu
Posts: 19

Rep: Reputation: 0
Troubles using cron+tar for automated backup


Good morning/day/evening.

I was looking into setting up an automatic backup system for a computer I have running Ubuntu Server 6.06 LTS.

I found an article discussing the use of tar to do this, and decided to go for that.

I have one command to backup my system, and another command to backup my /home directory. I put them into a simple script, which looks like this:

Backup script:
Code:
#Remove cached apt files.
apt-get clean

#Get date
DATE=`date +'%Y.%m.%d'

#command to backup system files
tar -cvpzP --exclude=/home --exclude=/proc --exclude=/lost+found --exclude=/mnt --exclude=/sys -f /home/shares/allusers/Server-Files/backups/$DATE\_sys.tgz /

#command to backup home direcory
tar -cvpzP --exclude=/home/shares/allusers/Server-Files/backups -f /home/shares/allusers/Server-Files/backups/$DATE\_home.tgz /home
When I run this script manually (as root using sudo), it works fine, and I end up with two new files:

ls -l gives me:
Code:
-rw-r--r-- 1 root  root   84649846 2007-02-13 17:17 2007.02.13_home.tgz
-rw-r--r-- 1 root  root  203552424 2007-02-13 17:17 2007.02.13_sys.tgz
So, I added my new script into my root crontab, with it scheduled to run every week, like this:

Code:
1 04 * * 4     /home/shares/allusers/Server-Files/backups/backup_script
Now, the problem I am having is that the cron job runs. It creates the files as expected, but they are incomplete:

ls -l on files created by cron job:
Code:
-rw-r--r-- 1 root  root         20 2007-02-22 04:01 2007.02.22_home.tgz
-rw-r--r-- 1 root  root    6320482 2007-02-22 04:01 2007.02.22_sys.tgz
As you can see, the files are much smaller than they were when I ran the command manually. I have tried listing the contents of these archives with tar -tvf:

For the system archive, tar lists several files, but then suddenly exists with:
Code:
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
And for the /home archive, there is no output at all.

So, does anybody know why this is happening? I thought that maybe there was a time limit on cron jobs which my script was exceeding, and that it was getting killed at this point.. but I failed to find any info confirming this, and it would not explain why the second tar archive is created.

I am at a total loss. Would appreciate any help or suggestions.

Cheers.
 
Old 02-26-2007, 07:11 AM   #2
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,450

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Hi

Maybe tar gets problems when reading "/dev"? (Just a guess...)
 
Old 02-26-2007, 07:18 AM   #3
magli
LQ Newbie
 
Registered: Nov 2005
Location: South Tirol, Italy
Distribution: gentoo/ubuntu
Posts: 19

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Guttorm
Maybe tar gets problems when reading "/dev"? (Just a guess...)
I guess this is possible, though I do not understand why it only has this problem when it is being run as a cron job (if I run it manually, it works fine).

It also does not explain why it is failing to backup my /home directory (which runs as a different tar command than the one which archived /dev)

You just made me realize that it may be of interest to know what the last items listed in the tar archive is - when I do

Code:
ls -tvf 2007.02.22_sys.tgz
[...]
-rw-r--r-- root/root         0 2007-02-10 14:13:02 /var/lib/apt/lists/it.archive.ubuntu.com_ubuntu_dists_dapper-backports_restricted_binary-i386_Packages
-rw-r--r-- root/root    195085 2007-02-10 14:13:02 /var/lib/apt/lists/it.archive.ubuntu.com_ubuntu_dists_dapper-backports_universe_binary-i386_Packages
-rw-r--r-- root/root     25437 2007-02-10 14:13:02 /var/lib/apt/lists/it.archive.ubuntu.com_ubuntu_dists_dapper-backports_multiverse_binary-i386_Packages
-rw-r--r-- root/root     13018 2007-02-10 14:17:31 /var/lib/apt/lists/it.archive.ubuntu.com_ubuntu_dists_dapper-backports_main_source_Sources
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
Thanks.
 
Old 02-26-2007, 07:25 AM   #4
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,450

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
You're right. Running it from a command should not work then.

Another idea is to check the error messages. Normally they would be emailed to root. You can also add a line like "MAILTO=username" first in the cron file you get when you type "sudo crontab -e".

Or does the tar command hang?
 
Old 02-26-2007, 07:25 AM   #5
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
There is nothing wrong in your script, apparently. Nor the crontab imposes a time limit on the scheduled jobs. First of all, you can check for any standard error and/or output from your cron job. To do this, login as the user to which crontab belongs (root?) and issue the mail command. On most Linux systems the output and error from cron jobs are sent to the system mail, unless they are redirected to a file or device.
 
Old 02-26-2007, 07:45 AM   #6
magli
LQ Newbie
 
Registered: Nov 2005
Location: South Tirol, Italy
Distribution: gentoo/ubuntu
Posts: 19

Original Poster
Rep: Reputation: 0
No luck.

I had to install mailx to get the mail command. However, it reports that there is no mail for root, or my user.

How can I check if the output is being redirected?

Should I add MAILTO=root to the first line of my crontab to make sure root gets emailed the errors?
 
Old 02-26-2007, 07:56 AM   #7
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
If you don't have the mail command there is no clue to retrieve standard error and output from there (very strange indeed, the mail command should be always available). As an alternative you can redirect both standard error and output from your cronjob, appending the following to the command in the crontab
Code:
> $HOME/cron.log 2>&1
where $HOME/cron.log is a totally arbitrary filename.
 
Old 02-26-2007, 07:58 AM   #8
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,450

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Hi

Well, the default is "MAILTO=root" for cron running as root, so there's no need to specify this. Since cron doesn't set the environment like when you're using a shell, above your line in the cron file, you can add stuff like:
SHELL=
PATH=
MAILTO=
HOME=

But if you're haveing trouble with email, another way would be to simply log to file:
tar -cvpzP --exclude=/home --exclude=/proc --exclude=/lost+found --exclude=/mnt --exclude=/sys -f /home/shares/allusers/Server-Files/backups/$DATE\_sys.tgz / &>/tmp/tar.log

Edit: Another duplicate
 
Old 02-26-2007, 08:02 AM   #9
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
Quote:
Originally Posted by Guttorm
Edit: Another duplicate
Yeah! Since you posted first, I can retire, my captain!
 
Old 02-26-2007, 08:08 AM   #10
magli
LQ Newbie
 
Registered: Nov 2005
Location: South Tirol, Italy
Distribution: gentoo/ubuntu
Posts: 19

Original Poster
Rep: Reputation: 0
Thanks for the suggestions guys. I will redirect the output in this way.

Had a couple questions though: what does "2>&1" do in the example command colucix gave?,
Also, now that I have installed the mail command, should I get mail in the future? (In case you are interested, Ubuntu stopped including mail by default after hoary: http://www.ubuntuforums.org/showthread.php?t=102941)
 
Old 02-26-2007, 08:21 AM   #11
magli
LQ Newbie
 
Registered: Nov 2005
Location: South Tirol, Italy
Distribution: gentoo/ubuntu
Posts: 19

Original Poster
Rep: Reputation: 0
On another note, while I have your attention -

Do you wise fellows see any potential problems with this backup method? Should I be doing it a different way?

Cheers.
 
Old 02-26-2007, 08:30 AM   #12
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
Quote:
Originally Posted by magli
Had a couple questions though: what does "2>&1" do in the example command colucix gave?
In simple words, this tells to redirect the standard error to the same place where standard output is redirected. More technical: 2 is the file descriptor for standard error, 1 is the file descriptor for standard output. The >& is the symbol to perform such a redirection.
Quote:
Also, now that I have installed the mail command, should I get mail in the future?
Yes, if the mail service is correctly configured. I guess it is, if you have installed with apt-get or synaptic!
Quote:
In case you are interested, Ubuntu stopped including mail by default after hoary: http://www.ubuntuforums.org/showthread.php?t=102941
Thank you for the link. I never heard about this, before. Maybe, because I am not using Ubuntu?!
 
Old 02-26-2007, 08:33 AM   #13
magli
LQ Newbie
 
Registered: Nov 2005
Location: South Tirol, Italy
Distribution: gentoo/ubuntu
Posts: 19

Original Poster
Rep: Reputation: 0
Problem mysteriously solved

I added redirected the output of my script, and changes the cron schedule from thursday morning at 04:01 to Monday 15:20 (current time here), and it ran fine - no errors. The archives are full.

This is very frustrating, as the problem I described above did not only occur once, it happened every time my script ran as a cron job, until this very moment.

I am going to change the schedule back to sometime at night - maybe 03:00 this time.

Could it be, that there is some other system-maintenance task scheduled for 04:00, which was accessing (blocking?) some of the that my script was trying to archive?

Thanks for your help.
 
Old 02-26-2007, 08:39 AM   #14
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
Nope. The crontab jobs should be indipendent from each other. I suggest to leave the scheduled crontab as is (4:01) and see the cron.log which will be created after that. Just for debugging purposes. In any case you have a full backup, right now!
 
Old 02-26-2007, 08:49 AM   #15
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,450

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Hi again

The important thing is that you check your backup and see if the files you want backup of really are there. Making a backup script, and then not checking it is a common mistake.

And having root email go to a real email account that you read is a good thing. It will email you different messages every once in a while - it's better to have cron output emailed, than being put in a log file. You can also get emails about different problems.

"2>&1" means send all error messages to standard output. I just used a little shortcut. "2>&1 >somefile.log" is the same as "&>somefile.log".

Another tip, add a line like in the beginning of your script:
dpkg --get-selections >/etc/deb.packages.txt

This makes a text file called deb.packages.txt in /etc. If your computer totally dies, it would not be so easy to get a new computer running with all the files you have in the tar files. If you have that "deb.packages.txt" file available, you can just do a minimum (server) install on whatever computer, then do a
dpkg --set-selections </etc/deb.packages.txt

Also, a good test, if you have the time: Imagine the computer you take backup of is not available anymore (stolen or whatever). Find a spare computer and get it up. You only have those tar files from the server (and perhaps an Ubuntu CD). A practical test can often reveal problems you didn't even think about.

Last edited by Guttorm; 02-26-2007 at 08:51 AM.
 
  


Reply

Tags
backup, cron, tar


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
BackUp & Restore with TAR (.tar / .tar.gz / .tar.bz2 / tar.Z) asgarcymed Linux - General 5 12-31-2006 02:53 AM
Automated HD backup General Linux - Hardware 3 05-11-2006 08:39 PM
mysql automated backup dkc_ace General 12 10-02-2004 08:19 AM
Automated Backup via FTP KePSuX Linux - Newbie 3 02-11-2004 08:25 AM
automated FTP backup josephswagner Linux - Software 2 06-06-2003 04:42 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 03:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration