Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I'm having real trouble trying to do an incremental tar. My understanding is that I keep using the same list file, and the first time it should get created, after which it should create smaller tars with only new changes.
What happens in practice is that although the list file exists and nothing has changed, it still creates a full backup. The output of tar is not very helpful. I modded the script to echo the tar command, and this is what I get:-
Basename: Public-2010-03
Listname: /mnt/storage/Backups/tmpPublic-2010-03.snar
Destname: /mnt/storage/Backups/tmpPublic-2010-03-5.tar.bz2
List file exists.
List file is readable.
tar --verbose --create --ignore-failed-read --listed-incremental="/mnt/storage/Backups/tmp/Public-2010-03.snar" --bzip2 --file "/mnt/storage/Backups/tmp/Public-2010-03-5.tar.bz2" "/mnt/storage/Public"
Your script is creating a new name for the .snar file which contains the date. For a differential backup, you create a working copy of the .snar file created from the first full backup. Then all files modifies since the full backup will be archived. This allows you to restore by restoring the full backup and the last differential backup.
If you reuse the same snar file, it will be updated after each dump so you will be creating an incremental backup. The incremental backups will only backup files since the last incremental backup.
Section 5.2 of the tar info manual is where this is explained.
---
FYI, I downloaded the tar package source and run "./configure && make pdf" to create a more readable book version of the info manual for tar. Some things like backups are important enough where I will print out the documentation. I find I can learn and retain the info better. (Maybe I'm showing my age)
Your script is creating a new name for the .snar file which contains the date.
Thanks for the reply. That's not quite correct, I believe. The .snar file is named with the year and the week number. The tar.bz2 file is named with the year, week number and week day (number). If you look at the output in my original post, the files are Public-2010-03.snar and Public-2010-03-5.tar.bz2, The idea is that the .snar file should keep the same name for a week, meaning that I should get one full backup per week and 6 incremental daily backups.
As you can see, I put the check in to echo if the .snar file exists and is readable when the script runs, and sure enough it does.
This backed files up to a locally mounted external drive and replicated files on a remote location. I did test that only newer files were backed up during an incremental backup. You could do something like:
tar
You did test that the .snar file was present and readable. But you didn't test that it was not empty. Also the script is deleting the snar file in the working directory. Why? It is using the local directory for the snar file in the tar command. I don't trust what is happening with the snar file, or that the snar file is valid when you run the tar command.
I have in the past used something like:
tar -C <base_dir> --create --listed=incremental=<snar file> -f - <dir list> | tee /mnt/backups/<backup_name> | ssh user@host tar -C <remote_base_dir> -xvf - >logfile
You could do something like:
tar --verbose --create --ignore-failed-read --listed-incremental="${working_dir}/${listname}" --bzip2 \
--file - "$src_dir" |
ssh ${user}@${dest_addr} cat >${dest_dir}/${destname}
This way you don't need to have space available to create a temporary archive.
In any event, I would keep the .snar file available locally and not remove it.
the snar file is getting deleted after the files are copied to the destination. The .snar file is modified when you run tar. You need to reuse the modified .snar file when performing the next incremental backup.
Given the line above, I don't trust the results of the tests. They test for the existence of the .snar file, and its readability, but it could be empty or not updated.
"Don't touch that, you don't know where it's been."
---
You might consider running tar like:
tar -C $base_dir --create --bz2 --listed-incremental=<snar_file> - <dir_list> | ssh <user@host> cat > ${remote_dir}/{backup_name}
This will pipe the file to the destination instead of creating a temporary backup locally and copying the files to the destination.
Otherwise you need at least the size of a full backup free locally.
In any case, I wouldn't delete the *.snar files locally.
Hi Jschiwal, and thanks for all the time you put into this. I've removed all the quoting around file names in the script because I don't actually need it in this case (no spaces in the paths) and because I wanted to see if it was causing a problem.
Actually the command you pointed out had a bug in (because of the quotes), but one which meant it was doing nothing at all.
It should have read:-
rm -f ${working_dir}/!(${basename}).snar
Quoting breaks the globbing character, turned on by the "shopt -s extglob" line. The above line deletes any snar *apart* from the current snar file. In other words, it cleans up old ones. It seems to work as intended now.
The tar still behaves as before though, creating a full backup each time.
Only two things I can think of that could cause a full backup every time. First is if there isn't a snar file. Second is if somehow the timestamps are being updated.
You can check the latter by logging tar with the -vv option and redirecting the output.
I would try commenting out the line that deletes the snar files just to make sure.
---
I tested out using !(pattern) manually and in a script:
Code:
cat testscr
shopt -s extglob
ls bash-4.0/!(${basename}).pdf
set bash-4.0/!(${basename}).pdf
echo $*
jschiwal@qosmio:~/Documents> ls bash-4.0/!(${basename}).pdf
bash-4.0/bash.pdf bash-4.0/bashref.pdf bash-4.0/rose94.pdf
jschiwal@qosmio:~/Documents> cat testscr
shopt -s extglob
ls bash-4.0/!(${basename}).pdf
set bash-4.0/!(${basename}).pdf
echo $*
jschiwal@qosmio:~/Documents> ./testscr
bash-4.0/article.pdf bash-4.0/bash.pdf bash-4.0/bashref.pdf bash-4.0/rose94.pdf
bash-4.0/article.pdf bash-4.0/bash.pdf bash-4.0/bashref.pdf bash-4.0/rose94.pdf
enabling extglob and entering "ls bash-4.0/!(${basename}).pdf" excludes article.pdf.
Inside a script it includes it.
This same behavior would delete your current snar file in your script, causing a full backup every time.
I haven't looked into it further. Is extglob for interactive shells?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.