LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-01-2009, 10:36 AM   #1
arashi256
Member
 
Registered: Jan 2008
Location: Brighton, UK
Distribution: Ubuntu 12.04 / CentOS 6.5
Posts: 394

Rep: Reputation: 61
Question Incremental backups?


I've got our shiny new Samba server (well, not really shiny or new - I built it out of bits from our storeroom and threw in a 500GB drive) fired up. Anyway....I need to backup the Samba share directory (and all sub-directories) to an external drive - which is mounted - once a month. This bit is easy. The hard bit is that the boss wants incremental daily backups of files that have changed in this directory tree to be copied to a daily sub-directory named with todays date on the external drive.

Can I use rdiff-backup for this? Or something else? Or some sort of scripting? The directory struture is rather complex. If there is no other way, can I use Bash or Perl or something to cycle through directories and compare the file timestamp on the Samba share to the one on the same file on the backup drive and copy it that way?

What is the easiest way to achieve this?

Thanks for any advice.
 
Old 07-01-2009, 10:42 AM   #2
JulianTosh
Member
 
Registered: Sep 2007
Location: Las Vegas, NV
Distribution: Fedora / CentOS
Posts: 674
Blog Entries: 3

Rep: Reputation: 90
BackupPC is the most awesome backup software (linux of course) I've used to date. Very easy install. Very easy maintenance and configuration. Restore process couldnt be easier.

Love it!
 
Old 07-01-2009, 10:48 AM   #3
arashi256
Member
 
Registered: Jan 2008
Location: Brighton, UK
Distribution: Ubuntu 12.04 / CentOS 6.5
Posts: 394

Original Poster
Rep: Reputation: 61
Thanks for the reply - I'll look into BackupPC but ideally (sorry, forgot to mention this) I'd like a command-line solution.
 
Old 07-01-2009, 02:14 PM   #4
xeleema
Member
 
Registered: Aug 2005
Location: D.i.t.h.o, Texas
Distribution: Slackware 13.x, rhel3/5, Solaris 8-10(sparc), HP-UX 11.x (pa-risc)
Posts: 987
Blog Entries: 4

Rep: Reputation: 252Reputation: 252Reputation: 252
Post A CLI-based Alternative for Incremental Backups

Greetingz!
If you're looking for something you can dump into a script and then slip into a crontab, you may want to consider a cp/rsync/ssh trifecta;


First, you want to use the "find" command to display and create a list of all files modified within the past 24hrs;


$ find /directory/to/backup -type f -mtime -1 -print |\
> tee ~/filelist.wri


Create a directory to store the daily backup;

$ mkdir /path/to/backup_directory/backup-`date +%a`

(NOTE: the date command is surrounded by "backticks", not single-quotes. The end-result is a directory named similarly to "backup-Wed". See the "date" manpage for more tricks.)

Now copy the files listed in our filelist.wri file into the backup directory;

$ for filename in $(cat ~/filelist.tmp)
> do
> cp "$filename" /path/to/backup/directory/backup-`date +%a`
> done


(NOTE: if you're doing this in a script you may want to set a Variable for "backup-`date +%a`". Otherwise, if the above for-loop is started before midnight, then right after the stroke of midnight, all of the remaining copies will fail)

Now, you'll have to wipe out the previous "Wednesday" backup before you do this, else you're just going to add files into that directory.

So all in all, you might have a script that looks something like this
Code:
#!/bin/sh
## Incremental Backup - Script that copies whole files to a separate
##    directory for safe keeping.  Only keeps seven days of files.
######################################################
## Set the Day (Mon, Tues, Wed, Thurs, Fri, etc, etc)
##   NOTE: if you *really* want all lower-case letters
##   in the date, comment out the short ${DAY} and uncomment
##   the second version.
DAY=`date +%a`
#DAY=`date +%a | tr '[:upper:]' '[:lower:]'`
## Set the full path to the directory that we'll store backups in.
TARGET=/path/to/backup/directory
## Find and wipe out the previous $DAY's backup.
if [ -r ${TARGET}/backup-${DAY} ]; then
   rm -rf ${TARGET}/backup-${DAY}
fi
## Find all files that aren't older than 24 hours
find /directory/to/backup -type f -mtime -1 -print | \
tee ~/filelist.tmp
## Create our backup directory and copy our files over to it
##    Note that we use double-quotes around our $filename because
##    Samba users like to use spaces in their filenames, which
##    freaks out our shells (Thanks MSFT!)
##    Sub-Note - check the manpage for 'cp', "-a" should be
##    defined. If not, use "-dpR".
mkdir ${TARGET}/backup-`date +%a`
for filename in $(cat ~/filelist.wri)
do
cp -a "$filename" ${TARGET}/backup-`date +%a`
done
## Maybe get paranoid, maybe do a little post-backup file
##    integrety checking just in case the boss thinks our
##    backup didn't backup the file correctly (it helps
##    to be able to do a quick 'md5sum -c filelist.md5'
##    just to be sure our data is intact).
for filename in $(cat ~/filelist.wri)
do
md5sum "$filename" | tee ${TARGET}/backup-${DAY}/filelist.md5
done

## Clean up after ourselves
mv ~/filelist.wri ${TARGET}/backup-${DAY}/filelist.md5
rm ~/filelist

## End of File.

Last edited by xeleema; 09-03-2010 at 06:23 PM. Reason: Added code tags.
 
Old 07-01-2009, 02:48 PM   #5
arashi256
Member
 
Registered: Jan 2008
Location: Brighton, UK
Distribution: Ubuntu 12.04 / CentOS 6.5
Posts: 394

Original Poster
Rep: Reputation: 61
That is fantastic, thank you! A very detailed reply.
 
Old 07-01-2009, 04:57 PM   #6
xeleema
Member
 
Registered: Aug 2005
Location: D.i.t.h.o, Texas
Distribution: Slackware 13.x, rhel3/5, Solaris 8-10(sparc), HP-UX 11.x (pa-risc)
Posts: 987
Blog Entries: 4

Rep: Reputation: 252Reputation: 252Reputation: 252
You're welcome! Let me know if that works!
(Most of my freehand scripting does, however YMMV).
 
Old 07-02-2009, 07:53 AM   #7
arashi256
Member
 
Registered: Jan 2008
Location: Brighton, UK
Distribution: Ubuntu 12.04 / CentOS 6.5
Posts: 394

Original Poster
Rep: Reputation: 61
Thumbs up

I modified it a little bit, but essentially that working brilliantly. I had to tweak it so that it created the destination directory structure as well, as some files that might be modified were in different directories but had the same filename, but aside from that - awesome.
 
Old 07-02-2009, 09:51 PM   #8
xeleema
Member
 
Registered: Aug 2005
Location: D.i.t.h.o, Texas
Distribution: Slackware 13.x, rhel3/5, Solaris 8-10(sparc), HP-UX 11.x (pa-risc)
Posts: 987
Blog Entries: 4

Rep: Reputation: 252Reputation: 252Reputation: 252
Glad to hear it!
 
Old 07-06-2009, 10:27 AM   #9
arashi256
Member
 
Registered: Jan 2008
Location: Brighton, UK
Distribution: Ubuntu 12.04 / CentOS 6.5
Posts: 394

Original Poster
Rep: Reputation: 61
This version is even better - it only creates the directory structure for newly modified files if there are any files that have been modified (i.e. any files listed in FILE-LIST.tmp), so I don't end up with a bunch of empty directories (had to figure out sed to make this work - nightmare!)

Code:
#!/bin/sh
DAY=`date +%F`
BACKUP_TARGET=/media/LaCie/BACKUP
find /home/sambashare/ -type f -mtime -1 -print > ~/FILE-LIST.tmp
LINECOUNT=`wc -l ~/FILE-LIST.tmp | sed -e 's/[A-Za-z/./-]*//g'`
if [ "$LINECOUNT" -gt "0" ]; then
	mkdir ${BACKUP_TARGET}/backup-${DAY}
	for filename in $(cat ~/FILE-LIST.tmp)
	do
		DIRECTORY=${filename%/*}
		mkdir -p ${BACKUP_TARGET}/backup-${DAY}${DIRECTORY}
		cp -a "$filename" ${BACKUP_TARGET}/backup-${DAY}${DIRECTORY}/
	done
	mv ~/FILE-LIST.tmp ${BACKUP_TARGET}/backup-$backup${DAY}/FILE-LIST.tmp
fi
I'm liking Bash scripting...
 
Old 07-06-2009, 09:46 PM   #10
xeleema
Member
 
Registered: Aug 2005
Location: D.i.t.h.o, Texas
Distribution: Slackware 13.x, rhel3/5, Solaris 8-10(sparc), HP-UX 11.x (pa-risc)
Posts: 987
Blog Entries: 4

Rep: Reputation: 252Reputation: 252Reputation: 252
"I'm liking Bash scripting..."

The great part is, notice the "#!/bin/sh" at the top of the script? If you keep a close eye on your syntax, you should be able to move that script around to various UNIX distributions. The "sh" in there implies the Bourne Shell (not the Bourne Again Shell, a.k.a Bash), which has been a long running shell for several decades.

One trick to remember, specify the full path to your binaries, you never know what the $PATH environment variable is gonna have. (Also, that's just good practice for scripts that run as root...)

(And I do realize that I didn't do it in my version *sheepish_grin*)

Happy scripting!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Differential and incremental backups mikieboy Linux - General 14 07-06-2008 10:27 PM
Rsync for incremental backups Meson Linux - General 1 10-30-2007 10:44 AM
Encrypted incremental backups? DaneelGiskard Linux - Security 3 07-25-2007 07:57 AM
Rsync for incremental backups? Phaethar Linux - Software 3 12-04-2003 02:27 PM
Incremental tar backups Pepe Linux - General 5 03-18-2002 04:31 AM


All times are GMT -5. The time now is 11:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration