LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 03-09-2007, 06:37 AM   #1
mkhan919
Member
 
Registered: Jan 2007
Location: Valbonne, France
Distribution: Mandriva 2008 (Free)
Posts: 135

Rep: Reputation: 15
rsync question


Hi,

I am using Mandriva 2007. I wanted to get a basic working solution for backing up my data. So i am using Rsync to take backups of my internal drive on my external USB drive.

i am using rsync with --delete switch, which removes the files from the backup, that have been removed from the source. Now i was wondering, if it is possible to configure rsync to instead of deleting such files, it should move them to a separate folder.

Like for example, it should synchornize the source with destination, and the files that have been deleted, should be moved to a "deleted files" directory.

Any suggestions on how it may be done.
 
Old 03-09-2007, 06:43 AM   #2
Clemente
Member
 
Registered: Aug 2003
Distribution: Debian, Ubuntu
Posts: 188

Rep: Reputation: 30
Hi,

don't think that rsync provides such feature.

Don't know exactly, but it could be possible that the ouput of a dry rsync run provides enough information to extract the files that would become deleted by a real run.
You can move them by script before starting the real rsync run.
Not that handy, I know ;-)
 
Old 03-09-2007, 08:06 AM   #3
mkhan919
Member
 
Registered: Jan 2007
Location: Valbonne, France
Distribution: Mandriva 2008 (Free)
Posts: 135

Original Poster
Rep: Reputation: 15
well it is an idea nevertheless.. i'll try that tonight ...
 
Old 03-09-2007, 08:55 AM   #4
tuxrules
Senior Member
 
Registered: Jun 2004
Location: Chicago
Distribution: Slackware64 14.1
Posts: 1,138

Rep: Reputation: 55
Like the above poster said, you can diff command to compare directories, move the directories that don't match between source and destination and then run rsync.
 
Old 03-09-2007, 09:11 AM   #5
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,115

Rep: Reputation: 218Reputation: 218Reputation: 218
Hi

Another option is to install the program "rdiff-backup". Instead of deleting the files that have been deleted (or changing the changed files) it automaticly puts diffs so you can recreate backup from any time. Most distros should have rdiff-backup, you just need to install it.

To make a backup, e.g.:
rdiff-backup /home /mnt/usbdisk

/mnt/usbdisk/home will be a plain copy /home but it will have a special directory called "rdiff-backup-data" in it with diffs for files deleted or changed.

If you run that command a few times, you can try:
rdiff-backup -l /mnt/usbdisk
This will give you a list of all the times you have taken backups. Any of them can be restored using
rdiff-backup --restore-as-of TIME /mnt/usbdisk /tmp

Because the usb disk will fill up with diffs if the files change a lot, you need to delete very old diffs. This will remove diffs older than 32 days:
rdiff-backup --remove-older-than 32D

In sum: I think rdiff-backup is really neat - having all the benefits of rsync plus the ability to keep files even if they are changed or deleted.
 
Old 03-09-2007, 09:51 AM   #6
makyo
Member
 
Registered: Aug 2006
Location: Saint Paul, MN, USA
Distribution: {Free,Open}BSD, CentOS, Debian, Fedora, Solaris, SuSE
Posts: 718

Rep: Reputation: 72
Hi.
Quote:
... if it is possible to configure rsync to instead of deleting such files, it should move them to a separate folder.
I use rsync to keep a directory "current" in sync with my home directory. If there are extant files in current" that will be over-written, they are first copied to a directory which has the current date as part of the name. I keep these incremental backups around for 30 days or so.

The option that controls this is:
Quote:
--backup-dir=DIR
In combination with the --backup option, this tells rsync to
store all backups in the specified directory. This is very use-
ful for incremental backups. You can additionally specify a
backup suffix using the --suffix option (otherwise the files
backed up in the specified directory will keep their original
filenames). If DIR is a relative path, it is relative to the
destination directory (which changes in a recursive transfer).
Here is a copy of the log for today, as an example:
Code:
 Backup date =  Fri Mar 9 02:05:02 CST 2007
 Creating archive directory /backup/xandros/makyo/2007-03-09
 Using directory name 2007-03-09, path /backup/xandros/makyo/2007-03-09
 Backing up from /home/makyo to /backup/xandros/makyo/current,
 excluding patterns from file /backup/xandros/makyo/exclude,
 and archiving older versions into /backup/xandros/makyo/2007-03-09
 After backup, the size of /backup/xandros/makyo/current is 117750
Here is a copy of the specific rsync command (apologies for the heavy parameterization):
Code:
rsync -abv $mode --backup-dir=$BACKUPDIR --delete \
--exclude-from=$BB/exclude --delete-excluded \
$FROMDIR $CURRENT > $BACKUPDIR/$DATE.log \
2> $BACKUPDIR/$DATE.err
You may need to fiddle with other options to get precisely what you want. I usually make a few test cases and base my production on that. I been using this scheme for around a year and it seems to have worked well so far.

Best wishes ... cheers, makyo
 
Old 03-09-2007, 10:48 AM   #7
Clemente
Member
 
Registered: Aug 2003
Distribution: Debian, Ubuntu
Posts: 188

Rep: Reputation: 30
If the goal are incremental backups (instead of a directory containing all deleted files as the OP wrote), you can use tools like dirvish (command line) or backuppc (daemon with cgi-interface).
Both use rsync (backuppc knows other "treansports", too) to keep a remote dir in sync, storing only changed files.
Files that did not change become hardlinked and don't need additional space on the disk. You get "snapshot like incrementals" in a comfortable manner.
 
Old 03-09-2007, 12:37 PM   #8
makyo
Member
 
Registered: Aug 2006
Location: Saint Paul, MN, USA
Distribution: {Free,Open}BSD, CentOS, Debian, Fedora, Solaris, SuSE
Posts: 718

Rep: Reputation: 72
Hi, Clemente.

My scheme does not capture deleted files, only the originals for files scheduled to be replaced. Apologies if I mislead anyone.

It does appear to me that option -b, --backup is intended to keep transferred or deleted files:
Quote:
With this option, preexisting destination files are renamed as
each file is transferred or deleted ...
If I get some time, I'll investigate that ... cheers, makyo
 
Old 03-09-2007, 10:29 PM   #9
makyo
Member
 
Registered: Aug 2006
Location: Saint Paul, MN, USA
Distribution: {Free,Open}BSD, CentOS, Debian, Fedora, Solaris, SuSE
Posts: 718

Rep: Reputation: 72
Hi.

Indeed, rsync is able to copy files that would be deleted. The heart of this is a command such as:
Code:
rsync -abv --delete --backup-dir=$B --suffix=.orig $S/ $D/
which will compare the contents of the source, $S, and the destination, $D. All files that would be deleted (because they are in $D but not in $S) and all older versions of files will be collected in the directory $B. The $B directory is tucked into the $D directory. In this case, the files will have .orig tacked onto the end of their names.

If this is of interest, I can post the test script and the results ... cheers, makyo
 
Old 03-12-2007, 11:01 AM   #10
mkhan919
Member
 
Registered: Jan 2007
Location: Valbonne, France
Distribution: Mandriva 2008 (Free)
Posts: 135

Original Poster
Rep: Reputation: 15
Thanks a lot guys....

And yes makyo, i would really appreciate if you can post the test scripts etc. That would really be helpful.

And soory for replying late, but my net was down for the past two days...

Last edited by mkhan919; 03-12-2007 at 11:02 AM.
 
Old 03-12-2007, 11:42 AM   #11
makyo
Member
 
Registered: Aug 2006
Location: Saint Paul, MN, USA
Distribution: {Free,Open}BSD, CentOS, Debian, Fedora, Solaris, SuSE
Posts: 718

Rep: Reputation: 72
Hi, mkhan919.

OK, here it is.

There are some debugging aids built-in so that you can turn on and off explanatory messages.

A number of files are created in the source directory. These are copied to the destination directory. The dates of files in the latter are adjusted by the odd call to touch so that they will appear to be older to rsync, and thus be candidates for copying.

One file in source will be deleted to create the situation where rsync will schedule the file for deletion in the destination.

The final result is that all files are updated, the two directories are synchronized, and the originals and the one deleted file will be placed in the backup directory. To emphasize that placement, I added a suffix for the filenames. A recursive diff shows that the only item different is the backup directory.

The output from the script should illustrate these points.

Best wishes ... cheers, makyo

Code:
#!/bin/sh

# @(#) s1       Demonstrate rsync copy from source to destination.

debug=":"
debug="echo"

# Remove previous debris.

S=source
D=destination
B=backup

rm -rf $S $D
mkdir $S $D

cd $S
echo one >t1
echo two >t2
echo thr >t3
echo fou >t4
echo fiv >t5
cd ..
cp $S/* $D
rm $S/t3

# Back date the destination.

backdate=$( date --date="5 minutes ago" )
$debug " now = $(date);  backdate = :$backdate:"

touch --date="$backdate" $D/*

echo
echo " $S:"
ls -l $S

echo
echo " $D:"
ls -l $D

rsync -abv --delete --backup-dir=$B --suffix=.orig $S/ $D/

echo
echo " $D, final"
ls -l $D

echo
echo " $D/$B"
ls -l $D/$B

echo
echo " Diff of $S and $D"
diff -r $S $D

exit 0
Which, when run, produces:
Code:
% ./s1
 now = Mon Mar 12 11:40:13 CST 2007;  backdate = :Mon Mar 12 11:35:13 CST 2007:

 source:
total 16
-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t1
-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t2
-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t4
-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t5

 destination:
total 20
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t1
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t2
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t3
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t4
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t5
building file list ... done
deleting t3
t1
t2
t4
t5

sent 308 bytes  received 100 bytes  816.00 bytes/sec
total size is 16  speedup is 0.04

 destination, final
total 16
drwxr-xr-x  2 makyo makyo 168 Mar 12 11:40 backup
-rw-r--r--  1 makyo makyo   4 Mar 12 11:40 t1
-rw-r--r--  1 makyo makyo   4 Mar 12 11:40 t2
-rw-r--r--  1 makyo makyo   4 Mar 12 11:40 t4
-rw-r--r--  1 makyo makyo   4 Mar 12 11:40 t5

 destination/backup
total 20
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t1.orig
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t2.orig
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t3.orig
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t4.orig
-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t5.orig

 Diff of source and destination
Only in destination: backup
( edit 1: typo )

Last edited by makyo; 03-12-2007 at 11:43 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync question nazs Linux - Software 1 10-23-2006 07:31 PM
Rsync Question stefaandk Linux - General 2 05-12-2006 11:12 PM
rsync question juanb Linux - General 10 01-28-2006 03:18 PM
Another Rsync question phatboyz Linux - Networking 2 07-02-2004 08:46 AM
rsync question falco365 Programming 4 05-25-2003 09:24 AM


All times are GMT -5. The time now is 02:24 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration