LinuxQuestions.org - rsync question

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - rsync question (https://www.linuxquestions.org/questions/linux-newbie-8/rsync-question-535957/)

Hi,

I am using Mandriva 2007. I wanted to get a basic working solution for backing up my data. So i am using Rsync to take backups of my internal drive on my external USB drive.

i am using rsync with --delete switch, which removes the files from the backup, that have been removed from the source. Now i was wondering, if it is possible to configure rsync to instead of deleting such files, it should move them to a separate folder.

Like for example, it should synchornize the source with destination, and the files that have been deleted, should be moved to a "deleted files" directory.

Any suggestions on how it may be done.

Hi,

don't think that rsync provides such feature.

Don't know exactly, but it could be possible that the ouput of a dry rsync run provides enough information to extract the files that would become deleted by a real run.
You can move them by script before starting the real rsync run.
Not that handy, I know ;-)

well it is an idea nevertheless.. i'll try that tonight ...

Like the above poster said, you can diff command to compare directories, move the directories that don't match between source and destination and then run rsync.

Hi

Another option is to install the program "rdiff-backup". Instead of deleting the files that have been deleted (or changing the changed files) it automaticly puts diffs so you can recreate backup from any time. Most distros should have rdiff-backup, you just need to install it.

To make a backup, e.g.:
rdiff-backup /home /mnt/usbdisk

/mnt/usbdisk/home will be a plain copy /home but it will have a special directory called "rdiff-backup-data" in it with diffs for files deleted or changed.

If you run that command a few times, you can try:
rdiff-backup -l /mnt/usbdisk
This will give you a list of all the times you have taken backups. Any of them can be restored using
rdiff-backup --restore-as-of TIME /mnt/usbdisk /tmp

Because the usb disk will fill up with diffs if the files change a lot, you need to delete very old diffs. This will remove diffs older than 32 days:
rdiff-backup --remove-older-than 32D

In sum: I think rdiff-backup is really neat - having all the benefits of rsync plus the ability to keep files even if they are changed or deleted.

Hi.

Quote:

... if it is possible to configure rsync to instead of deleting such files, it should move them to a separate folder.

I use rsync to keep a directory "current" in sync with my home directory. If there are extant files in current" that will be over-written, they are first copied to a directory which has the current date as part of the name. I keep these incremental backups around for 30 days or so.

The option that controls this is:

Quote:

--backup-dir=DIR
In combination with the --backup option, this tells rsync to
store all backups in the specified directory. This is very use-
ful for incremental backups. You can additionally specify a
backup suffix using the --suffix option (otherwise the files
backed up in the specified directory will keep their original
filenames). If DIR is a relative path, it is relative to the
destination directory (which changes in a recursive transfer).

Here is a copy of the log for today, as an example:

Code:

 Backup date =  Fri Mar 9 02:05:02 CST 2007

 Creating archive directory /backup/xandros/makyo/2007-03-09

 Using directory name 2007-03-09, path /backup/xandros/makyo/2007-03-09

 Backing up from /home/makyo to /backup/xandros/makyo/current,

 excluding patterns from file /backup/xandros/makyo/exclude,

 and archiving older versions into /backup/xandros/makyo/2007-03-09

 After backup, the size of /backup/xandros/makyo/current is 117750

Here is a copy of the specific rsync command (apologies for the heavy parameterization):

Code:

rsync -abv $mode --backup-dir=$BACKUPDIR --delete \

--exclude-from=$BB/exclude --delete-excluded \

$FROMDIR $CURRENT > $BACKUPDIR/$DATE.log \

2> $BACKUPDIR/$DATE.err

You may need to fiddle with other options to get precisely what you want. I usually make a few test cases and base my production on that. I been using this scheme for around a year and it seems to have worked well so far.

Best wishes ... cheers, makyo

If the goal are incremental backups (instead of a directory containing all deleted files as the OP wrote), you can use tools like dirvish (command line) or backuppc (daemon with cgi-interface).
Both use rsync (backuppc knows other "treansports", too) to keep a remote dir in sync, storing only changed files.
Files that did not change become hardlinked and don't need additional space on the disk. You get "snapshot like incrementals" in a comfortable manner.

Hi, Clemente.

My scheme does not capture deleted files, only the originals for files scheduled to be replaced. Apologies if I mislead anyone.

It does appear to me that option -b, --backup is intended to keep transferred or deleted files:

Quote:

With this option, preexisting destination files are renamed as
each file is transferred or deleted ...

If I get some time, I'll investigate that ... cheers, makyo

Hi.

Indeed, rsync is able to copy files that would be deleted. The heart of this is a command such as:

Code:

rsync -abv --delete --backup-dir=$B --suffix=.orig $S/ $D/

which will compare the contents of the source, $S, and the destination, $D. All files that would be deleted (because they are in $D but not in $S) and all older versions of files will be collected in the directory $B. The $B directory is tucked into the $D directory. In this case, the files will have .orig tacked onto the end of their names.

If this is of interest, I can post the test script and the results ... cheers, makyo

Thanks a lot guys....

And yes makyo, i would really appreciate if you can post the test scripts etc. That would really be helpful.

And soory for replying late, but my net was down for the past two days...

Hi, mkhan919.

OK, here it is.

There are some debugging aids built-in so that you can turn on and off explanatory messages.

A number of files are created in the source directory. These are copied to the destination directory. The dates of files in the latter are adjusted by the odd call to touch so that they will appear to be older to rsync, and thus be candidates for copying.

One file in source will be deleted to create the situation where rsync will schedule the file for deletion in the destination.

The final result is that all files are updated, the two directories are synchronized, and the originals and the one deleted file will be placed in the backup directory. To emphasize that placement, I added a suffix for the filenames. A recursive diff shows that the only item different is the backup directory.

The output from the script should illustrate these points.

Best wishes ... cheers, makyo

Code:

#!/bin/sh



# @(#) s1      Demonstrate rsync copy from source to destination.



debug=":"

debug="echo"



# Remove previous debris.



S=source

D=destination

B=backup



rm -rf $S $D

mkdir $S $D



cd $S

echo one >t1

echo two >t2

echo thr >t3

echo fou >t4

echo fiv >t5

cd ..

cp $S/* $D

rm $S/t3



# Back date the destination.



backdate=$( date --date="5 minutes ago" )

$debug " now = $(date);  backdate = :$backdate:"



touch --date="$backdate" $D/*



echo

echo " $S:"

ls -l $S



echo

echo " $D:"

ls -l $D



rsync -abv --delete --backup-dir=$B --suffix=.orig $S/ $D/



echo

echo " $D, final"

ls -l $D



echo

echo " $D/$B"

ls -l $D/$B



echo

echo " Diff of $S and $D"

diff -r $S $D



exit 0

Which, when run, produces:

Code:

% ./s1

 now = Mon Mar 12 11:40:13 CST 2007;  backdate = :Mon Mar 12 11:35:13 CST 2007:



 source:

total 16

-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t1

-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t2

-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t4

-rw-r--r--  1 makyo makyo 4 Mar 12 11:40 t5



 destination:

total 20

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t1

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t2

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t3

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t4

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t5

building file list ... done

deleting t3

t1

t2

t4

t5



sent 308 bytes  received 100 bytes  816.00 bytes/sec

total size is 16  speedup is 0.04



 destination, final

total 16

drwxr-xr-x  2 makyo makyo 168 Mar 12 11:40 backup

-rw-r--r--  1 makyo makyo  4 Mar 12 11:40 t1

-rw-r--r--  1 makyo makyo  4 Mar 12 11:40 t2

-rw-r--r--  1 makyo makyo  4 Mar 12 11:40 t4

-rw-r--r--  1 makyo makyo  4 Mar 12 11:40 t5



 destination/backup

total 20

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t1.orig

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t2.orig

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t3.orig

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t4.orig

-rw-r--r--  1 makyo makyo 4 Mar 12 11:35 t5.orig



 Diff of source and destination

Only in destination: backup

( edit 1: typo )