rsync keeps backing up same folders over and over

Red Squirrel · 01-04-2009, 02:00 PM

I have a backup server with some rsync scripts that mount a samba share, back it up, then unmount it. (does windows and linux boxes)

I've noticed that at random, it seems to always copy certain folders even though they have not been altered. Anyone ever see something like this before?

It's weird. I can run the same job twice in a row, you'd think the second time wont copy anything, but wrong, it copies certain folders again even though they were not touched. These are data folders and not system folders. For example one particular folder it does this with is a folder with nintendo emulators and roms. I have not touched that in like a year yet it always gets backed up even though the copy on the backup drive matches.

AuroraCA · 01-04-2009, 05:39 PM

What is the rsync command with options that you are using? Please post the actual command line.

Red Squirrel · 01-04-2009, 07:32 PM

It's actually a shell script where everything gets passed to it but the end result is this:

Code:

rsync --exclude-from=/store1/exclude -vru /source /destination

AuroraCA · 01-04-2009, 09:36 PM

You should change your script rsync command to:

Code:

rsync --exclude-from=/store1/exclude -avu /source /destination

adding the "a" option will copy only files that have changed since the last backup. The "v" displays information about the backup as it is being done. The "u" option does not change any files in the backup which have dates newer than the files you are copying. The "r" option is not necessary since it is implied in the "a" option.

Code:

-a, --archive
    This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and
    want to preserve almost everything.

    Note however that -a does not preserve hardlinks, because finding multiply-linked
    files is expensive. You must separately specify -H.

-v, --verbose
    This option increases the amount of information you are given during the transfer. 
    By default, rsync works silently. 
    A single -v will give you information about what files are being transferred 
    and a brief summary at the end. 
    Two -v flags will give you information on what files are being skipped and
    slightly more information at the end.
    More than two -v flags should only be used if you are debugging rsync.

-u, --update
    This forces rsync to skip any files for which the destination file already
    exists and has a date later than the source file.

-r, --recursive
    This tells rsync to copy directories recursively. 
    If you don't specify this then rsync won't copy directories at all.

For further explanations of rsync command options see: http://www.ss64.com/bash/rsync_options.html

billymayday · 01-04-2009, 10:00 PM

rsynch only copies changed files by default. The "a" option doesn't impact that behavior.

Quote:

Rsync finds files that need to be transferred using a "quick check" algorithm (by default)that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.

RS, I'd guess that some process is modifying the last modified time of the relevant files. Can you perhaps take some random "stats" (ie stat filename) of likely culprits and compare before and after? Not easy I know, but perhaps you could write a script to collect the necessary info into a file for later use. Your untouched directory would be a good candidate.

AuroraCA · 01-04-2009, 10:22 PM

Quote:

Originally Posted by billymayday

rsynch only copies changed files by default. The "a" option doesn't impact that behavior.

Thanks billymayday. I was under a false impression about the way the rsync command worked. I appreciate your correction.

Red Squirrel · 01-04-2009, 10:39 PM

Quote:

Originally Posted by billymayday

rsynch only copies changed files by default. The "a" option doesn't impact that behavior.

RS, I'd guess that some process is modifying the last modified time of the relevant files. Can you perhaps take some random "stats" (ie stat filename) of likely culprits and compare before and after? Not easy I know, but perhaps you could write a script to collect the necessary info into a file for later use. Your untouched directory would be a good candidate.

When I get the chance I'll go through my backup logs in my cron account, they go back a few years. What's odd is it seems random.

Actually could a defrag cause a file to appear as changed? Would not explain why it's always the same directories that seem to be affected mind you.

Also I was thinking maybe the time on the backup server was off but I checked and it's ok. I have a local NTP server and it syncs to it before running the backup job and also has the ntpd service running.

billymayday · 01-04-2009, 11:21 PM

You don't defrag a samba share do you?

According to the docs (see above), rsync works off mod time and size. Assuming the size isn't changing, that leaves the modified time. Moving, copying, etc will change that.

Red Squirrel · 01-04-2009, 11:34 PM

well the samba share is on windows boxes and those are scheduled to be defragged every now and then.

billymayday · 01-04-2009, 11:37 PM

So do you mean that the drive space is physically on a windows box? That would probably explain it then.

There is a windows version of rsync btw, I tried it out recently. Not sure how it would deal with this one.