LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
LinkBack Search this Thread
Old 01-04-2009, 02:00 PM   #1
Red Squirrel
Member
 
Registered: Dec 2003
Distribution: FC9 on main server
Posts: 652

Rep: Reputation: 33
rsync keeps backing up same folders over and over


I have a backup server with some rsync scripts that mount a samba share, back it up, then unmount it. (does windows and linux boxes)

I've noticed that at random, it seems to always copy certain folders even though they have not been altered. Anyone ever see something like this before?

It's weird. I can run the same job twice in a row, you'd think the second time wont copy anything, but wrong, it copies certain folders again even though they were not touched. These are data folders and not system folders. For example one particular folder it does this with is a folder with nintendo emulators and roms. I have not touched that in like a year yet it always gets backed up even though the copy on the backup drive matches.
 
Old 01-04-2009, 05:39 PM   #2
AuroraCA
Member
 
Registered: Jul 2008
Location: Northern CA USA
Distribution: Ubuntu, Slackware, Gentoo, Fedora, Red Hat, Puppy Linux
Posts: 370

Rep: Reputation: 35
What is the rsync command with options that you are using? Please post the actual command line.
 
Old 01-04-2009, 07:32 PM   #3
Red Squirrel
Member
 
Registered: Dec 2003
Distribution: FC9 on main server
Posts: 652

Original Poster
Rep: Reputation: 33
It's actually a shell script where everything gets passed to it but the end result is this:


Code:
rsync --exclude-from=/store1/exclude -vru /source /destination
 
Old 01-04-2009, 09:36 PM   #4
AuroraCA
Member
 
Registered: Jul 2008
Location: Northern CA USA
Distribution: Ubuntu, Slackware, Gentoo, Fedora, Red Hat, Puppy Linux
Posts: 370

Rep: Reputation: 35
You should change your script rsync command to:

Code:
rsync --exclude-from=/store1/exclude -avu /source /destination
adding the "a" option will copy only files that have changed since the last backup. The "v" displays information about the backup as it is being done. The "u" option does not change any files in the backup which have dates newer than the files you are copying. The "r" option is not necessary since it is implied in the "a" option.

Code:
-a, --archive
    This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and
    want to preserve almost everything.

    Note however that -a does not preserve hardlinks, because finding multiply-linked
    files is expensive. You must separately specify -H.

-v, --verbose
    This option increases the amount of information you are given during the transfer. 
    By default, rsync works silently. 
    A single -v will give you information about what files are being transferred 
    and a brief summary at the end. 
    Two -v flags will give you information on what files are being skipped and
    slightly more information at the end.
    More than two -v flags should only be used if you are debugging rsync.

-u, --update
    This forces rsync to skip any files for which the destination file already
    exists and has a date later than the source file.

-r, --recursive
    This tells rsync to copy directories recursively. 
    If you don't specify this then rsync won't copy directories at all.
For further explanations of rsync command options see: http://www.ss64.com/bash/rsync_options.html
 
Old 01-04-2009, 10:00 PM   #5
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 120Reputation: 120
rsynch only copies changed files by default. The "a" option doesn't impact that behavior.

Quote:
Rsync finds files that need to be transferred using a "quick check" algorithm (by default)that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
RS, I'd guess that some process is modifying the last modified time of the relevant files. Can you perhaps take some random "stats" (ie stat filename) of likely culprits and compare before and after? Not easy I know, but perhaps you could write a script to collect the necessary info into a file for later use. Your untouched directory would be a good candidate.
 
Old 01-04-2009, 10:22 PM   #6
AuroraCA
Member
 
Registered: Jul 2008
Location: Northern CA USA
Distribution: Ubuntu, Slackware, Gentoo, Fedora, Red Hat, Puppy Linux
Posts: 370

Rep: Reputation: 35
Quote:
Originally Posted by billymayday View Post
rsynch only copies changed files by default. The "a" option doesn't impact that behavior.
Thanks billymayday. I was under a false impression about the way the rsync command worked. I appreciate your correction.
 
Old 01-04-2009, 10:39 PM   #7
Red Squirrel
Member
 
Registered: Dec 2003
Distribution: FC9 on main server
Posts: 652

Original Poster
Rep: Reputation: 33
Quote:
Originally Posted by billymayday View Post
rsynch only copies changed files by default. The "a" option doesn't impact that behavior.


RS, I'd guess that some process is modifying the last modified time of the relevant files. Can you perhaps take some random "stats" (ie stat filename) of likely culprits and compare before and after? Not easy I know, but perhaps you could write a script to collect the necessary info into a file for later use. Your untouched directory would be a good candidate.
When I get the chance I'll go through my backup logs in my cron account, they go back a few years. What's odd is it seems random.

Actually could a defrag cause a file to appear as changed? Would not explain why it's always the same directories that seem to be affected mind you.

Also I was thinking maybe the time on the backup server was off but I checked and it's ok. I have a local NTP server and it syncs to it before running the backup job and also has the ntpd service running.
 
Old 01-04-2009, 11:21 PM   #8
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 120Reputation: 120
You don't defrag a samba share do you?

According to the docs (see above), rsync works off mod time and size. Assuming the size isn't changing, that leaves the modified time. Moving, copying, etc will change that.
 
Old 01-04-2009, 11:34 PM   #9
Red Squirrel
Member
 
Registered: Dec 2003
Distribution: FC9 on main server
Posts: 652

Original Poster
Rep: Reputation: 33
well the samba share is on windows boxes and those are scheduled to be defragged every now and then.
 
Old 01-04-2009, 11:37 PM   #10
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 120Reputation: 120
So do you mean that the drive space is physically on a windows box? That would probably explain it then.

There is a windows version of rsync btw, I tried it out recently. Not sure how it would deal with this one.

Last edited by billymayday; 01-05-2009 at 12:22 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Rsync not backing up MySQL cabron Linux - Server 3 12-26-2007 07:56 AM
Backing up Folders tommytomato Linux - General 5 07-14-2006 09:34 AM
backing up to the net with ftp or rsync rblampain Linux - General 2 04-27-2006 12:40 AM
rsync and backing up danimalz Linux - Software 12 10-24-2005 12:01 PM
RSync or another way? Backing Up Off-site. kemplej Linux - Software 0 08-19-2004 04:55 PM


All times are GMT -5. The time now is 10:10 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration