[SOLVED] rsync keeps overwriting some unmodified files

figure002 · 01-07-2010, 01:29 PM

Hello people.

I'm having this strange problem with rsync. I'm trying to synchronize my music collection with the music collection on my mp3 player (Cowon iAUDIO). The mp3 player has a harddisk with the FAT32 file system. The problem is that it keeps overwriting unmodified files on the mp3 player.

I use this command to synchronize:

Code:

rsync -avhpu --delete --progress --exclude-from "/home/serrano/backup-filters-iaudio.txt" "/windows/D/music/" "/media/IAUDIO/music/"

The above command should work, because I use the very same command (with different paths) to backup my linux home folder to an NTFS external harddisk. And that works just fine.

I've done some test to get a view of what's going on:

*Connect the IAUDIO*
*run the rsync command*
See attachment: rsync_result1.txt
*run the rsync command again*
See attachment: rsync_result2.txt (so far so good)
*Disconnect the IAUDIO*
*Connect the IAUDIO*
*run the rsync command again*
See attachment: rsync_result3.txt (overwrites unmodified files)

First, everything seems to work just fine. As seen in rsync_result2.txt, rsync skips the files that have already been synchronized.
You would expect the same result once I reconnect the mp3 player. But as seen in in rsync_result3.txt, some unmodified files are overwritten again. What also is very strange, is that it keeps overwriting the same files while it skips others without an apparent reason.

Could someone please explain to me what's happening?

System information:
openSUSE 11.2
rsync version 3.0.6 protocol version 30

rweaver · 01-07-2010, 01:41 PM

Have you tried md5suming the files before and after rsync3?

figure002 · 01-07-2010, 01:57 PM

Quote:

Originally Posted by rweaver

Have you tried md5suming the files before and after rsync3?

80d5ee2fa4823f7877aa56a991724687 01. Arctic Monkeys - My Propeller.mp3
*reconnect mp3*
80d5ee2fa4823f7877aa56a991724687 01. Arctic Monkeys - My Propeller.mp3
*run rsync command* ("01. Arctic Monkeys - My Propeller.mp3" is being overwritten)
80d5ee2fa4823f7877aa56a991724687 01. Arctic Monkeys - My Propeller.mp3
*run rsync command again* ("01. Arctic Monkeys - My Propeller.mp3" is ignored)
80d5ee2fa4823f7877aa56a991724687 01. Arctic Monkeys - My Propeller.mp3

So it indeed overwrites unmodified files. Could this have anything to do with the file dates?

rweaver · 01-08-2010, 11:04 AM

Code:

core:~/test/test20$ touch fred-fred-burger
core:~/test/test20$ ls -al fred-fred-burger
-rw-r--r-- 1 core core 0 2010-01-08 12:00 fred-fred-burger
core:~/test/test20$ md5sum fred-fred-burger
d41d8cd98f00b204e9800998ecf8427e  fred-fred-burger

core:~/test/test20$ touch fred-fred-burger
core:~/test/test20$ ls -al fred-fred-burger
-rw-r--r-- 1 core core 0 2010-01-08 12:01 fred-fred-burger
core:~/test/test20$ md5sum fred-fred-burger
d41d8cd98f00b204e9800998ecf8427e  fred-fred-burger

core:~/test/test20$ touch fred-fred-burger
core:~/test/test20$ ls -al
total 12
drwxr-xr-x  2 core core 4096 2010-01-08 12:00 .
drwxr-xr-x 27 core core 4096 2010-01-08 12:06 ..
-rw-r--r--  1 core core   46 2010-01-08 11:45 datafile
-rw-r--r--  1 core core    0 2010-01-08 12:07 fred-fred-burger

core:~/test/test20$ mkdir ../test21
core:~/test/test20$ rsync -avhe ./* ../test21
core:~/test/test20$ ls -al ../test21
total 12
drwxr-xr-x  2 core core 4096 2010-01-08 12:07 .
drwxr-xr-x 27 core core 4096 2010-01-08 12:06 ..
-rw-r--r--  1 core core   46 2010-01-08 12:07 datafile
-rw-r--r--  1 core core    0 2010-01-08 12:07 fred-fred-burger

core:~/test/test20$ ls -al
total 12
drwxr-xr-x  2 core core 4096 2010-01-08 12:00 .
drwxr-xr-x 27 core core 4096 2010-01-08 12:06 ..
-rw-r--r--  1 core core   46 2010-01-08 11:45 datafile
-rw-r--r--  1 core core    0 2010-01-08 12:07 fred-fred-burger


core:~/test/test20$ touch fred-fred-burger
core:~/test/test20$ ls -al
total 12
drwxr-xr-x  2 core core 4096 2010-01-08 12:00 .
drwxr-xr-x 27 core core 4096 2010-01-08 12:06 ..
-rw-r--r--  1 core core   46 2010-01-08 11:45 datafile
-rw-r--r--  1 core core    0 2010-01-08 12:11 fred-fred-burger

core:~/test/test20$ rsync -avhe ./* ../test21
sending incremental file list
fred-fred-burger

sent 80 bytes  received 31 bytes  222.00 bytes/sec
total size is 0  speedup is 0.00

core:~/test/test20$ ls -al
total 12
drwxr-xr-x  2 core core 4096 2010-01-08 12:00 .
drwxr-xr-x 27 core core 4096 2010-01-08 12:06 ..
-rw-r--r--  1 core core   46 2010-01-08 11:45 datafile
-rw-r--r--  1 core core    0 2010-01-08 12:11 fred-fred-burger

core:~/test/test20$ ls -al ../test21
total 12
drwxr-xr-x  2 core core 4096 2010-01-08 12:11 .
drwxr-xr-x 27 core core 4096 2010-01-08 12:06 ..
-rw-r--r--  1 core core   46 2010-01-08 12:07 datafile
-rw-r--r--  1 core core    0 2010-01-08 12:11 fred-fred-burger

core:~/test/test20$ md5sum ./fred-fred-burger
d41d8cd98f00b204e9800998ecf8427e  fred-fred-burger
core:~/test/test20$ md5sum ../test21/fred-fred-burger
d41d8cd98f00b204e9800998ecf8427e  ../test21/fred-fred-burger

core:~/test/test20$ md5sum ./datafile
65d20a3372dc1d4d9deac22e0d6a987d  ./datafile
core:~/test/test20$ md5sum ../test21/datafile
65d20a3372dc1d4d9deac22e0d6a987d  ../test21/datafile

So yes, it will resend each time the time stamp changes. You can use --modify-window to avoid some of that behavior. Check the manpage.

figure002 · 01-08-2010, 01:16 PM

Quote:

Originally Posted by rweaver

So yes, it will resend each time the time stamp changes. You can use --modify-window to avoid some of that behavior. Check the manpage.

rsync manpage:

Code:

       --modify-window
              When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than  the  modify-window  value.
              This  is  normally  0  (for  an  exact  match),  but  you  may  find it useful to set this to a larger value in some situations.  In
              particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a  2-second  resolution),  --mod-
              ify-window=1 is useful (allowing times to differ by up to 1 second).

That did the trick! So it's the FAT filesystem that's causing this. Thank you very much rweaver!

So for everyone who's having the same problem, here's my modified rsync command:

Code:

rsync -avhpu --delete --progress --modify-window 1 --exclude-from "/home/serrano/backup-filters-iaudio.txt" "/windows/D/music/" "/media/IAUDIO/music/"