LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 04-17-2012, 08:02 AM   #1
amityweb
LQ Newbie
 
Registered: Apr 2012
Location: Caerphilly, Wales
Distribution: Ubuntu, Kubuntu, Centos, Fedora
Posts: 4

Rep: Reputation: Disabled
rsync is downloading the same unchanged files everytime


Hi

I am using rsync for backups. I use it with --link-dest to compare to the previous download.

I am rsyncing with about 40,000 files. It works fine apart from about 10 files, which it re-download everytime.

These 10 files are just images, .jpg files. They are in a folder with thousands of other images, all of which do not get re-downloaded... just these same 10 everytime.

Funnily I have an exact copy of this system, in a sub-folder on the server, and those same 10 files and also re-downloaded everytime in the sub-folder location too!

So I think something must be up with these files, and I wondered if anyone had any ideas what it might be, or where I should look.

Thanks a lot
 
Old 04-17-2012, 08:07 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,489

Rep: Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453
please show us the commands you executed: which flags were used. Are you using rsyncd also?
 
Old 04-17-2012, 08:12 AM   #3
amityweb
LQ Newbie
 
Registered: Apr 2012
Location: Caerphilly, Wales
Distribution: Ubuntu, Kubuntu, Centos, Fedora
Posts: 4

Original Poster
Rep: Reputation: Disabled
Oh yes good point... its taken me ages to cobble this together from various scripts online, to do it how I want to do it (which is folder name by date/time to keep copies rather than one copy)


Quote:
source="user@myhost:/mysource/"
destination="/mydest"

mkdir -p "$destination/Progress"

rsync -aPx --exclude '*cache' -e 'ssh' --delete --link-dest="$destination/Latest" "$source" "$destination/Progress" && date=`date "+%Y-%m-%d-%H%M%S"` && mv "$destination/Progress" "$destination/$date" && ln -sfh "$date" "$destination/Latest"
Oh, my server is Linux, my client machine is Mac
 
Old 04-17-2012, 08:48 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,489

Rep: Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453
I would check the times (also in link-dest dir), maybe that causes the transfer. Also I would try the --update flag.
also see the flag --size-only and the explanation:
Quote:
This modifies rsync's "quick check" algorithm for finding files that need to be transferred, changing it from the default of transferring files with either a changed size or a changed last-modified time to just looking for files that have changed in size. This is useful when starting to use rsync after using another mirroring system which may not preserve timestamps exactly.
 
Old 04-17-2012, 11:09 PM   #5
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,048

Rep: Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799Reputation: 1799
If the destination is on an FAT file system, the timestamp resolution is 2 seconds, so if the source file has a modification time that contains an odd number of seconds the destination timestamp will never match. The rsync command has a "--modify-window" option that helps deal with this.
 
Old 04-18-2012, 02:52 AM   #6
cliffordw
Member
 
Registered: Jan 2012
Location: South Africa
Posts: 501

Rep: Reputation: 189Reputation: 189
The "--itemize-changes" option may help you identify why the files are being transferred.
 
Old 04-18-2012, 03:14 PM   #7
amityweb
LQ Newbie
 
Registered: Apr 2012
Location: Caerphilly, Wales
Distribution: Ubuntu, Kubuntu, Centos, Fedora
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thanks for these ideas, I will try them when I get a chance, been busy lately. I got my backups working in the meantime so thats the important thing.

For info, its not Fat32, its Mac OSX Extended Journaled, and I am downloading from a Centos Linux box. But I assume it must be file related as its just these same 10 files.

I also deleted them on the client machine today, so rsync cant compare to anything and download them in full again, but did not have time to run rsync again to see if did the same thing.

I read about --modify-window and did not assume it would be as big as 2 seconds, so only tried it with 1 second which did not work, so will try with 2.

Will update you if anything works or not.

Thanks

Last edited by amityweb; 04-18-2012 at 03:20 PM.
 
Old 04-24-2012, 07:54 AM   #8
amityweb
LQ Newbie
 
Registered: Apr 2012
Location: Caerphilly, Wales
Distribution: Ubuntu, Kubuntu, Centos, Fedora
Posts: 4

Original Poster
Rep: Reputation: Disabled
the images still download with --modify-window=2

Not sure what --itemize-changes is telling me, I can see the images are still downloaded

I dont really want to rely on --size-only, there may be a change and the file will be the same size

--update does not work either

so very strange
 
Old 04-24-2012, 08:04 AM   #9
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,489

Rep: Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453Reputation: 3453
you can test if --size-only solves your problem, at least we will know if it helps. You may check the "current" time on both hosts. Maybe they are not synced. another tip: --omit-dir-times.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Downloading specific files from an rsync site Skaperen Linux - Software 2 11-30-2011 01:35 PM
rsync can not rsync files with include filter... xiutuo Linux - Server 2 07-23-2010 03:10 AM
need to rsync only selected files (--files-from) also need to delete files on dest. ? BrianK Linux - General 5 10-22-2009 10:52 PM
Could I run rsync to download files from a server without rsync daemon? Richard.Yang Linux - Software 1 09-18-2009 05:08 AM
[rsync] get the differences between the source files and the existing files djgerbavore Linux - Networking 2 06-04-2008 01:05 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 03:03 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration