Ok, so this one has me flummoxed more than somewhat.

I'm supposed to sync a folder (including sub-folders and contents) between one machine and another. Simple, and straightforward, right? Well, here's where it gets fun. rsync is supposed to sync the difference between the two folders (and their sub-folders and contents).

What this rsync command seems to do is re-sync all folders and contents irrespective of what is happening.

Note: As this is used to keep images (which aren't small things at the best of times) in sync between web servers, it's set in a cron job to run every minute, but fails to complete in less than four minutes. The server load creeps up, the server falls over, etc.

Here's the command I'm running:
And this is the result every time.
I've checked file ownership, and indeed set file ownership to be identical across both servers. And now the time stamps have been set to within seconds of each other, and it's just made things worse! HELP!

If the timestamps are not identical, then both the source and destination have to spend time generating and comparing checksums to determine how much changed data needs to be sent (just 9320 bytes for the case you presented). If the destination is on any variety of FAT filesystem, its timestamps have a resolution of 2 seconds, and any source timestamp that happens to be an odd number of seconds can never match. You can use rsync's "--modify-window=NUM" option to compare modification times with reduced accuracy. Setting that to 2 seconds is generally sufficient, but you might have to go to 3602 seconds to get around problems with Daylight Savings Time changes.

2 members found this post helpful.

Are either the source or destination filesystems FAT32?

1 members found this post helpful.

Hi rknichols and AlucardZero.

Thanks for both of your contributions. Thankfully, or unfortunately depending on your viewpoint, neither of the servers run FAT filesystems. They are both ext3. I have tried with the --modify-window set to 3602 just in case we had some stupid with the timezones going on, but that isn't the issue. I also tried (purely to see if there was an issue with the timezones) setting them both to be on UTC and then using NTP to get the clocks in sync. None of the above actions has helped.

Keep those suggestions rolling in, please! :-)

How far these machines are?? And how the data is being transferred?? I mean over a LAN or WAN?

Do the timestamps match after rsync has done its thing? That should happen regardless of whether the two clocks are in sync since rsync will use the utime() system call to copy the timestamp from the source to the destination.

You could try using the "--itemize-changes" option to see what differences rsync believes exist. (You'll need the manpage to interpret the result.)

1 members found this post helpful.

Hi

chandhokshashank - LAN connection although I'm not sure of the relevance
rknichols - it does appear to be the same, so no issues there. However, I'll try the --itemise-changes option with the man page as suggested. This week seems to be rsync week for me, so nothing to lose, etc.

Many thanks for suggestions so far, and not to appear ungrateful, but do keep them coming in, cos I read every one of them. I really do appreciate it, and I'll keep you all posted.

1. although you can specify the start time of a cron job down to a minute, creating a new process every minute hammers the system as it has to create an entire new env each time.
Its also quite common to end up having them trip over each other (as you've seen).
2. My rule of thumb is to write a daemon for anything more frequent than every 5 mins and just have it sleep at the bottom of the loop for however long you'd like.
This dramatically lessens the load and also prevents the problem of multiple copies running.

2 members found this post helpful.

As rknichols observed, the literal data sent is just 9320 bytes, so rsync is definitely not "re-syncing all folders and contents irrespective" as you suggested. However, it's comparing about 20% of the 80,000 files so the question is why. --itemize-changes will give you a clue, I suppose.

If it's not the time stamp, my guess is that the owner or group for some of the files is not getting transferred - perhaps because of permissions on the target.

Maybe it takes 3 minutes just to scan through the 80,000 files.

1 members found this post helpful.

quick reply as yesterday was spent managing server performance due to rsync issues (told you it was one of those weeks! ).

chrism01, your point sounds very interesting. I don't suppose you'd have a link to hand about creating daemons, would you? I'd surely appreciate it.

SecretCode and rknichols, now that these servers seem more stable, I'm going to do an in-depth investigation of --itemise-changes to understand this better and move myself a step closer to being an rsync guru!

Kudos and ratings to all of you wonderful contributors, and as before, keep them coming...

Further update now that I'm back and looking at it again:

Major props to rknichols and SecretCode for re-prompting - now I understand --itemize-changes. However having done the following:
and
to both servers, I'd have thought that this would have meant the timestamps were now the only issue, but on running the latest
I find that there are still files within this folder that say there are permissions issues, and are therefore rsyncing on the basis of timestamps and permissions.

What the hell am I missing here?

well, I've done the above and now run rsync twice. I'm most definitely confused by this - any help truly appreciated.

I'm not convinced that this is part of the cause of the problem, but I thought I'd throw it in there. Possible red herring alert

User is called owner, group is group.

On server server1 owner is listed in /etc/passwd as owner:x:1000:1000::/home/owner
On the other server owner is listed in /etc/passwd as owner:x:1000:441::/home/owner

Both these GIDs are for the group group
Is it possible that this discrepancy is the cause of the files with permission issues?

Could well be ... I'll bet that rsync compares numeric group ids not group names, and (unless you have an LDAP system in place) there's no guarantee these match between different hosts.

Try a test with (I think) --no-g

1 members found this post helpful.

No, as long as the same name exists on both machines, rsync defaults to comparing the names, not the numbers, with the exception of UID 0 and GID 0. See the section for "--numeric-ids" in the manpage.

2 members found this post helpful.