Hi

I am trying to rsync the big files, all bigger than 20GB from one directory to another in the same server. The file gets touched every hour or it might just make minute change to files but when the rsync is run, it will again rsync all the big files from beginning for a long time even when little have changed. I have tried rsync with all the option I can imagine --inplace, -c, -no-whole-file but have not been able to do it faster to copy the files from one directory to another. How do I do the delta-transfer in same server. Thank you for your input. :-)

I don't know the rsync algorithm in detail, but I guess the source file needs to be scanned to compare it to the destination file? In essence, it needs to be read in its entirety to figure out which blocks were modified. If I am right, you don't gain anything. rsync is useful when you transfer your files over a link that is slower than your disks.

Disclaimer: I am just speculating.

Of course it has to re-calc the delta each time, its not magic you know
As you say, for a local disk-to-disk or on the same disk/partition, a straight/full copy each time might be quicker.
You'd really have to do some tests to check.

I think, from memory (and I am pointing out that this is a wildly unreliable resource!) that this is in area in which Unison can be more efficient than rsync. Well, if you have huge files, even if, conceptually, you diff them and send the differences compressed that can be problematic, too, for most applications the conduit between the source and destination locations is the bottleneck, and, in that case, unison can be slicker.

I'd put this in to the category 'worth checking out', but YMMV.

Edit

I'd be in dereliction of my duty (...whatever that is...) if i didn't point out that I've just noticed that there is a running thread 'Rsync vs Unison'

http://www.linuxquestions.org/questi...on-4175488585/

OTOH, don't get your hope up that high, as it probably isn't going to be as useful to you as you might think from the title, as the implied use case is quite different from yours.

HTH

Just something to evolv. Maybe lvm snapshots could help here. Starting from a synced point you activate lvm snapshot on the source directory. Then after some time you grab that snapshot and put the changes onto the destination. Then start a new snapshot on source. Have no clue if this is even remotly possible but this way you would at least have only the delta somewhere.