Backup Application with pen drive as data carrier between 2 computers.
 

Problem:
I am having computers A and B having same copy of data (around 20 gigabytes) and a pen drive (~2G capacity).
The changes will be made in data of computer A.

I want to sync data (~20G) from computer A to B using pen drive (can not connect two computers directly).

Since pen drive ~2G capacity < data ~20G.
This implies, I can't use
rsync -av /data/ /mnt/pendrive
(will copy 20G to pen drive, hence overflow).

What I need:
Application which has feature:
1) Store changes to host side, hence doesn't require connecting computer B to computer A for finding changes.
2) Dump only changes made in data (between computer A and B) like rsync, to pen drive (which is initially empty).

so that I would be able to use 2G pen drive for actually synching 20G data between both computers.

Depending on the way the data is, you might be able to use the way some distro's send the changes to data. I forget what it is now. It just sends the changes.
You see it once in a while when downloading iso's. I can't find it now.

I get the feeling rsync can do it too somehow.


Any chance you could compress that 20G down that far? Or possibly split the file?

Actually splitting/compressing wouldn't serve my purpose as I want to sync the changes made in computer A to B on daily basis.
This requires only transferring delta (changes) from A to B.

Splitting 20G data into 10 or 11 pieces and manually backing up daily would be waste of time (certainly would be a headache for me, you know :) ).

As far as I know about rsync it connects to destination (in my case it would be computer B) then first get md5sum of all files on it (md5sum of blocks of files), then compare these with md5sum of source files (in my case it would be computer A).
Hence for rsync to work, it needs to connect to computer B first.
Which I can't do.

In terms of rsync command, I can say

What rsync does:
Note Before: /source and /destination has almost same data.
rsync -av /source /destination
(only deltas/changes are passed to destination, not whole source.)

What I need:
rsync /source /temp
(Now temp has only delta/changes not whole source.)
then

rsync /temp /destination
(deltas from /temp are used with already existing /destination data to update new data on /destination.)

(If rsync doesn't do it, then I want to know which command can do?)

Would it be cheaper in time and effort to just purchase a 32GB flash drive? Can you assume the changes will always be <2GB?

The --itemize-changes and -v options might work in terms of providing a list of files that were transferred so you could save that output to a file. A bash script could parse the file and then copy those that were transferred (between /source and /temp) to the flash drive.

Another approach would be to use the find command to locate modified files and them pipe them to the tar command saving it to the flash drive.

You could possibly use xdelta. You would probably need to compress your data into one file on machine A, run xdelta, then apply the delta to the same compressed file on machine b. A distro I work on does this with changes to .iso files. we move changes 18mb to 100mb at a time instead of 700mb. I really don't know how to use it to make the .delta file, but it sounds similar to what you want.

Maybe cpio + tar's split fuction?
http://rightsock.com/~kjw/Ramblings/tar_v_cpio.html

32G flash drive would shorten this exercise considerably. ;)

@michaelk, @Habitual
Actually the changes in my data to be backed up would not go more than ~1G in a week,
and yeah
I would love to buy an external hard drive someday.:)

@dolphin_oracle, @Habitual
I would certainly look into that.
But meanwhile I have got rsync option --compare-dest

Although I have not tried it practically for my main backup, but I have experimented as follows..
This worked for me and only dumped changed files into /pendrive.
Also
syncs changed files to the destination.

Thus these two commands provide me what I need, i.e.
This I can also combine with sanpshots backup feature with utility like rsnapshot, backintime etc. (using a snapshot as /destination). More on this I will post later when I will implement it practically.

Since the solution command is obtained, so I am now marking this as solved.