LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-12-2014, 06:31 AM   #1
s.verma
Member
 
Registered: Oct 2013
Distribution: Debian Sid, Gentoo, Arch, Debian
Posts: 184
Blog Entries: 4

Rep: Reputation: 25
Backup Application with pen drive as data carrier between 2 computers.


Problem:
I am having computers A and B having same copy of data (around 20 gigabytes) and a pen drive (~2G capacity).
The changes will be made in data of computer A.

I want to sync data (~20G) from computer A to B using pen drive (can not connect two computers directly).

Since pen drive ~2G capacity < data ~20G.
This implies, I can't use
rsync -av /data/ /mnt/pendrive
(will copy 20G to pen drive, hence overflow).

What I need:
Application which has feature:
1) Store changes to host side, hence doesn't require connecting computer B to computer A for finding changes.
2) Dump only changes made in data (between computer A and B) like rsync, to pen drive (which is initially empty).

so that I would be able to use 2G pen drive for actually synching 20G data between both computers.
 
Old 02-12-2014, 04:35 PM   #2
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,129

Rep: Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430Reputation: 3430
Depending on the way the data is, you might be able to use the way some distro's send the changes to data. I forget what it is now. It just sends the changes.
You see it once in a while when downloading iso's. I can't find it now.

I get the feeling rsync can do it too somehow.


Any chance you could compress that 20G down that far? Or possibly split the file?

Last edited by jefro; 02-12-2014 at 04:38 PM.
 
Old 02-13-2014, 04:20 AM   #3
s.verma
Member
 
Registered: Oct 2013
Distribution: Debian Sid, Gentoo, Arch, Debian
Posts: 184

Original Poster
Blog Entries: 4

Rep: Reputation: 25
Actually splitting/compressing wouldn't serve my purpose as I want to sync the changes made in computer A to B on daily basis.
This requires only transferring delta (changes) from A to B.

Splitting 20G data into 10 or 11 pieces and manually backing up daily would be waste of time (certainly would be a headache for me, you know ).

As far as I know about rsync it connects to destination (in my case it would be computer B) then first get md5sum of all files on it (md5sum of blocks of files), then compare these with md5sum of source files (in my case it would be computer A).
Hence for rsync to work, it needs to connect to computer B first.
Which I can't do.

In terms of rsync command, I can say

What rsync does:
Note Before: /source and /destination has almost same data.
rsync -av /source /destination
(only deltas/changes are passed to destination, not whole source.)

What I need:
rsync /source /temp
(Now temp has only delta/changes not whole source.)
then

rsync /temp /destination
(deltas from /temp are used with already existing /destination data to update new data on /destination.)

(If rsync doesn't do it, then I want to know which command can do?)
 
Old 02-13-2014, 08:30 AM   #4
michaelk
Moderator
 
Registered: Aug 2002
Posts: 21,773

Rep: Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246Reputation: 4246
Would it be cheaper in time and effort to just purchase a 32GB flash drive? Can you assume the changes will always be <2GB?

The --itemize-changes and -v options might work in terms of providing a list of files that were transferred so you could save that output to a file. A bash script could parse the file and then copy those that were transferred (between /source and /temp) to the flash drive.

Another approach would be to use the find command to locate modified files and them pipe them to the tar command saving it to the flash drive.
 
Old 02-13-2014, 09:55 AM   #5
dolphin_oracle
MX Linux
 
Registered: Dec 2013
Posts: 402

Rep: Reputation: Disabled
You could possibly use xdelta. You would probably need to compress your data into one file on machine A, run xdelta, then apply the delta to the same compressed file on machine b. A distro I work on does this with changes to .iso files. we move changes 18mb to 100mb at a time instead of 700mb. I really don't know how to use it to make the .delta file, but it sounds similar to what you want.
 
Old 02-13-2014, 11:26 AM   #6
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Maybe cpio + tar's split fuction?
http://rightsock.com/~kjw/Ramblings/tar_v_cpio.html

32G flash drive would shorten this exercise considerably.
 
Old 02-14-2014, 04:12 AM   #7
s.verma
Member
 
Registered: Oct 2013
Distribution: Debian Sid, Gentoo, Arch, Debian
Posts: 184

Original Poster
Blog Entries: 4

Rep: Reputation: 25
Quote:
Would it be cheaper in time and effort to just purchase a 32GB flash drive? Can you assume the changes will always be <2GB?
Quote:
32G flash drive would shorten this exercise considerably.
@michaelk, @Habitual
Actually the changes in my data to be backed up would not go more than ~1G in a week,
and yeah
I would love to buy an external hard drive someday.

Quote:
You could possibly use xdelta..
Quote:
Maybe cpio + tar's split fuction?
@dolphin_oracle, @Habitual
I would certainly look into that.
But meanwhile I have got rsync option --compare-dest

Although I have not tried it practically for my main backup, but I have experimented as follows..
Code:
rsync -vcrlDh --stats --progress --compare-dest=/destination/ /source/ /pendrive
This worked for me and only dumped changed files into /pendrive.
Also
Code:
rsync -vcrlDh --stats --progress /pendrive /destination
syncs changed files to the destination.

Thus these two commands provide me what I need, i.e.
Quote:
What I need:
rsync /source /temp
(Now temp has only delta/changes not whole source.)
then

rsync /temp /destination...
This I can also combine with sanpshots backup feature with utility like rsnapshot, backintime etc. (using a snapshot as /destination). More on this I will post later when I will implement it practically.

Since the solution command is obtained, so I am now marking this as solved.

Last edited by s.verma; 02-14-2014 at 04:13 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Software required for recovering data from corrupted pen drive shakes82 Linux - Software 2 07-09-2010 06:11 AM
DSL on a Pen Drive with home data abaksh1 DamnSmallLinux 2 02-04-2010 04:10 PM
how can i browse data from pen drive in redhat linux 9 dhir_dhiru Linux - Newbie 3 01-05-2009 02:22 AM
preventing from copy data to a pen drive varunkant Ubuntu 2 12-01-2008 12:56 PM
data missing in usb pen drive when plugout/plugin Jacksytoh Linux - Hardware 10 11-17-2003 01:51 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 11:07 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration