LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Blogs > rainbowsally
User Name
Password

Notices


Rate this Entry

Incremental backup using diff for BIG projects

Posted 06-09-2013 at 01:46 AM by rainbowsally
Updated 06-09-2013 at 05:52 AM by rainbowsally

Note: This blog entry will to go away soon. Another version is in the works.

Note also that having the entire files rather than just a code snippet like a patch makes it easy see as much context as you need in order to recreate a change log by looking at ONLY the files that were changed from one build to another.

KDE's 'kompare' is an excellent utility for viewing changes.

So for now, since this is a little buggy (not serious but needs tweeking), consider it a puzzle.

It's still interesting and might inspire you to create something even better. :-)

Here's where the adventure begins....
-----------
[Update: The README in the installer's TREE says to use 'make-init'. Should be 'diff-init'. We'll fix that in a moment.]

I wanted to upload this as a collection of source files but it was a bit too large for the blog, so I created a self-extractor using 'new.sfxz' which is available here at the blog.

Have you ever been working on a huge collection of files and finding yourself having to revert so often that your backups blow up to hundreds of megs? Me too. In fact that's what's happening right now, so I wrote a utility that uses 'diff' to figure out just which files were changed, not the changes in the files, and using a base fileset as a reference for the bulk of the restored data, inserts, adds, or deletes files in the currently working build so that the exact same version can be restored with only a few files and a simple data base (text files).

Whether or not you have any use for this, it's interesting to look at. And there are no dependencies on anything except base64 (core utils) and lzma (the xz file compression) for the self extractor, and diff (a very common utility) and tar. Optionally, if you have the lq-qt/mc2 files installed, the tool called 'choose' is used to allow you to press Y/N for whether or not to bump the 'build' number.

How to use it is probably pretty self evident after a trial run, but if not, the diff-init shell script has a help switch that will get you started.

But take a look at how this might work for you, with this application installer as an unrealistically simple example.

Once the self-extractor runs you'll have a directory tree looking something like this

Run 'make install' to create the mirror image of the TREE in your own HOME/bin folder, which is probably already in your PATH so that you can use it immediately*.

-----------------------
* If you have to, you can add $HOME/bin to your PATH and either log out and back in, or for a quick test, run sh explicitly in a terminal to force reloading the shell rc file which is usually ~/.bashrc.
-----------------------

So now we're ready to try it out.
Code:
You are Here
|
`-- new-diff-backups-installer-v1.0
  |-- Makefile
  |-- TREE
  |   `-- bin
  |       `-- src
  |           `-- LQ-projects
  |               `-- new-diff-backups
  |                   |-- diff-backups
  |                   |   |-- Makefile
  |                   |   |-- README
  |                   |   |-- diffs
  |                   |   |-- mc2.def
  |                   |   `-- tools
  |                   |       |-- diff-backup
  |                   |       |-- diff-init
  |                   |       |-- diff-init-help
  |                   |       |-- diff-next
  |                   |       |-- diff-restore
  |                   |       |-- diff-revert
  |                   |       `-- test
  |                   |-- new.diff-backups
  |                   `-- test
  |-- dirs.dat
  |-- files.dat
  |-- links.dat
  |-- mains.dat
  |-- mk_postinstall
  `-- mk_postremove
To make a new diff-backups folder (or whatever you choose to rename it) for this file set, type
Code:
new.diff-backups
to create the backup folder along side the files you'll be backing up so you have this setup.

Now...

Code:
You are Here
|
|-- diff-backups                          * The place to make the backups
`-- new-diff-backups-installer-v1.0       *The Fileset's name
   `-- the rest of the files and folders
Then cd into the diff-backups folder.
Code:
cd diff-backups
At this point, if you try to make a backup you'll receive an error message that 'build number 001' can't be found. The build number is the last three digis of the directory name following a dash. So we create our first fileset by changing the name of the folder to *-001, like so:

Code:
You are Here
|
|-- diff-backups                      * The place to make the backups
`-- <anyName>.<anyVersion>-001        *The Fileset's name + "-001" is the first build number
This too can be done from the commandline.
Code:
mv new-diff-backups-installer-v1.0 new-diff-backups-installer-v1.0-001
The name "new-diff-backups-installer-v1.0-001" is a ligit fileset and build name because "-NNN" only occurs once in the name.

Now diff-init, which runs in the same folder as the Makefile as 'tools/diff-init', will create the initial fileset after prompting you for the name of the directory once. Obviously copy/paste can come in handy.

Type
Code:
tools/diff-init
and paste in the directory name without the '../' path prefix.

Then build 001 will exist and the Makefile will begin working.

diff-init will fall through to run 'make backup' and from now on you can use the Makefile which won't work until the fileset.txt is initialized with some kind of data.

So now...

Type
Code:
make targets
to get a quick overview of functions you're likely to need.

One such function you'll want to run immediately is 'make next-build'.

Type
Code:
make next-build
if it doesn't run automatically (which requires mc2).

And change the build number of the fileset to match.

Then change the stupid code to remove break points and stuff, if I forgot to fix that. And make your next backup.

In the build folder (the new fileset) in "diff-backup" comment out the echo line like so:
Code:
bp() 
{ 
  # echo "$1 (Press ENTER to proceed)"; read key; 
  printf "" # noop
}
And reinstall.

And then back up build #2 and bump the build number again, in case you do any more edits. :-)

Turns out the README has an obsolete name in it. Your next edit might be to fix that. Change 'make-init' to 'diff-init' and you're now on build #003. :-)

You can restore any build version non-destructively. The 'restore' function creates a tarball in the diff-backups folder.

Take a look at the diffs folder and unpack the diffs to see what ends up being able to recreate your build version.

It's quite efficient.

Again, this is for HUGE builds with lots of problems where you may need to revert many many times to previous builds one or even several levels below the current build number, before you can proceed again. But it's also kind of fun. :-)


file: new-diff-backups-installer-v1.0.sfxz
purpose: utility (executable)
Code:
#!/bin/sh
# base64 self extractor
# file: new-diff-backups-installer-v1.0

filename=new-diff-backups-installer-v1.0

# don't allow user to click on the file
if ! tty >/dev/null; then
  xmessage -center "
  $(basename $0)
  must be run in a terminal 
  "
  exit
fi

# create and/or clear out tmp dir
mkdir -p $HOME/tmp/sfxz
rm -rf $HOME/tmp/sfxz/*
base64 -d << _BEOF >$HOME/tmp/sfxz/$filename.xz
/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4Mf/FAtdABcLykbiXeoaTtyqfO8W+McR+caRY2ix2MTc
5L3t40EfYkdH3eJYDbJByDoURtuoFn6lEyLbgExXxi11bmXvt/K/cQU2ICR5GTj2BTbOL9wgFXLS
MRvYRVBT1ADB0lMg5ukehzJwNGPdpXTHLPMCZUiQLGWD6cTIfO8t4tW0XBbqiuhOTwhtsKcVd2KR
rknB3TO3ahYra7eGVjiT7LP+wrdAWnxrJ+2D5XtjlkVxt6e758rQapwiwqVpWUXWwMR3p997h14a
lSIq1oOSZ/5pFDAY52zl9pin7ej15gVo4x+hhwAV+hBWmWPYZ7AXkR+E8gi4+64hvNnMMlBkFEy7
b3wtoTHUb+AQKJyz2tRN5ElYcZ86guaYfG5m6p8XhtziwKc8cferf3yY8mi1P/H+f5NGjQQ9C26h
2gtJJj5dCDOvlpf/eSxnERlsvoY8WDb+l0Kp5kw/vMG47NtsKhzmatMnPwJrd846istsSj4DFC+w
UCwXUzaN15ANisOGAT2XKTdBHwlyq+6B3VAmPwLxcSktjwGZPlGoOL1LPgsquwTvntEXHpzEwtXN
D+rnyX7REAwvQhrFAatj53Ph71ceuBC5VSVtuvEI+tXTDk77CWdASKCrX2WXliLgqmUIrN79jghm
813JspaTgvD43x2MBUn+KL1WJ3a5B0vGicLD+gh4jaW3hOOgzVE7YIDgMsFpbf7sFqhKrV1mYUGV
8GhPWJIlgFy9VOlGTA6+hP+vOPBRlGdnx5TlsyaI8rQqQkxGGvBmGUIY/mE/sFjCgw4suVLjBKT2
2WoYHBT7Lne/9kLMYaDLggqh16OqLHdqPU0InxElnBUF208AJv9Ipg2aYU4d5ttXtkQWlMvDQ91h
sx/ODnqWl3ehZ7dxbLdnODOj3wtyF18JLno+pY9/lB2RD85HvptK+eDFuSEpoc8sTuIG2LvaJm9c
yIDxDt5DFZ9Y9442mKJJb65rTXgGiw964xobv3T3yd2pKy7UT+VvS5dxftcYwirdD/TddV6cxKD2
SV4kfHXolJnMrKpWP4vlDS2T9KsmMD/Tsg5hXWZwz6bsD92nYmIBi9UZR8BGO9b5RleGwvl6MN8K
0jLUAj8S9CrjLR3IBAXFQx9lqyp+3KIT9jwH6K+uLY+jgB9DFznS5r6yzseWZnVaSdQRCzayGEG9
i6DSOklUgbHbukHyLoJhhZGtALg23MfveTYmeYaKB5vRJQItDKdzk9KWTjD0/QOghh2sRq0PRZ95
JOE2jkB+pSbH4wCKsKUgh95tbDVkKAiCItYRuc1jSN277YBwdw8vmsdW4PS9QJLWQAnZpzp1yWoF
fAMoa1POEGLY2kkPYVWo/4cMs5K/dpZ6i2x9TD8NEmXoPkqpCjbj/kjP0IMdc91xCIVbKuzCr+xA
O16xlF3zbF+NNcrjBwQA5CYJI1HKzYGCa40fz3gh+/AXZsVEw+uhesWSOLjmSvwEAc3+0OL+4c0f
ovxuhjxBCcN/hkLowaLEN+r/xwJKGdMKtRcmoZYSUYCx/MYMFSRsjvdxFNs4+xUPJQ3JngVHc2yb
5bjoR9cJhCl8mPDoeiBI/ZNHGjnp8OaGv8mmCauY8TBSxFy2NMK8peoDTC3+DYh1O6F0CUtsyifI
Q1IlR1/sRLF80B6SEuxEbm8DXoxNd4mDfSIOyfdCHNlF9Q79Ki7NwGbymI527AR+4mDv52hxV0m/
JdW1f+xJjPZjkjF7Nn+UgKIF4aWxxDyE33VFOkgBB2wXFR4ArV5EQe6bEp93fBE2Do9RznUI1LJ5
TtLBK7P2pQYkasiZJcoKfKnduvIPXCy/DUKG6E/yN0wKcdb1zev66y6SphMsHh64Ykyaonpp/B17
sk76IqfPpTYlW9yiwn9jtw9kYeB3FbYUkfLkfCCgLQC1/U1x2bI6toKw05Qb7xqQQnYto/GVrCpe
7slr2LXTRF03zMjuSTIXpmx/HSXxscp0Kpa+GNPqOqBnx7Uoa92b1FFJIb3mwDI7p8+XKPTy/4om
wWAGsexHN6ooFAxCg0EVjD7QLM7eyQann+ZXpySq7992L8S4ooGkwVm39wJM5wY1oaWK0GlmkKm6
61C0sHOy4goY0IkSqbSKJad9qbObOjQIH2QxD3R1IbX+ehUMehBc7hrH1CI33OzsLjXmrWKBFaU8
AVA+Oth1uzETY166xRx8+M/T061EJtRMDw5e/VigQROohNqkMVJeosGsZbpXdrXxHNZw/7TXAiD8
hA58YyODVpyOwOkyjV6kVZprG3+ui7/pqLvU2CI0iGW9oRsOD6hq3CV2KpYzwwpHKex4v1KRUenc
MBLnmmnzoyV0SkUu4C4vLQ7luFLBXZ5Xwcex0KzOxzQSHdhmOfmg5GfR/x/vos6JRA9UxQ5MoYN4
VAa8Rysko0+y1olc69qABhSRFxiceNUw0HNlBfigvYjGiGUm0OY0GzIozjwbv19l125WLzkBkoSW
ZbAp1CYs3G4fvWDihbK7gpAfY+via07eqrixXSz1D6deiXnWJaHdVDo6L3XRtE7E8YUyRE8Nb0zX
Fyx4O062RaT+tGJz9kM0D7trcelFK4B6wgSinGyjfD7DaJDf1eTndhCLC67YuNr7hq7UhXS6tzN2
TJGgmdeBH/x5EvlK46odV/uFy4kQ+nHegz10WRTR4IRxImZHh473l6pECT5Zz52T8QiS5QzIZcNU
wtb+Xdqp8PDiGId0x35EiBmUlt8v8wL0J+BNMkKuMvUKNYzX+UM8BeXt0UWRB9uE8QsL+VIG3TiJ
jcaHXGVbOOU7R/xXEnBANvmuJ0UnuGDf8b/9xZq6fsEdxZzofZfJp254tZI6m688kdGE99XOywme
QC1q9YbHmwdyC6tGHH1rGpyz+Q+v7qmGGrnKodvZwCBjo/VjpwikTlAt771yc00mAQmr5LYQKDrw
YhRSnLzm4zSFyTvN3o001SonbDjkoshyfUPHNPjL9bI2wcSHziNnVDYC+PeJPkTDvk6hKz5mlknR
H3DGMAj5nlFMNJt/AhMjZw/3vJKPItGcyI5G7I1o+EUsdIHNj7iLpEUZPoDVDuFUZif0ZPWznX1L
OfbC2JfMr0ZOvw4hkmag6cCO7ftFUHMfxkfwkBmAsWvQqjGxTTDZygyBgxc2X7L6Wgcvismp1NQY
v9bNN2FYipdpQmzzpdHTQGzdSBU0IjI2y8YvheOqN2llXjbprhTGoDV96Sa5AmsPoofs/uZvujX7
1PrWDOgFXc5yeyq5kA+HktPgbfPINGR8kGIOY++nKZxUY7vJEX0IkAPMMrvGEu5ONt73QaAi02WN
4LtyKLeC1bXzbK5SmMT07tWHf505dgwYuCZesIiAw3X6AyJaOBnVNnF+XeEJU9PWudEmS0zl6LbU
CyOPO5dtjVFCnJLtL0YXNtFPy98q6rZerXtUwHTNlNVIg1FB2II1+U7qPMJoQfazNBkReZxsXz4A
Bu53v5XYnOq3EF1aDoFUebNSeMn/OU/+TjypeohnGB1vu4VJlNmYLle4vqLgxtsl9Bc6OEb5NLt3
dmTtToU3qdbPl2ex6Sy0cb78n9QHDgkCIV9RowgY8BHgyiMHblxbfBF01D5TTQ/2JTMwQr46a7NB
G/A1a+7wCBBNuPj8jz6c1M9jxv0ZeVO6Epgi7Q6GYsgnJ7zKjyFG76Iwu7VVN7J3yPVJvBpoXfak
xc8hf5bvkElNEGrUZkwj36NF5YFFHseo75PK2zcWtPy42YWWC3RpA76cEPdC/LIUIJCt/zGappPp
naXUzLgXKH9Yd1uxZID4B0RAHzLjuUlJ/Mzn7rKziQfvnDXaeUqfrfkJmoufEYnZPgYbIRtmvm6D
RZiF34JmCGl9jB6UCweD7pJUaPYjvO5agQJ9mdZnPN/Duz4z3qMyxHEKY3sxavDcwekHvIfoh/71
/parjvGBG12HOHI9AYdf98lQzbtEcMzcC3E/EyrVYbKRIOX8AOhHxVa/YeTA5sAneQbKTQj44wsL
p9H5ayItJZkdLBh8A1R2B0qC9+au1yEViN/3qaniGx0l+DIV6eo+x7AF0ezvX8j9O8YaN2z1syOw
OGI9BFoye5zBfrmOltoSGKuQcI39SOI8Nk2DALH7XivFbMQ95001IPk8T7Yb+d1nX4oNzCHrcWch
26ffmI5lwMUAg6UWWBFkrn6tE6yzmwnxAmpEeMQjvEJlyhd7dN9zsUCz4sF/Th3XxechvS5ohNpK
f5ennqHhNc8aquPZ+dxn+wV4nSpXOOh3G/3IITx/zkH1pVFf1qjN0FPMZAk6f62eenQEYHpPMY9N
+FDwixHbEYY1QKRvLbfyh1v2YV1fO+qdL/QMa09oSaM//R7yu57h/r5PCkKrv1RYqFfw+A32PeAp
YFB+8LsjpWJQF8w+CrF72ivgc/QWHnYD/11/k42wr/kZhS6f5Ynn+TGSEAYDf1vZAVl7RoFgb7km
bPEbJScuSU4ycPUITi+3+o/IsSTZ84C142eVMI+499yp2rydJtMs9gd3OvbBtGmXHFGBsBesCtv1
Tu42wyA+jTLRbs5yV2UD+SESVfRlm6sfA4PsxXDjZRttYkoxsYCozaYMW9OEakB4+e8lux73YknA
GBBgmWJI/jfR13wyVCYcLK4FNf628AHxEe98XtxDr+Q/to7XmZ3KgBfU7eqAXjTpAeUTvzGFwEjw
q0jdQW0gW2Y/5pGR2pKJ/hwpX0AvVKkBb6c8SNgXj+JQiEBxiMP1eEr8Eo2+srBZeUz/2zmKSKek
AORV0h1eIokm9+KVD5iAa1vPNuWcp6a7r/4cIEXYK5oveEmZLILR1ls0CMW5dNwhW/R/cTJUwdkh
zQSJiEvfR5cNAMhVK45gQSzlWNnSXsPuDHOpWVYWO0HK1D53JzkZaEkXtJkpDA2Rq5i6OeSoLvuu
jzPYaN38bcf+V8gDzIs89GG0TpohsQydpvTAqhlvERLh97i+uffyqOCHHEDpD8asfzhrgSU5uTes
r9f0iS+4PJ2BYFkSEZy2azsqH9wjg5xdyf9ZucRLqqi+xj7QbvA9aiD4aT+TnaA5C5RNo0cZ242H
LNbOa4+Z1NkzAcnpVjnhJJSek6JrWTxOAdCFcHcSCOj+7zdR0R+zBG41XTZJ6JY+aQgGRu/kIdMq
rhc82h3EHWXyLO3JgGZ/2SVtJMuL6bS4F7pLjH5i2G5yeXTSkW8w19rqdj5dUrY6Ik6NVfRTmn7y
vfdd8DCNUplnSd2mOzhZtuN6EInlmO8G3Xmv4Gm6dxQv5DiZm5FXnBawqjLbx6Hs9bd38Yr0jyMk
NeS9v58+oAgodgw8Evqcj+FS7K+1v+RBpVMIKRHQ9Y/dG+95a6BsevBe2L63JEx8poRkuNkW/yM0
vLukywNixChvBABjR1BscmVW0X1jnZ5HB4DLmAaR0cuxOWBN39BhmB9plOye3v2o90ftAe5COusJ
mzQ6juw9V1gnd8kCdYb1gvC5btNm0I+Y8fIlPLrysQqa5/eLbv0H8CN48wkN1EmFHHEksFjAErn2
lo6pgpJHDOwVEJ8QhngpYx7fw7XKV15octeTMqUNxR80JSud3ZtLwYTqXisDSzJegIX6dUxQ3ovY
cg6oIRsR7VincXhGhJPHwKV7aaENK/j2Ksg6JhFUMH6z07XQvW5zXAUtO1PfIhNktf5P6Q9p28Me
NV0b9qlQEnGk4QND+XEMWxjTCEjLnqhFwCjU3znRAV4k+9hr0oWrAN6wZWYJowYOGegOrF6cAVE9
6vZ7UcbzlQrrjJWjKVL2hlDg/Cq22sgjoqt04N7HodrnC6c3yYxpa9CRz6fzE31kIT0sUl44gcmo
RNwoEHLoVfQ5iNz/6e1pWJFaNXMUmLEXe7tPT+IO5o4YhJyzquknfqJRhJhX5CVCgdWmd4nETwV9
xP1PZIYBADr4KVvAjRAB0YTTlFvE84cNkRcuKnWATy5d/LEz9zyEggBxj10lRldT371cOilQQJ3e
RkC4BDvwa3JFC9qtFy3IGWZvptBIGtMdQdrjocvjVYpw7ijYFo24AWhYrMn5o+217wY/+Y51K2eg
RCYGsXt4680Z9dxDZ7VJRgxO9n7m82K0+opl9n9WgCFtl32Gy3BBIbjuliabWfLU8sR5loB2oom/
+eZ1Xwa0O0FNt3F4k3Y/WhaduusaFGKwS0P6yUIuaJJBCAoa0eM1G8ttAUGxo4fO2RV293mS1/vA
ytS5xnOhYb0SnzxppQXIGU4QnLQmZYe9M8msbP048lWam21IGAvjeOE4hbEcKKBTmtIG8Xa3h7gn
Om5X8SPGx1VkadJcXXG6T7xOSX/mqhZbbdxEeXzRifcvO+J+a3FqdJVuF+0eT0yHj1SGfuP5QgUm
XplnsZzB88U3euC/ZDbDnz6fRUSrPynNW9P1zB4PFEr9za+2vrpvpgnF7runc3vRXIThRkknTDyP
2643fWpeFsu9vmq9GRCXvYIYvF8tIAfr9tc8fLIyVdcBVdhJ1Gxh0NxWxbGHcF6ZaTk7/T0TMyRr
F+fp5oN12rPDaw/A/3iytEGrseAWggX5v6qyc3OXOa6ZVHGa2qhudL9rmr1rOUgxhx1U0USG26My
xtT7qEIklng/Q5+dYtALvOglisAYbnRHRZaMANWjvBiFPb2aYZS82ykE7Vep1lpnOrWpb1gz48mQ
QC4ldhfPfBkMrO2FBzOLWsjIxQ1UVBPYVB2FNME0orvmt6NpXOLNuVCd+kZ5WUT/DHgqJkkQBAm8
QJb9Z/gal9BQ95FOcm2/WQFqihpohl+38WKRMtdRUk31D7AyxLlH7S0vG/1/ICh1ZNJXi7K/D/5V
mF3K8lkksnQWPeSr8+tSor+x8lh/4dn15VnYn41eQBAAAP41CXgY2yCKAAGnKICQAwBGzJoNscRn
+wIAAAAABFla
_BEOF

(cd $HOME/tmp/sfxz && tar -xaf new-diff-backups-installer-v1.0.xz)
rm -f $HOME/tmp/sfxz/new-diff-backups-installer-v1.0.xz
 
cat << _BEOF > $HOME/tmp/sfxz/post-extract
#!/bin/sh

is_yes() # returns OK if first char is Y or y
{
  local key=`echo $1 | cut -b1 | sed 's/y/Y/; /Y/!d'`
  [ "$key" == 'Y' ] && return 0 # true
  return 1 # false
}

if [ -e new-diff-backups-installer-v1.0 ]; then
  printf "
  Overwrite existing file(s)? [N/y]:"
  read key
  if ! is_yes $key ;then
  echo "Aborting.."
    exit 0
  fi
fi

mv $HOME/tmp/sfxz/new-diff-backups-installer-v1.0 .
_BEOF

sh $HOME/tmp/sfxz/post-extract
rm -rf $HOME/tmp/sfxz/*
rmdir $HOME/tmp/sfxz 2>/dev/null || true
rmdir $HOME/tmp 2>/dev/null || true
PS. I could update the self-extractor to include these corrections, but let's call this "a tutorial".

- The Computer Mad Science Team

:-)
Posted in Uncategorized
Views 916 Comments 0
« Prev     Main     Next »
Total Comments 0

Comments

 

  



All times are GMT -5. The time now is 04:21 AM.

Main Menu
Advertisement
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration