Backup only the files which have changed?

mike11 · 05-18-2011, 08:23 PM

Hi All,

Can someone please explain if there an application which does the following:

1. I have a laptop with an internal 200GB HD.
2. I run the application & it creates a list of all files (size & time-stamp) without actually storing them. Let's call this the "snapshot list".
3. I update some of the files on the laptop.
4. Now I run the application & it only copies the files which have changed on the laptop, that have different size/time-stamp from the snapshot list, onto some external media, such as a memory card. Of course, the files should be copied onto their proper location in the directory tree & not just pile up in one place.

Why is this useful? although the laptop has a 200GB HD I typically only update a small number of files, whose total size is maybe 10MB or so. If I could only backup those which have changed, I could do this with a tiny SD card instead of lugging around an external usb HD.

Thanks,
Mike.

smturner1 · 05-18-2011, 10:21 PM

If I am understanding this correctly, you would like a script that takes a list of recently updated files and have them copied to a flash drive of some kind, right?

The way you are wording this is a little confusing.

Quote:

2. I run the application & it creates a list of all files (size & time-stamp) without actually storing them. Let's call this the "snapshot list".

There is definitely a way to create a script to do this. First you would want to call out the directory, list the files with the sizes and last date changed, run a couple of conditional statements, point the output to copy to your flash media, and voila you have your application.

man ls
man cut
man cp
google if statements
google storing variables..........etc.

I am sure there are some great programmers that could give you better hints than I, but I think this might be a classroom assignment. I could be wrong, but..........?

Hope that points you into the right direction.

mike11 · 05-18-2011, 10:36 PM

Thanks for replying. This is not a classroom assignment, just sometime which I think can be useful.

it is probably doable by hand but it may be incredibly inefficient, especially when examining HD of 100s of GB. I'll try to rephrase the scenario:

1. I"ve got a laptop with an internal HD, say 200GB.
2. The application scans over all HD files and stores their current size & time-stamp.
3. I work on the laptop, and update some files.
4. I re-run the application and it now compares each file to its stored size & time-stamp. any files which have been changed are copied into another media for backup. however, they are stored in their proper directory location and not just dumped into one directory.

Hope this is clearer. aren't there application out there already which do this kind of smart-backuping?.

TIA.

Nylex · 05-18-2011, 11:29 PM

rsync? Your distribution probably has a package for it, if it's not installed already.

mike11 · 05-18-2011, 11:47 PM

Thanks. I am familiar with rsync though I can see an option to create a list of files with their current size and time-stamp without copying the entire file. or am I missing something?.

j1alu · 05-19-2011, 12:15 AM

never did it, but man says:
--list-only list the files instead of copying them

I bet if you run "-n" or "--dry-run" it will get stored somewhere.

smturner1 · 05-19-2011, 09:57 PM

There is another way to approach this, have you ever heard of checksum? Checksum is an algorithm that looks at files and compares their sizes to make sure you have a good copy, that there are no malicious programs attached to it, and that you have the most recent copy. Of course, the tool is only as good as the datum.

What I would do is use rsync, it uses the checksum methodology/algorithm. It will reduce the amount of commands in your shell by removing the cp, checksum, and ls, because if it is used properly it will do all of it (locally and remotely).

You are going to have to write the script, but the people on this forum are incredibly helpful. There are two reasons they are helpful, they will not give you the fish, they will teach you to fish!

Remember, have fun! There is nothing more rewarding in UNIX/Linux than creating a working tool that automates tasks.

astromime · 05-19-2011, 11:10 PM

i have an...extensive option

Code:

#!/bin/bash

function direc {
        if [ ! -d backup ]
        then
                mkdir backup
        fi
}

copy () {
if [ ! -d backup ]
then
        echo "The backup folder does not exist."
else
for file in *
do
        if [ $file -nt backup/$file.bak ] && [ -f $file ]
        then
                cp -v $file destination/$file
        else
                echo "The file $file was not modified since last copy."
        fi
done
fi
}

backup () {
direc
for files in *
do
        if [ -f $files ]
        then
                cp -v $files backup/$files.bak
        fi
done
}

while getopts cb opts
do
        case $opts in
        c) copy;;
        b) backup;;
        esac
done

obviously, this will work if you want to back the files up also, but you said you didnt want to do that...you would have to change "destination" to wherever your media is mounted on (df -h)

but if you decide to use this, just call it with the appropriate options, so if you name it "filecopy.sh" then youd call it with "filecopy.sh -b" to create the backup, or renew them, and "filecopy.sh -c" to copy the new files to your drive

mike11 · 05-24-2011, 01:30 AM

Thanks all for the suggestions.