Latest LQ Deal: Linux Power User Bundle
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 09-09-2011, 07:14 AM   #1
LQ Newbie
Registered: Sep 2011
Posts: 4

Rep: Reputation: Disabled
Back up of all files AND Ubuntu itself


Fairly new to Ubuntu (10.04). I'm working on a 1TB disk and I'd like to back up all my files and the OS onto another 1TB external disk. Ideally just recording the changes in files and OS (package upgrades/ configs) each time. I think it is a kind of external mirror I guess but done when I plug in the drive.

Can anyone recommend a tool to use? I have seen a few for files but I'd really like everything backed up so I could recover my system entirely by booting from the external backup disk?

Hopefully this is possible. Any help would be greatly appreciated.

Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 09-09-2011, 11:30 AM   #2
Senior Member
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,541

Rep: Reputation: 1060Reputation: 1060Reputation: 1060Reputation: 1060Reputation: 1060Reputation: 1060Reputation: 1060Reputation: 1060
It's certainly possible -- you would want to use the dd utility to back up the entire system:
log in as root
# init 1              (put the system into single user mode, no GUI running, most stuff shut down)
# dd if=/dev/sda of=/dev/sdb
where "if=/dev/sda" is your system disk and "of=/dev/sdb" is your external drive (whatever its name actually is, probably not /dev/sdb though).

Note that this is going to be slow, real slow (and, no, there really isn't a go-fast switch).

Now, let's talk about this for a while.

What you're doing with dd is, for all practical purposes, creating a mirror of your system. By putting the system into single-user mode (that's what init 1 does), you're stopping almost all the daemons that are running, stopping your GUI (so you'll have a black screen with a log in prompt), stopping everything so nothing will interfere with the copy you're making. You would do this after you've installed and configure your distribution -- a clean, bright, shiny level 0 snapshot (as it were) of the system.

When you're done with this initial mirroring, you simply enter
init 6
to reboot.

From that "level 0," you would want to incrementally back up anything that has changed -- but not everything that has changed. You don't really care about the system logs, for example, but you do care (a great deal) about the /etc directory (that's where configurations are recorded), the /hom tree (that's all your user stuff), possibly the /opt tree (that's where things like or go) and maybe a few others depending upon where applications have written configuration files and perhaps data files. You may also, as you indicated above, want to back up any changed system-level files (like when patches and upgrades are applied).

So, how do you do that?

The find utility is probably your best bet. Here's an example of how you would use find on multiple directories:
# find /etc /opt /home /usr/local (and some options)
OK, so how do you locate only those things that have changed since sometime or other?
# find dirname -type f -mtime -10
would find all files (-type f that have been modified in the last 10 days.

Well, OK, what about files that have changed in the last six months, two years, whatever?

Let's say that you installed your distribution and got it all the way you wanted on, oh, 15 July 2010. If you create a time stamp file, you can use the newer option in find:
# touch -d "15 jul 2010 03:00:00" date_marker
# find dirname -newer date_marker
The -d option is important -- it parse STRING and use it instead of current time; i.e., it sets the file time stamps to the date you tell it to.

By doing this, you don't need to calculate the number of days since some date. Now we need to fiddle with this a little to get only files we do care about and not get files we don't care about:
# touch -d "15 jul 2010 03:00:00" date_marker
# find /etc /opt /usr/local /home -type f -newer date_marker
Now, bear in mind that you want to change that date_marker to the last date and time you did an incremental back up (otherwise it's going to get slower and slower as you go): you only want to back up files that have changed since some time. It's usually a good idea to use 03:00:00 on a Sunday as your back up time (and, why 03:00:00? so you miss daylight savings time switches).

Also, if you're doing this on a regular schedule, you don't need to create the time stamp file; you can simply use -mtime -7 for the preceding week. Let's keep it as simple as possible (but the example below do use the date_marker file, go figure).

Now, if you're going to use your external disk dive you don't need to use dd to copy the changed files to it, you can simply use cp in a pipe:
# touch -d "15 jul 2010 03:00:00" date_marker
# cp -pr `find /etc /opt /usr/local /home -type f -newer date_marker` /dev/sdb
again, where /dev/sdb is the name or mount-point of your external drive.

You do want to pay attention to the list of directories -- those above may not meet your needs and you may want to add or delete to the list. Bear in mind that you don't need to copy the entire system every time you do this, only the changes (and you really don't care about copying the system logs and that kind of stuff or the /tmp directory). If you don't really care about how long it takes just do something like this:
cp -pr `find / -type f -mtime -7` /dev/sdb
and get everything that's changed in the last 7 days.

By the way, the -pr option to cp instructs it to do a recursive copy of directories; so, if you find a file or group of files that are in a new directory the directory and all its content will get created for you.

Also give some though to using other media; e.g., CD-ROM or DVD, for your incremental back ups; you can plug in a CD-ROM or DVD and write to it then stick it in file (with a date indicated on the envelope!). This has a certain advantage in that you may want to recover back to a certain date which you will not be able to do if you continuously overwrite that disk drive with changed files. Doing so is a little more fooling around (you can't just write to a DVD with cp) but it may be worth thinking about.

Hope this helps some.
2 members found this post helpful.
Old 09-09-2011, 06:46 PM   #3
Senior Member
Registered: Jul 2006
Location: North Carolina
Distribution: CentOS 6, CentOS 7 (with Mate), Ubuntu 16.04 Mate
Posts: 1,799

Rep: Reputation: 125Reputation: 125
Hi kw032,

Thinking ahead for backup and restore - Good for you!!!

How do you have your hard drive partitioned? If you have it basically in one big lump, mounted as / then backup is going to be a slow and tedious process. Here is some material from another reply I posted recently
I have always followed an approach like this for my home PCs.

sda1 - / 10 GB formatted as ext3
sda2 - /home 10 GB formatted as ext3
sda3 - (about = RAM) swap although with 8 GB of RAM I don't really need swap
sda4 - /data (all the rest of the drive) formatted ext4

And the reasons for my madness

I like to be able do do a cold backup of the OS. I use g4l (the program formerly known as Ghost for Linux). So I keep the OS partition reasonably small. I use ext3 because g4l does not support ext4 at the moment.

I also do a monthly cold backup of /home. I don't store data there if I can avoid it so a small home is fine.

/data is where I store my DATA. The important stuff ranging from email to documents, databases, etc. etc. etc. This I hot backup nightly with a script to a second hard drive. For really critical data I maintain a 7 day rolling backup. For other thing such as downloaded bank statements and items which can be downloaded or created again I maintain a single backup on the second drive. I backup selected subdirectories from /data to CD or DVD monthly.

Bottom line - I design my system layout to facilitate backup and recovery.
You might want to have a look at g4l and Clonezilla. They provide an interactive approach to backup and restore. If you need to modify your partition layout you can do so with Gparted without having to do a reinstall of Linux. All three of these utilities are booted from a CD and allow the PC to be backed up or restored without the PC's operating system running.

If you are interested in doing a script based backup of selected data to your external drive let me know. I will be happy to share my backup and housekeeping script.

1 members found this post helpful.
Old 09-09-2011, 06:50 PM   #4
Senior Member
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora
Posts: 3,935
Blog Entries: 5

Rep: Reputation: Disabled
Originally Posted by taylorkh
You might want to have a look at ... Clonezilla.

That's what I was going to suggest.
Old 09-11-2011, 06:48 AM   #5
LQ Newbie
Registered: Sep 2011
Posts: 4

Original Poster
Rep: Reputation: Disabled
Wow thank you all so much for your help. I think I might try it with the raw dd commands and make up a shell script. So I will pay particular attention to you notes tronayne. Ken does your script use the dd commands? If so I would be interested to see it. It's also a good thought to use my file system in a way which makes back up and restore easier, I hadn't thought of that. Clonezilla looks good but I'm keen to do it with Linux commands now I know it's possible!

Thanks again.
Old 09-11-2011, 08:50 AM   #6
Senior Member
Registered: Jul 2006
Location: North Carolina
Distribution: CentOS 6, CentOS 7 (with Mate), Ubuntu 16.04 Mate
Posts: 1,799

Rep: Reputation: 125Reputation: 125
It may not be pretty nor sophisticated with loops, substitutions, sed and awk and so forth. Rather it is just a pick and shovel script to do the tasks as I would do them by hand. Some notes about the directories I am backing up:

/data is the partition used for data

/data/data and its subdirectories contain active data ( files, database files, other files that get UPDATED periodically) This included my Firefox and Thunderbird profiles which I have moved from ~/.mozilla etc. Actually the profiles were transfered from my old Windows machine when I switched to Linux for real - I simply point to them from ~/ken/.mozilla/firefox/profiles.ini and ~/ken/.thunderbird/profiles.ini

/data/static and its subdirectories contain data files which just accumulate. For example if I see a news article on the web which I want to save - in the old days I might cut the article from a newspaper. Now I print it to a pdf file and save that file under /data/static. I also file bank and other financial statements, downloaded documents such as Full Circle magazine, manuals and parts lists for tools and appliances, pictures from my digital camera etc.

echo Starting data mirror >> /quitelarge/_mirror/mirror.log
echo $(date +%y/%m/%d_%r) >> /quitelarge/_mirror/mirror.log
# some subrirectories only accumulate new files - a single backup is enough. Also use -u to copy only newer files
echo mirror static >> /quitelarge/_mirror/mirror.log
cp /data/static -R -p -u -f -v /quitelarge/_mirror >> /quitelarge/_mirror/mirror.log
# other subdirectories contain files which are changed periodically - keep 7 daily itterations
echo roll forward the 7 iteration subdirectories >> /quitelarge/_mirror/mirror.log
rm -R -f /quitelarge/_mirror/mirror7 2>> /quitelarge/_mirror/mirror_error.log
mv /quitelarge/_mirror/mirror6 /quitelarge/_mirror/mirror7 2>> /quitelarge/_mirror/mirror_error.log
mv /quitelarge/_mirror/mirror5 /quitelarge/_mirror/mirror6 2>> /quitelarge/_mirror/mirror_error.log
mv /quitelarge/_mirror/mirror4 /quitelarge/_mirror/mirror5 2>> /quitelarge/_mirror/mirror_error.log
mv /quitelarge/_mirror/mirror3 /quitelarge/_mirror/mirror4 2>> /quitelarge/_mirror/mirror_error.log
mv /quitelarge/_mirror/mirror2 /quitelarge/_mirror/mirror3 2>> /quitelarge/_mirror/mirror_error.log
mv /quitelarge/_mirror/mirror1 /quitelarge/_mirror/mirror2 2>> /quitelarge/_mirror/mirror_error.log
mkdir /quitelarge/_mirror/mirror1 2>> /quitelarge/_mirror/mirror_error.log
mkdir /quitelarge/_mirror/mirror1/data 2>> /quitelarge/_mirror/mirror_error.log
echo mirror data to mirror1 >> /quitelarge/_mirror/mirror.log
cp -R -p -f -v /data/data /quitelarge/_mirror/mirror1 2>> /quitelarge/_mirror/mirror_error.log
# Backup my home subdirectory (but first clean up some crap)
rm /home/ken/.thumbnails/normal/*.*
# more cleanup added 12/2/2010
rm -R /home/ken/.thumbnails/*
rm /home/ken/.recoll/mboxcache/*
rm -R /home/ken/.dvdcss/*
rm /home/ken/.local/share/gvfs-metadata/*
rm /home/ken/.gconf/apps/nautilus/desktop-metadata/*
rm /home/ken/.wine/drive_c/users/Public/Application Data/DVD Shrink/*
rm -R /home/ken/.gconf/apps/nautilus/desktop-metadata/*
rm "/home/ken/.wine/drive_c/users/ken/Local Settings/Application Data/QuickPar"/*
rm "/home/ken/.wine/drive_c/users/Public/Application Data/DVD Shrink"/*
# clean these files - eliminate problem with gvfsd-metadata hogging cpu 12/5/2010
rm -rf ~/.local/share/gvfs-metadata

echo tarring home directories to mirror1 >> /quitelarge/_mirror/mirror.log
#tar -cvpzf /quitelarge/_mirror/mirror1/home-ken.gz /home/ken 2>> /quitelarge/_mirror/tar_error.log
tar -cvpzf /quitelarge/_mirror/mirror1/home-ken.gz /home/ken > /quitelarge/_mirror/ken_tar.log 2>> /quitelarge/_mirror/tar_error.log
tar -cvpzf /quitelarge/_mirror/mirror1/home-patsy.gz /home/patsy 2>> /quitelarge/_mirror/tar_error.log
#cp -R -p -f -p /home/ken /quitelarge/_mirror/mirror1/home 2>> /quitelarge/_mirror/mirror_error.log
# Make a backup of the recoll index directory
echo mirror recoll index >> /quitelarge/_mirror/mirror.log
cp -R -p -f -v -u /data/xapiandb /quitelarge/_mirror 2>> /quitelarge/_mirror/mirror_error.log
echo Mirror finished >> /quitelarge/_mirror/mirror.log
echo $(date +%y/%m/%d_%r) >> /quitelarge/_mirror/mirror.log
echo ---------------------------------------- >> /quitelarge/_mirror/mirror.log
# Add a step to update recoll index
recollindex 2>> /data/xapiandb/recollindexerror.log
A couple of other notes:

>>2 writes errors from a command to a file - a handy trick. Make sure NOT to leave a space between 2 and >
I also do some privacy cleanup in the script.
And I also update the indexes for recoll (a full search text search tool).

Please feel free to ask questions about the script. I have probably overlooked something or flubbed something and your questions will help. I cannot prof read what I write or code



Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Tilde / back-tick key not giving me a tilde or back-tic. Ubuntu 10.04 on Macbook Pro BrianK Linux - Laptop and Netbook 3 09-01-2015 10:14 PM
LXer: Nautilus Elementary is Back! PPAs for Ubuntu 11.04 Natty, Ubuntu 10.10 Maverick Updated LXer Syndicated Linux News 0 03-23-2011 10:41 PM
[SOLVED] I want to use Ubuntu 9.10 to access all Windows files and back them up. Advice Pro Ubuntu 9 12-01-2010 02:20 PM
How to get back removed files? sacoskun Linux - Newbie 19 01-23-2007 01:45 AM
How to get back my --linux-.--- files?? dufferin Slackware 2 06-28-2004 02:30 AM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:12 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration