LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 01-24-2013, 09:11 AM   #1
TiMMay333
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Rep: Reputation: Disabled
Question Best way to do image based backups on linux


Hello all,

I would just like to know what people use as an image based solution to backing up Linux. Not really specifying a distro, just would like to know what people use / philosophies.

Thanks for the helpful replies!
 
Old 01-24-2013, 09:18 AM   #2
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,148
Blog Entries: 2

Rep: Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886
If I really have to make a complete image I use dd in conjunction with a compressor, like bzip2:
1. Write zeroes to the unused parts of a disk, using dd to write zeroes to a file until the disk is full.
2. Use dd to read the partition, pipe the output to the compressor and redirect it to a file on the backup medium.

Since any distro comes with dd and i can choose the compressor this is the most distro agnostic approach, I would think.
Of course there are other approaches with more functionality, like Clonezilla, but I prefer the simple things.
 
1 members found this post helpful.
Old 01-24-2013, 09:52 AM   #3
GazL
LQ Veteran
 
Registered: May 2008
Posts: 6,897

Rep: Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019
Never been one for image backups. I prefer to take a set of tar archives, one per filesystem, that I can use to restore the complete system state, or just individual files/filesystems as necessary. Combine those with the saved output of "sfdisk -d", "vgcfgbackup", and "cryptsetup luksHeaderBackup" and it should be possible to restore everything to its current state from scratch, using only the tools on my slackware installation media.
 
Old 01-24-2013, 03:14 PM   #4
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,985

Rep: Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626
I can't say I have any good choice. I long for what I perceive as the good old days of ghost.

In the real world we only have a few ideas. One is a bit by bit copy that I use a lot. It has many faults and tips and tricks with it but tried and true.

Second is the file by file method. Many tools can copy off files and a few apps and live cd's work well with this method but again, not me(fool)proof

I work on so many systems and OS's that I use what I can in each place, even if it isn't the best I use what works.
 
Old 01-24-2013, 04:01 PM   #5
Janus_Hyperion
Member
 
Registered: Mar 2011
Location: /
Distribution: Fedora (typically latest release or development release)
Posts: 372

Rep: Reputation: Disabled
I have tried with clonezilla. But, now-a-days, I do not image based backups anymore. I just rsync to my external HDs. perhaps, it helps that I run no applications that I cannot reinstall. I do backup configuration files as well.
 
Old 01-24-2013, 08:20 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,359

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
You could try www.mondorescue.org; does a warm backup as opposed to Clonezilla which I believe requires a shutdown as it does a cold backup.
Depends on your requirements.
Databases need special handling if not done cold.

Many (most) prod systems can not be shutdown, so a combination of Having the OS install disks, then something like eg Mondo or NetBackup https://en.wikipedia.org/wiki/NetBackup, with DB hotbackup techniques is usual.
Of course if you have hot swap mirror disks, that works too
 
Old 01-26-2013, 10:38 AM   #7
TiMMay333
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thumbs up

Thank you all for the replies!

Im curious about the DD along with bzip2, how would one go about restoring?

The reason why im asking, is that in a complete disaster scenario, I would like to restore the system "at a point in time" and let it resotre an image, than take the time to copy the files, because in a disaster, there are probably other things I need to do while its restoring.

Ill also take a look at mondo rescue, seems like a flexible solution as well.

Thank you all, I appreciate the insight!
 
Old 01-26-2013, 11:52 AM   #8
rokytnji
LQ Veteran
 
Registered: Mar 2008
Location: Waaaaay out West Texas
Distribution: antiX 23, MX 23
Posts: 7,112
Blog Entries: 21

Rep: Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474Reputation: 3474
What works for me

http://www.linuxquestions.org/questi...s-work-895225/
 
Old 01-26-2013, 01:41 PM   #9
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,148
Blog Entries: 2

Rep: Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886
Quote:
Originally Posted by TiMMay333 View Post
Im curious about the DD along with bzip2, how would one go about restoring?
Take the image (asuming that the partition in question is /dev/sda1 and the backu medium is /dev/sdb1):
1. Boot from a live medium.
2. Mount the partition to image:
Code:
mount /dev/sda1 /mnt
3. Fill the unused parts of the partition with zeroes, so that the compressed image is smaller:
Code:
dd if=/dev/zero of=/mnt/zeroes bs=16M
rm /mnt/zeroes
Depending on size of the partition and speed of the disk this can take quite some time.
4. Unmount the partition to image and mount the backup medium:
Code:
umount /dev/sda1
mount /dev/sdb1 /mnt
5. Take the image and compress it:
Code:
dd if=/dev/sda1 bs=16M | bzip2 | dd of=/mnt/image bs=16M
6. Unmount the backup medium and you are done:
Code:
umount /dev/sdb1
Restore the image:
1. Boot from a live medium.
2. Mount the backup medium:
Code:
mount /dev/sdb1 /mnt
3. Read the image, uncompress it and write it to the partition:
Code:
dd if=/mnt/image bs=16M| bzip2 -d | dd of=/dev/sda1 bs=16M
4. Unmount the backup medium (see step 6 above) and you are done.

You may want to also make a copy of the output of
Code:
fdisk -l
so that you can have information on the exact partition size if needed.

Note that I use a blocksize of 16MB for the dd operations, because I found that blocksizes in the range between 8MB and 32MB (depending on your hardware) give me in general a better performance than the default value.

Last edited by TobiSGD; 01-26-2013 at 08:55 PM.
 
1 members found this post helpful.
Old 01-26-2013, 08:02 PM   #10
TiMMay333
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Original Poster
Rep: Reputation: Disabled
@TobiSGD

wow, thank you very much for the detailed instructions. Ill test this right away!

I really appreciate it!
 
Old 01-27-2013, 09:45 PM   #11
TiMMay333
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Original Poster
Rep: Reputation: Disabled
Smile

So I tried out @TobiSGD method for using DD and i was able to backup and restore a whole CentOS 6 machine with one image file, very awesome!

There are a few things I wanted to clarify with you though:

The way I did the test, i didnt have to boot into a live media in order to do the backup, i just ran:

Code:
dd if=/dev/sda bs=16M | bzip2 | dd of=/mnt/image bs=16M
and it did the trick, even without writing zeroes, it only backed up the actual usage of the partitions.

also, i didnt specify the partitions of the disk, I just took the whole drive (sda instead of sda1)

When restoring, i ran your other command back to an un-partitioned hard drive with an Ubuntu 12.04 Desktop live CD and it worked, it restored the partitions as well.

Code:
dd if=/mnt/image bs=16M| bzip2 -d | dd of=/dev/sda bs=16M
again only specifying the hard drive, not a specific partion (sda instead of sda1)

does this make sense?

Thank you everyone for your input!
 
Old 01-28-2013, 12:05 AM   #12
jschiwal
LQ Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682
Zeroing out the free space first allows for a smaller compressed backup. Also consider using ddrescue which might be faster. It will retry if it reaches some bad blocks. It also varies the block size depending on the health of the drive. Bad blocks are zeroed out in the image. Using dd without options will simply fail.
 
Old 01-28-2013, 03:21 AM   #13
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,148
Blog Entries: 2

Rep: Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886
Quote:
Originally Posted by TiMMay333 View Post
The way I did the test, i didnt have to boot into a live media in order to do the backup, i just ran:

Code:
dd if=/dev/sda bs=16M | bzip2 | dd of=/mnt/image bs=16M
Do you mean you have made an image of the running system? This is not reliable, since the file-systems can change during the backup, which will leave you with an inconsistent backup. Backups that are possibly inconsistent are worthless.

Quote:
Originally Posted by jschiwal
Also consider using ddrescue which might be faster. It will retry if it reaches some bad blocks. It also varies the block size depending on the health of the drive. Bad blocks are zeroed out in the image. Using dd without options will simply fail.
I disagree. Of course you can use ddrescue, but if you have to use it because of a faulty disk it is already to late for a backup, the data is already potentially damaged. If there are already bad blocks on the disk you are not making a reliable backup anymore, you are creating an image for rescue purposes.
 
Old 01-28-2013, 03:24 AM   #14
Thad E Ginataom
Member
 
Registered: Mar 2011
Distribution: Ubuntu 12.04 with KXStudio, MATE & Compiz
Posts: 46

Rep: Reputation: 8
Thanks: I was wondering, why the zeros?
Quote:
Originally Posted by TiMMay333 View Post
I would like to restore the system "at a point in time" and let it resotre an image, than take the time to copy the files, because in a disaster, there are probably other things I need to do while its restoring.
That's not really a valid comparison. If you have a tar backup, the backup itself is one file, and can be restored with just one command. As it works at the file level, not the block device level, it gives you enormous flexibility to restore all or part, and to do so to any location (assuming that the backup was made using ./this/that/the-other relative paths) on any device or any point within the directory tree[s].

dd is a lower-level technical tool for when you need a bit-identical, not data-identical, copy. Which I suppose is what "image" means <Blush>. dd will also do stuff like read or write a given number of bits, convert block sizes, etc.

Tar is your swiss-army knife: dd is your leatherman multi-tool.
 
Old 01-28-2013, 03:24 AM   #15
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware64-15.0
Posts: 6,371

Rep: Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750Reputation: 2750
The problem with running dd on a live file system is that things can change while the backup is running e.g. changes to log files. Also you will backup the the contents of /sys and /proc which are automatically generated at boot.

Personally, I like dd images of my Windows machines. If you restore from the image after a virus invasion it is comfortable to know that the sucker has been completely eliminated from the disk.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Automated backups, full disk image Eppo Linux - Server 6 01-24-2012 12:26 PM
Swap on Linux-based PXE image cccc Linux - Networking 3 07-29-2011 07:55 PM
LXer: Disk Based Backups With Amanda On Debian Etch LXer Syndicated Linux News 1 09-23-2007 01:07 PM
Image and Incremental backups with rsync kaplan71 Linux - Software 2 08-13-2007 03:42 PM
Floppy based linux for hard drive backups AP81 Linux - General 2 01-11-2007 02:02 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 03:57 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration