LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-17-2012, 02:05 AM   #1
strungoutfan78
Member
 
Registered: Mar 2011
Location: San Diego, CA
Distribution: Debian
Posts: 46

Rep: Reputation: 0
How can I increment a disk image created with dd?


Is it even possible? I've searched all over the place and I've seen people hinting at the notion that rsync can perform incremental backups on a disk image created with dd, but I can't find any hard facts regarding this. Ultimately I'd like to cron a weekly full dd image and a daily rsync to maintain that this image. Am I dreaming? Should I just stick to rsync for the entire process?

Also, as a side note, how the heck do I get my cron jobs to stop emailing me???? It's driving me batty and I can't figure out how to turn them off.
**EDIT**
Nevermind this second part. I just realized my redirect syntax was incorrect.

Last edited by strungoutfan78; 12-17-2012 at 02:16 AM.
 
Old 12-17-2012, 07:36 AM   #2
linosaurusroot
Member
 
Registered: Oct 2012
Distribution: OpenSuSE,RHEL,Fedora,OpenBSD
Posts: 982
Blog Entries: 2

Rep: Reputation: 244Reputation: 244Reputation: 244
Are you mounting the dd image with "-o lo" and using rsync into that?
 
Old 12-17-2012, 07:38 AM   #3
linosaurusroot
Member
 
Registered: Oct 2012
Distribution: OpenSuSE,RHEL,Fedora,OpenBSD
Posts: 982
Blog Entries: 2

Rep: Reputation: 244Reputation: 244Reputation: 244
Why use dd and not dump (assuming an ext* format)?
 
Old 12-17-2012, 03:36 PM   #4
jefro
Moderator
 
Registered: Mar 2008
Posts: 19,514

Rep: Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017Reputation: 3017
I think you'd have to mount the image and then use conventional means to rsysnc it or such.

I also don't get the reason or use of dd for this task.
 
Old 12-24-2012, 01:58 PM   #5
strungoutfan78
Member
 
Registered: Mar 2011
Location: San Diego, CA
Distribution: Debian
Posts: 46

Original Poster
Rep: Reputation: 0
Sorry it took so long to respond. Holidays and whatnot.

Well I guess I'm not real sure why I want to use this method other than the fact that dd makes a copy of the entire drive and not just the contents, i.e. MBR, etc. I realize I could just use rsync and use dd to backup just the MBR. I was just wondering if this is possible really. Judging by the responses I'd say, if it is, it's not very practical? I'm not very knowledgeable on file system structure backup techniques. I guess my real question then would be the advantage of dd over rsync and vice versa. There may be a better solution for me than either. I'm just not sure.

I've been using luckybackup, which is just a frontend for rsync as far as I know, but it has issues running as scheduled in my crontab, which leads to me forgetting to run my backups constantly. So, I thought I would go for a more low-level CLI approach, where I have more control over the whole thing. This is just for my laptop, which has about 365Gb of data on it. Bottom line is I want to be able to make an image that I can do a bare metal restore with, if the need should ever arise. It sounds like I'm looking at two entirely different types of backups though. Please enlighten me on the differences between the two.
 
Old 12-24-2012, 03:30 PM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,361

Rep: Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001
You can certainly update the contents of the drive image by mounting the filesystems therein and using rsync to update them. There are a few steps to perform, but whether you see that as "practical" is your decision.
Code:
# losetup -f --show /path/to/whole/disk/image
/dev/loop0
# kpartx -a -v /dev/loop0
add map loop0p1 ...
add map loop0p2 ...
add map loop0p3 ...
The losetup command associates a loop device with your file and tells you which device it used. Then you run kpartx on that loop device to create device maps for the individual partitions. You can now mount the filesystems in those partitions just like mounting a disk partition (e.g., "mount /dev/mapper/loop0p1 /mnt/tmp") and use rsync to update the contents. Just don't forget to run "kpartx -d /dev/loop0" and "losetup -d /dev/loop0" after umounting the filesystems or you will soon use up all of your available loop devices.

That could all be scripted pretty easily.

BTW, none of that is going to work with a compressed image file.

Last edited by rknichols; 12-24-2012 at 03:49 PM. Reason: add note about compressed image
 
Old 12-24-2012, 04:25 PM   #7
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,003
Blog Entries: 1

Rep: Reputation: 847Reputation: 847Reputation: 847Reputation: 847Reputation: 847Reputation: 847Reputation: 847
How much of an entire disk is unused space? About the only reason I could see to use dd would be security reasons in an effort to undelete deleted files. And you can't really mount it unless you dd partitions, not entire disks. The MBR is really only useful for partition data, the rest of it gets changed on occassion. And is easy to change / replace. And you don't really need the partition data if you dd partitions. Do you really need to dd swap space?

Tar would likely serve you better as it only copies the actual not-deleted files. And works about the same as dd if you don't add -z or -j for compression. Although you lose the mounting option that way. You could potentially save a ton on disk usage for the tar images over what would result from dd. And a ton of time if you're not using more than 50% of a disk.

What are you trying to accomplish with these backups and incremental changes? Most of the useful data is in /home/ from a user perspective. And other places depending on what you use (mysql / apache). Is this for usable backups? Security concerns (we've been hacked, what changed)? Or just syncing multiple devices with the same user data? Unless you're into video editing or 3D rendering, /home/ tends to be a fraction of the actual disk usage (at least initially).
 
Old 12-24-2012, 09:18 PM   #8
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,361

Rep: Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001
Lest anyone think my procedure in #6 is in any way an endorsement of this as a backup solution, rest assured it is not. You are saving all the free space, swap space, tmp space, etc., and unless you run that full-image dd command while booted from separate media you are saving active file systems in an inconsistent state that should be fsck-ed before trying to do any updates. It is, however, a quick and dirty solution that can be done quickly while you research better backup methods, and, as they say, any backup is better than no backup.
 
1 members found this post helpful.
Old 12-25-2012, 12:50 AM   #9
strungoutfan78
Member
 
Registered: Mar 2011
Location: San Diego, CA
Distribution: Debian
Posts: 46

Original Poster
Rep: Reputation: 0
Quote:
How much of an entire disk is unused space?
The hard disk is 500Gb, so it's well more than %50 full.

I appreciate all the info and the different perspectives. I can see how a dd image including /proc, /tmp, swap space, etc. could cause problems. Especially since I'm making a backup of a running system. Most of the info I need is found in /home, /etc, /usr/local, and /opt. I'm certain rsync or tar is a better option at this point. However, having never had the need to perform a restore from a backup, how different are the restore processes between restoring from a dd image (made on a non-running disk) vs an archive made with rsync or tar?

Quote:
The MBR is really only useful for partition data, the rest of it gets changed on occassion. And is easy to change / replace.
Shadow_7 could you elaborate on this?
 
Old 12-25-2012, 08:06 PM   #10
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,003
Blog Entries: 1

Rep: Reputation: 847Reputation: 847Reputation: 847Reputation: 847Reputation: 847Reputation: 847Reputation: 847
# apt-get update
# apt-get dist-upgrade

If the update includes an update to grub, that will likely mean that the MBR gets updated. Like an upgrade from grub-legacy (v.1) to grub2. Not every time mind you. If you run lilo, you basically update the MBR everytime you upgrade the kernel. Plus that whole windows tendency to overwrite the MBR everytime you install it. Although I'm not up to date on that since I haven't really run (or installed) windows since 2002. And yet in close proximity to all that is the partition table that identifies how a disk is allocated.

As far as dd backups from a live and running system. Really bad idea. /proc, /dev, and /sys are meant to be virtual. And /tmp is meant to be cleared at boot. All of which will have a "population" from a backup done from a live system. And with dd you might even capture a filesystem mid-write which could leave all or a portion of the filesystem corrupt at best, when restored. All of which could cause considerably more down time than taking the system down for a non-live backup. Mess up /boot/ or /*bin/ and things could get real messy, real fast.
 
Old 12-25-2012, 11:08 PM   #11
strungoutfan78
Member
 
Registered: Mar 2011
Location: San Diego, CA
Distribution: Debian
Posts: 46

Original Poster
Rep: Reputation: 0
Thanks for that. Ironically, during this conversation, my laptop hd has become corrupt. Just my luck. I just deleted my only backup too.

I'm typing this from my Knoppix USB drive. (Thank God for Knoppix!) I've mounted all my hd partitions and I'm trying to extract everything I can with good ol' rsync. So far so good. I did forget to exclude /sys though. I'll have to remember to delete that.

This now leads me to a new question:
I'm running Debian Stable, up to date; What is the best way to recover this once I get a new HD?

Should I make a list of installed packages, reinstall the OS, then just recover my configs? Or should I reinstall the OS then just copy all the backup data back to the clean install?
 
Old 12-26-2012, 09:14 AM   #12
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,361

Rep: Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001
Quote:
Originally Posted by Shadow_7 View Post
/proc, /dev, and /sys are meant to be virtual.
The content of these virtual filesystems does not exist on disk, do dd would not pick it up.
 
Old 01-02-2013, 10:45 PM   #13
strungoutfan78
Member
 
Registered: Mar 2011
Location: San Diego, CA
Distribution: Debian
Posts: 46

Original Poster
Rep: Reputation: 0
Well I guess I've determined that this is indeed an impractical solution for me (or anyone rather). Thanks for weighing in on it. I think I'm going to give rsnapshot a whirl. It seems to be the option that provides me with a solution that fits what I'm looking to do the best. As a side note, anyone experienced with bacula? It sounds fantastic, but more for an enterprise/network backup scenario, not my laptop. It also seems very difficult to set up and configure. Any thoughts?

P.S
I was able slap a new hard disk into my BlacX esata dock and successfully transfer my entire faulty HD in place, while simultaneously upgrading from 500GB to 1TB. No need to reinstall anything. Had to do a little chroot'ing to fix some things but otherwise good old "cp --preserve=all" worked great where dd kept failing. Only had to reinstall one library (libnss3 I think) because it had too many symbolic links.
 
Old 01-02-2013, 11:34 PM   #14
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD, Raspbian, Arch
Posts: 2,283

Rep: Reputation: 354Reputation: 354Reputation: 354Reputation: 354
Quote:
Originally Posted by strungoutfan78 View Post
Thanks for that. Ironically, during this conversation, my laptop hd has become corrupt. Just my luck. I just deleted my only backup too.

I'm typing this from my Knoppix USB drive. (Thank God for Knoppix!) I've mounted all my hd partitions and I'm trying to extract everything I can with good ol' rsync. So far so good. I did forget to exclude /sys though. I'll have to remember to delete that.
rsync is fine for copying files from a non-corrupted filesystem. But if it's corrupted as you say, rsync is not your best solution. Depending on what is corrupted, you might be able to get some stuff off by using rsync. But you might leave a lot of important stuff behind too. This is where you would want to use dd to copy EVERYTHING over to a backup image, corruption, swap space, and all. Then you would work with this backup image to pull out corrupted stuff that might not be reachable with rsync. For example, you might want to use "photorec", which does not depend on any filesystem.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Trouble mounting a filesystem image created using 'dd' Klaus_K Linux - General 2 10-23-2012 01:52 PM
Using 'dd' and 'gzip' to create a disk image. Restoring image: can't find lvm_root. skiman1979 Linux - Desktop 2 02-25-2012 06:42 AM
clonezilla clone disk to image, when restore from image to destination hdd problem. hocheetiong Linux - Newbie 3 05-06-2008 08:59 PM
Mounting a qemu created root disk image Valhalla Linux - Software 4 10-26-2005 09:34 PM
Adding more space to image created by dd fyanardi Linux - Software 0 08-31-2005 11:25 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:05 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration