LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 08-15-2016, 02:44 AM   #1
hack3rcon
Senior Member
 
Registered: Jan 2015
Posts: 1,432

Rep: Reputation: 11
Post How to use "dd" for create an image from used parts?


Hello.
I have an external HDD with 1TB capacity and 500GB of this HDD is used. I just like to create an image from used parts and not whole Disk. Is it possible?

Thank you.
 
Old 08-15-2016, 08:58 AM   #2
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,457
Blog Entries: 13

Rep: Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553
I think more would be helpful to know. For instance I had to do something like this with an ext2 file system, but it was not the boot or kernel. I was able to create a maximum file system somewhere else, copy from the source to that temporary file system, and then adjust the new file system to be the minimum. This was however only data as I say.

Another option I'm thinking about is to use gparted to move all used data to a partition which is just below 500G, or just compress the partition that is there to however small it can fit, then dd off of that drive and then re-expand the partition once completed.

The harder option is to "know" what blocks are used, and then you can do a very detailed dd command specifying start and stop blocks and sizes. That's not the way I'd prefer to do that.
 
Old 08-15-2016, 09:35 AM   #3
hack3rcon
Senior Member
 
Registered: Jan 2015
Posts: 1,432

Original Poster
Rep: Reputation: 11
Can you show me how can I use "dd" for clone a HDD in specific blocks?
 
Old 08-15-2016, 09:58 AM   #4
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,457
Blog Entries: 13

Rep: Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553Reputation: 4553
How about looking at the manual page for dd? There is a bs attribute for controlling the number of blocks or bytes you copy, there are seek and skip attributes which control skipping a number of bytes before you copy, and there's a count attribute to control the number of blocks or bytes you choose to copy. I've not always had great success with it, hence why I don't recommend it. Plus given the possible fragmentation of things, same recommendation. I think it'll be messy. I'd stage some tests to verify you can do this, but still expect it to be involved.
 
1 members found this post helpful.
Old 08-15-2016, 10:04 AM   #5
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by hack3rcon View Post
Can you show me how can I use "dd" for clone a HDD in specific blocks?
You can't.

Yes, you can copy specific blocks... but the result is unusable because the filesystem contains a structure that defines how each block is used.

If you ONLY want to copy used blocks, try tar, rsync, cp... These use the filesystem to identify the important blocks, the order of those blocks and to extract the data (as a block may not be fully used...)

The only reason to use dd to copy disks is to clone the disk/partition. It has to copy even the unused blocks as dd can't tell which are used. dd has also been used to try and recover from a bad sector by copying all blocks up to the error, then copy blocks following the error (directing dd to skip the beginning and the size of the bad part) which makes an image that usually can have data retrieved from - minus the contents of the bad part.

dd can be used to copy single files - but it is not designed for that purpose.

Last edited by jpollard; 08-15-2016 at 10:08 AM.
 
Old 08-15-2016, 10:29 AM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,592

Rep: Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114
Copying just the used parts of a filesystem is a job for specialized tools, such as Clonezilla (among others).
 
1 members found this post helpful.
Old 08-16-2016, 05:09 AM   #7
hack3rcon
Senior Member
 
Registered: Jan 2015
Posts: 1,432

Original Poster
Rep: Reputation: 11
Quote:
Originally Posted by rknichols View Post
Copying just the used parts of a filesystem is a job for specialized tools, such as Clonezilla (among others).
I guess "Clonezilla" Use "dd" too????
 
Old 08-16-2016, 06:59 AM   #8
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Only if clonzilla can't identify the filesystem on the partition being copied. And when it does, it copies everything as "used"/"unused" space cannot be determined.
 
Old 08-16-2016, 03:46 PM   #9
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,274

Rep: Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458
Normally if you want to use dd to clone off you do a trick. You zero out the free area and then use compression to make the resulting image much smaller than 1Tb.

You can use clonezilla as noted or Gparted to copy off the partition in file format as opposed to bit by bit. Consider any of the other ways to copy file by file and only use dd to copy the boot area. rsync, cpio, tar and such may work fine.

I tend to use a very old program G4U.
 
Old 08-16-2016, 04:23 PM   #10
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,398
Blog Entries: 8

Rep: Reputation: 412Reputation: 412Reputation: 412Reputation: 412Reputation: 412
dd is a crude low level tool, so no (as others have noted).

The solution to your problem depends on what, precisely, you wish to do. For most purposes, copying the files rather than making an actual disk image is what you really want. The big advantage of that is that rsync can be used to incrementally keep the backup up to date (it only copies over the new/moved/changed files). The big disadvantages are:

1) No boot sector (although you can use dd to create a very tiny image of just the boot/partition sectors).

2) Special files don't get copied over correctly by rsync (I'm not sure if there are options to make them work ... cp -vax works). This is an issue for copying over special OS files.

3) UUID won't be the same.

Now, if you want to create a snapshot in time and you truly want a disc image, then my experience is that this is the best:

Step 1) Use gparted to shrink the big partition.

Step 2) Use dd to create a disk image slightly bigger than the end sector used. Something like this:

dd if=/dev/sdb of=/home/user/bigimage20160816 bs=32768 count=1234567

(The count should be the total byte size desired divided by 32768.)

When recovering from this backup, the steps would be something like:

Step 1) Use dd to copy from the disk image:

dd if=/home/user/bigimage20160816 bs=32768

Step 2) Use gparted to expand the big partition.

This works best if there's no swap partition, or the swap partition is before the big data partition. If you have to move the swap partition around with gparted, the moved partition may have a new UUID, which you'll have to account for with /etc/fstab.

Now, it does indeed take time for gparted to resize a partition, but my experience is that the time saved during the dd steps greatly exceeds the time required to do the resizing steps. So, even if this process didn't save any disk space (obviously, it does save a lot), it would still be worth it just for the time savings.
 
Old 08-17-2016, 02:24 PM   #11
RockDoctor
Senior Member
 
Registered: Nov 2003
Location: Minnesota, US
Distribution: Fedora, Ubuntu, Manjaro
Posts: 1,780

Rep: Reputation: 425Reputation: 425Reputation: 425Reputation: 425Reputation: 425
It's not dd, and might not do enough of what you want, but you might want to look at fsarchiver.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Help about "xen-create-image". hack3rcon Linux - Virtualization and Cloud 5 06-13-2016 10:01 AM
Of "libvte2_90.so.9" not existing on a Peppermint Six machine and GCC "being unable to create executables" DerAxeEffekt Linux - Newbie 3 12-22-2015 04:34 AM
[SOLVED] What is "Internal" and "External" parts in DNS configuration? hack3rcon Linux - Server 10 09-01-2015 03:00 AM
How to create an ISO image of a "live" Fedora ? ajaygarg Linux - Newbie 3 06-12-2012 04:58 AM
Can I create a custom "Backtrack" image? sl33p Linux - Newbie 4 08-22-2009 04:47 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:22 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration