LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-16-2016, 01:55 AM   #1
patufet99
Member
 
Registered: Dec 2005
Location: Switzerland
Distribution: Debian/Ubuntu
Posts: 37

Rep: Reputation: 15
Create partition compressed image with dd & gzip and see the progress with pv.


What would be the correct way to create a partition compressed image with dd and gpzip and see the progress with pv?

In the hard disk I have several partitions and I only want to clone the mbr and the first two partitions. With fdisk -l i get the End block of the second partition. So XXX is the obtained end block + 1

I have tried:

Quote:
# dd if=/dev/sdb bs=4096 count=XXX | pv | gzip > sdbimage.img.gzip
but then dd does not stop to the second partition but seems to continue and backup the complete hd. What I am doing wrong?
 
Old 09-16-2016, 02:52 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,970

Rep: Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633Reputation: 3633
dd will do as it is told - implies that your arithmetic is wrong.
 
Old 09-16-2016, 03:10 AM   #3
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,398
Blog Entries: 8

Rep: Reputation: 412Reputation: 412Reputation: 412Reputation: 412Reputation: 412
fdisk will report the end block using a block size of 512 bytes.

dd will use a block size of whatever the heck you tell it. In this case, 4096 bytes. The number you use here is unimportant unless you are using a tape backup device or something like that. Basically, it just tells dd how many bytes to process at a time. Using a higher number helps performance, up to a point of diminishing returns.

But here's where you likely stumbled-you put in the block count based on what fdisk told you, but the disk block size isn't what dd uses or thinks of when you specify "count". dd doesn't care about the hard drive's block size. It only cares about the bs value you give it.

So, to get the desired count=XXX, you need to do some arithmetic. I like to use python for this arithmetic, but you can use a pencil and paper to calculate it if you like. Basically, you want to multiply by 512 to get the desired byte count, and then you divide by bs.
 
Old 09-16-2016, 07:53 AM   #4
patufet99
Member
 
Registered: Dec 2005
Location: Switzerland
Distribution: Debian/Ubuntu
Posts: 37

Original Poster
Rep: Reputation: 15
I see now. By dividing the number of blocks obtained with fdisk by 8 (bs: 4096/ fdisk logical sector size: 512) it works as expected.
Thanks for your help.
 
Old 09-16-2016, 08:12 AM   #5
Emerson
LQ Sage
 
Registered: Nov 2004
Location: Saint Amant, Acadiana
Distribution: Gentoo ~arch
Posts: 7,231

Rep: Reputation: Disabled
FYI you do not need pv any more, dd can show progress with status=progress option.
 
1 members found this post helpful.
Old 09-16-2016, 10:25 AM   #6
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,398
Blog Entries: 8

Rep: Reputation: 412Reputation: 412Reputation: 412Reputation: 412Reputation: 412
Nice! I have used dd for years, and I never even looked for that option. I have just adopted an attitude of, "It takes as long as it takes." But no more, thanks!

[edit added:]

Oh well, the version of dd in Debian Stable does not have the progress option. No big deal. It takes as long as it takes.

Last edited by IsaacKuo; 09-16-2016 at 10:29 AM.
 
Old 09-16-2016, 10:33 AM   #7
Emerson
LQ Sage
 
Registered: Nov 2004
Location: Saint Amant, Acadiana
Distribution: Gentoo ~arch
Posts: 7,231

Rep: Reputation: Disabled
Sorry, forgot to mention it is a new feature.
 
Old 09-16-2016, 02:45 PM   #8
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,176

Rep: Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436
Thoughts.

Could look at changing default compression level. The default is middle, your use may vary.

There are also file by file ways that may do just as well or better.
 
Old 09-17-2016, 05:37 AM   #9
vonbiber
Member
 
Registered: Apr 2009
Distribution: slackware 14.1 64-bit, slackware 14.2 64-bit, SystemRescueCD
Posts: 491

Rep: Reputation: 110Reputation: 110
Quote:
Originally Posted by patufet99 View Post
What would be the correct way to create a partition compressed image with dd and gpzip and see the progress with pv?

In the hard disk I have several partitions and I only want to clone the mbr and the first two partitions. With fdisk -l i get the End block of the second partition. So XXX is the obtained end block + 1
Have you tried
1. just the MBR+partition table (first 512 bytes of the disk):
Code:
dd if=/dev/sdb of=mbr.bin bs=512 count=1
2. images of 1st and 2nd partitions:
Code:
dd if=/dev/sdb1 bs=1M | gzip -9 > sdb1.img.gz
dd if=/dev/sdb2 bs=1M | gzip -9 > sdb2.img.gz
 
1 members found this post helpful.
Old 09-18-2016, 04:28 PM   #10
patufet99
Member
 
Registered: Dec 2005
Location: Switzerland
Distribution: Debian/Ubuntu
Posts: 37

Original Poster
Rep: Reputation: 15
Thank you for your comments.

Quote:
There are also file by file ways that may do just as well or better.
dd is great to make snapshots of a whole partition, so that in case of disk crash, by just restoring on another hd you get quickly a working system.
To backup the data I agree that solutions that work by files such as rsync are more adequate.

Quote:
Could look at changing default compression level.
I did some tests with bzip2 or changing the compressing level, but the compression time increased a lot. The scope was in my case to save some space but not to obtain the smallest possible file, so that the gz default was good for me.
 
Old 09-19-2016, 04:20 PM   #11
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,176

Rep: Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436Reputation: 3436
Yes, you could use a number of compression tools that would suit your data better. I meant more of an option number behind the compression. If you want speed I think it is 1 and full space is 9.

Just thoughts, you may know all this already, can't tell.
 
  


Reply

Tags
clone, compression, dd


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Using 'dd' and 'gzip' to create a disk image. Restoring image: can't find lvm_root. skiman1979 Linux - Desktop 2 02-25-2012 05:42 AM
Piping dd through gzip to image Win2k Partition sr_25 Linux - Newbie 2 07-08-2007 02:43 PM
how to create partition imaqe compressed and mountable? Anti_bug Linux - Software 5 01-10-2006 03:17 AM
create symlink with an option,e.g. gunzip & gzip,how? l_9_l Linux - General 1 05-15-2002 03:59 AM
gzip compressed files jamaso Linux - Newbie 4 01-13-2002 04:59 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 01:37 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration