LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-24-2017, 11:45 AM   #16
NotionCommotion
Member
 
Registered: Aug 2012
Posts: 761

Original Poster
Rep: Reputation: Disabled

Quote:
Originally Posted by pan64 View Post
but I still do not understand, why dd | gzip?
Code:
# to create:
ssh user@machine2 "dd if=/dev/sda" | gzip -1 > image.gz
# to restore:
gzip -d -c image.gz | ssh user@machine2 "sudo dd of=/dev/sda bs=256k"
Or am I missed something?
see man gzip, gzip -# (digit): number is the compression level (1 fastest, 9 best)
Hopefully, I am not mistaken, but my understanding is that it will pipe dd stdout (using the specified input file and byte size) to gzip's stdin for compression using the fastest but least effective algorithm.

Just executed the following 982 seconds ago, and am 11GB out of 32GB complete.

Code:
ssh michael@machine2 "dd if=~/path/to/my/images/RPi2a.img.gz bs=4M" | gzip -cd | sudo dd of=/dev/sda bs=4M status=progress
 
Old 12-24-2017, 12:35 PM   #17
NotionCommotion
Member
 
Registered: Aug 2012
Posts: 761

Original Poster
Rep: Reputation: Disabled
Nope... Any thoughts?

Code:
michael@gb-stretch-4:~ $ ssh michael@myserver.net "dd if=~/gateway/RPi_client_images/RPi2a.img.gz bs=4M" | gzip -cd | sudo dd of=/dev/sda bs=4M status=progress
michael@myserver.net's password:
29700259840 bytes (30 GB, 28 GiB) copied, 2413.01 s, 12.3 MB/s825+1 records in
825+1 records out
3463047295 bytes (3.5 GB) copied, 2404.77 s, 1.4 MB/s
31094996992 bytes (31 GB, 29 GiB) copied, 2526.01 s, 12.3 MB/s
dd: error writing '/dev/sda': No space left on device
0+721223 records in
0+721222 records out
31104958464 bytes (31 GB, 29 GiB) copied, 2539.8 s, 12.2 MB/s
michael@gb-stretch-4:~ $
 
Old 12-24-2017, 01:06 PM   #18
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,360

Rep: Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001
Quote:
Originally Posted by NotionCommotion View Post
Nope... Any thoughts?

Code:
michael@gb-stretch-4:~ $ ssh michael@myserver.net "dd if=~/gateway/RPi_client_images/RPi2a.img.gz bs=4M" | gzip -cd | sudo dd of=/dev/sda bs=4M status=progress
michael@myserver.net's password:
29700259840 bytes (30 GB, 28 GiB) copied, 2413.01 s, 12.3 MB/s825+1 records in
825+1 records out
3463047295 bytes (3.5 GB) copied, 2404.77 s, 1.4 MB/s
31094996992 bytes (31 GB, 29 GiB) copied, 2526.01 s, 12.3 MB/s
dd: error writing '/dev/sda': No space left on device
0+721223 records in
0+721222 records out
31104958464 bytes (31 GB, 29 GiB) copied, 2539.8 s, 12.2 MB/s
michael@gb-stretch-4:~ $
That's simple. The image was of a disk larger than the 31GB /dev/sda. How big was it?
Code:
gzip -dc <~/gateway/RPi_client_images/RPi2a.img.gz | wc -c
It looks like you made /dev/sda just large enough to hold the compressed image file, and that's way too small for the decompressed image.

Last edited by rknichols; 12-24-2017 at 01:10 PM.
 
Old 12-25-2017, 12:51 AM   #19
NotionCommotion
Member
 
Registered: Aug 2012
Posts: 761

Original Poster
Rep: Reputation: Disabled
rknichols, Per your recommended command, the image was 31914983424 bytes. I thought it would have been the same size as the file was created from a disk of the same size.

Code:
sudo dd bs=4M if=/dev/sda | gzip > ~/RPi2a.img.gz

Last edited by NotionCommotion; 12-25-2017 at 07:46 AM.
 
Old 12-25-2017, 09:12 AM   #20
_roman_
Member
 
Registered: Dec 2017
Location: _Austro_Bavaria_
Distribution: gentoo / linux mint
Posts: 433

Rep: Reputation: 29
Quote:
I thought it would have been the same size as the file was created from a disk of the same size.
No, you copy over empty space also. And than you use gzip to compress it
 
Old 12-25-2017, 09:32 AM   #21
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,000
Blog Entries: 1

Rep: Reputation: 846Reputation: 846Reputation: 846Reputation: 846Reputation: 846Reputation: 846Reputation: 846
You could be more verbose.

dd if=/dev/sda of=/dev/stdout bs=1M | gzip .....

Although it's probably an assumed default (until something changes). If you're not that worried about encryption and there's an open port, you can use netcat instead of ssh. Each with it's set of caveats.

$ nc -l -w 300 -p 5900 > got_it.file

$ nc 192.168.2.99 5900 < sent_it.file

Where the -l one listens (run it first). And the sent_it.file could be something like $(dd .....). With the caveat being that you won't know when to stop the sender or receiver until you have to whole file and it hasn't updated for a while. File size, date/time stamp, md5sum's, and other ways to manually check. Like giving it it's own interface and checking traffic with something like speedometer, when the traffic stops you likely have the whole file. Some versions of netcat have an option to quit when done, but not a default, and not always there.
 
Old 12-25-2017, 10:06 AM   #22
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,360

Rep: Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001
Quote:
Originally Posted by rknichols View Post
That's simple. The image was of a disk larger than the 31GB /dev/sda. How big was it?
Code:
gzip -dc <~/gateway/RPi_client_images/RPi2a.img.gz | wc -c
It looks like you made /dev/sda just large enough to hold the compressed image file, and that's way too small for the decompressed image.
Quote:
Originally Posted by NotionCommotion View Post
rknichols, Per your recommended command, the image was 31914983424 bytes. I thought it would have been the same size as the file was created from a disk of the same size.

Code:
sudo dd bs=4M if=/dev/sda | gzip > ~/RPi2a.img.gz
That is nothing like my recommended command, which would have reported how large the decompressed file would be.

I have to ask, what is the point of compression if the resulting file is the same size as the original? That's just absurd.
 
Old 12-25-2017, 10:24 AM   #23
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 6,230

Rep: Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724
I fail to see any advantage with using dd rather than tar, rsync, scp, cp etc for copying whole partition accross systems

- no test for file integrity
- wasting time copying emtpy space
- will keep files fragmented
 
Old 12-25-2017, 12:52 PM   #24
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,576

Rep: Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341Reputation: 4341
dd does not really care about the filesystem, just the content of the partition. Sometimes better (especially when you want to save a damaged/corrupted filesystem).
 
Old 12-26-2017, 12:38 AM   #25
NotionCommotion
Member
 
Registered: Aug 2012
Posts: 761

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
That is nothing like my recommended command, which would have reported how large the decompressed file would be.

I have to ask, what is the point of compression if the resulting file is the same size as the original? That's just absurd.
Sorry, my response was unclear. I followed your recommended command to determine the size which I then documented in my response, and then showed the command which I came up with to create the image. Two very different commands!

The point? After reading your response, I tend to believe I am missing something. I assumed if I take the bits from a device using dd, compress them,send them on their way, and decompress them when they arrive, this is a good thing. Maybe I am assuming without knowing?
 
Old 12-26-2017, 12:43 AM   #26
NotionCommotion
Member
 
Registered: Aug 2012
Posts: 761

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by keefaz View Post
I fail to see any advantage with using dd rather than tar, rsync, scp, cp etc for copying whole partition accross systems

- no test for file integrity
- wasting time copying emtpy space
- will keep files fragmented
Just trying to clone a device in its entirety.
 
Old 12-26-2017, 04:45 AM   #27
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 10,991

Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
Personally, I find rsync to be a good way of cloning a device, although tar & scp work perfectly well. Tar gives you the option of zipping with a simple program option.

What dd gives you extra is the boot record, the partition table, and the directory structure throughout the drive which in most cases is disk specific, and useless or even unwanted if you restore to a different disk.

Let's say you have a disk crash, and mark some bad sectors, but the rest is fine. Are you sure the disk won't write those sectors anyhow? It doesn't check anything, or even mount the disk.Even if it skips them, won't you then run out of space?
 
Old 12-26-2017, 10:31 AM   #28
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,360

Rep: Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001
Quote:
Originally Posted by NotionCommotion View Post
Sorry, my response was unclear. I followed your recommended command to determine the size which I then documented in my response, and then showed the command which I came up with to create the image. Two very different commands!
OK. That size (31914983424) was similar enough to the place that the dd command ran out of space (31104958464) that I didn't notice the difference.

But, /dev/sda is indeed more than 800MB smaller than uncompressed source image. It just isn't going to fit.
 
Old 12-27-2017, 01:13 AM   #29
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,000
Blog Entries: 1

Rep: Reputation: 846Reputation: 846Reputation: 846Reputation: 846Reputation: 846Reputation: 846Reputation: 846
dd can copy the WHOLE device. With partx and other routes to treat the resulting file like a device with mountable partitions.

I've had issues with tar on 32 bit systems with resulting files in excess of 4GB. I generally rsync my installs for backing up, because it saves a lot of space and doesn't need a lot of know how to access individual files. Various cp routes with the wrong options can otherwise remove permissions, change date/time stamps, and other not quite a backup of a usable system things. In short, if you're not well versed, and have the space, dd is the simpler and safer option IMO. Especially if you just want an undo option and are otherwise dealing with quirky OSes like windows.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Executing multiple commands on multiple Linux machines earlfox Linux - General 13 07-13-2012 02:47 PM
multiple linux machines bong.mau Linux - Software 3 06-09-2007 09:54 AM
Rsync between multiple machines Maeltor Linux - General 7 08-01-2006 12:22 AM
Multiple Machines Compiling RemusX2 Linux - General 2 07-06-2005 05:44 AM
dhcp and multiple machines xthel Linux - Networking 7 06-16-2004 09:32 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 01:03 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration