LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 09-09-2013, 11:08 PM   #796
GMHilltop
LQ Newbie
 
Registered: Oct 2010
Posts: 7

Rep: Reputation: 0
Use dd & rysnc to image an os to a smaller/larger partition?


In the past I have used software like Acornis to image partitions and then copy them back to other partitions of various sizes.
ie:
  1. Load Windows 7 and all drivers on sda1 (70GB in size)
  2. Image with Acronis and copy to sda2 (40Gb in size)
  3. Change the uuid number of sda2
    Code:
    sudo dd if=/dev/urandom of=/dev/sda2 bs=8 count=1 seek=9
  4. Load Ubuntu or Linux mint to sda3 and let grub do the rest to make a multi boot system

I'd like to learn how to do this using dd & rysnc.

I have made an image of the first partition to my storage partition (/media/C6003810003809C7) using:
Code:
dd if=/dev/sda1 conv=sync,noerror bs=8M | gzip -c  > /media/C6003810003809C7/sda1.70GB.img.gz
That seemed to work quite well - the image compressed down to about 5.8GB (I did copy zeros to all the blank space by running
Code:
dd if=/dev/zero of=/dev/sda1/bigfile.file bs=8M; rm /dev/sda1/bigfile.file
first) It was a fresh install of Windows 7 with all the drivers and updates installed.

That is about all that I really know how to do.
I know if I try to copy that image directly back to the 40GB partition it won't work, and if I try to copy it to a larger partition (say a 100GB partition) that I'll loose about 30GB's of space.

I was thinking of proceeding like this but I am not sure of how to write the code:
  1. mount this image using 'loop' ===> I have read about it but I am not sure how to do that as the image is compressed.
  2. rysnc the files back to sda2 ===> is there a way to do that that verifies they were copied correctly?
  3. If need be, as I don't know what rysnc will do yet ==> Change the uuid number of sda2
    Code:
    sudo dd if=/dev/urandom of=/dev/sda2 bs=8 count=1 seek=9
  4. Load Ubuntu or Linux mint to sda3 and let grub do the rest to make a multi boot system

Would this work?

Thanks for the help.

Last edited by GMHilltop; 09-09-2013 at 11:10 PM. Reason: correct a spelling error
 
Old 09-14-2013, 11:41 AM   #797
GMHilltop
LQ Newbie
 
Registered: Oct 2010
Posts: 7

Rep: Reputation: 0
IS there a way to "Mount" the compressed image so I can rsync it to another partition without having the original image size affect the new partitions capacity?
 
Old 09-14-2013, 12:16 PM   #798
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,213

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
Quote:
Originally Posted by GMHilltop View Post
IS there a way to "Mount" the compressed image so I can rsync it to another partition without having the original image size affect the new partitions capacity?
yes, you would loop mount the partition and copy the files you want.
 
Old 09-14-2013, 05:56 PM   #799
qrange
Member
 
Registered: Jul 2006
Location: Belgrade, Yugoslavia
Distribution: Debian
Posts: 750

Rep: Reputation: 29
@GMHilltop

I'd stick with Acronis, because unlike Linux, Windows has some copy protections.
its not a matter of simply copying files or sectors, registry and boot.ini need some modifications, iirc.
 
Old 09-15-2013, 02:53 PM   #800
GMHilltop
LQ Newbie
 
Registered: Oct 2010
Posts: 7

Rep: Reputation: 0
qrange,

Thanks for the suggestion, but I am actually not really able to do that, and I'd like to see if I can work it from Linux side of things.

The particular hardware that I am running has some strange characteristics that I have never encountered before.

The short of it (If I am able to do so in a way that makes sense) is as follows

1 - Hardware.
  • Sager NP-8275-S
  • Optimus Technology ===> which is part of the problem
  • UEFI capable ====> which is also causing issues

2 - Software
  • Windows 7 Ultimate Install DVD will not boot to install screen with UEFI enabled - Linux mint 15 installs flawlessly.
  • Turn off UEFI in BIOS, Windows installs - Linux Will install (in compatibility mode) but does not boot after that without installing Bumblebee drivers (Why UEFI would have any bearing on that is still beyond me).
  • No rescue software like ACRONIS, Hiren's.BootCD.15.2, systemrescuecd-x86-3.7.1, gparted-live-0.16.1-1-i486 - will boot - the process starts and then just hangs - UEFI enabled or disabled does not matter.
  • Only the boot-repair-disk-64bit would boot from a USB device, WHEN told to boot from the BIOS using the boot override function (F7 and choosing the which device to boot from does not work), AND WHEN you have loaded it using unetbootin-windows-584.exe only. Pendrivelinux does not work.

. . . yes I've wasted a lot of time with this, but this is where I am at.

So, I thought I'd waste a little more time (which it isn't if I can get it to work) and see if I couldn't do it as mentioned above.

I have changing the UUID's coverd using the old BIOS method - GPT Partitioned disks will be a new challenge I look forward to, but first things first.

Right or wrong, I'd like to see if I simply can copy files from a gziped image to a partition using rysnc without having to unpack the image first.

Suggestions anyone?
 
Old 09-22-2013, 11:02 PM   #801
GMHilltop
LQ Newbie
 
Registered: Oct 2010
Posts: 7

Rep: Reputation: 0
Hi schneidz,

Can you point me at a link that would help guide me through how to do that?

Again, Thanks for the help.
 
Old 09-23-2013, 10:18 AM   #802
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,213

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
Code:
sudo mount -o loop image.iso /whatever/floats/your/boat
 
Old 11-13-2013, 05:50 PM   #803
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Arrow

Quote:
Originally Posted by GMHilltop View Post
In the past I have used software like Acornis to image partitions and then copy them back to other partitions of various sizes. I'd like to learn how to do this using dd & rysnc.

I have made an image of the first partition to my storage partition (/media/C6003810003809C7) using:
Code:
dd if=/dev/sda1 conv=sync,noerror bs=8M | gzip -c  > /media/C6003810003809C7/sda1.70GB.img.gz
That seemed to work quite well - the image compressed down to about 5.8GB (I did copy zeros to all the blank space by running
Code:
dd if=/dev/zero of=/dev/sda1/bigfile.file bs=8M; rm /dev/sda1/bigfile.file
first) It was a fresh install of Windows 7 with all the drivers and updates installed.

I know if I try to copy that image directly back to the 40GB partition it won't work, and if I try to copy it to a larger partition (say a 100GB partition) that I'll loose about 30GB's of space.

I was thinking of proceeding like this but I am not sure of how to write the code:
  1. mount this image using 'loop' ===> I have read about it but I am not sure how to do that as the image is compressed.
  2. rysnc the files back to sda2 ===> is there a way to do that that verifies they were copied correctly?
  3. If need be, as I don't know what rysnc will do yet ==> Change the uuid number of sda2
    Code:
    sudo dd if=/dev/urandom of=/dev/sda2 bs=8 count=1 seek=9
  4. Load Ubuntu or Linux mint to sda3 and let grub do the rest to make a multi boot system

Would this work?
NO, you must uncompress the image file first. Acronis says it makes a drive image, but that depends on how one defines image. An actual image is an image of the original file system. Acronis, Semantec, and everyone else uses an intermediary, proprietary file system that can't be read by Windows, only by the imaging program.

That's great, because the program images only valid data, and it puts everything back almost exactly the way it was. It can use compression and allow browsing the image for files.

But it's not a real image. It's just another type of file system used exclusively by a certain imaging program. If you ever can't get that program, the data is lost.

Dd has been around since the 1970s, at least, possibly the 1960s! And it will probably be around in fifty more years! But dd only does one thing, it makes a bitstream image with optional conversion functions.

Nothing Windows works like that, because it doesn't use a device file file-system. In linux you can write to devices like you write to files.

But I'm not aware of any way to mount a gzipped image. You can uncompress it by piping through gunzip:

dd if=/file.bin | gunzip | dd of=/drive/

For the other post about booting UEFI, grub 2 is not completely GPT compatible. You have to use gdisk and muck around a bit with the partition table to get grub to UEFI boot.

Optical disks don't have GPT partition tables, so I would think you must turn UEFI off and use BIOS extensions instead. But I could be wrong on that one, so look it up.
 
1 members found this post helpful.
Old 11-14-2013, 03:16 AM   #804
ronin21
LQ Newbie
 
Registered: Nov 2013
Location: Australia
Distribution: Linux, Windows 7
Posts: 2

Rep: Reputation: Disabled
This post about 'dd" command is very detailed and useful. I learned few tricks and still absorbing more out of it.
Tried to implement my understanding by doing one simple task:
1. SanDisk Cruze 2GB to iso file on linux Kali machine.
2. Keeping clone of my USB flash disk and later on try to keep updated clone
of my BT Live USB Disk with persistence on my laptop.
I am stuck on step 1 and wont move to next until i confirm my skills.

dd if=/dev/sdc of=usb.iso conv=sync,notrunc #successfully done!
tried to mount
mkdir /mnt/usb1
mount usb.iso /mnt/usb1
Error: special device usb.iso does not exist # i guess i need to mount on loop
Tried loopback
# mkdir -p /mnt/usb1
# mount -o loop usb.iso /mnt/usb1
but Error : you must specify the filesystem type

Also i tried to mount in my windows 7 with Daemon Tools but it shows non-supported format !!

Anyone can point out my wrong step or i need to correct my method.

Regards

Ron
 
Old 11-14-2013, 04:15 AM   #805
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
The flash drive might have U3 on it. Try this:

Wipe the flash drive:

dd if=/dev/zero of=/dev/sdc bs=2048 conv=sync,noerror count=5000000

Then pull the drive out of the usb. Then, plug it back in, hit the up arrow in terminal to bring back the dd command line, and run it again. But don't write a back-up image back to the drive or you'll have the same problem again. You have to make the flash drive all over again. But it's worth it. I hope this helps.
 
Old 11-14-2013, 05:55 AM   #806
ronin21
LQ Newbie
 
Registered: Nov 2013
Location: Australia
Distribution: Linux, Windows 7
Posts: 2

Rep: Reputation: Disabled
Quote:
Originally Posted by AwesomeMachine View Post
The flash drive might have U3 on it. Try this:

Wipe the flash drive:

dd if=/dev/zero of=/dev/sdc bs=2048 conv=sync,noerror count=5000000

Then pull the drive out of the usb. Then, plug it back in, hit the up arrow in terminal to bring back the dd command line, and run it again. But don't write a back-up image back to the drive or you'll have the same problem again. You have to make the flash drive all over again. But it's worth it. I hope this helps.

You are right even after runing dd for two times still i had U3 showing in windows 7 pc.
I searched online and SanDisk website got U3 removal tool that formated it and clean it out.
After that i used same commaed as i used in the very beginning to make my mirror iso and it mounted on loop device without problemo.
No filesystem error anymore.
Salute to your advise and efforts.

But i regarded "dd" command so high and it still didnt weedout this "U3" on my USB disk!!
****
Next day i did little more digging on U3 that's what i found

http://www.linuxquestions.org/questi...sh-how-410539/

I guessed nothing can survive dd sector by sector sweep but this U3 is quite a legendary piece of work.
Anyone have this on their USB disk please google it out and there is quite a lot on it, U3 hack, U3 removal tool etc. But linux fails to temper it, what a shame?

Hope it helps another bugger!

Last edited by ronin21; 11-14-2013 at 07:08 PM.
 
1 members found this post helpful.
Old 12-13-2013, 02:46 AM   #807
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
U3 might have changed a bit. People should try this to get rid of it:

dd if=dev/zero of=/dev/sdb conv=sync,noerror bs=2k

and repeat. Also, Windows 7 might detect a flash drive without U3 and rewrite it to the drive. Whereas a Linux flash drive cannot be tracked, a U3 infected drive can. Open-source antivirus should detect U3 as malware and clean it off the drive.
 
Old 04-02-2014, 06:09 AM   #808
mfsc
LQ Newbie
 
Registered: Nov 2010
Posts: 6

Rep: Reputation: 6
Copying a partition, Grub 2 hazard

When I used dd 5 years ago I regularly made a drive copy to a second HD, suddenly I found that just trying a Live CD resulted in trashing the partition table of both disks. It was repeatable but it took me a while to realise that Grub 2 uses uuid as partition identifier and to have two 'unique' id s on one system is asking for trouble. I suggest in the second example of using dd there is an added warning that it will result in the copy having the same uuid, unless it is intended to be stored in a safe, away from the original it is as well to generate a new uuid for the copy, easily done with gparted.
 
1 members found this post helpful.
Old 04-02-2014, 06:59 AM   #809
Drakeo
Senior Member
 
Registered: Jan 2008
Location: Urbana IL
Distribution: Slackware, Slacko,
Posts: 2,658
Blog Entries: 3

Rep: Reputation: 224Reputation: 224Reputation: 224
http://u3-tool.sourceforge.net/ use linux
 
Old 04-02-2014, 07:26 AM   #810
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Smile UUIDs and DD

Quote:
Originally Posted by mfsc View Post
When I used dd 5 years ago I regularly made a drive copy to a second HD, suddenly I found that just trying a Live CD resulted in trashing the partition table of both disks. It was repeatable but it took me a while to realise that Grub 2 uses uuid as partition identifier and to have two 'unique' id s on one system is asking for trouble. I suggest in the second example of using dd there is an added warning that it will result in the copy having the same uuid, unless it is intended to be stored in a safe, away from the original it is as well to generate a new uuid for the copy, easily done with gparted.
You can also put an entry in /etc/fstab to mount with the no-uuid option. Although that might not be available for every different file system format. I know xfs includes it. But you are correct, use of dd for imaging is complicated by uuid identification.

But UUIDs make life a lot more livable. Thanks for the heads-up.

-Awesome
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 05:22 AM
The best way to learn? iz3r Programming 7 02-07-2005 12:00 AM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 04:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 02:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 02:02 AM


All times are GMT -5. The time now is 04:01 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration