LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-03-2017, 01:50 PM   #16
yankeenewbie
LQ Newbie
 
Registered: Jan 2010
Posts: 10

Original Poster
Rep: Reputation: 0

Quote:
Originally Posted by AwesomeMachine View Post
You can compress a dd image:
Code:
$ dd if=/dev/sda | gzip > image.file.gz
Today (3 July 17) I followed the process suggested by JJJCR except I ran from my Clonezille Live CD throughout; did not copy Clonezilla onto the new drive and then run thru RAM. The clone appears to boot normally and seems to be a complete and working dupe. of my original "sda" drive so I am happy.

The clone does not ask for a password which puzzles me--why would that be the case?
 
Old 07-03-2017, 02:30 PM   #17
erik2282
Member
 
Registered: May 2011
Location: TX
Distribution: Debian
Posts: 762

Rep: Reputation: 220Reputation: 220Reputation: 220
Does not ask a password for what?
 
Old 07-03-2017, 03:05 PM   #18
JeremyBoden
Senior Member
 
Registered: Nov 2011
Location: London, UK
Distribution: Debian
Posts: 1,552

Rep: Reputation: 314Reputation: 314Reputation: 314Reputation: 314
Quote:
Originally Posted by AwesomeMachine View Post
Dd doesn't have a clone command. Did you see the link I posted above. And, you can run dd on a live system. You don't have to boot with a live CD. Drives can be mounted or not. It doesn't matter with dd. But with the size of today's drives, dd can be slow.
It's a bit dangerous to run dd on a active system that is in the midst of writing transactions to disk. You don't know what state your "clone" is in.
If you boot from a live USB with the HD partitions unmounted you will get a consistent clone.
 
Old 07-03-2017, 03:32 PM   #19
jefro
Moderator
 
Registered: Mar 2008
Posts: 19,217

Rep: Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926
Just in case I didn't read it. Be sure you boot to some live media to use dd command. It won't work correctly within the subject booted hard drive.
 
Old 07-03-2017, 03:54 PM   #20
tofino_surfer
Member
 
Registered: Aug 2007
Posts: 356

Rep: Reputation: 102Reputation: 102
Quote:
My problem is that when I run the command "sudo fdisk -1" I get an error message that "-1" is invalid. The command is supposed to display all connected harddrives and is vital for correct identification of source and destination drives.
fdisk and gdisk require you to specify /dev/sdX device when using -l


Code:
04:31 PM $ sudo gdisk -l /dev/sdb
GPT fdisk (gdisk) version 0.8.6

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 488397168 sectors, 232.9 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 57F23261-A25A-49FC-8CAB-599CBCE36AB5
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 488397134
Partitions will be aligned on 2048-sector boundaries
Total free space is 263055661 sectors (125.4 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS_Boot-Main
   2        52408320        53432319   500.0 MiB   EF00  
   3       104861696       188747775   40.0 GiB    8300  
   4       188747776       209719295   10.0 GiB    8300  
   5         1026048        52406271   24.5 GiB    8300  
   6       209719296       225343487   7.5 GiB     8200  swap
   7            4096         1026047   499.0 MiB   EF00  
   8        52406272        52408319   1024.0 KiB  EF02  BIOS_Boot-Fedora_24
   9        53432320       104861695   24.5 GiB    8300  

04:31 PM $ sudo gdisk -l 
GPT fdisk (gdisk) version 0.8.6

Problem opening -l for reading! Error is 2.
The specified file does not exist!
From the gdisk manpage.

Quote:
OPTIONS
-l List the partition tables for the specified devices and then exit.
It is parted that has an -l option that lists all attached drives.

sudo parted -l works. You may have been confusing {f,g}disk with parted.

sudo parted -l will list all actual /dev/sdX devices as well as all LVM and mdraid virtual devices. It lists everything. You should use sudo parted -l instead as it is what you are looking for.
 
Old 07-03-2017, 05:44 PM   #21
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 19.1 MATE
Posts: 8,016
Blog Entries: 5

Rep: Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861Reputation: 2861
Quote:
Originally Posted by tofino_surfer View Post
fdisk and gdisk require you to specify /dev/sdX device when using -l
Not so for fdisk.

Code:
sudo fdisk -l
will list all devices.

As per man fdisk:

Code:
fdisk [options] device

fdisk -l [device...]

Last edited by hydrurga; 07-03-2017 at 05:45 PM.
 
Old 07-03-2017, 07:19 PM   #22
JeremyBoden
Senior Member
 
Registered: Nov 2011
Location: London, UK
Distribution: Debian
Posts: 1,552

Rep: Reputation: 314Reputation: 314Reputation: 314Reputation: 314
This seems to be a sometimes thing! No device is necessary!
Code:
sudo fdisk -l
gives the following output
Code:
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 83E6C9B0-EB46-4C66-8518-8A63ADE7F56A

Device         Start        End   Sectors   Size Type
/dev/sdb1       2048     614400    612353   299M Microsoft basic data
/dev/sdb2     616448  546654207 546037760 260.4G Linux filesystem
/dev/sdb3  546654208 1439582207 892928000 425.8G Linux filesystem

Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 488DCBD9-3D4B-4C43-9E1D-165B9F3DC055

Device         Start        End    Sectors  Size Type
/dev/sda1       2048    2099199    2097152    1G EFI System
/dev/sda2    2099200   43059199   40960000 19.5G Linux filesystem
/dev/sda3   43059200   51447807    8388608    4G Linux swap
/dev/sda4   51447808  366020607  314572800  150G Linux filesystem
/dev/sda5  366020608 1594820607 1228800000  586G Linux filesystem
Code:
sudo parted -l
gives more readable information.

Last edited by JeremyBoden; 07-03-2017 at 07:21 PM.
 
Old 07-03-2017, 08:29 PM   #23
tofino_surfer
Member
 
Registered: Aug 2007
Posts: 356

Rep: Reputation: 102Reputation: 102
I was unaware that there was this difference between fdisk and gdisk (GPT fdisk). I use gdisk as I have all GPT partitions except for an old data drive. However parted -l does give more information.

To the OP there is absolutely no need for any screenshot such as in post #11 when the information is textual such as in a terminal window. It is wasteful and actually shows less information than a scrollable code block. Text should simply be copied and pasted into a code block, which is scrollable if the contents are long. Text can also be copied and referred to in future posts.

In addition you gave a full screen grab when your terminal window is about a quarter the area of the whole screen. If you ever do need to make a screen grab to describe a GUI problem do it only for the window in question and not the full screen.

That said in your unneeded screen grab there appears to be a full set of partitions on /dev/sdb. Using dd with the target of sdb would overwrite the whole drive which would be unnecessary. Either the cloning was already done or sdb is in fact the original drive or you should just use rsync with each partition for the cloning. Since you made an unnecessary screen grab I can't scroll up to see what the partition table for sda looks like.
 
Old 07-03-2017, 09:09 PM   #24
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004
It works fine.

Quote:
Originally Posted by jefro View Post
Just in case I didn't read it. Be sure you boot to some live media to use dd command. It won't work correctly within the subject booted hard drive.
I've used dd dozens of times on a live system. The image is is identical to the original. I think it's an urban myth not to use dd on a live system.

Or, perhaps, if one is making a forensic duplicate, and the machine is the shut down state, you could boot from the Helix live CD in order to avoid changes to the subject file system.

EDIT: I read through the discussion again. It is true that you might not get an exact image using dd on a live system, because of disk writes that may occur after dd has copied a file(s).

But that is also fairly inconsequential, because the image is only useful for back-up once it completes. You can't delete files you might need, while the image is being made, and expect them to be in the image.

Last edited by AwesomeMachine; 07-05-2017 at 10:56 PM.
 
Old 07-04-2017, 05:48 AM   #25
yankeenewbie
LQ Newbie
 
Registered: Jan 2010
Posts: 10

Original Poster
Rep: Reputation: 0
Final CloneZilla clone boots properly and appears to be complete,. Only odd thing is it opens without asking for a password, which is good!!. Many thanks for all your pointers and comments and apologies for confusing my screenshots.

dd never did perform; refused to take any action. CloneZillia needs a thorough editing and simplification; I basically followed the defaults, running from a booting CD and ignored all the storm and thunder and warnings and came out OK.
 
Old 07-05-2017, 03:46 PM   #26
jefro
Moderator
 
Registered: Mar 2008
Posts: 19,217

Rep: Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926Reputation: 2926
Not an urban myth. Technical fact. It is so easy to boot to a live media and use dd it isn't worth the effort to try it any other way.
 
Old 07-05-2017, 04:49 PM   #27
jviray2011
LQ Newbie
 
Registered: Jul 2017
Distribution: Kali, Ubuntu, CentOS
Posts: 8

Rep: Reputation: Disabled
i use most on Deft Zero for cloning, special cases on malware stuff.. plain and simple, be careful on encrypted(LUKS) drives, i suggest to unencrypt it first.

Last edited by jviray2011; 07-05-2017 at 04:50 PM.
 
Old 07-05-2017, 10:28 PM   #28
mlp68
Member
 
Registered: Jun 2002
Location: NY
Distribution: Gentoo,RH
Posts: 333

Rep: Reputation: 40
Hm, yes, the above methods work ... often, but that's not guaranteed and depends on the disk geometry to be identical to be bootable and giving an identical partition layout. Keep in mind that anything that's dd-based is a byte-by-byte copy of the entire device, so even if you have just 10% of the capacity used on a 500G drive, you still transfer the entire 500G, which, as was pointed out, takes a long time.

I have been using variants of a different scheme for many years. It allows to clone to a disk that has a size different from the original, and also allows to make relatively small-sized backups that can serve as restore points.

1) The idea is that you re-create the same partition layout on the target disk. This even allows you to adjust the partition sizes in the process as long as they are large enough to hold the original content. This is useful if you clone your system to a smaller disk and want to be less generous with the sizes.

2) We then format the new partitions on the target disk the same way.

3) We mount the new partitions somewhere, and

4) we copy (preferably with rsync) the content of each data partition to the equivalent one of the target drive

5) we make the new disk bootable.


For example, here is a partition layout on one of my servers:

Quote:
Device Boot Start End Sectors Size Id Type
/dev/sda1 2048 12584959 12582912 6G 83 Linux
/dev/sda2 12584960 488397167 475812208 226.9G 5 Extended
/dev/sda5 12587008 54530047 41943040 20G 83 Linux
/dev/sda6 54532096 71309311 16777216 8G 82 Linux swap / Solaris
/dev/sda7 71311360 92282879 20971520 10G 83 Linux
/dev/sda8 92284928 488397167 396112240 188.9G 83 Linux
that I obtained with the "fdisk -l" command that started this thread.

Using fdisk or parted, create the same layout on the target disk. If you do not want to change anything, and if your disk is below 2G, you can use sfdisk to streamline that process - make sure you don't confuse source and target disk!

sfdisk -d /dev/sda | sfdisk /dev/sdb


(for peace of mind, you might try adding the -n option to make a "dry run" first and see that all is ok:
sfdisk -d /dev/sda | sfdisk -n /dev/sdb
)

You should end up with the exact same layout on /dev/sdb as you have on /dev/sda.

Note: I rarely use this because you can end up with unused space. It's just a few keystrokes to accomplish that layout interactively and subtly adjust sizes so you fill the entire disk. Also, I always make it so that the last /dev/sda8 partition is mounted as /home, and I increase or shrink the space here and /home takes the "overflow", if any.

Here is what is mounted as what on my system:

Quote:
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
/dev/sda5 on /usr type ext4 (rw,relatime,data=ordered)
/dev/sda7 on /var type ext4 (rw,relatime,data=ordered)
/dev/sda8 on /home type ext4 (rw,relatime,data=ordered)
and /dev/sda6 is the swap partition.

I format the new partitions on sdb the same way:

Quote:
mkfs -t ext4 /dev/sdb1
mkfs -t ext4 /dev/sdb5
mkfs -t ext4 /dev/sdb7
mkfs -t ext4 /dev/sdb8
mkswap /dev/sda6
Before going on, here are the actually used sizes:

Quote:
/dev/sda1 6061632 1985996 3744680 35% /
/dev/sda5 20511356 10307220 9139176 54% /usr
/dev/sda7 10190136 5010396 4639068 52% /var
/dev/sda8 194815596 52783900 132112508 29% /home
You see that the relatively large (20G) /usr partition (sda5) is only 54% used, so if we were to clone to a significantly smaller disk (often happens if you switch from spinning disk to SSD), we could recoup some space by reducing the size to, say, 15G only.

Now let's make the corresponding mount points:

Quote:
mkdir /mnt/n{root,usr,var,home}
(the "n" stands for "new"). You mount the target partitions:

Quote:
mount /dev/sdb1 /mnt/nroot
mount /dev/sdb5 /mnt/nusr
mount /dev/sdb7 /mnt/nvar
mount /dev/sdb8 /mnt/nhome
and then we rsync the file systems over one by one:

Quote:
rsync -ax / /mnt/nroot
rsync -ax /usr/ /mnt/nusr
rsync -ax /var/ /mnt/nvar
rsync -ax /home/ /mnt/nhome
(note that each source in rsync ends with a slash (/) - important. The "x" says not to cross file system boundaries, which is needed here.)

How to make the target disk bootable depends on whether you use grub or grub2, but I usually chroot to the new root partition and make it bootable there. (I can provide more info if you don't know how).

At this point, we have a guaranteed-to-work clone in hand.

This method is super-versatile. I use this scheme for many other applications, and as you may have guessed, I have this all scripted. I often install dozens of machines in one fell swoop, and I use this method to set up each system from a "restore point" - the ultimate backup. In this way I get 100% identical systems, and all it takes is to customize a few configuration files to give the new clone its new identity (that is automated, too). Ip address, hostname, ssh host keys, and the like.

How to get such a restore point?

Staying with my partition layout, I make a tarball of each partition's contents:

Quote:
cd /
tar -c --one-file-system . | gzip -3c > /some_external_backupdisk/root.tar.gz

cd /usr
tar -c --one-file-system . | gzip -3c > /some_external_backupdisk/usr.tar.gz

cd /var
tar -c --one-file-system . | gzip -3c > /some_external_backupdisk/var.tar.gz

cd /home
tar -c --one-file-system . | gzip -3c > /some_external_backupdisk/home.tar.gz
If my harddisk crashes one day, I boot from a CD and do the same make partitions/format/mount steps, only instead of the rsync commands above I populate the partitions from the tarballs. Make the new disk bootable as before, done.

I hope this helps and shows a more versatile way.

Last edited by mlp68; 07-05-2017 at 10:32 PM.
 
Old 07-05-2017, 11:08 PM   #29
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004
dd is written to be used on live systems.

Quote:
Originally Posted by jefro View Post
Not an urban myth. Technical fact. It is so easy to boot to a live media and use dd it isn't worth the effort to try it any other way.
But it isn't dangerous either. The entire purpose of dd is to use it on a live system. That's why it was written. That's the way it's been done for almost fifty years!

We didn't always have live CDs.

But not to belabor the point, if anyone feels better using a live CD to clone, far be it from me to contradict such a thing. I still cannot find a single documented instance in which cloning a live system caused a problem.

The US Department of Justice has done extensive testing with dd on live systems. Many times it is the preferred method!
 
Old 07-06-2017, 09:20 AM   #30
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,297

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Quote:
Originally Posted by AwesomeMachine View Post
But it isn't dangerous either. The entire purpose of dd is to use it on a live system. That's why it was written. That's the way it's been done for almost fifty years!
If you use dd to copy an active read/write filesystem, what you get is an image of what you would have if you yanked the power plug on the running system.** There may be partially written open files, inodes using space but with no link in any directory, block usage bitmaps that are not consistent with the blocks actually claimed by inodes, and various other inconsistencies in the filesystem metadata. It's probably nothing that a quick preen by fsck can't patch up, but it's a rather foolish way to clone a system.

**Somewhat worse than that, really, since the dd operation takes significant time, so there can be inconsistencies between parts of the filesystem that were copied earlier and parts that had subsequent changes that were seen later in the operation.

Last edited by rknichols; 07-06-2017 at 09:24 AM.
 
2 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Hard Drive Cloning junquesea Linux - Newbie 3 07-15-2012 10:19 PM
Cloning Hard Drive FredGSanford Linux - Software 5 08-28-2008 05:37 AM
Cloning a hard drive! Shioni Linux - General 6 03-19-2006 11:39 AM
Hard drive cloning sabeel_ansari Programming 2 02-14-2006 01:40 PM
Cloning a hard drive davcefai Debian 3 11-28-2005 05:36 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:03 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration