LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 09-24-2012, 06:43 AM   #766
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179

Quote:
Originally Posted by apjena View Post
... can anyone give a reason why dd if=...iso of=/dev/sdb to create a bootable flash works sometime and sometime not ...
Adapting from my post in another LQ thread ...

I'm very hazy on the details but ...

For a USB Flash Drive to be bootable it needs the same as a HDD does, that is something for the BIOS to load from the first 512 bytes (a master boot record, MBR) which can load a more capable boot loader which can (directly or indirectly) load an operating system.

In the case of .iso files which, when copied by dd to a UFD, create a bootable UFD, the start of the ISO 9660 image must be such an MBR.

ISO 9660 allows for this. From Wikipedia: "The first 32768 bytes of the disk are unused by ISO 9660 data structure, and therefore available for other use".

Quote:
Originally Posted by AwesomeMachine View Post
U3 -Awesome
Do you mean this U3, AwesomeMachine? If so, isn't it a Windows-specific thing?
 
1 members found this post helpful.
Old 09-24-2012, 07:09 AM   #767
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Yes, that's it. It is a windows thing, but it messes up Linux DIY stuff. You must write zeroes over the first 100,000 sectors or so, a few times.

dd if=/dev/zero of=/dev/sdb bs=32768 count=1562

and hit the up arrow after, two or three times. It sometimes helps to pull the drive out of the USB port after the first dd run. Before you put an operating system on the drive, you have to partition and format. Unless you use mkfs.xfs, which makes its own partition before formatting. But I'm not sure if xfs works for /boot with grub.

All flash drives are NOT infected by U3, but many of them are. It doesn't hurt to try the method I've described. Just don't tell anyone else! Not really. But it's a good secret to know.

Catkin has a point, but most machines will boot a do an el torito boot (CD-type boot method) off a USB drive. However, you can't write to iso9660 after you boot it, even if the media is writable. CD file systems have no random-access write provision, because CD media is not random-access writable. But I make a reservation on that in case someone sees this reply and cites the one bizarre exception.

Last edited by AwesomeMachine; 09-24-2012 at 07:23 AM.
 
3 members found this post helpful.
Old 11-15-2012, 05:00 PM   #768
andywebsdale
Member
 
Registered: Jan 2005
Location: Lewisham,London,UK
Distribution: Debian Wheezy AMD64
Posts: 87
Blog Entries: 2

Rep: Reputation: 23
Great informative post. It really highlights the advantages of Free software. There's no Windows-style hand-waving (caused by the closed nature of the software), everything is clearly stated, there's no mystery. The exchange above concerning U3 & isos is a good example where an issue that could be cited by MS apologists as an instance of Windows being superior, is explained in a straightforward way that makes it clear that any problem lies with closed proprietary executables "infecting" USB keys.
 
1 members found this post helpful.
Old 11-15-2012, 05:34 PM   #769
OriginalCrazyone
LQ Newbie
 
Registered: Oct 2012
Posts: 23

Rep: Reputation: Disabled
wow... very through explanation of the dd (disk destroyer :P) command! I love seeing this, it is far better than any man file I have seen and I would love to see more of these info files on more commands!!! When I am trying to figure out how to do something, I usually find a fix for the issue, but the commands used in the fix are usually not explained, so I don't know why it works, and that is what I want is something like this that explains everything... after I read this 4 or 5 times, and set up a drive to play with it I feel like I will truely understand this, thanks to you!

-OcO-
 
1 members found this post helpful.
Old 12-23-2012, 06:04 AM   #770
lostlinlinux
LQ Newbie
 
Registered: Jan 2012
Distribution: Ubuntu
Posts: 2

Rep: Reputation: Disabled
Gents, ive been reading for hours and not quite seen what im after, can anyone help
i have a dreamplug which runs off and sd. the original card was pre loaded with an image which was fine until the system crashed. cut a long story short when i re tried to install the 4gb image to the 4Gb disk it failed to operate using

dd if=imagefile.img of=/dev/somecardref bs=512

again cutting the story short, i found by installing a 2gb image onto the disk it worked, but now ive tweaked the install i want to back it up using DD but ignoring the empty/ unused partition at the end of the drive, because ths then creates a 4gb image file which includes the empty unused partition.

Which brings me back to the problem in hand in that i think there may be a few bad sectors on the drive (hence it wont accept the original 4gb image)

will the notrunc option resolve this.

aternatively can i dd each partition in turn and stick them together somehow or even extract the used partitons and re install them seperatley one after the other back to the disk, but hen how do i manage the mbr which is surely seperate
 
Old 12-23-2012, 07:22 AM   #771
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Main: Gentoo Others: What fits the task
Posts: 15,653
Blog Entries: 2

Rep: Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097Reputation: 4097
If it is a classical MBR then just backup the MBR (including the partition-table) with
Code:
dd if=/dev/SDCARD of=~/mbr bs=512 count=1
The partition can be backed up with
Code:
dd if=/dev/PARTITION_ON_THE_SDCARD of=~/image
where you of course use the partition name, for example /dev/sdb1.
Copying the image of the partition/card back to the SD with the bs=512 parameter can cause serious slowdowns and wearout on the card, you should try to use a bs parameter that matches the cards erase block size.

By the way, if the card has bad sectors, just replace it, they are cheap nowadays.
 
1 members found this post helpful.
Old 12-24-2012, 11:31 PM   #772
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Smile Disks and partitions

Quote:
Originally Posted by lostlinlinux View Post
Gents, ive been reading for hours and not quite seen what im after, can anyone help
i have a dreamplug which runs off and sd. the original card was pre loaded with an image which was fine until the system crashed. cut a long story short when i re tried to install the 4gb image to the 4Gb disk it failed to operate using

dd if=imagefile.img of=/dev/somecardref bs=512

again cutting the story short, i found by installing a 2gb image onto the disk it worked, but now ive tweaked the install i want to back it up using DD but ignoring the empty/ unused partition at the end of the drive, because ths then creates a 4gb image file which includes the empty unused partition.

Which brings me back to the problem in hand in that i think there may be a few bad sectors on the drive (hence it wont accept the original 4gb image)

will the notrunc option resolve this.

aternatively can i dd each partition in turn and stick them together somehow or even extract the used partitons and re install them seperatley one after the other back to the disk, but hen how do i manage the mbr which is surely seperate
The amazing thing about dd is not dd, but *nix kernel virtual device-files. Addressing devices as files solves the problem of applying permissions to hardware.

Disks are addressed as disks or file-systems. /dev/sda is a disk. /dev/sda1 is a file system. You can copy disks to disks, or file-systems to file-systems (or empty partitions).

Disk to disk images (verb) all the file-systems, file-system to file-system images one file-system (partition).

Dd will write a disk to a useable image directly to another disk. But, a file-system requires a partition on the destination, sort of, for sanity's sake.

So, you can copy only one partition, but it must have partition on the destination to be written to. Otherwise, you'll have no MBR, or a messed-up one!

I've never actually tried:

dd if=/dev/sda1 of=/dev/sdb

but it could eventually cause problems even under ideal conditions.
 
3 members found this post helpful.
Old 12-25-2012, 04:06 AM   #773
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
Quote:
Originally Posted by AwesomeMachine View Post
/dev/sda is a disk. /dev/sda1 is a file system.
[pedant]/dev/sda1 is a partition. It may hold a file system, as may /dev/sda.[/pedant]
 
Old 03-06-2013, 03:01 PM   #774
xl
Member
 
Registered: Feb 2013
Location: Germany
Distribution: Wheezy
Posts: 46

Rep: Reputation: 7
I haven't checked many pages and the search is bad with finding "dd" so sorry AwesomeMachine.
Maybe you could post how continue dd if stopped (besides paused with Ctrl+z and fg).

When stopped with Ctrl+c use dd's informations "records in" and "records out":
Code:
33706002+0 records in
33706002+0 records out
If you had to reboot and remember the "records" just give them at the next invocation:
Code:
dd if=/dev/zero of=/dev/sdX seek=33706002
Pay attention that the block size is the same. dd's default is 512 bytes.
So if you want to use a 1k block size you have to divide seek by 2.

"Records" are [about the same as] sectors shown in e.g. fdisk:
Code:
fdisk -lu /dev/sdX
Total capacity in bytes divided by block size.

To find out with hexdump what's not zeroed out play around with the skip option:
Code:
dd if=/dev/sdX skip=33706002 | hexdump -C | grep [^00]
or
Code:
dd if=/dev/sdX bs=1k skip=16853001 | hexdump -C | grep [^00]
For zeroing the direct flag could also prove faster.
Code:
dd if=/dev/zero of=/dev/sdX bs=32k oflag=direct

Last edited by xl; 03-06-2013 at 03:13 PM.
 
Old 03-09-2013, 12:59 AM   #775
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
axl, I like the way you think. You're industrious and resourceful. The direct flag sounds like it would be faster, but it's not. There are kernel constraints that prevent dd from performing actual direct I/O. There are heavy restrictions to using direct. Presently, I advise against it for Linux users. It's only within the last year or two that direct could be used at all with the Linux kernel.

The reply is good. Keep on hacking!

-Awesome
 
1 members found this post helpful.
Old 03-09-2013, 08:37 AM   #776
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,214

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
Quote:
Originally Posted by AwesomeMachine View Post
axl, I like the way you think. You're industrious and resourceful. The direct flag sounds like it would be faster, but it's not. There are kernel constraints that prevent dd from performing actual direct I/O. There are heavy restrictions to using direct. Presently, I advise against it for Linux users. It's only within the last year or two that direct could be used at all with the Linux kernel.

The reply is good. Keep on hacking!

-Awesome
i wonder if zerofree is faster ?
 
Old 03-10-2013, 05:17 AM   #777
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Zerofree may or may not actually work. And it might work sometimes but not others. If you want to zero free space, you can:

dd if=/dev/zero of=/home/user/file.file

And dd will fill overwrite yhe free space on the partition that's mounted on /home, or the entire / partition if there's no separate /home partition, and then you have to:

sync

rm -f /home/user/file.file


One footnote: to make a sparse file (for a virtual-machine image) it is proper to use dd in an incorrect way:

dd if=/dev/zero of=~/sparse.file bs=1k seek=2000000 count=0

mkfs.ext4 sparse.file


Dd moves forward 2.0 GB and writes nothing. That will reserve the 2.0 GB sparse file, but it will grow in size only as files are added.
 
1 members found this post helpful.
Old 03-28-2013, 08:59 AM   #778
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,689

Rep: Reputation: Disabled
Quote:
Originally Posted by AwesomeMachine View Post
...
dd if=/dev/zero of=/dev/sdb bs=32768 count=1562

and hit the up arrow after, two or three times. It sometimes helps to pull the drive out of the USB port after the first dd run...
Regrettably doesn't work for me . On the other hand I have a "ReadyBoostPerfTest.tmp" file on the stick, so maybe this is a different sort of abomination.

Question: I want to restore an image from an USB-stick back to the harddisk as an .iso file like it originally was. I tried "dd if=/dev/sdb of=image.iso" and got all 8 GB of the stick into the file. Originally it was just about 2.7 GB. How can I discard the empty "tail"?

Last edited by JZL240I-U; 04-02-2013 at 02:39 AM.
 
Old 03-28-2013, 09:18 AM   #779
mddesai
Member
 
Registered: Mar 2013
Location: Bengaluru, India
Distribution: Redhat, Arch
Posts: 485

Rep: Reputation: 133Reputation: 133
Superb
 
Old 03-29-2013, 04:54 AM   #780
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Post Restoring ISO files to hard drive images

Quote:
Originally Posted by JZL240I-U View Post
Regrettably doesn't work for me . On the other hand I have a "ReadyBoostPerfTest.tmp" file on the stic, so maybe this is a different sort of abomination.

Question: I want to restore an image from an USB-stick back to the harddisk as an .iso file like it originally was. I tried "dd if=/dev/sdb of=image.iso" and got all 8 GB of the stick into the file. Originally it was just about 2.7 GB. How can I discard the empty "tail"?
It depends how you wrote the iso file. If you wrote it as a file then you can use dd to write the file back to disk as an iso image.

If you wrote the iso file to a partition or an entire drive, you must mount the drive, use rsync to copy the files into a directory tree on the HDD, and then use genisoimage to write the directory tree back to an iso image.

Otherwise, you might get a much larger file than you want.
 
2 members found this post helpful.
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 05:22 AM
The best way to learn? iz3r Programming 7 02-07-2005 12:00 AM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 04:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 02:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 02:02 AM


All times are GMT -5. The time now is 06:42 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration