Adapting from my post in another LQ thread ...

I'm very hazy on the details but ...

For a USB Flash Drive to be bootable it needs the same as a HDD does, that is something for the BIOS to load from the first 512 bytes (a master boot record, MBR) which can load a more capable boot loader which can (directly or indirectly) load an operating system.

In the case of .iso files which, when copied by dd to a UFD, create a bootable UFD, the start of the ISO 9660 image must be such an MBR.

ISO 9660 allows for this. From Wikipedia: "The first 32768 bytes of the disk are unused by ISO 9660 data structure, and therefore available for other use".

Do you mean this U3, AwesomeMachine? If so, isn't it a Windows-specific thing?

Yes, that's it. It is a windows thing, but it messes up Linux DIY stuff. You must write zeroes over the first 100,000 sectors or so, a few times.

dd if=/dev/zero of=/dev/sdb bs=32768 count=1562

and hit the up arrow after, two or three times. It sometimes helps to pull the drive out of the USB port after the first dd run. Before you put an operating system on the drive, you have to partition and format. Unless you use mkfs.xfs, which makes its own partition before formatting. But I'm not sure if xfs works for /boot with grub.

All flash drives are NOT infected by U3, but many of them are. It doesn't hurt to try the method I've described. Just don't tell anyone else! Not really. But it's a good secret to know.

Catkin has a point, but most machines will boot a do an el torito boot (CD-type boot method) off a USB drive. However, you can't write to iso9660 after you boot it, even if the media is writable. CD file systems have no random-access write provision, because CD media is not random-access writable. But I make a reservation on that in case someone sees this reply and cites the one bizarre exception.

Great informative post. It really highlights the advantages of Free software. There's no Windows-style hand-waving (caused by the closed nature of the software), everything is clearly stated, there's no mystery. The exchange above concerning U3 & isos is a good example where an issue that could be cited by MS apologists as an instance of Windows being superior, is explained in a straightforward way that makes it clear that any problem lies with closed proprietary executables "infecting" USB keys.

wow... very through explanation of the dd (disk destroyer :P) command! I love seeing this, it is far better than any man file I have seen and I would love to see more of these info files on more commands!!! When I am trying to figure out how to do something, I usually find a fix for the issue, but the commands used in the fix are usually not explained, so I don't know why it works, and that is what I want is something like this that explains everything... after I read this 4 or 5 times, and set up a drive to play with it I feel like I will truely understand this, thanks to you!

-OcO-

Gents, ive been reading for hours and not quite seen what im after, can anyone help
i have a dreamplug which runs off and sd. the original card was pre loaded with an image which was fine until the system crashed. cut a long story short when i re tried to install the 4gb image to the 4Gb disk it failed to operate using

dd if=imagefile.img of=/dev/somecardref bs=512

again cutting the story short, i found by installing a 2gb image onto the disk it worked, but now ive tweaked the install i want to back it up using DD but ignoring the empty/ unused partition at the end of the drive, because ths then creates a 4gb image file which includes the empty unused partition.

Which brings me back to the problem in hand in that i think there may be a few bad sectors on the drive (hence it wont accept the original 4gb image)

will the notrunc option resolve this.

aternatively can i dd each partition in turn and stick them together somehow or even extract the used partitons and re install them seperatley one after the other back to the disk, but hen how do i manage the mbr which is surely seperate

If it is a classical MBR then just backup the MBR (including the partition-table) with The partition can be backed up with where you of course use the partition name, for example /dev/sdb1.
Copying the image of the partition/card back to the SD with the bs=512 parameter can cause serious slowdowns and wearout on the card, you should try to use a bs parameter that matches the cards erase block size.

By the way, if the card has bad sectors, just replace it, they are cheap nowadays.

Disks and partitions
 

The amazing thing about dd is not dd, but *nix kernel virtual device-files. Addressing devices as files solves the problem of applying permissions to hardware.

Disks are addressed as disks or file-systems. /dev/sda is a disk. /dev/sda1 is a file system. You can copy disks to disks, or file-systems to file-systems (or empty partitions).

Disk to disk images (verb) all the file-systems, file-system to file-system images one file-system (partition).

Dd will write a disk to a useable image directly to another disk. But, a file-system requires a partition on the destination, sort of, for sanity's sake.

So, you can copy only one partition, but it must have partition on the destination to be written to. Otherwise, you'll have no MBR, or a messed-up one!

I've never actually tried:

dd if=/dev/sda1 of=/dev/sdb

but it could eventually cause problems even under ideal conditions.

[pedant]/dev/sda1 is a partition. It may hold a file system, as may /dev/sda.[/pedant]

I haven't checked many pages and the search is bad with finding "dd" so sorry AwesomeMachine.
Maybe you could post how continue dd if stopped (besides paused with Ctrl+z and fg).

When stopped with Ctrl+c use dd's informations "records in" and "records out":
If you had to reboot and remember the "records" just give them at the next invocation:
Pay attention that the block size is the same. dd's default is 512 bytes.
So if you want to use a 1k block size you have to divide seek by 2.

"Records" are [about the same as] sectors shown in e.g. fdisk:
Total capacity in bytes divided by block size.

To find out with hexdump what's not zeroed out play around with the skip option:
or
For zeroing the direct flag could also prove faster.

axl, I like the way you think. You're industrious and resourceful. The direct flag sounds like it would be faster, but it's not. There are kernel constraints that prevent dd from performing actual direct I/O. There are heavy restrictions to using direct. Presently, I advise against it for Linux users. It's only within the last year or two that direct could be used at all with the Linux kernel.

The reply is good. Keep on hacking!

-Awesome

i wonder if zerofree is faster ?

Zerofree may or may not actually work. And it might work sometimes but not others. If you want to zero free space, you can:

dd if=/dev/zero of=/home/user/file.file

And dd will fill overwrite yhe free space on the partition that's mounted on /home, or the entire / partition if there's no separate /home partition, and then you have to:

sync

rm -f /home/user/file.file

One footnote: to make a sparse file (for a virtual-machine image) it is proper to use dd in an incorrect way:

dd if=/dev/zero of=~/sparse.file bs=1k seek=2000000 count=0

mkfs.ext4 sparse.file

Dd moves forward 2.0 GB and writes nothing. That will reserve the 2.0 GB sparse file, but it will grow in size only as files are added.

Regrettably doesn't work for me :(. On the other hand I have a "ReadyBoostPerfTest.tmp" file on the stick, so maybe this is a different sort of abomination.

Question: I want to restore an image from an USB-stick back to the harddisk as an .iso file like it originally was. I tried "dd if=/dev/sdb of=image.iso" and got all 8 GB of the stick into the file. Originally it was just about 2.7 GB. How can I discard the empty "tail"?

Superb

Restoring ISO files to hard drive images
 

It depends how you wrote the iso file. If you wrote it as a file then you can use dd to write the file back to disk as an iso image.

If you wrote the iso file to a partition or an entire drive, you must mount the drive, use rsync to copy the files into a directory tree on the HDD, and then use genisoimage to write the directory tree back to an iso image.

Otherwise, you might get a much larger file than you want.