Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
For a USB Flash Drive to be bootable it needs the same as a HDD does, that is something for the BIOS to load from the first 512 bytes (a master boot record, MBR) which can load a more capable boot loader which can (directly or indirectly) load an operating system.
In the case of .iso files which, when copied by dd to a UFD, create a bootable UFD, the start of the ISO 9660 image must be such an MBR.
ISO 9660 allows for this. From Wikipedia: "The first 32768 bytes of the disk are unused by ISO 9660 data structure, and therefore available for other use".
Quote:
Originally Posted by AwesomeMachine
U3 -Awesome
Do you mean this U3, AwesomeMachine? If so, isn't it a Windows-specific thing?
1 members found this post helpful.
Click here to see the post LQ members have rated as the most helpful post in this thread.
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
Yes, that's it. It is a windows thing, but it messes up Linux DIY stuff. You must write zeroes over the first 100,000 sectors or so, a few times.
dd if=/dev/zero of=/dev/sdb bs=32768 count=1562
and hit the up arrow after, two or three times. It sometimes helps to pull the drive out of the USB port after the first dd run. Before you put an operating system on the drive, you have to partition and format. Unless you use mkfs.xfs, which makes its own partition before formatting. But I'm not sure if xfs works for /boot with grub.
All flash drives are NOT infected by U3, but many of them are. It doesn't hurt to try the method I've described. Just don't tell anyone else! Not really. But it's a good secret to know.
Catkin has a point, but most machines will boot a do an el torito boot (CD-type boot method) off a USB drive. However, you can't write to iso9660 after you boot it, even if the media is writable. CD file systems have no random-access write provision, because CD media is not random-access writable. But I make a reservation on that in case someone sees this reply and cites the one bizarre exception.
Last edited by AwesomeMachine; 09-24-2012 at 06:23 AM.
Great informative post. It really highlights the advantages of Free software. There's no Windows-style hand-waving (caused by the closed nature of the software), everything is clearly stated, there's no mystery. The exchange above concerning U3 & isos is a good example where an issue that could be cited by MS apologists as an instance of Windows being superior, is explained in a straightforward way that makes it clear that any problem lies with closed proprietary executables "infecting" USB keys.
wow... very through explanation of the dd (disk destroyer :P) command! I love seeing this, it is far better than any man file I have seen and I would love to see more of these info files on more commands!!! When I am trying to figure out how to do something, I usually find a fix for the issue, but the commands used in the fix are usually not explained, so I don't know why it works, and that is what I want is something like this that explains everything... after I read this 4 or 5 times, and set up a drive to play with it I feel like I will truely understand this, thanks to you!
Gents, ive been reading for hours and not quite seen what im after, can anyone help
i have a dreamplug which runs off and sd. the original card was pre loaded with an image which was fine until the system crashed. cut a long story short when i re tried to install the 4gb image to the 4Gb disk it failed to operate using
dd if=imagefile.img of=/dev/somecardref bs=512
again cutting the story short, i found by installing a 2gb image onto the disk it worked, but now ive tweaked the install i want to back it up using DD but ignoring the empty/ unused partition at the end of the drive, because ths then creates a 4gb image file which includes the empty unused partition.
Which brings me back to the problem in hand in that i think there may be a few bad sectors on the drive (hence it wont accept the original 4gb image)
will the notrunc option resolve this.
aternatively can i dd each partition in turn and stick them together somehow or even extract the used partitons and re install them seperatley one after the other back to the disk, but hen how do i manage the mbr which is surely seperate
If it is a classical MBR then just backup the MBR (including the partition-table) with
Code:
dd if=/dev/SDCARD of=~/mbr bs=512 count=1
The partition can be backed up with
Code:
dd if=/dev/PARTITION_ON_THE_SDCARD of=~/image
where you of course use the partition name, for example /dev/sdb1.
Copying the image of the partition/card back to the SD with the bs=512 parameter can cause serious slowdowns and wearout on the card, you should try to use a bs parameter that matches the cards erase block size.
By the way, if the card has bad sectors, just replace it, they are cheap nowadays.
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
Disks and partitions
Quote:
Originally Posted by lostlinlinux
Gents, ive been reading for hours and not quite seen what im after, can anyone help
i have a dreamplug which runs off and sd. the original card was pre loaded with an image which was fine until the system crashed. cut a long story short when i re tried to install the 4gb image to the 4Gb disk it failed to operate using
dd if=imagefile.img of=/dev/somecardref bs=512
again cutting the story short, i found by installing a 2gb image onto the disk it worked, but now ive tweaked the install i want to back it up using DD but ignoring the empty/ unused partition at the end of the drive, because ths then creates a 4gb image file which includes the empty unused partition.
Which brings me back to the problem in hand in that i think there may be a few bad sectors on the drive (hence it wont accept the original 4gb image)
will the notrunc option resolve this.
aternatively can i dd each partition in turn and stick them together somehow or even extract the used partitons and re install them seperatley one after the other back to the disk, but hen how do i manage the mbr which is surely seperate
The amazing thing about dd is not dd, but *nix kernel virtual device-files. Addressing devices as files solves the problem of applying permissions to hardware.
Disks are addressed as disks or file-systems. /dev/sda is a disk. /dev/sda1 is a file system. You can copy disks to disks, or file-systems to file-systems (or empty partitions).
Disk to disk images (verb) all the file-systems, file-system to file-system images one file-system (partition).
Dd will write a disk to a useable image directly to another disk. But, a file-system requires a partition on the destination, sort of, for sanity's sake.
So, you can copy only one partition, but it must have partition on the destination to be written to. Otherwise, you'll have no MBR, or a messed-up one!
I've never actually tried:
dd if=/dev/sda1 of=/dev/sdb
but it could eventually cause problems even under ideal conditions.
I haven't checked many pages and the search is bad with finding "dd" so sorry AwesomeMachine.
Maybe you could post how continue dd if stopped (besides paused with Ctrl+z and fg).
When stopped with Ctrl+c use dd's informations "records in" and "records out":
Code:
33706002+0 records in
33706002+0 records out
If you had to reboot and remember the "records" just give them at the next invocation:
Code:
dd if=/dev/zero of=/dev/sdX seek=33706002
Pay attention that the block size is the same. dd's default is 512 bytes.
So if you want to use a 1k block size you have to divide seek by 2.
"Records" are [about the same as] sectors shown in e.g. fdisk:
Code:
fdisk -lu /dev/sdX
Total capacity in bytes divided by block size.
To find out with hexdump what's not zeroed out play around with the skip option:
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
axl, I like the way you think. You're industrious and resourceful. The direct flag sounds like it would be faster, but it's not. There are kernel constraints that prevent dd from performing actual direct I/O. There are heavy restrictions to using direct. Presently, I advise against it for Linux users. It's only within the last year or two that direct could be used at all with the Linux kernel.
axl, I like the way you think. You're industrious and resourceful. The direct flag sounds like it would be faster, but it's not. There are kernel constraints that prevent dd from performing actual direct I/O. There are heavy restrictions to using direct. Presently, I advise against it for Linux users. It's only within the last year or two that direct could be used at all with the Linux kernel.
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
Zerofree may or may not actually work. And it might work sometimes but not others. If you want to zero free space, you can:
dd if=/dev/zero of=/home/user/file.file
And dd will fill overwrite yhe free space on the partition that's mounted on /home, or the entire / partition if there's no separate /home partition, and then you have to:
sync
rm -f /home/user/file.file
One footnote: to make a sparse file (for a virtual-machine image) it is proper to use dd in an incorrect way:
and hit the up arrow after, two or three times. It sometimes helps to pull the drive out of the USB port after the first dd run...
Regrettably doesn't work for me . On the other hand I have a "ReadyBoostPerfTest.tmp" file on the stick, so maybe this is a different sort of abomination.
Question: I want to restore an image from an USB-stick back to the harddisk as an .iso file like it originally was. I tried "dd if=/dev/sdb of=image.iso" and got all 8 GB of the stick into the file. Originally it was just about 2.7 GB. How can I discard the empty "tail"?
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
Restoring ISO files to hard drive images
Quote:
Originally Posted by JZL240I-U
Regrettably doesn't work for me . On the other hand I have a "ReadyBoostPerfTest.tmp" file on the stic, so maybe this is a different sort of abomination.
Question: I want to restore an image from an USB-stick back to the harddisk as an .iso file like it originally was. I tried "dd if=/dev/sdb of=image.iso" and got all 8 GB of the stick into the file. Originally it was just about 2.7 GB. How can I discard the empty "tail"?
It depends how you wrote the iso file. If you wrote it as a file then you can use dd to write the file back to disk as an iso image.
If you wrote the iso file to a partition or an entire drive, you must mount the drive, use rsync to copy the files into a directory tree on the HDD, and then use genisoimage to write the directory tree back to an iso image.
Otherwise, you might get a much larger file than you want.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.