LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-28-2007, 08:07 PM   #271
saikee
Senior Member
 
Registered: Sep 2005
Location: Newcastle upon Tyne UK
Distribution: Any free distro.
Posts: 3,398
Blog Entries: 1

Rep: Reputation: 112Reputation: 112

AwesomeMachine,

There has been an ineresting debate on dd in this thread.

The debate is on whether dd needs to read one full sector of 512 bytes to give us one byte when we set bs=1.

This is probably based on reading/writing a file the smallest unit is one full sector of 512 bytes and that has been taken as the minimum unit in any read/write operation of a hrd disk. dd does not care about the filing structure but does it still need to read one full sector? It has to carry some kind of additional intelligence to strip out the 511 bytes to give the exact one byte we want if it does read one sector at a time.

Care to comment?
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 02-06-2007, 11:51 PM   #272
champagne4kc
LQ Newbie
 
Registered: Feb 2007
Posts: 1

Rep: Reputation: 0
Hi All,

I would like to know how to capture the messages displayed by dd when it is done transfering data (number of records in and out, and data transfer rate). I need to do this in my script, then grep the capture to find out the data rate. I tried the following:

dd if=/dev/zero of=datafile bs=32k count=32768 | grep 'bytes/sec'

But it did not work. Seems the messages from dd were not piped to grep. I then tried:

dd if=/dev/zero of=datafile bs=32k count=32768 > output.txt

It still did not work. output.txt ended with 0 byte in it.

What is the proper way to capture the output from dd?

Any help will be greatly appreciated.

Thanks.
 
Old 02-08-2007, 01:21 AM   #273
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Smile Stdout

Quote:
Originally Posted by champagne4kc
Hi All,

How to capture the messages displayed by dd. records in records out transfer rate in a script. What is the proper way to capture the output from dd?

Any help will be greatly appreciated.

Thanks.
Dd is a little bit tricky here. The command sends the error output to STDOUT, which is the terminal. When you pipe dd to another command you are piping the bitstream dd creates to another command. For example:

dd if=/dev/sda | hexdump -C | grep 'I love you, Thor.'

Will check all of:

/dev/sda

on all file systems, regardless of what security is used, to check if your girlfriend is in love with someone else named 'Thor', and she used her computer to convey this information to someone, possibly Thor himself. But, don't ever use a computer without permission. And, trust me, if a girl cheats on you and you find out, pretend like you don't know, and suck it in. Walk away, and never look back. But, leave with your dignity. If she thinks you never knew she will respect you, even though you left her. Never let a woman know what you are thinking, and tell her everything you think she wants to hear. Like if she asks, "If I was burned over 90% of my body in a car wreck, would you still love me?", say yes. If she looks at you in the morning with her make-up all smeared, caustic breath, and a rats nest for hair you don't need to be honest with her when she asks, "Do you still think I'm beautiful without your beer goggles on?". Women always look beautiful, no matter what.

Ok, try this page to redirect STDOUT to a file using:

'exec'

and then direct it back to the terminal after you capture the dd output:

http://tldp.org/LDP/abs/html/x15794.html

on tldp.org
 
Old 02-08-2007, 12:57 PM   #274
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Smile I didn't see this right away

Quote:
Originally Posted by saikee
AwesomeMachine,

There has been an ineresting debate on dd in this thread.

The debate is on whether dd needs to read one full sector of 512 bytes to give us one byte when we set bs=1.

This is probably based on reading/writing a file the smallest unit is one full sector of 512 bytes and that has been taken as the minimum unit in any read/write operation of a hrd disk. dd does not care about the filing structure but does it still need to read one full sector? It has to carry some kind of additional intelligence to strip out the 511 bytes to give the exact one byte we want if it does read one sector at a time.

Care to comment?
I made my comments. I think the whole concept will be easier to understand now.
 
Old 02-08-2007, 01:01 PM   #275
saikee
Senior Member
 
Registered: Sep 2005
Location: Newcastle upon Tyne UK
Distribution: Any free distro.
Posts: 3,398
Blog Entries: 1

Rep: Reputation: 112Reputation: 112
Yes you did. Great stuff and essential to the general understanding of the subject.

I know we can leave the problem to be sorted out between the OS and the hard disk controller but a little lecture by the expert comes in handy.
 
Old 02-16-2007, 01:31 PM   #276
sidder
LQ Newbie
 
Registered: Feb 2007
Distribution: sid64
Posts: 2

Rep: Reputation: 0
I using dd to backup ntfs partitions.
Up untill now i was cloning a partition to a partition but i decided to dump to a file this time.

so to backup sda1 (50gb partition with 3gb used space):
dd if=/dev/sda1 of=/.../9gbfile bs=512
but:
mkfs.ntfs -f /dev/sda1
dd if=/.../9gbfile of=/dev/sda1 bs=512 conv=notrunc
or
dd if=/.../9gbfile of=/dev/sda1 bs=512
(I used also bs=4b)
18106848+0 records in
18106848+0 records out
9270706176 bytes (9.3 GB) copied, 11600.9 seconds, 799 kB/s

don't recreate sda1 as it was. I can mount the 9gbfile fine as a loop device (loop device content and original sda1 content match).
Can i use dd to restore directly from the 9gbfile or first mount the 9gbfile and then use dd? 9gbfile is in a reiserfs 3.6 filesystem.
 
Old 02-17-2007, 04:17 AM   #277
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 64
FYI:
I stuck this thread (I thought it was stuck before?) because I think it's a valuable resource. It is certainly a long thread, but the data is mostly pertinent, and is therefore also valuable.

Cool
 
Old 02-17-2007, 05:08 PM   #278
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Smile smaller to larger

Quote:
Originally Posted by sidder
I using dd to backup ntfs partitions.
Up untill now i was cloning a partition to a partition but i decided to dump to a file this time.

so to backup sda1 (50gb partition with 3gb used space):
dd if=/dev/sda1 of=/.../9gbfile bs=512
but:
mkfs.ntfs -f /dev/sda1
dd if=/.../9gbfile of=/dev/sda1 bs=512 conv=notrunc
or
dd if=/.../9gbfile of=/dev/sda1 bs=512
(I used also bs=4b)
18106848+0 records in
18106848+0 records out
9270706176 bytes (9.3 GB) copied, 11600.9 seconds, 799 kB/s

don't recreate sda1 as it was. I can mount the 9gbfile fine as a loop device (loop device content and original sda1 content match).
Can i use dd to restore directly from the 9gbfile or first mount the 9gbfile and then use dd? 9gbfile is in a reiserfs 3.6 filesystem.
Hi,

Dd copies file systems or files. If you make the file system smaller, and bitstream it to a file, when you bitstream it back to a bigger partition, the file system will shrink. I don't know how you could dd a 50 GB partition to a 9 GB file using dd. Dd will not ordinarily do that. Dd writes a file system. If you only want to back up what is used use:

cp -r /* /wherever-you-copy-to

That will copy files. You can also pipe dd to gzip and the resulting file will be quite a bit smaller. That method is contained somewhere in this thread.
 
Old 02-18-2007, 03:55 AM   #279
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,066
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Quote:
Originally Posted by MasterC
FYI:
I stuck this thread (I thought it was stuck before?) because I think it's a valuable resource. It is certainly a long thread, but the data is mostly pertinent, and is therefore also valuable.

Cool
And I've un-sticked it per our e-Mail communication,
thanks mate. :}

I still think that awesome machine should submit it as
a linux-answer, and that dd isn't sought by newbies
often enough to warrant the sticky thread for it.


Cheers,
Tink

Last edited by Tinkster; 02-18-2007 at 03:57 AM.
 
Old 02-23-2007, 01:29 AM   #280
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
This thread was designed from the ground up to be used by newbies. Everything is taught by example. Dd provides an insurance policy against losing a good install, and allows the newbie to experiment without fear of having to reinstall. This marks a remodeling an renovation in which a 1980's look was updated to third millennium, and obsolete, little used information was removed from the post. I hope the new motif is more suited to LinuxQuestions.org, and users can now get the information they need, quickly and easily.

Last edited by AwesomeMachine; 02-23-2007 at 02:36 AM.
 
Old 02-23-2007, 02:26 AM   #281
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,066
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Nice change. The new look is *much* more pleasing and sensible,
easier to read. Well done.




Cheers,
Tink
 
Old 03-06-2007, 08:52 PM   #282
CyberCod
Member
 
Registered: Jul 2006
Posts: 35

Rep: Reputation: 15
I wanted to start out by saying thank you for taking so much time and effort to document such a useful and complicated command. I've read through most of the previous posts on this thread and I'm not sure if I can do what I need to do with dd or not. I'm made a new thread here so that I wouldn't hijack anyone's thread, but you guys in here really seem to know your stuff, and if I can get you to look at my problem, I'm hopeful that one of you can give me a clue. I'm not sure if it is doable with dd or not, though some part of it will probably include dd. If you have a moment, please come by and check it out, and forgive the intrusion on such a wonderfully cared-for thread.

Last edited by CyberCod; 03-06-2007 at 08:54 PM.
 
Old 03-09-2007, 12:13 AM   #283
iframe
LQ Newbie
 
Registered: Mar 2007
Posts: 15

Rep: Reputation: 0
Bootable USB thumb drive

I did install the debian based insert.iso into my memory stick, unfortunatelly my machine is not able to boot from a usb device.
Now my usb memory appears to be 59.2MB instead of the 256 MB that it really is.
How can i restore my memory stick? i.e., erase the insert.iso file and restore the original size?

Thanks in advance

Last edited by iframe; 03-09-2007 at 12:16 AM.
 
Old 03-09-2007, 12:23 AM   #284
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Squeeze 2.6.32.9 SMP AMD64
Posts: 3,275

Rep: Reputation: 126Reputation: 126
AwsomeMachine, can I ask you consider adding one specific example to your first page: i.e. copy the MBR and first partition from one drive to another without wrecking the partition table on the second drive. I think I can do this now, but it wasn't clear to me a few months ago, and I've badmouthed "dd" as a result of not being able to add it all up to make a working method. I doubt I'm the only one that would have trouble putting this particular 1+1 together to make 2.
 
Old 03-09-2007, 06:13 AM   #285
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Write 256 MB to the drive with dd.

Quote:
Originally Posted by iframe
I did install the debian based insert.iso into my memory stick, unfortunatelly my machine is not able to boot from a usb device.
Now my usb memory appears to be 59.2MB instead of the 256 MB that it really is.
How can i restore my memory stick? i.e., erase the insert.iso file and restore the original size?

Thanks in advance
256 MB is 1024 * 1000 * 256 bytes = 256 MB:

dd if=/dev/zero of=/dev/hdb1 bs=1024 count=256000

Make your file system of choice with the appropriate command:

mkisofs
mak2fs
makreiserfs
format

You might need to partition first. Or you can use gui tools like:

gparted
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 05:22 AM
The best way to learn? iz3r Programming 7 02-07-2005 12:00 AM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 04:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 02:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 02:02 AM


All times are GMT -5. The time now is 06:52 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration