LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 06-22-2010, 09:42 AM   #631
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 231Reputation: 231Reputation: 231

Quote:
Originally Posted by AwesomeMachine View Post
0x100h = 1 0000 0000b - 1b = 0xffh = 1111 1111b = 255d. So, 0xffh is hex notation for bitwise 1111 1111. In the example above, dd reads 32k of zeroes (0x00h) from /dev/zero. Sed replaces every 32k of hex zeroes with 32k of hex ff.
If anyone has any doubts, run this as a test:
Code:
dd if=/dev/zero bs=32k count=1  | sed 's,\x00,\xff,g'  | hd  | less

Quote:
Originally Posted by AwesomeMachine View Post
...the sed perl expression, ...
Two quibbles:
  1. 's,\x00,\xff,g' is not a perl expression, it's a regular expression ("regex" or "RE"), & not even a perl RE -- sed can't do those. Nor a grep/POSIX extended regex -- there's no "-r" option. For a quick take on the "flavors" of RE's, see the grep man page. For a more full & confusing discussion, see the regex man page.

  2. The default RE separator, "/", often makes RE's visually confusing: Quick, what does '/\/\/\/\//\/\/\//' do?
    I was taught to use "," instead; unless of course my RE contains one, then I go to "#" or "!". BTW this works in awk & perl also.
 
Old 06-22-2010, 11:55 AM   #632
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Arch,

My terminology might not be technically correct. I just call 'em perl expressions, because you use 'em with perl. I have found certain instances, with complex expressions, where / was not a good choice, simply because it didn't work. I'll remember what you wrote.
 
Old 06-23-2010, 07:01 AM   #633
jpantano88
LQ Newbie
 
Registered: Jun 2010
Posts: 6

Rep: Reputation: 0
Quote:
Originally Posted by AwesomeMachine View Post
Hi jpan,

Welcome to LinuxQuestions.org. This should work for you:

dd if=/dev/zero bs=32k | sed 's/\x00/\xff/g' | dd of=/dev/target

0x100h = 1 0000 0000b - 1b = 0xffh = 1111 1111b = 255d. So, 0xffh is hex notation for bitwise 1111 1111. In the example above, dd reads 32k of zeroes (0x00h) from /dev/zero. Sed replaces every 32k of hex zeroes with 32k of hex ff.

This would not work without the \x symbols in the sed perl expression, because without them sed would replace ascii 00 (0x3030h) with ascii ff (0x6666h). Sed would not find ascii 00 in the dd bit stream, so dd would pipe through sed unchanged.

-Awesome

Amazing. Thank you!!
 
Old 06-23-2010, 04:57 PM   #634
hanu7man
LQ Newbie
 
Registered: May 2008
Posts: 1

Rep: Reputation: 1
Talking Sublime thread!

Gives us newbies hope for a bright future!

Hey, Rick Rolling is cool...
I think I'll give Electric Avenue a try...

I'm gonna rock down to...Electric Avenue!
 
1 members found this post helpful.
Old 06-25-2010, 08:30 AM   #635
jpantano88
LQ Newbie
 
Registered: Jun 2010
Posts: 6

Rep: Reputation: 0
So I have another question for you guys. If I do something like
Code:
dd if=/dev/sdb bs=4k | hexdump -C > logfile
Then I can see the raw data that is on that drive.
I understand that this ignores and partitioning or file system allocations or anything else that might be on the disk (such is the power of dd).

The hex dump looks something like:
Code:
12345678   00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00 |................|
Now, my question is this:
Is this data related to the physical space on the drive, or is it the logical address seen by the OS?
In other words, does the address above (12345678) correspond to a physical media such as an individual notch on a hard drive or a single transistor on a SSD... or does it represent data as interpreted by the LBA of the drive?
Is there any way to check this?

I hope I'm being concise. Thanks again for the help.
 
Old 06-28-2010, 05:33 AM   #636
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
The number on the far left is the hex offset of the leftmost byte in that line. The hex offset is absolute, in relation to the starting point. If you start at the beginning of a physical disk, such as from, /dev/sdb, the offset is the exact number of bytes from the beginning of the disk, starting at zero.

Each sector of disk is 512 bytes, or 0x200h bytes. The beginning offset of a sector always ends with a zero, and the last byte of each sector ends with 0xfh. It's the same with each line in hexdump. To convert to decimal offset, you use a calculator with logical functions.

Since dd uses device files to read and write physical media, it is using information from the Linux kernel, not directly reading the physical hardware. The kernel handles system bios information for the physical drives, and programs use the kernel APIs to communicate with hardware. One notable exception is hdparm, which will communicate directly with the drive in certain instances.
 
Old 06-28-2010, 07:46 AM   #637
jpantano88
LQ Newbie
 
Registered: Jun 2010
Posts: 6

Rep: Reputation: 0
Talking

Quote:
Originally Posted by AwesomeMachine View Post
Since dd uses device files to read and write physical media, it is using information from the Linux kernel, not directly reading the physical hardware. The kernel handles system bios information for the physical drives, and programs use the kernel APIs to communicate with hardware. One notable exception is hdparm, which will communicate directly with the drive in certain instances.
Thanks again!
 
Old 07-05-2010, 09:21 AM   #638
RichTJ99
LQ Newbie
 
Registered: Jul 2010
Posts: 5

Rep: Reputation: 0
Hi,

I am looking to copy everything on sda to sdb1 using the bzip2 command & DD. I have sdb1 mounted properly.

What i type is:

Code:
dd if=/dev/sda | bzip2 -9 >/media/sdb1/disk_image.img.bz2
What I get is:

Code:
Invalid command line: Not enough files given. Aborting...
When I do this I do get a file created on sdb1 however, its just an empty file.

If I replace bzip2 with gzip, it seems to be working.

I was told that bzip2 -9 has better compression than gzip -9. The drive is a 2TB drive with mostly empty space (which I need backed up as a full image as well).

Is there a way to get bzip2 working? I am using the latest http://partedmagic.com/ boot CD.

Thank you very much for your help,
Rich
 
Old 07-23-2010, 07:10 PM   #639
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Welcome to LinuxQuestions.Org

You need spaces on either side of >, like this:

dd if=/dev/sda | bzip2 -9space>space/media/sdb1/disk_image.img.bz2 ,

but don't actually put the word, 'space'. Just use the space bar.

If that doesn't work, there's something wrong with the CD. I also place spaces on either side of a pipe.

Last edited by AwesomeMachine; 07-23-2010 at 07:12 PM.
 
Old 07-30-2010, 05:10 AM   #640
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 231Reputation: 231Reputation: 231
I seem to only be posting corrections, sorry. ...

From the bash man page (search for "metacharacter"):

Quote:
metacharacter
A character that, when unquoted, separates words. One of the following:
| & ; ( ) < > space tab
This means that the '>' separates words on the CLI the same as a space does, & therefore putting spaces around a '>' is NOT necessary for the command to work.

Furthermore, that is also my personal experience.

Anyone who wants to can test this by comparing the results of the following 2 commands:
Code:
>space_test0
> space_test1
Each will create an empty file in the current directory.

I DO generally put a space before '>' for clarity, especially if it's in a code snippet that I'm going to post. (See my post #361 above.)
 
Old 07-30-2010, 11:06 AM   #641
Sumguy
Member
 
Registered: Jul 2010
Location: Rural Kentucky, USA.
Distribution: Slackware; Crunchbang.
Posts: 403
Blog Entries: 2

Rep: Reputation: 97
I'm a newbie...but this info will come in handy one day, I'm sure.
 
Old 08-03-2010, 05:57 AM   #642
theterran
LQ Newbie
 
Registered: Aug 2008
Posts: 3

Rep: Reputation: 0
dd has saved my bacon many times. I use it most frequently to make a backup of a USB flash drive for embedded firewalls running pfSense. First thing to do after building a new firewall is shut down, pull the flash drive, and run dd to create an image onto a second flash drive. If (when!) the first one fails, just pop in the second one and boot.

Thanks for the great reference!
 
Old 08-03-2010, 08:09 AM   #643
pgroover
Member
 
Registered: Sep 2005
Location: Colorado
Distribution: Ubuntu
Posts: 55

Rep: Reputation: 16
I've read this numerous times and I have to say that it's always a life saver, thanks!
 
Old 08-03-2010, 02:01 PM   #644
lupusarcanus
Senior Member
 
Registered: Mar 2009
Location: USA
Distribution: Arch
Posts: 1,022
Blog Entries: 19

Rep: Reputation: 146Reputation: 146
This should be a sticky... immensely useful.

Thanks AwesomeMachine!
 
Old 08-24-2010, 01:27 AM   #645
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,674

Rep: Reputation: Disabled
I'd have guessed(!) either
Code:
dd if=/dev/sda | "bzip2 -9" > /media/sdb1/disk_image.img.bz2
or
Code:
dd if=/dev/sda | 'bzip2 -9' > /media/sdb1/disk_image.img.bz2
Can't check it right now, though.
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 04:22 AM
The best way to learn? iz3r Programming 7 02-06-2005 11:00 PM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 03:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 01:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 01:02 AM


All times are GMT -5. The time now is 04:17 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration