LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-25-2004, 07:33 AM   #1
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE Tumbleweed-KDE, Mint 17.3+18.3+19.0, MX-17
Posts: 4,198

Rep: Reputation: Disabled
How "low-level" is the dd-command? (Backups)


I read man and info and searched here but could not find definite answers.

Reason for asking is this:

/dev/hda2 is WinNT with NTFS. Can dd read any and all bits from that NTFS-partition for a complete backup? (It's been absolutely dastardly to set up, don't want to have to redo it when or if necessary).

I want to do

Code:
#backup
dd if=/dev/hda2 | bzip2 -zk > /backup/file/of/NT

#restore
bzip2 -zkdfc /backup/file/of/NT | dd of=/dev/hda2
Will each and every bit of /dev/hda2 be back not only in the files, but also the partition table, system information of NTFS etc. so that I have a complete "new" and useable install on /dev/dha2?

PS.: I formatted but forgot to set /dev/hda2 on binary zeroes before installing, so bzip2 won't work with maximum achievable compression. Anybody know a tool unter windows (like "dd if=/dev/null of=/file/with /zeroes) to create a large file containing binary zeroes?

PPS.: Sometimes dd gives in a summary information like 12345+1 records written ... huh? what "+1" record (or sometimes "+0" at that) ... any ideas?

Last edited by JZL240I-U; 05-25-2004 at 07:47 AM.
 
Old 05-25-2004, 08:01 AM   #2
whansard
Senior Member
 
Registered: Dec 2002
Location: Mosquitoville
Distribution: RH 6.2, Gen2, Knoppix,arch, bodhi, studio, suse, mint
Posts: 3,240

Rep: Reputation: 60
the +1 means part of a record.
dd will only get what you give it access to. if you tell it to get hda2, it will get all and only hda2. the partition table is at the beginning of the disk, not in the partition itself, except in logical drives, sort of. and you would have to extract it to a partition that was exactly the same size or bigger, or it wouldn't work. only bootloaders within the partition itself would be restored.

you can use dd under windows with cygwin, and the command is
dd if=/dev/zero of=/filename

a little cleaner

dd if=/dev/hda2 | bzip2 > /filename.dd.bz2

use split if you need to if anything gives you grief from over > 2 gig files

dd if=/dev/hda2 |bzip2 |split --bytes 2000m - /1/hda2.dd.bz2-

bzip2 -cd /filename.dd.bz2 > /dev/hda2

Last edited by whansard; 05-25-2004 at 08:12 AM.
 
Old 05-25-2004, 08:10 AM   #3
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE Tumbleweed-KDE, Mint 17.3+18.3+19.0, MX-17
Posts: 4,198

Original Poster
Rep: Reputation: Disabled
Quote:
Originally posted by whansard
the +1 means part of a record.
Ah thank you.

Quote:
Originally posted by whansard
dd will only get what you give it access to. if you tell it to get hda2, it will get all and only hda2. the partition table is at the beginning of the disk, not in the partition itself, except in logical drives, sort of.
Even though the data reside on NTFS?

Oh sh*t! No writing to NTFS. My whole scheme down the drain ...
Now I'll have to use partimage, and that's so inartistic .

Quote:
Originally posted by whansard
and you would have to extract it to a partition that was exactly the same size or bigger, or it wouldn't work.
That's why I would use bzip2.

Quote:
Originally posted by whansard
only bootloaders within the partition itself would be restored.
Good point. Boot loader and boot.ini file to be added to backup.

Quote:
Originally posted by whansard
you can use dd under windows with cygwin, and the command is
dd if=/dev/zero of=/filename
I'll have to look into cygwin, then.

Thanks whansard. Anybody else? Please?
 
Old 05-25-2004, 08:18 AM   #4
whansard
Senior Member
 
Registered: Dec 2002
Location: Mosquitoville
Distribution: RH 6.2, Gen2, Knoppix,arch, bodhi, studio, suse, mint
Posts: 3,240

Rep: Reputation: 60
quote:Originally posted by whansard
dd will only get what you give it access to. if you tell it to get hda2, it will get all and only hda2. the partition table is at the beginning of the disk, not in the partition itself, except in logical drives, sort of.

Even though the data reside on NTFS?

no, no,
the filesystem information is backed up by dd also, just not the partition table. no format or mount. just write to a partition that's big enough to hold it all. when you access a partition with dd, it doesn't care what the filesystem is. it just copies bytes. that includes everything there. you can copy any filesystem with it, as long as it's the same size place you restore it to. and the windows boot loader inside the partition requires that you restore it to the same numbered partition.
 
Old 05-25-2004, 08:27 AM   #5
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE Tumbleweed-KDE, Mint 17.3+18.3+19.0, MX-17
Posts: 4,198

Original Poster
Rep: Reputation: Disabled
Okay, so I'd use
Code:
sfdisk -d /dev/hda2 > /backup/file/of/NT-partition-table
(learned that just today ). And for the rest, bzip2 compresses the partition contents after all, so size is not so bad a restriction. Lastly, I'd restore to the original partition with the original size of course.

And /devhda2 is a primary partition.

And the boot loader resides on /dev/hda1 (C: ).

[Edit]
Um, come to think of it, using dd if=/dev/hda2 really should include the partition table in its input, shouldn't it?!?

Last edited by JZL240I-U; 05-25-2004 at 08:34 AM.
 
Old 05-25-2004, 03:04 PM   #6
whansard
Senior Member
 
Registered: Dec 2002
Location: Mosquitoville
Distribution: RH 6.2, Gen2, Knoppix,arch, bodhi, studio, suse, mint
Posts: 3,240

Rep: Reputation: 60
you could get the partition table with dd if=/dev/hda of=/table.dat bs=512 count=1

say your hda2 goes from cylinder 2000-4000
dd if=/dev/hda2 of=backup, gets EVERYTHING from cylinder 2000-4000, nothing else. the partition table is in the first cylinder, first sector of the drive. that's before hda1 starts.
 
Old 05-27-2004, 02:54 AM   #7
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE Tumbleweed-KDE, Mint 17.3+18.3+19.0, MX-17
Posts: 4,198

Original Poster
Rep: Reputation: Disabled
As I understand the disk layout, dd if=/dev/hda of=/table.dat bs=512 count=1 should give me the MBR (Master Boot Record). Partition table should be in the next record(s?), shouldn't it?

Now, to the read-only state of NTFS partitions, what would happen should I re-format /dev/hda2 as ext2 and then restore with dd my old WinNT backup, originally from NTFS. Would that change the file system type too?

[edit]
Is the mount status relevant at all with the dd command? Or can I write/output to any device using dd without prior mounting?

Last edited by JZL240I-U; 05-27-2004 at 07:02 AM.
 
Old 05-27-2004, 09:12 AM   #8
whansard
Senior Member
 
Registered: Dec 2002
Location: Mosquitoville
Distribution: RH 6.2, Gen2, Knoppix,arch, bodhi, studio, suse, mint
Posts: 3,240

Rep: Reputation: 60
the partition table is part of the master boot record. i actually have a 180 gig hard drive that i keep 2 full uncompressed backups of my 80 gig hard drive on. i use 2 different partition tables for that and i restore them that way. dd or cat, it doesn't matter. like cat table1>/dev/hde puts the first partition table on that drive.

http://www.boot-us.com/gloss04.htm

yes, the dd restore would put hda2 back as ntfs. i have my copies of solaris 8 and freebsd and netbsd and openbsd backed up just that way. they all exist on my hard drive from cylinder 261-999. ( or they did anyway. not right now.) and i just dd the backup to a partition existing there, and it's back to any of those OS's with their filesystems intact.

don't forget you could actually mount the backup file as a loopback filesystem. if you mounted something the kernel could write to ok, like ext2 or reiserfs, then you could write to the big file and make changes in it with it mounted.
mount -o loop /backup.file /mountpoint
cd /mountpoint
ls
as long as the filesystem isn't compressed anyway, i think. i'll show you.

/# fdisk -l
Disk /dev/hda: 81.9 GB, 81964302336 bytes
255 heads, 63 sectors/track, 9964 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 * 1 260 2088418+ 6 FAT16
/dev/hda2 261 1500 9960300 c W95 FAT32 (LBA)
/dev/hda4 1501 9964 67987080 85 Linux extended
/dev/hda5 1501 3500 16064968+ 83 Linux
/dev/hda6 3501 3700 1606468+ 83 Linux
/dev/hda7 3701 4000 2409718+ 83 Linux
/dev/hda8 4001 9964 47905798+ 83 Linux
/# dd if=dev/hda6 of=/hda6.file bs=1M
1568+1 records in1568+1 records out
/# mount -o loop hda6.file /3
/# cd /3
/3# ls
total 326
4 . 4 Win4Lin 4 floppy 4 nap 4 usr
2 .. 4 bin 4 hda1 4 nuke 4 util
4 1 0 bmp 4 hda2 4 opt 4 var
4 10 68 bookmark.htm 4 hda3 4 proc 4 varia
4 2 4 boot 4 home 4 root 4 will
16 2.2.14 4 burnup 4 jpg 4 sbin 4 win1
16 2.2.16 4 cdrom 4 lib 4 scdrom 4 work
12 2.4 32 dev 4 linux 4 tangeman 4 xc
4 21 4 dos 4 list 4 test
4 22 4 download 4 lost+found 4 test.txt
4 3 4 etc 4 mnt 4 tmp
/3# df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/hda5 16064472 9683024 6381448 61% /
none 452364 0 452364 0% /dev/shm
/dev/hda1 2088128 1228192 859936 59% /hda1
/dev/hda2 9950560 1167216 8783344 12% /hda2
/hda6.file 1581200 1322232 178648 89% /3

/3# cd /
/# umount /3
/# fsck /hda6.file
fsck 1.33 (21-Apr-2003)
e2fsck 1.33 (21-Apr-2003)
/hda6.file: clean, 62957/200928 files, 336875/401617 blocks
/#

crazy stuff huh.



you do not want to have a filesystem mounted when you are overwriting it with dd. you can mess it up, and will likely not be able to umount it either. you don't want it mounted when you are backing up either.

you can actually use a disk editor, like norton disk editor, if you had to for this type of stuff. it's just harder since you don't have the linux devices to write to, which make this stuff easy. you would have to figure out the start and ending sectors for that.

Last edited by whansard; 05-27-2004 at 09:22 AM.
 
Old 05-27-2004, 09:51 AM   #9
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE Tumbleweed-KDE, Mint 17.3+18.3+19.0, MX-17
Posts: 4,198

Original Poster
Rep: Reputation: Disabled
Quote:
Originally posted by whansard
the partition table is part of the master boot record.
Didn't know that. I thought the MBR just starts a known boot loader. What happens with a big partition table causing an overflow of the 512 bytes of the MBR? Or is there a limit to the size of the partition table? I'll read the link you provided to find the answers there... thanks by the way.

Quote:
Originally posted by whansard
.... i use 2 different partition tables for that and i restore them that way.
Huh? Wouldn't that set your system to wobbling -- I mean, the partition table better reflects the physical layout of your drive(s) or you can't successfully mount...

Quote:
Originally posted by whansard
.... i have my copies of solaris 8 and freebsd and netbsd and openbsd backed up just that way. they all exist on my hard drive from cylinder 261-999. ( or they did anyway. not right now.) and i just dd the backup to a partition existing there, and it's back to any of those OS's with their filesystems intact.
Slick trick

Quote:
Originally posted by whansard
.... don't forget you could actually mount the backup file as a loopback filesystem.
Won't help with NTFS, though...

Quote:
Originally posted by whansard
.... i'll show you.
Thanks again crazy stuff indeed.

Quote:
Originally posted by whansard
.... you do not want to have a filesystem mounted when you are overwriting it with dd. you can mess it up, and will likely not be able to umount it either. you don't want it mounted when you are backing up either.
Oh. I'll keep that in mind.

Thank you for an excellent and helpful discussion.

Last edited by JZL240I-U; 05-27-2004 at 09:52 AM.
 
Old 05-27-2004, 10:11 AM   #10
whansard
Senior Member
 
Registered: Dec 2002
Location: Mosquitoville
Distribution: RH 6.2, Gen2, Knoppix,arch, bodhi, studio, suse, mint
Posts: 3,240

Rep: Reputation: 60
it could help with ntfs. you could mount it read-only and copy all the files out of it if you had to. you could make a fat32 partition, mount the ntfs image as read-only, and copy all the files from the image to the new partition.

yes, there is a limit to the size of the partition table. that link explains it well, under partition table format i think.
 
Old 05-28-2004, 05:13 AM   #11
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE Tumbleweed-KDE, Mint 17.3+18.3+19.0, MX-17
Posts: 4,198

Original Poster
Rep: Reputation: Disabled
To sum it up: dd definitely is "low level", enough so that it doesn't even need a device to be mounted to be able to write to it. It doesn't care what file system exists for read and write operations. (It can read / write to mounted devices / file systems though, using a file name for the "if=" or "of=" parameter(s).) I conclude from this, that dd can get direct access to the devices with the blessings of the kernel.

Powerful and dangerous -- and capable of doing exactly what I want it to do.

Thanks for a most instructive discussion, whansard .
 
Old 05-28-2004, 11:30 AM   #12
whansard
Senior Member
 
Registered: Dec 2002
Location: Mosquitoville
Distribution: RH 6.2, Gen2, Knoppix,arch, bodhi, studio, suse, mint
Posts: 3,240

Rep: Reputation: 60
yeah. it's really just a data reader-copier-manipulator. it's the linux devices that let you do the cool stuff with it. you can even copy a hard drive with "cat" . cat /dev/hda > /dev/hdb.

also, i like to be able to watch dd's progress, so i edited the source code, and added a counter, and i changed the default block size to 1 meg, so i could watch the meg's copied.
it was only a few lines of code, and a quick compile. i named that version ddd, and i have the standard and my version both. sometimes you can be copying stuff, and you need to know how long it's going to take, or how fast it's going. maybe dma mode got turned off or something, but you can't tell, because dd doesn't tell you anything until it's done.
this is the area in dd.c i changed. about line 475. the part commented wills hacks.
i can't remember exactly what i changed, but i found where dd was keeping track of the number of blocks copied, and outputted that variable to the screen.

if (nwritten != output_blocksize)
{
error (0, errno, _("writing to %s"), quote (output_file));
if (nwritten > 0)
w_partial++;
quit (1);
}
else
w_full++;
fprintf (stderr, _("%s records out\r"), /* wills hacks */
human_readable (w_full, buf[0], 1, 1));
oc = 0;

Last edited by whansard; 05-28-2004 at 11:39 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell Script: Find "Word" Run "Command" granatica Linux - Software 5 07-25-2007 08:42 AM
"stalled" problem in " scp" command gentleshark Linux - Software 1 08-30-2005 03:08 AM
"stalled" problem in " scp" command gentleshark Linux - Networking 1 08-29-2005 01:41 PM
Tiny Sofa 2.0 - I thought "halt", "reboot" were only root command ?? sorcerer Linux - Distributions 1 08-21-2004 04:28 PM
"segmentation error" when issuing "useradd" command through terminal with RH 9.0 kaihuang Linux - General 0 10-22-2003 12:47 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration