LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 03-22-2006, 03:35 AM   #91
dement
LQ Newbie
 
Registered: Mar 2005
Distribution: SUSE 10.1
Posts: 8

Rep: Reputation: 0

i must say i found this thread quite useful
but i have a problem as i'm trying to copy an image of a disk to a fat partition.. which, as you may know, has a 4gb file size limitation. and the partition that i want imaged is sth like 17,9gb big.. although only 4,3 is actual data.

so is there a way to write the image to, let's say... a few different files?
or am i forced to use a fs that has no such limitations (at least not that low )?
 
Old 03-22-2006, 11:05 PM   #92
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
Smile

Quote:
Originally Posted by dement
i must say i found this thread quite useful
but i have a problem as i'm trying to copy an image of a disk to a fat partition.. which, as you may know, has a 4gb file size limitation. and the partition that i want imaged is sth like 17,9gb big.. although only 4,3 is actual data.

so is there a way to write the image to, let's say... a few different files?
or am i forced to use a fs that has no such limitations (at least not that low )?
Dd will copy the format with the data. I'm not sure if dd will change the partition type word, which is 2 bytes in each partition table entry, if you copy to a partition of a different file system type. So, if you copy an NTFS partition to a FAT volume you'll wind up with a NTFS format and a type 06 partition, probably. These don't match, and the file system won't end up FAT, but the partition table will probably say it is FAT.

See this link for partition types:

http://www.win.tue.nl/~aeb/partition...n_types-1.html
 
Old 03-23-2006, 01:15 AM   #93
fdhin
LQ Newbie
 
Registered: Mar 2006
Location: Europe
Posts: 5

Rep: Reputation: 0
To make the backup, gzip it and split it in 4GB files:
dd if=/dev/hda1 | gzip -c | split -b 4096m - /mnt/hdc1/backup.gz

To restore this backup:
cat /mnt/hdc1/backup.gz.* | gzip -dc | dd of=/dev/hda1
 
Old 03-23-2006, 04:06 PM   #94
nerdful1
LQ Newbie
 
Registered: Jun 2005
Location: greater Portland Maine
Posts: 13

Rep: Reputation: 0
FDHIN, or anyone, I noticed you have both /mnt/<drive> and /dev/<anotherdrive>. This was confusing me a lot when trying to run DD on a live Knoppix 4.02 live cd.
I'm working with a terminal window open to the DDing drives on my live cd booted from external usb/dvd.

I end up right clicking the shortcuts to mount/unmount or just changing mnt to dev in window, vice versa, etc, and eventually the drives see each other.

Is there a thread explaining mounting, unmounting, permissions etc? In the case of using DD?

And Awesome, thanks for the howto wish I saw it weeks ago.

Last edited by nerdful1; 03-23-2006 at 04:28 PM.
 
Old 03-24-2006, 12:14 AM   #95
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
Angry Not Correct

Quote:
Originally Posted by fdhin
To make the backup, gzip it and split it in 4GB files:
dd if=/dev/hda1 | gzip -c | split -b 4096m - /mnt/hdc1/backup.gz

To restore this backup:
cat /mnt/hdc1/backup.gz.* | gzip -dc | dd of=/dev/hda1
I don't know what this post pertains to, but it is wrong. The way it is written, dd would write to the screen, and even if it did write to a drive, it still won't work right. You cannot use the "cat" command for anything but text. It doesn't work with bitstreams. Fortunately, nothing bad will happen even if you do enter these commands. The first one will fail. So, the second one has to fail.
 
Old 03-24-2006, 12:22 AM   #96
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
Smile Mount, umount

Quote:
Originally Posted by nerdful1
FDHIN, or anyone, I noticed you have both /mnt/<drive> and /dev/<anotherdrive>. This was confusing me a lot when trying to run DD on a live Knoppix 4.02 live cd.
I'm working with a terminal window open to the DDing drives on my live cd booted from external usb/dvd.

I end up right clicking the shortcuts to mount/unmount or just changing mnt to dev in window, vice versa, etc, and eventually the drives see each other.

Is there a thread explaining mounting, unmounting, permissions etc? In the case of using DD?

And Awesome, thanks for the howto wish I saw it weeks ago.
Dd uses device files. It doesn't matter if they are mounted or not. You don't have to worry about mount with dd. Dd will make a bitstream duplicate regardless of whether or not the file system is mounted. I usually do not mount a file system when using dd. If it happens to be mounted already, so be it. But I don't intentionally mount a device's file system in the linux file tree just to use dd.

I have a question for you. What are you doing with dd, and a Knoppix CD in a usb drive. You can't do an installation to a hard drive that way. You cannot dd the CD to the hard drive and have it be a Knoppix hard drive after that. There is a way to install knoppix to a hard drive, but dd isn't it.

Your welcome for the tutorial.
 
Old 03-24-2006, 08:12 PM   #97
nerdful1
LQ Newbie
 
Registered: Jun 2005
Location: greater Portland Maine
Posts: 13

Rep: Reputation: 0
1. Interesting about the dev/mnt info looking at my notes I am sure I used /dev/somedriveorfile to dd to /dev/anotherdriveorfile, but would only work when I used the combination of mnt and dev. Anyway I am learning. I'm use dmesg or just seeing which item lists its files, then I go from there.

2. Not installing Knoppix, just using a livecd do do the drive xfer operations, mark them as read only, permissions, mount umount or not. I'll learn the cli for all this eventually, but I am also able to check here and elswhere on the web without running to my other computers.

3. I am backing up an irreplacable (softwarewise) seagate drive, 8.6gig from 1999 in my HDTV broadcast simulator. This generates an off the air signal from large HDTV bitstream files to attach to an HDTV tuner/tv. It uses w98. Not even SE I think. I already cloned to a spare drive, it works ok, but I want to dd everything to cdr/dvd including boot info for a permanent off site backup.
After mastering this, I want to use DD fluently to back up to safe media, xp, mac, old dos, etc. Also my HDTV off air for free recordings are huge, 20-30gigs for 1 olympic broadcast day in 1080i for example. So need to learn splitting them to files for xfer to DVD, just split .nuv data files... DVD for tv burning is another matter that I have solutions for.

I am sick of ghost, Acronis and similar costing me a lot so far, and peeing their marks all over my drives. I have pre Norton Ghost also that was the bees knees for years, but it is limited in todays world. Acronis would not do my fedora core 3 mythtv system either. So its Linux to stay. I like cli anyway, being a dos veteren.
Also all this copyright and DRM b.s. is so fragile to back up or transfer around legally on my own stuff.
 
Old 03-24-2006, 08:24 PM   #98
nerdful1
LQ Newbie
 
Registered: Jun 2005
Location: greater Portland Maine
Posts: 13

Rep: Reputation: 0
Awesome, just noticed your reply to fdhin regarding cat command...This is what I have been using to put split files back to my drives. They are not text obviously, so what do I use to put my split files of drive imaged by dd split gzip back together again? This may be answer to my problem.
 
Old 03-25-2006, 02:38 AM   #99
fdhin
LQ Newbie
 
Registered: Mar 2006
Location: Europe
Posts: 5

Rep: Reputation: 0
Quote:
Originally Posted by AwesomeMachine
I don't know what this post pertains to, but it is wrong. The way it is written, dd would write to the screen, and even if it did write to a drive, it still won't work right. You cannot use the "cat" command for anything but text. It doesn't work with bitstreams. Fortunately, nothing bad will happen even if you do enter these commands. The first one will fail. So, the second one has to fail.
Nope, you are plain wrong on all accounts. Now instead of talking about it, try this example out. It is proof of concept so to say.

Make two files with random bytes in them:
dd if=/dev/urandom of=~/sourcefile bs=200M count=1
dd if=/dev/urandom of=~/targetfile bs=200M count=1

Load the loopback kernel driver and associate each file with a loop device. They will from here on function as a block device:
modprobe loop
losetup /dev/loop0 ~/sourcefile
losetup /dev/loop1 ~/targetfile

make an ext2 filesystem on loop0 (which is the sourcefile):
mke2fs /dev/loop0

Make a mountpoint and mount the sourcefile:
mkdir /mnt/sourcemountpoint
mount -t ext2 /dev/loop0 /mnt/sourcemountpoint

Now copy 200MB of assorted crap to /mnt/sourcemountpoint. I copied some MP3 files.

Make the backup with the commands I suggested:
dd if=/dev/loop0 | gzip -c | split -b 1m - /tmp/testbackup.gz
You will have a lot of 1MB testbackup.gz files in /tmp now.

Now put these on loop1 device (the target file)
cat /tmp/testbackup.gz* | gzip -dc | dd of=/dev/loop1

Let's check now if it worked:
mkdir /mnt/targetmountpoint
mount -t ext2 /dev/loop1 /mnt/targetmountpoint

So.. I can see and play MP3 files now from the /mnt/targetmountpoint, so the commands work. Both of them.

However there was one little flaw in the first set of commands I gave. split does not put a . in the file, so it is cat /tmp/testbackup.gz* and not cat /tmp/testbackup.gz.*

Thus the commands will be:
dd if=/dev/blockdevice | gzip -c | split -b 1m - /tmp/backup.gz

And to restore this backup
cat /tmp/backup.gz* | gzip -dc | dd of=/dev/blockdevice


And finally, to clean up after this example if you try it:
umount /dev/loop0
umount /dev/loop1
losetup -d /dev/loop0
losetup -d /dev/loop1
rm /tmp/testbackup.gz*
rm ~/sourcefile ~/targetfile
rmdir /mnt/targetmountpoint/
rmdir /mnt/sourcemountpoint/

Last edited by fdhin; 03-25-2006 at 02:40 AM.
 
Old 03-25-2006, 04:46 AM   #100
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
When you add all the other stuff

Quote:
Originally Posted by fdhin
Nope, you are plain wrong on all accounts. Now instead of talking about it, try this example out. It is proof of concept so to say.

Make two files with random bytes in them:
dd if=/dev/urandom of=~/sourcefile bs=200M count=1
dd if=/dev/urandom of=~/targetfile bs=200M count=1

Load the loopback kernel driver and associate each file with a loop device. They will from here on function as a block device:
modprobe loop
losetup /dev/loop0 ~/sourcefile
losetup /dev/loop1 ~/targetfile

make an ext2 filesystem on loop0 (which is the sourcefile):
mke2fs /dev/loop0

Make a mountpoint and mount the sourcefile:
mkdir /mnt/sourcemountpoint
mount -t ext2 /dev/loop0 /mnt/sourcemountpoint

Now copy 200MB of assorted crap to /mnt/sourcemountpoint. I copied some MP3 files.

Make the backup with the commands I suggested:
dd if=/dev/loop0 | gzip -c | split -b 1m - /tmp/testbackup.gz
You will have a lot of 1MB testbackup.gz files in /tmp now.

Now put these on loop1 device (the target file)
cat /tmp/testbackup.gz* | gzip -dc | dd of=/dev/loop1

Let's check now if it worked:
mkdir /mnt/targetmountpoint
mount -t ext2 /dev/loop1 /mnt/targetmountpoint

So.. I can see and play MP3 files now from the /mnt/targetmountpoint, so the commands work. Both of them.

However there was one little flaw in the first set of commands I gave. split does not put a . in the file, so it is cat /tmp/testbackup.gz* and not cat /tmp/testbackup.gz.*

Thus the commands will be:
dd if=/dev/blockdevice | gzip -c | split -b 1m - /tmp/backup.gz

And to restore this backup
cat /tmp/backup.gz* | gzip -dc | dd of=/dev/blockdevice


And finally, to clean up after this example if you try it:
umount /dev/loop0
umount /dev/loop1
losetup -d /dev/loop0
losetup -d /dev/loop1
rm /tmp/testbackup.gz*
rm ~/sourcefile ~/targetfile
rmdir /mnt/targetmountpoint/
rmdir /mnt/sourcemountpoint/
This is quite different than you original post. If you go through all that trouble it will work, but why would anyone ever do that? You don't have to mount a file as loopback device to use dd. Dd will make a file you can later mount as loop device, and people have been doing backups with dd since the 1970's. If you are going to backup you are never going to do it on the same drive. Furthermore, in the other person's post, he wasn't trying to do a backup. If he was trying to do a backup he would have just made a file in any file system type. It wouldn't matter.

I know you think you are really special, and you are. But, read my post again and learn how to use dd. There are all sorts of weird, useless things you can do with anything. I'm trying to teach constructive methods to solve real world problems with dd. I compliment you on your astute ability. It is impressive that you figured out that command. But, it was totally impertinent to what you were replying to. You are needlessly confusing people in an otherwise quality thread. I don't mean to get down on you, but if you go find me some good info on dd, I'll edit the post and put in it what you turn me on to about dd. But, post a link. Do not post the whole web page in a reply. I will look at the link and edit the post. Then I will post a post thanking you. Would that be ok?
 
Old 03-25-2006, 10:37 AM   #101
fdhin
LQ Newbie
 
Registered: Mar 2006
Location: Europe
Posts: 5

Rep: Reputation: 0
What I posted above is adequate proof that this will work:

To make the backup, gzip it and split it in 4GB files:
dd if=/dev/hda1 | gzip -c | split -b 4096m - /mnt/hdc1/backup.gz

To restore this backup:
cat /mnt/hdc1/backup.gz* | gzip -dc | dd of=/dev/hda1

I posted the above as a method to prove it, without anyone having to risk any data in the proces.

I'll happily analyze it now, so that everyone understands why it works. So, the first command:
dd if=/dev/hda1 | gzip -c | split -b 4096m - /mnt/hdc1/backup.gz

AwesomeMachine writes "dd would write to the screen". Actually not so. dd will write to stdout which as default happens to be the screen. We can pipe the stdout somewhere else with the | character, so that the dd output is sent on to the gzip commands stdin instead.
So what happens here is that block for block dd is reading hda1, passing it on to gzip which is set to pipe the compressed blocks to stdout instead of a file with the -c switch. From gzip the compressed blocks are passed from its stdout to the stdin of the split command, again with the | character. Split does just what its name implies: It splits files up into smaller fragments. With split -b we determine the size of the chunks and set them to be 4096MB, and the second - (minus) tells split that it should get its input from stdin. If we did not put that then split would start looking for a file called /mnt/hdc1/backup.gz and split that, which is not what we want here.

If you some day desire to restore this backup on hda1, then you can use this command:
cat /mnt/hdc1/backup.gz* | gzip -dc | dd of=/dev/hda1

This command takes the backup.gz files in the correct order and pass them on to gzip. The gzip command is set to decompress with -d and again set to pipe to stdout. The stdout is piped on to the dd command, which then uses hda1 as outfile and writes the contents.

AwesomeMachine writes "You cannot use the "cat" command for anything but text. It doesn't work with bitstreams." - In fact cat can do that. Try to cat any character device in /dev/, and cat also works for the above.


Anyway, I dont want to discuss anymore whether this works or not. I use it, and it works.
 
Old 03-25-2006, 04:34 PM   #102
nerdful1
LQ Newbie
 
Registered: Jun 2005
Location: greater Portland Maine
Posts: 13

Rep: Reputation: 0
So awesome, before I confuse myself with the cat process discussed above, I would like the equivalent of using cat but using dd to recombine split files. Do I do dd if=splitfile.gz* of=recombinedfilename to that effect?
I will go back and read your tutorial now and if I find answer, I'll come back here soon and edit/delete this post.

Last edited by nerdful1; 03-25-2006 at 04:36 PM.
 
Old 03-25-2006, 06:04 PM   #103
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,053

Rep: Reputation: 484Reputation: 484Reputation: 484Reputation: 484Reputation: 484
"If your only tool is a hammer, everything looks like a nail"

Yes, eventually you CAN do anything with DD. I guess that using NETCAT and DD in a smart way, you can even share files over the network. BUT... NFS was invented for that.

DD is an awesome tool, and awesomemachine did the best job I ever saw to describe it.

However, DD is not the only tool in Linux.

I am still convinced that if you do want to copy N bits from here to there in the broadest sense then use DD. If it is more complicated, you should look for a more appropriate solution. If you want to split into multiple volumes, compress, and copy to a different file system on a different sized medium, it is insane to insist to use DD. Or you do it just for fun, but then it is no use to confuse other people in this thread.

So, if you want to split into multiple volumes, compress, and copy to a different file system on a different sized medium, use the appropriate tool. TAR or STAR is one of them.

jlinkels

Program names in CAPS are actually lowercase names.

Last edited by jlinkels; 03-25-2006 at 06:06 PM.
 
Old 03-25-2006, 07:26 PM   #104
nerdful1
LQ Newbie
 
Registered: Jun 2005
Location: greater Portland Maine
Posts: 13

Rep: Reputation: 0
So can you answer my question I asked at 5:34pm today? how to use DD to recombine files split using dd and pipe to split. Not using cat. I am not looking for shortcuts. I want to learn all the proper dd stuff. thanks

Last edited by nerdful1; 03-26-2006 at 07:28 PM.
 
Old 03-27-2006, 06:54 PM   #105
nerdful1
LQ Newbie
 
Registered: Jun 2005
Location: greater Portland Maine
Posts: 13

Rep: Reputation: 0
Is there anyone out there?
repeat:So can you answer my question I asked at 5:34pm 2 days ago? how to use DD to recombine files split using dd and pipe to split. Not using cat. I am not looking for shortcuts. I want to learn all the proper dd stuff. thanks
clip from the past:So can you answer my question I asked at 5:34pm today? how to use DD to recombine files split using dd and pipe to split. Not using cat. I am not looking for shortcuts. I want to learn all the proper dd stuff. thanks

Last edited by nerdful1; 03-27-2006 at 06:56 PM.
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 04:22 AM
The best way to learn? iz3r Programming 7 02-06-2005 11:00 PM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 03:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 01:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 01:02 AM


All times are GMT -5. The time now is 08:56 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration