LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 06-04-2008, 04:56 AM   #466
BT+1
LQ Newbie
 
Registered: Aug 2007
Posts: 11

Rep: Reputation: 0

Quote:
Originally Posted by JZL240I-U View Post
Out of curiosity: If you used a "bs=1M" parameter with dd, wouldn't that work?
The problem is that unless dd resizes the NTFS file system within the partition (as the True Image products do when necessary) by changing its superblock-type information and the content of its free-space bitmap the faithfully-reproduced old file system size will not match the new (changed, due to the different alignment) partition size. Partition managers (and for all I know the file system itself) think that this is an error (because when a partition is formatted the file system is sized to fill it entirely) - and I wouldn't want to bet my system on how they may decide to 'fix' such 'errors' (though my experience with Partition Magic suggests that it handles them reasonably - unlike the case where it itself copies a partition from a disk with one geometry to a disk with another, when it can really screw things up).

It slipped my mind earlier that Paragon Hard Disk Manager V8 and Paragon Partition Manager V8.5 may still have free versions available as well. In general, when any Windows systems are involved I prefer to use such commercial products for handling partitioning: some Windows software is kind of picky about things like having the order of any logical partitions on the disk the same as the order of their links in the extended-partition chain, having all logical partitions confined within the 'extended' partition, or having only a single 'extended' partition, while open-source utilities haven't always been as strict in such regards (and some, like the Ranish Partition Manager, do other innovative but non-standard things that can result in data loss if some conventional partition manager is then used to operate on the disk). Also, I have a bit more confidence that they'll handle any associated file system resizing correctly, though my impression is that some of the open-source options now available are pretty reliable in this area too, even for NTFS - and when copying/moving a partition they're usually able to move only the actual data within it, whereas using dd would presumably copy the free space as well (which could take quite a bit of extra time if the partition weren't very full).
 
Old 06-04-2008, 05:07 AM   #467
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,686

Rep: Reputation: Disabled
Quote:
Originally Posted by BT+1 View Post
The problem is that unless dd resizes the NTFS file system within the partition (as the True Image products do when necessary) by changing its superblock-type information and the content of its free-space bitmap the faithfully-reproduced old file system size will not match the new (changed, due to the different alignment) partition size. ...
But why is resizing necessary? In your original post you wrote that the problem arises from partition boundaries not aligned with the cylinders but along megabytes. Okay, so maybe it is not only the "bs=1MB" but also a "count=xy" parameter necessary with xy the actual size of the non-cylinder-aligned partition. Would that work or where am I missing your point? Is it due do changed writing of the output of "dd"?

Last edited by JZL240I-U; 06-04-2008 at 05:08 AM.
 
Old 06-04-2008, 02:20 PM   #468
BT+1
LQ Newbie
 
Registered: Aug 2007
Posts: 11

Rep: Reputation: 0
Quote:
Originally Posted by JZL240I-U View Post
But why is resizing necessary? In your original post you wrote that the problem arises from partition boundaries not aligned with the cylinders but along megabytes. Okay, so maybe it is not only the "bs=1MB" but also a "count=xy" parameter necessary with xy the actual size of the non-cylinder-aligned partition. Would that work or where am I missing your point? Is it due do changed writing of the output of "dd"?
The problem is that the partition changes size but the file system which resides in it does not (unless you modify its internals as the commercial products which I cited do, and while the open-source utilities g4u, Partimage, and SelfImage can do some of the same things the commercial products do they cannot resize NTFS - or any other - file systems within the partition, though DriveImage XML may be able to).

The resizing is necessary because other partitioning software (and possibly the native file system itself) will consider it an error if it's not done, and do God only knows what to try to 'fix' it. Whereas a product that was *designed* to resize file systems seems to me to have a better shot at doing it right (and indeed has done so in the many cases when I've asked it to).

- bill

Last edited by BT+1; 06-04-2008 at 02:31 PM. Reason: added SelfImage and DriveImage XML to examples
 
Old 06-05-2008, 07:15 AM   #469
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,686

Rep: Reputation: Disabled
Quote:
Originally Posted by BT+1 View Post
The problem is that the partition changes size but the file system which resides in it does not...
And that is precisely what I don't understand. Why should it do that when you tell "dd" to copy each and all 1MB sized chunks, which belong to the original partition? And that includes the beginning, too, which holds the partition data of the source partition? Both source and destination are NTFS and should be identical down to the last random bit pattern in unused parts...

I (think I) understand the problem with the necessity of same-sized file system and partition. What I don't get is the change in size of the partition during use of "dd". I could understand it, if it was "fdisk" being unable to create partitions not on cylinder boundaries, but this is "dd"...

Last edited by JZL240I-U; 06-05-2008 at 07:17 AM.
 
Old 06-05-2008, 01:38 PM   #470
BT+1
LQ Newbie
 
Registered: Aug 2007
Posts: 11

Rep: Reputation: 0
Quote:
Originally Posted by JZL240I-U View Post
And that is precisely what I don't understand. Why should it do that when you tell "dd" to copy each and all 1MB sized chunks, which belong to the original partition? And that includes the beginning, too, which holds the partition data of the source partition?
No, it doesn't: if it's a primary partition, the partition size data is in the MBR (which resides at the start of the disk, not the start of the partition); if it's a logical partition, the partition size data is in the PBR (which *precedes* the start of the partition).

And if dd actually *did* copy the partition size information (e.g., if you gave it disk offsets to work with rather than in-partition addresses) then the partition would wind up ending somewhere other than on a logical cylinder boundary on the destination disk, in which case you'd be back to the situation where other software would think that it was corrupted and try to fix it (not to mention the fact that its start address would also not match its physical address on the destination).

- bill
 
Old 06-06-2008, 02:23 AM   #471
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,686

Rep: Reputation: Disabled
Quote:
Originally Posted by BT+1 View Post
No, it doesn't: if it's a primary partition, the partition size data is in the MBR (which resides at the start of the disk, not the start of the partition); if it's a logical partition, the partition size data is in the PBR (which *precedes* the start of the partition).
Ahhh, so that's it. My bad, I wrongly assumed that the partition data were actually part of the partition -- thanks for clarifying that. Wikipedia says that this is a speciality of NTFS: http://en.wikipedia.org/wiki/Partition_Boot_Record so that's why it is new for me.

Quote:
Originally Posted by BT+1 View Post
And if dd actually *did* copy the partition size information (e.g., if you gave it disk offsets to work with rather than in-partition addresses) then the partition would wind up ending somewhere other than on a logical cylinder boundary on the destination disk, in which case you'd be back to the situation where other software would think that it was corrupted and try to fix it (not to mention the fact that its start address would also not match its physical address on the destination).

- bill
Well this is confusing again. If vista is creating its file system that way, why should it be confused if it gets a new partition built exactly like it does build them itself?
 
Old 06-06-2008, 05:09 AM   #472
BT+1
LQ Newbie
 
Registered: Aug 2007
Posts: 11

Rep: Reputation: 0
Quote:
Originally Posted by JZL240I-U View Post
Ahhh, so that's it. My bad, I wrongly assumed that the partition data were actually part of the partition -- thanks for clarifying that. Wikipedia says that this is a speciality of NTFS: http://en.wikipedia.org/wiki/Partition_Boot_Record so that's why it is new for me.
No: it has nothing to do with NTFS. The volume boot record (which resides *within* a partition at its start and contains the intermediate boot code specific to the operating system therein) is not the same as the MBR (which resides at the start of the disk outside any partition and contains both the primary partition layout information and initial boot code to transfer control to the active target partition) or the PBR (which *precedes* each logical partition within the extended partition and contains that partition's layout information, the link to the next logical partition, if any, and possibly - at least in the case of a logical partition at the very beginning of the extended partition - boot code for a boot manager such as GRUB to which the MBR may pass control for a secondary layer of generic boot management if that boot manager has not been installed in the MBR itself).

Quote:
Well this is confusing again. If vista is creating its file system that way, why should it be confused if it gets a new partition built exactly like it does build them itself?
This has nothing to do with the way Vista creates its file system: it's all about how Vista handles its partitions and how this differs from how the rest of the world has handled partitions for the past several decades. Vista may be smart enough not to screw up a disk on which partitions are laid out in the traditional manner, but if the disk began life with new-style partition alignment and was then (incompatibly) modified by an old-style partition manager I've read of cases where subsequently returning to Vista to create additional partitions destroyed the old-style partitions that had been created (especially within the extended partition); older existing software can't possibly understand how to work with Vista's newer partitioning strategies: it can only consider them corrupt, in which case it will hopefully at least ask before attempting to 'fix' them and otherwise refuse to perform additional partitioning operations.

Reportedly WinXP isn't as reluctant as it should be in this regard: it will perform additional partitioning operations on a Vista new-style-aligned disk (as just described above) and this too can result in entire lost partitions (especially within the extended partition). Partition Magic is more cautious and won't touch such a disk unless you allow it to 'fix' it. How other software behaves I couldn't say.

The bottom line is that you really don't need to understand *how* all this works: you just need to know that the problem exists and how to work around it if you want to mix Vista with other systems and/or other partitioning software. I hope I've provided enough information to help do that - though one thing I may not have mentioned is that *some* other systems may be able to be installed and function happily within new-style Vista partitions (though I remember reading that this is not always true for WinXP, even if you refrain from trying to use XP's Disk Management facilities to modify the partition layout on such a disk).

- bill
 
Old 06-06-2008, 05:14 AM   #473
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,686

Rep: Reputation: Disabled
Okay, thanks for the additional clarification, I think I got it now.
 
Old 06-10-2008, 03:01 AM   #474
lawrence_lee_lee
Member
 
Registered: May 2007
Posts: 141

Rep: Reputation: 16
Seems like I come a bit late, as all the above are in the year around 2004 and 2005, but it's now already 2008!

However, I still find this thread extremely useful! It nearly guides me from scratch to at least know how to use dd to perform basic operation! Wonderful!
 
Old 06-12-2008, 12:14 AM   #475
onebuck
Moderator
 
Registered: Jan 2005
Location: Midwest USA, Central Illinois
Distribution: SlackwareŽ
Posts: 11,441
Blog Entries: 4

Rep: Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503Reputation: 1503
Hi,

Not really, this thread is very helpful to a lot of people.
 
Old 06-12-2008, 03:22 AM   #476
lawrence_lee_lee
Member
 
Registered: May 2007
Posts: 141

Rep: Reputation: 16
I have some questions about the example use of dd comand by AwesomeMachine

1.) About the creation of RAMdrive.
1a.) In fact this is not related to the use of dd. When I created the ramdrive by:
Quote:
dd if=/dev/zero of=/dev/ram7 bs=... count=...
What if there are some important stuff stored in the memory location of /dev/ram07, this will flush them all away. Will this happen? Will a system hang result? I've tried this once myself but nothing catastrophic happened. However, I want to know a bit more.
1b.) In the above command, are there any restrictions on bs and count, specific for such use on the ram? To be general, I want to ask that is the choice of bs makes only a performance difference, or that it must be chosen to some values in some special cases like i/of=/dev/ram0(?) or i/of=/dev/sda(?) ?
1c.) How to release the memory back? I may be working on a 300 MB file using the ramdrive and definitely it would be bad if those lots of memory cannot be released after use.
1d.) Is the choice of ram0? not important? Is that whatever number will work?

2.) About copying the MBR or back up the root partition.
When doing so, say,
Quote:
dd if=/dev/sda2 of=root.image bs=1M
where sda2 is the partition containg my root partition. Is it much safer (or essential) to first boot into a live CD first and then perform the above? Is it possible to do the above when the sda2 (root partition) is mounted and system is running?

3.) About Rejuvenating a hard disk
Quote:
dd if=/dev/sda3 of=/dev/sda3
3a.) Will this destroy the filesystem originally on the drive sdb3? Is formatting required after this step?
3b.) Can this also be applied to flash drive?

4.) About making an encrypted drive
I have to provide a password in order to perform encryption. So, where is my password stored? (I suppose it must be stored somewhere, or how can the system, when a user want to decrypt the file, know whether the entered password is correct?) Is my stored password also encrypted to prevent it from being read?

5.) About the "conv=notrunc"
Is it used only when "of" is an existing file and the original file is larger in size than the output from dd? If "of" is not existing, "conv=notrunc" is meaningless?

6.) About "Overwriting all the free space on a partition"
Quote:
dd if=/dev/urandom of=/home/sam/fileconsumingallfreespace.file
AwesomeMachine suggested the above command to overwrite all the free space on a partition. However, how is that "file consuming all the free space of the partition" be created?


I know that's a bit too much questions, but please help me to know more about dd.

Thank you very very much for any answering!
 
Old 06-13-2008, 04:21 AM   #477
lawrence_lee_lee
Member
 
Registered: May 2007
Posts: 141

Rep: Reputation: 16
Can anyone answer my questions above?
 
Old 06-13-2008, 05:01 AM   #478
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,686

Rep: Reputation: Disabled
Quote:
Originally Posted by lawrence_lee_lee View Post
Can anyone answer my questions above?
Please be patient. Let me remind you that this site is operated by volunteers. AwesomeMachine usually looks in periodically and answers the amassed questions in a batch.

Quote:
Originally Posted by lawrence_lee_lee View Post
2.) About copying the MBR or back up the root partition.
When doing so, say, where sda2 is the partition containg my root partition. Is it much safer (or essential) to first boot into a live CD first and then perform the above? Is it possible to do the above when the sda2 (root partition) is mounted and system is running?
As I understand it: No. Your system will have some files opened and so you will get an inconsistent image which might have problems to start when you use it as a backup. Better use a live CD.


Quote:
Originally Posted by lawrence_lee_lee View Post
3.) About Rejuvenating a hard disk
Code:
dd if=/dev/sda3 of=/dev/sda3
3a.) Will this destroy the filesystem originally on the drive sdb3? Is formatting required after this step?
What now? sda3 <> sdb3 ... if you do such things in real life you'll end up with problems .

When you do a
Code:
dd if=/dev/sda3 of=/dev/sda3
you will just refresh the entire magnetic structure of this partition, nothing will be destroyed, all your data will stay in place and their longevity should be improved.

Quote:
Originally Posted by lawrence_lee_lee View Post
6.) About "Overwriting all the free space on a partition"
...
However, how is that "file consuming all the free space of the partition" be created?...
By "dd". It can write either disks, partitions or files.

As far as I can see, there remains one thing to be kept in mind: some file systems (like the ext family) reserve some space (default about 5% AFAIR) for emergency operations by root. That space remains unaccessible for normal users. So you'd have to blank your disk as root.

Last edited by JZL240I-U; 06-13-2008 at 05:05 AM.
 
Old 06-15-2008, 05:59 AM   #479
lawrence_lee_lee
Member
 
Registered: May 2007
Posts: 141

Rep: Reputation: 16
Thanks a lot! That already answered many of my doubts!

By the way, I am very sorry about my impatience. I didn't mean to be rude. If my words offended anyone in this community, I want to say "sorry" and please accept my apology.

English is not my mother tongue. I don't know how to express my apology clearer....
 
Old 06-19-2008, 09:56 AM   #480
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Post You want to know my secrets ...

Quote:
Originally Posted by lawrence_lee_lee View Post
I have some questions about the example use of dd comand by AwesomeMachine

1.) About the creation of RAMdrive.
1a.) In fact this is not related to the use of dd.
For the purposes of this thread, if dd is used to perform a task, dd is related to the task. Other definitions I hadn't been aware of may exist for, 'related to'.

Quote:
Originally Posted by lawrence_lee_lee View Post
When I created the ramdrive by:

What if there are some important stuff stored in the memory location of /dev/ram07, this will flush them all away. Will this happen? Will a system hang result? I've tried this once myself but nothing catastrophic happened. However, I want to know a bit more.
1b.) In the above command, are there any restrictions on bs and count, specific for such use on the ram? To be general, I want to ask that is the choice of bs makes only a performance difference, or that it must be chosen to some values in some special cases like i/of=/dev/ram0(?) or i/of=/dev/sda(?) ?
1c.) How to release the memory back? I may be working on a 300 MB file using the ramdrive and definitely it would be bad if those lots of memory cannot be released after use.
1d.) Is the choice of ram0? not important? Is that whatever number will work?
I admire your inquisitiveness. The maximum size of a ram disk is determined by values compiled into the kernel. You can't make a 300 MB ram drive. 16 MB would be typical maximum. If you try to fill the ram disk more than that, dd will report no space left on device.

The linux kernel is designed so that memory currently in use is not overwritten by other commands. If this were possible, it would be the operating system equivalent of anyone being able to move into anyone else's house, whenever they felt like it, by killing the current residents.

Dd only fills the ram disk. It doesn't matter how one counts zeroes. 16384000 always equals 16384000, no matter how one divides them up. The amount is always the same.

Tinkering with some file system parameters may imperceptibly improve performance of a ram drive. Usually people wild enough to use a ram drive are satisfied with rebooting, should they need that 16 MB of ram.

You can use any /dev/ram* device.
Quote:
Originally Posted by lawrence_lee_lee View Post
2.) About copying the MBR or back up the root partition.
When doing so, say,

where sda2 is the partition containg my root partition. Is it much safer (or essential) to first boot into a live CD first and then perform the above? Is it possible to do the above when the sda2 (root partition) is mounted and system is running?
Dd isn't fussy about whether or not a file system is mounted. If you feel better using a live CD, that's what's really important. In my experience, duplicating the root partition to an image file will work, as long as you don't tell dd to make the image file in the same partition you are imaging.

Quote:
Originally Posted by lawrence_lee_lee View Post
3.) About Rejuvenating a hard disk

3a.) Will this destroy the filesystem originally on the drive sdb3? Is formatting required after this step?
3b.) Can this also be applied to flash drive?
This sounds like a veiled insult. Operations that result in destruction of the file system cannot properly be referred to as rejuvenating. Flash drives wear as they are written many times. You will wear out the flash media using this method. Flash media does not need rejuvenating.
Quote:
Originally Posted by lawrence_lee_lee View Post

4.) About making an encrypted drive
I have to provide a password in order to perform encryption. So, where is my password stored? (I suppose it must be stored somewhere, or how can the system, when a user want to decrypt the file, know whether the entered password is correct?) Is my stored password also encrypted to prevent it from being read?
I offer this in all humility, for your benefit, and not to have readers think less of you. You are naive to the point that encryption would almost inevitably result in data loss for you.
Quote:
Originally Posted by lawrence_lee_lee View Post
5.) About the "conv=notrunc"
Is it used only when "of" is an existing file and the original file is larger in size than the output from dd? If "of" is not existing, "conv=notrunc" is meaningless?
So you want to have a go with the one on top of the hill. What I have written I have written. It is not for me to force a man, but only to Light the way to Truth.
Quote:
Originally Posted by lawrence_lee_lee View Post
6.) About "Overwriting all the free space on a partition"

AwesomeMachine suggested the above command to overwrite all the free space on a partition. However, how is that "file consuming all the free space of the partition" be created?
The authors of linux coreutils, Paul Rubin, David MacKenzie, and Stuart Kemp, and authors of other amazing nix type tools would tell you dd creates files specified on the command line, 'automagically'. Trust is the key. All things are pure to the pure. Suspicion is the device of a guilty mind.

He who is wise is he who knows nothing.
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 05:22 AM
The best way to learn? iz3r Programming 7 02-07-2005 12:00 AM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 04:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 02:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 02:02 AM


All times are GMT -5. The time now is 11:28 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration