conv=notrunc?
 

I've been following this thread for years (2006 or 2007?). Excellent job on dd and extremely useful (also interesting!)--Many Thanks!

My question is, in the command to zero-out a drive:
dd if=/dev/zero of=/dev/sdx bs=4096 conv=notrunc

Is the conv=notrunc necessary (in this application)? That is, can you just as well run it as
dd if=/dev/zero of=/dev/sdx bs=4096
without the notrunc option?

Discussion/experiment:

It seems that running it without conv=notrunc does the job of writing zeros to drive sdx.

On a 1 GB flash drive (/dev/sdc), I ran
dd if=/dev/sdc | hexdump -C | grep [^00]
and found tons of non-zero lines (finally had to Control+c to stop it).

Then I ran (without notrunc)
dd if=/dev/zero of=/dev/sdc
dd: writing to `/dev/sdc': No space left on device
2015233+0 records in
2015232+0 records out
1031798784 bytes (1.0 GB) copied, 379.062 s, 2.7 MB/s

Then looked for non-zero lines and found none:
dd if=/dev/sdc | hexdump -C | grep [^00]
2015232+0 records in
2015232+0 records out
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
3d800000
1031798784 bytes (1.0 GB) copied, 92.9287 s, 11.1 MB/s

And finally, I checked a random part of the flash drive and found only zeros:
To the screen:
dd if=/dev/sdc bs=4096 skip=100000 count=100 | hexdump -C
mike@mike-desktop:~$ sudo dd if=/dev/sdc bs=4096 skip=100000 count=100 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00064000
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.057579 s, 7.1 MB/s


-> In general, I'm a little unclear about conv=notrunc (or its opposite), even after reading (I think) everything I can google on dd. But, specifically, today, I'm concerned about its use/non-use in zero-ing out a drive. You will see different sources doing it differently, some using notrunc, and some not using it.

-- Thanks, Mike.

Notrunc
 

In some instances the result of zeroeing out a drive is the same whether or not one uses conv=notrunc. If the default block size of bs=512 is used, and the sector size of the disk is 512 bytes, you can just as well leave out notrunc.

I'll change the OP during my next update, because it seems conv=notrunc,sync is a better method for writing zeroes.

Many thanks for your response.


Come to think about it, it seems I do keep using the default bs=512 when doing these zero-fills (and my disk sector size is 512 B).
I have a feeling, though, that it would also work with, for example bs=4k; I'll try that soon on my example above.


However, it seems you are saying, that, notwithstanding, to be safe and to cover all cases, it would be best to run with conv=notrunc,sync (both options enabled), right?


The reason I got into this (which may look like a naive post above), was because of this statement that we see around, regarding notrunc:
conv=notrunc
Do not truncate the output file.
Tells dd not to abbreviate blocks of all zero value, or multiple adjacent blocks of zeros; if notrunc is omitted, dd will abbreviate ANY size block of zeros with five asterisks *****
[This may be paraphrased, I can't recall, but the essence is consistent with the various man pages.]



For example, I ran into this:
http://www.justlinux.com/forum/showthread.php?p=887950

Wipe your USB stick clean...
dd if=/dev/zero of=/dev/sdX bs=4k conv=notrunc

and the best part is what the author then says about the "why":

"Say you have 10,000 blank, adjacent sectors, in the input file. Without conv=notrunc, dd will write 5 asterisks where the first sector should have been written in the output file, and leave out the rest of the 9,999 sectors. With conv=notrunc, all 10,000 sectors will be reproduced in the target.
In our case, we need to keep pumping zeros into the USB stick till it runs out of space. Without the conv=notrunc option, all we'll push in is a string of 5 asterisks."


Maybe my case (i.e., where bs=512=the disk sector size) is the only safe case for omitting conv-notrunc. As I sort of said, I'm just confused, uncertain, about it. As you already said, the issue may be where that condition is not satisfied (i.e., where bs differs from the sector size and is other than 512).

(btw, elsewhere around the net, so it seems, it is not uncommon to see a basic formula for doing these zero-fills: dd if=/dev/zero of=/dev/sdx , and that's all! no other parameters specified)


Thank you again for your response.

This thread evolved ...
 

Yes. Even I am not sure EXACTLY how dd works. I have taken information deemed to be correct, without testing every feasible possibility. I err on the side of caution. If you read a little past the five asterisks part of the discussion thread, I stated that dd does not abbreviate adjacent zero blocks with five asterisks.

I think conv=sync is what you are looking for, because sync will pad the last block with zeroes, if the drive doesn't divide into a whole number of blocks. I.e. if the block size is bs=4k, and there is 2k left at the end of the disk, conv=sync will pad the remaining 2k with zeroes.

In total answer, I officially recommend conv=notrunc,sync for zeroing out drives; just for safety sake. You can't be too careful. And then checking for nonzero values with grep doesn't hurt.

It's good to know people are really using this thread.

-Awesome

Thanks for the tip regarding conv=sync (to pad partial end blocks with zeros).
And thanks for offering your "position"statement:
"In total answer, I officially recommend conv=notrunc,sync for zeroing out drives; just for safety sake. You can't be too careful. And then checking for nonzero values with grep doesn't hurt."
That's what I will do from now on.
(Starman makes a good point in his work: it is important to zero-out a drive but then it is very important to verify! Your tips on piping to grep are very helpful.)


"It's good to know people are really using this thread."

Oh, gosh, YES! Sometimes I feel that this has become a hobby for me. I mainly use dd to zero-out drives, flash drives, for cloning flash drives, that sort of thing, and do it all the time for "privacy cleanup." Your tip on http://www.softpanorama.org/Tools/dd.shtml is good; although a little quirky at times, the site does have good links. Somehow, I bounced around until I got to this:
http://smartmontools.sourceforge.net/badblockhowto.html. Although about fixing bad disk sectors, it is also good for showing a few neat examples of using dd, using offsets to get down to specific bad sectors, and so on, similar to what you have done using offsets to explore an area on disk.


Again, AwesomeMachine, many thanks for your thread and for your help here.
I appreciate it.

Just for fun and profit I like to write random data to an entire flash drive 3 times before I zero the drive to to make doubly sure no one can see my super spy files I put on there :P

Hope you guys don't mind me posting another test here. Maybe some of you will find this interesting and may want to run your own tests. This is a follow-up to some of the points above.

Test:

1 GB Kingston flash drive. Write random numbers to it.
Zero-out the flash drive using
bs=512, 4096, and 40000
and do it with and without using conv=notrunc,sync.
(Six tests.)

Results:
All six tests successfully wiped out the flash drive (all the way out to the "end" of the drive, as seen by dd and based on the offsets from hexdumps). Each test "copied" the same number of bytes (as printed in the final dd output). Running conv=notrunc,sync had no added effect on this good result. Also, note that 40000 is not a multiple of 512 or 4096 (just to shake things up a bit). Each test wrote 1,031,798,784 bytes (with and without conv=notrunc,sync).


Details and comments:

> I hate to make this post lengthy, so I'll abbreviate the details. Some of you dd-afficionados may want to play with this and you may want some details.
> This has made me realize that maybe I don't really understand the meaning of "Records in" and "Records out."
> Although dd seems to be doing what it should do, it's not clear to me why!
> The dd statements are the same as in my post above and come from the tutorial.

Printout of six tests:

dd if=/dev/zero of=/dev/sdc [with and without conv=notrunc,sync]
dd: writing to `/dev/sdc': No space left on device
2015233+0 records in (= 1,031,799,296 bytes)
2015232+0 records out (= 1,031,798,784 bytes)
1031798784 bytes (1.0 GB) copied, 379.062 s, 2.7 MB/s

dd if=/dev/zero of=/dev/sdc bs=4096 [with and without conv=notrunc,sync]
dd: writing `/dev/sdc': No space left on device
251905+0 records in (= 1,031,802,880 bytes)
251904+0 records out (= 1,031,798,784 bytes)
1031798784 bytes (1.0 GB) copied, 171.941 s, 6.0 MB/s

sudo dd if=/dev/zero of=/dev/sdc bs=40000 [with and without conv=notrunc,sync]
dd: writing `/dev/sdc': No space left on device
25795+0 records in (=1,031,800,000 bytes)
25794+0 records out (= 1,031,760,000 bytes)
1031798784 bytes (1.0 GB) copied, 187.974 s, 5.5 MB/s

Note that for bs=40000, records out = 25794 = 1,031,760,000 bytes
Thus, we will play with the end zone beyond 1,031,760,000 bytes to the end of the drive (at 1,031,798,784 bytes).

How to write numbers to the end zone beyond 1,031,760,000 bytes:
dd if=/dev/urandom of=/dev/sdc seek=10317600 bs=100
dd: writing `/dev/sdc': No space left on device
388+0 records in
387+0 records out
38784 bytes (39 kB) copied, 0.422035 s, 91.9 kB/s

How to check for non-zero lines beyond 1,031,760,000 bytes:
dd if=/dev/sdc skip=10317600 bs=100 | hexdump -C | grep [^00]
And you can check your offsets to see that they add up (or make sense). Here:
387+1 records in
387+1 records out
38784 bytes (39 kB) copied, 0.0103191 s, 3.8 MB/s
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00009780 = 38784 decimal! <--- offset
=> exact same ending offset as the urandom fill above; this time, all zeroed out.
Check: skip=10317600 (bs=100) PLUS 38784 = 1,031,798,784 bytes (again!)

(All this checking is done to confirm that we are getting all the way out to what we think is the end of the drive.)

If dd reports anything+0 blocks in and out, it means the last block was a complete block. If is says anything+1 block in and out, it means the last block was a partial block. If the drive divides into a whole number of blocks, that is +0 blocks in, +0 blocks out, notrunc,sync has no effect on zeroing the drive.

Yes, thanks. I finally created some toy text files and played around to see some of this (in's and out's) yesterday.

In the example above, the drive is an even multiple of the first two test bs values 512 and 4096, but not so with bs=40000. Yet, dd if=/dev/zero of=/dev/sdc bs=40000 did seem to clear out the drive all the way to the end.

This is what caught me eye:
sudo dd if=/dev/zero of=/dev/sdc bs=40000
dd: writing `/dev/sdc': No space left on device
25795+0 records in (=1,031,800,000 bytes)
25794+0 records out (= 1,031,760,000 bytes)
1031798784 bytes (1.0 GB) copied, 187.974 s, 5.5 MB/s

My tests (verifying with grep and hexdumps) showed the disk was cleared out all the way beyond 1,031,760,000 ("records out") to 1,031,798,784 (the "end" of the drive). (Just as bs=512 and 4096 also did.) And all the bs values worked like this with and without conv=notrun,sync.

All this is just for information/discussion, it's interesting. Suggests to me that the programming of dd may be taking care of more than a user can see from above (reading man dd, et. al.) or that man dd needs a re-write or expansion to explain dd's (new?) capabilities. (btw, right now, doing all this work in Kubuntu 8.04.3.)
Many thanks again.

Thinking about this, messing with a few experiments, the conclusion of my post #594 makes sense (to me, that is; given that it did, in fact, happen). In any reasonable world, it is what we'd expect dd to do when we write dd if=/dev/zero of=/dev/sdx.

First, looking at the target, it doesn't make sense to truncate a device file /dev/sdxn, especially when the source is infinite (arguing against notrunc in this case).
And second, coming at it from the other end (the source), since the source is infinite (/dev/zero), the notion of a partial block (and subsequently padding it) doesn't make sense (arguing against the need for sync).
This case is for
infinite source -> finite (device) target.

As in mathematics, finite cases behave entirely differently, the logic is different: finite source -> finite target (and then the subcases of source being smaller or larger than target, and with/without notrunc and sync operands).

The result of #594 does make sense; in fact, it is what we would expect to happen: dd is doing as it "should." There would be no practical or reasonable support for anything less. For example, while theoretically interesting, the case where /dev/sdx would fill up with zeros except for a small deficit due to an odd-sized, unpadded bs, would be ridiculous in any real-world application. The spirit of this conclusion is similar to that of BT+1 in the discussion #351 regarding how dd does the right thing with partial blocks when only bs= is used (and not ibs with obs).

I'm thinking that the dd programmers have already thought through all this.

QZmike,

I'd like for you to do your best in running your tests, but before you post more results, I'd like you to coherently compile the data in a concise reply. That's kind of the spirit I've given the OP. Put a lot in a little space. After you've run all your tests, post all the results.

Why 3X?
 

How did you come up with the magic number '3'? And are you aware of how this shortens the life of your flash drive? Do you realize that the flash driver will spread writes to the same logical address out over different physical locations to minimize wear on the flash cells?

I ask all these questions because I strongly suspect you are only shortening the life of your flash drive for no real benefit. Each flash cell will start to fail after a few hundred thousand writes-erase cycles.

This thread really helped, thanks.

I was wondering whether it is possible to write 'A' to one half of a USB and 'B' to another.
I have a 512MB USB stick and I want to write 'A' to 256MB of the USB and 'B' to the other 256MB.

Is this possible?


Thanks

great article :)

That depends if you want 0xB or ascii B. if you want hex B, open a hex editor and make a file the size you want containing all 0xBB. You have to type one line of BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB. Then copy and paste until you have 100 lines, then copy and paste that until you have at least 256 MB.

For ascii you use an ascii (text) editor. then you write the content of the files directly to the drive using dd. You can control the exact write length using bs= and count=.

It all depends what kind of editor you would view the drive with. Ascii B is 0x42, or 42h. So if you write ascii Bs, and view the drive using in hex, it will look like: 42 42 42 42 42 42 42 42 42 42 42 42 42. If you write BBh, or 0xBB to the drive, and you view the drive in ascii, it will look blank, because hex B is ascii vertical tab.