[SOLVED] Disable caching writes when using dd to erase disk

designator · 05-31-2016, 08:18 PM

I'm trying to use the following command to erase an externally attached (USB 3) drive:

Code:

dd if=/dev/zero of=/dev/sdc

Depending on the block size I use, this command quickly eats the RAM buffer and eventually crashes my laptop. 1GB block size will freeze the laptop within a few minutes. Default, 512b block size freezes it within 15 minutes or so.

I also tried the following command to try and disable caching with similar results:

Code:

dd if=/dev/zero of=/dev/sdc bs=1G conv=sync

Relevant details:

Code:

$ uname -a
Linux laptop 4.1.21-14-default #1 SMP PREEMPT Sun Apr 17 07:27:45 UTC 2016 (fc187c1) x86_64 x86_64 x86_64 GNU/Linux
$ free -m
             total       used       free     shared    buffers     cached
Mem:         11894       2776       9117        421         13        929
-/+ buffers/cache:       1833      10060
Swap:            0          0          0

This is what free reports right before my laptop freezes when (I think) it runs out of RAM:

Code:

$ free -m
             total       used       free     shared    buffers     cached
Mem:         11894      11824         69        411       7849       1131
-/+ buffers/cache:       2843       9050
Swap:            0          0          0

What's the correct way to prevent dd from using buffer?
Thanks!

rknichols · 05-31-2016, 09:22 PM

Quote:

Originally Posted by designator

What's the correct way to prevent dd from using buffer?
Thanks!

That would be, "oflag=direct".

syg00 · 05-31-2016, 10:21 PM

Read this.

rknichols · 06-01-2016, 08:40 AM

Quote:

Originally Posted by syg00

Read this.

Very interesting. Now I'm wondering why I've never experienced the problem. I see the writing process stall for a long time while a few gigabytes of buffer get flushed to a slow USB flash drive at around 10 MB/s, but the rest of the 64-bit system continues to run just fine.

designator · 06-02-2016, 11:54 AM

Quote:

Originally Posted by rknichols

That would be, "oflag=direct".

Worked perfectly. Thank you.
Speed went from ~50MB/s to ~4MB/s, but buffer stays unused.

Quote:

Originally Posted by syg00

Read this.

Thanks for the explanation. Unless I misunderstood that article though it seems like there is no fix. You can only delay the inevitable which wouldn't really help in this case anyway since the entire buffer fills up.

rknichols · 06-02-2016, 01:45 PM

Quote:

Originally Posted by designator

Worked perfectly. Thank you.
Speed went from ~50MB/s to ~4MB/s, but buffer stays unused.

Choosing a block size more compatible with the device might alleviate the performance hit.

designator · 06-02-2016, 02:12 PM

Quote:

Originally Posted by rknichols

Choosing a block size more compatible with the device might alleviate the performance hit.

I discovered that as well. I set the block size to 32mb suggested by fdisk as optimal I/O size and was able to dd the entire drive @ ~70MB/s.

Code:

I/O size (minimum/optimal): 512 bytes / 33553920 bytes

Thanks again.