[SOLVED] dd and status=progress

kjhambrick · 06-03-2016, 07:29 AM

All --

I've been burning a few USB thumb drives lately

and I've tried to invoke the `dd status=progress` arg.

It does not seem to work.

I expect to see an occasional progress indicator but instead nothing writes to stderr until the dd operation is complete, sometimes several minutes later.

Am I doing it wrong ?

Assume IsoFile and USBDrive are set to 'something relevant':

Code:

dd if=${IsoFile} of=${USBDrive} bs=1M status=progress

Not critical but just wondering ...

Thanks.

-- kjh

Skaendo · 06-03-2016, 09:16 AM

It works for me. This is how I write ISO to USB and it works every time for me:

Code:

dd if=/path/to/filename.iso of=/dev/sdb bs=4M status=progress && sync

Alien Bob · 06-03-2016, 09:27 AM

I think that depends on what version of Slackware you are trying this on.

Snapshot taken during a "dd" operation:

Code:

root@bear:/# dd if=/mirrors/slackware-live/latest/slackware-live-xfce-current.iso of=/tmp/DUMMY.iso status=progress
178830848 bytes (179 MB, 171 MiB) copied, 3.00001 s, 59.6 MB/s

On Slackware-current the "status=progress" works. Any release before that, dd did not support "status=progress".

kjhambrick · 06-03-2016, 09:29 AM

Quote:

Originally Posted by Skaendo

It works for me. This is how I write ISO to USB and it works every time for me:

Code:

dd if=/path/to/filename.iso of=/dev/sdb bs=4M status=progress && sync

Dang !

I must be doing it wrong

Thanks Skaendo.

I like the && sync that you appended ... I type it manually after a dd but your way is less error-prone.

-- kjh

Skaendo · 06-03-2016, 10:16 AM

Quote:

Originally Posted by Alien Bob

I think that depends on what version of Slackware you are trying this on.

Snapshot taken during a "dd" operation:

Code:

root@bear:/# dd if=/mirrors/slackware-live/latest/slackware-live-xfce-current.iso of=/tmp/DUMMY.iso status=progress
178830848 bytes (179 MB, 171 MiB) copied, 3.00001 s, 59.6 MB/s

On Slackware-current the "status=progress" works. Any release before that, dd did not support "status=progress".

I think that you are 100% correct as usual Eric. Another member was using 8.21 and "status-progress" didn't work, where I am on current and 8.25 it does work.

@kjh
I cannot take credit for the '&& sync'. I found it on another forum somewhere while looking for something else and said hey, that looks like it might be a good idea.

Paulo2 · 06-03-2016, 10:17 AM

As -stable user I put dd in background and 'kill -10' it to see the progress,
good to know that there is an option to show the progress now.

kjhambrick · 06-03-2016, 10:38 AM

Quote:

Originally Posted by Alien Bob

I think that depends on what version of Slackware you are trying this on.

Thanks as always Eric !

-- kjh

kjhambrick · 06-03-2016, 11:45 AM

This is kinda strange ...

edit -- same groups and dd behavior from the runlevel 3 console

I am logged into a KDE Konsole session on an up-to-date Current + Multilib System

If I leave off bs=1M or bs=4M I get a progress output:

Code:

$ dd if=slackware-current-64-B60603.iso of=/dev/sde status=progress

2766066176 bytes (2.8 GB, 2.6 GiB) copied, 630 s, 4.4 MB/s    
5410816+0 records in
5410816+0 records out
2770337792 bytes (2.8 GB, 2.6 GiB) copied, 636.599 s, 4.4 MB/s

When I add the bs= arg, not only do I NOT get progress output but I am unable to kill dd via [Ctrl][C]:

The only way 'out' is to unplug the Thumb drive.

bs=4M:

Code:

$ dd if=slackware-current-64-B60603.iso of=/dev/sde bs=4M status=progress          

660+1 records in                                                                                            
660+1 records out                                                                                           
2770337792 bytes (2.8 GB, 2.6 GiB) copied, 13.6832 s, 202 MB/s                                              
dd: closing input file 'slackware-current-64-B60603.iso': Bad file descriptor

bs=1M

Code:

$ dd if=slackware-current-64-B60603.iso of=/dev/sde bs=1M status=progress          
                                                                                                            
2642+0 records in                                                                                           
2642+0 records out                                                                                          
2770337792 bytes (2.8 GB, 2.6 GiB) copied, 16.0967 s, 172 MB/s                                              
dd: closing input file 'slackware-current-64-B60603.iso': Bad file descriptor

no bs=

Code:

$ dd if=slackware-current-64-B60603.iso of=/dev/sde status=progress

2766066176 bytes (2.8 GB, 2.6 GiB) copied, 630 s, 4.4 MB/s    
5410816+0 records in
5410816+0 records out
2770337792 bytes (2.8 GB, 2.6 GiB) copied, 636.599 s, 4.4 MB/s

I am a member of plugdev:

Code:

$groups 
users lp floppy audio video cdrom plugdev netdev scanner

But this is kinda strange ( note the different group lists ) ...

Code:

$ su -
Password: 

# groups konrad

konrad : users lp plugdev netdev

as is this:

Code:

# grep konrad /etc/group
lp:x:7:lp,konrad
plugdev:x:83:konrad
netdev:x:86:konrad

dd is Version 8.25:

Code:

$ dd --version

dd (coreutils) 8.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Rubin, David MacKenzie, and Stuart Kemp.

I am going to log out of KDE / try from a runlevel 3 session ... maybe it's a KDE thing ?

-- kjh

dab1414 · 06-03-2016, 11:52 AM

When I last installed current (a few days ago), I did the dd urandom on hardrives then in another tty

Code:

pkill -USR1 -n -x dd

It showed me the progress, then I weeped after realizing how long it would take for my 1Tb drive.

kjhambrick · 06-03-2016, 12:13 PM

Quote:

Originally Posted by dab1414

When I last installed current (a few days ago), I did the dd urandom on hardrives then in another tty

Code:

pkill -USR1 -n -x dd

It showed me the progress, then I weeped after realizing how long it would take for my 1Tb drive.

Yes. That's more-or-less the way I used to run dd:

Code:

# dd if=slackware-current-64-B60603.iso of=/dev/sde &                
[2] 31262

# note the PID=31262 ...

# kill -USR1 31262

484017+0 records in
484017+0 records out
247816704 bytes (248 MB, 236 MiB) copied, 48.8515 s, 5.1 MB/s

Interesting ... if I add bs=1M or bs=4M to the dd command, I get no status on a kill -USR1 PID and I can't kill -15 PID or kill -9 PID ( same as [Ctrl][C] on an interactive dd session )

Maybe I've got a bad USB Drive ... ???

This particular USB Drive did some odd things when I used it to test liveslak ... maybe it was the USB Drive all along.

-- kjh

rknichols · 06-03-2016, 12:15 PM

If you have a lot of memory, what can happen is that dd finishes writing the data very quickly, but it's all sitting in kernel buffers waiting for the slow device. Then dd calls close() on the output file, and that hangs until the buffers are flushed out. The dd program remains unresponsive (hung in the "D" state) until that all completes. With your 2.6GB of data and a slow USB flash drive, that can take several minutes.

You can use "oflag=direct" with dd to avoid the buffering, but performance will be sensitive to the output block size. Use the optimal I/O size reported by fdisk for best results.

kjhambrick · 06-03-2016, 12:37 PM

Quote:

Originally Posted by rknichols

If you have a lot of memory, what can happen is that dd finishes writing the data very quickly, but it's all sitting in kernel buffers waiting for the slow device. Then dd calls close() on the output file, and that hangs until the buffers are flushed out. The dd program remains unresponsive (hung in the "D" state) until that all completes. With your 2.6GB of data and a slow USB flash drive, that can take several minutes.

You can use "oflag=direct" with dd to avoid the buffering, but performance will be sensitive to the output block size. Use the optimal I/O size reported by fdisk for best results.

edit - marked as solved

We have a winner ! Thank you rknichols !

I've got 64GB of RAM so pretty much everything I read gets cached to RAM ...

And I saw the D State of the unresponsive DD program when I tried to kill -15 PID and kill -9 PID ...

My 'optimal size is 512 bytes ( below ) and dd now works with ANY bs= and status=progress as long as I add oflag=direct arg !

Thanks again !

-- kjh

Code:

# fdisk -l /dev/sde

Disk /dev/sde: 7.5 GiB, 8006074368 bytes, 15636864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
<<snip>>