LinuxQuestions.org - top looses track of dd process

- Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)

- - top looses track of dd process (https://www.linuxquestions.org/questions/linux-software-2/top-looses-track-of-dd-process-4175622594/)

top looses track of dd process

Howdy LQ:

I was writing random data over a partition with dd (I forgot to optimize the bs parameter, so it is taking forever--already two days now). I logged in again on a separate tty (all console, without x11), and ran top to monitor the process: I can walk by and see that dd is still at the top of the list, and know it hasn't finished yet... unless the screen goes to sleep. If I leave for a few hours and come back to it, such that the screen has gone to sleep for screen protection or power-saving, when I hit a key to wake it back up, top is still there tracking processes, except dd is no longer registering on it.

I think whoo hoo, its done. But when I switch ttys and view the dd command, it hasn't returned to command prompt :( Switching back to top, if I quit top, and restart it, dd is back on the list.

Why would top loose track of a process? Is this a bug?

(running on slackware64-14.2)

By default, the top command sorts the display by accumulated CPU time, and dd might not use enough CPU time to be among the displayed processes. You can use the "F" or "O" command in top to change the sort field.

Have you tried watch ps?

Example:

Code:

watch ps u -C <your process name here>

If you're using /dev/random, the excess time is due to that. If you use /dev/urandom, it will go much faster. When the entropy pool is empty, /dev/random blocks until more entropy is gathered, whereas /dev/urandom will just repeat the entropy bits.

dd probably drops off the list in top, because it is waiting for entropy to be gathered, so /dev/random can output more random bytes, and so the dd process becomes mostly inactive. For overwriting a disk you don't need the quality of entropy gotten from /dev/random. /dev/urandom will work perfectly well, at about 100x the speed.

It was sorted to display by %CPU, and when dd is listed, it is shown as using 99-100% cpu, throwing everything it's got at it; but that usage should keep dd at the top of tops list. If I start top with the -i flag, it only displays dd, top, and kworker--everything else is idle. I figured if dd isn't at the top, then it should be completed. There's 12GB ram, but no swap.

I never knew about watch ps, and so I'm trying that now to see if watch ps survives the default power / screen save cycles.

More exacerbating, is that this is intermittent, and sometimes dd is still at the top of top where it should be when I wake up the screen, and sometimes it's not.

A slackware installation readme I follow recommends writing random data over LVM partition for some kind of security gain. Probably wouldn't be wondering about any of top's quirky behavior, if dd would hurry up and finish.

Was using

Code:

dd if=/dev/urandom of=/dev/sda5

sda5 is 866.8G partion, type 8e00, linux lvm.

I figured the slow down was because I didn't specify a tuned bs= parameter, like bs=4096. And even going it's slowest, I didn't expect dd to last over two days: right now it's at 3,319 minutes, lol. I probably could cancel dd, redo it with better parameters, and be finished in a few hours; but I have a morbid curiosity about how long dd with its default opts will drag on.

this is a low budget ideapad that still has 5400 rpm slow i/o drive, and it's off to a slow start... hopefully the tortoise still gets there in the end.

The dd process still disappears from top, sorted by %cpu, and running on tty1; whereas the dd process remained, using, on tty2

Code:

watch ps u -C dd

. And the watch process still reports >99%cpu, so seems like it should be at the top of top... and I tried sorting by other fields, but can't find dd listed in the same active top session--watch ps is listed--but not dd.

There was a thread in the Slackware forum recently about the increased performance of /dev/urandom since the change to a ChaCha20 algorithm in newer kernels. (Whoever thought an Intel Core 2 Duo could dust an AMD Ryzen).
There is also a neat suggestion in post #5.

Deciding that three days was more than even morbid curiosity bargained for, I cancelled dd... whew good thing, because it wasn't even half way through yet:

Code:

root@spore:/# dd if=/dev/urandom of=/dev/sda5

^C702131966+0 records in

702131965+0 records out

359491566080 bytes (359 GB, 335 GiB) copied, 250006 s, 1.4 MB/S

maybe something is broke on this new machine: 1.4 MB/S -- really?

here's some info on cpu:

Code:

root@spore:~# cat /proc/cpuinfo | grep "model name"

model name    : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G

model name    : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G

model name    : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G

model name    : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G

Code:

root@spore:~# cat /proc/cpuinfo | grep MHz

cpu MHz        : 2500.000

cpu MHz        : 1300.000

cpu MHz        : 1300.000

cpu MHz        : 1300.000

Hmm... Why is the first cpu nearly twice as fast as the other three?

And, so I wonder about write speeds on Hard Disk. Besides using dd, what is a good way to benchmark or test if this brand new drive is trash on arrival? I tried smartctl -t short /dev/sda, but two minutes later, smartctl -a /dev/sda tells me that zero errors were reported. I'm currently running the "long" test.

Since this is weirder than I at first thought, here is some more info on my partition layout

/dev/sda1 260M EFI System
/dev/sda2 32G full install of slackware
/dev/sda3 32G mirror of slackware64-14.2 tree
/dev/sda4 512M boot partition (currently mounted at /boot)
/dev/sda5 866G Linux LVM

I am currently booted into the full install on /dev/sda2, though using /dev/sda4 for boot partition, and /dev/sda3 is mounted at /slac

I was proceeding to write random data on /dev/sda5 before running cryptsetup, pvcreate, lvcreate, etc, to create logical volumes for the rest of my installation. However, since the write speed on the disk is 1.4MB/S, I am concerned that the drive is not functioning properly. Even on some smaller 100M dd tests, with different bs= values, I am never getting over 2.5 MB/S.

smartctl -a /dev/sda does report one error, though not from its short test, and the long isn't finished yet; here's an abbreviated output:

Code:

root@spore:/tmp#  smartctl -a /dev/sda



=== START OF INFORMATION SECTION === 

Device Model:    TOSHIBA MQ01ABD100

Sector Sizes:    512 bytes logical, 4096 bytes physical

Rotation Rate:    5400 rpm

Form Factor:      2.5 inches



=== START OF READ SMART DATA SECTION ===

Self-test execution status:                    (249) Self-test routine in progress...



SMART Error Log Version : 1

ATA Error Count: 1

    CR = Command Register

    FR = Features Register

    SC = Sector Count Register

    SN = Sector Number Register

    CL = Cylinder Low Register

    CH = Cylinder High Register

    DH = Device/Head Register

    DC = Device Command Register

    ER = Error Register

    ST = Status register

Powered_Up_Time is measured from power on, and printed as 

DDd+hh:mm:SS.sss



Error 1 occured at disk power-on lifetime: 9 hours (o days + 9 hours)

  When the command that caused the error occurred, the device was active or idle.



After command completion occurred, registers were:



ER  ST  SC  SN  CL  CH  DH

--  --  --  --  --  --  --

04  51  ff  00  00  00  e0



Commands leading to the command that caused the error were:



CR  FR  SC  SN  CL  CH  DH  DC    Powered_Up_Time    Command/Feature_Name

--  --  --  --  --  --  --  --    ---------------    --------------------

f2  00  00  00  00  00  e0  00      00:00:57.168      SECURITY UNLOCK

ec  00  00  00  00  00  e0  00      00:00:55.777      IDENTIFY DEVICE

What's that SECURITY UNLOCK error about? Some kind of bios unlock? I disabled "secure boot" in bios before this install.

Any suggestions on how to further check this drive? Or is 2.5 MB/S normal on low-end laptops, even though new. I figured quad core AMD, with 12G ram, should suffice for basic needs. But taking a week to write a TB of random data, regardless of bs= parameter, seems like somethings messed up. I'll report what the "long" test finds when it's complete.

I posted here instead of in slackware, because I thought this was just a post about dd process disappearing into top--question about linux software... but now I'm finding it may be hardware related too: grr...

If you have a CPU frequency adjusting daemon running, it's normal to see some or all of the CPU cores running at their minimum frequency unless your system is heavily loaded.

If you don't have a recent kernel, the /dev/urandom pseudo-random number generator is quite slow -- not nearly as slow as /dev/random, but still slow. I'm sure someone can chime in here and tell what kernel version introduced the improved urandom generator.

EDIT: I've seen one report that kernel version 4.4.99 is "old" in this context.

Thoughts.

dd shouldn't be taking much cpu. In this case it seems that your math function may be taking a lot of resources for some reason??? As in how linux does that task or how the kernel is using processors both physical and logical. Might even disable HT and try again.

Other thoughts. https://serverfault.com/questions/64...to-dev-urandom

I never figured out why top loses track of dd, but if I were to guess, I think it is that dd with urandom took so much cpu and ram, that top might have needed swap space, which I hadn't set up yet.

I never got dd with urandom to speeds over 2.5MB/S, but rknichols' trick from the slackware thread brought it up to a whopping 88 MB/S, and it only took a few hours. Jefro's link was to a similar approach, though using openssl instead of cryptsetup.

During the cryptsetup method, dd only took 20% cpu, and the rest of the load seemed divided up among kworkers.

Just out of curiosity, how would one disable HT? I see ht as a cpu flag in /proc/cpuinfo. Could I just prefix my dd command like

Code:

CPU_FLAGS="same list as in /proc/cpuinfo, minus the ht" dd if=/dev/urandom of=/dev/sda5 bs=4096

?

Don't know if this thread will ever get solved, as far as top losing dd... but since my LVM partition has been prepared, I'll have to continue on.

Till next time, LQ! Much Thanks!

Quote:

Till next time, LQ! Much Thanks!

It is always gratifying when people report back that they have had success.

Disable HT in bios usually if there is an option. Might have different name too.