LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 01-28-2018, 06:25 PM   #1
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-current
Posts: 340

Rep: Reputation: 133Reputation: 133
top looses track of dd process


Howdy LQ:

I was writing random data over a partition with dd (I forgot to optimize the bs parameter, so it is taking forever--already two days now). I logged in again on a separate tty (all console, without x11), and ran top to monitor the process: I can walk by and see that dd is still at the top of the list, and know it hasn't finished yet... unless the screen goes to sleep. If I leave for a few hours and come back to it, such that the screen has gone to sleep for screen protection or power-saving, when I hit a key to wake it back up, top is still there tracking processes, except dd is no longer registering on it.

I think whoo hoo, its done. But when I switch ttys and view the dd command, it hasn't returned to command prompt Switching back to top, if I quit top, and restart it, dd is back on the list.

Why would top loose track of a process? Is this a bug?

(running on slackware64-14.2)
 
Old 01-28-2018, 07:27 PM   #2
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,340

Rep: Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989
By default, the top command sorts the display by accumulated CPU time, and dd might not use enough CPU time to be among the displayed processes. You can use the "F" or "O" command in top to change the sort field.
 
1 members found this post helpful.
Old 01-28-2018, 08:02 PM   #3
killingthemonkey
Member
 
Registered: Mar 2011
Location: Winston-Salem, NC
Distribution: Fedora, CentOS, Linux Mint
Posts: 181

Rep: Reputation: 19
Have you tried watch ps?

Example:
Code:
watch ps u -C <your process name here>
 
1 members found this post helpful.
Old 01-28-2018, 08:26 PM   #4
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,511

Rep: Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006Reputation: 1006
If you're using /dev/random, the excess time is due to that. If you use /dev/urandom, it will go much faster. When the entropy pool is empty, /dev/random blocks until more entropy is gathered, whereas /dev/urandom will just repeat the entropy bits.

dd probably drops off the list in top, because it is waiting for entropy to be gathered, so /dev/random can output more random bytes, and so the dd process becomes mostly inactive. For overwriting a disk you don't need the quality of entropy gotten from /dev/random. /dev/urandom will work perfectly well, at about 100x the speed.
 
1 members found this post helpful.
Old 01-28-2018, 08:49 PM   #5
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-current
Posts: 340

Original Poster
Rep: Reputation: 133Reputation: 133
It was sorted to display by %CPU, and when dd is listed, it is shown as using 99-100% cpu, throwing everything it's got at it; but that usage should keep dd at the top of tops list. If I start top with the -i flag, it only displays dd, top, and kworker--everything else is idle. I figured if dd isn't at the top, then it should be completed. There's 12GB ram, but no swap.

I never knew about watch ps, and so I'm trying that now to see if watch ps survives the default power / screen save cycles.

More exacerbating, is that this is intermittent, and sometimes dd is still at the top of top where it should be when I wake up the screen, and sometimes it's not.

A slackware installation readme I follow recommends writing random data over LVM partition for some kind of security gain. Probably wouldn't be wondering about any of top's quirky behavior, if dd would hurry up and finish.
 
Old 01-28-2018, 09:12 PM   #6
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-current
Posts: 340

Original Poster
Rep: Reputation: 133Reputation: 133
Was using

Code:
dd if=/dev/urandom of=/dev/sda5
sda5 is 866.8G partion, type 8e00, linux lvm.

I figured the slow down was because I didn't specify a tuned bs= parameter, like bs=4096. And even going it's slowest, I didn't expect dd to last over two days: right now it's at 3,319 minutes, lol. I probably could cancel dd, redo it with better parameters, and be finished in a few hours; but I have a morbid curiosity about how long dd with its default opts will drag on.

this is a low budget ideapad that still has 5400 rpm slow i/o drive, and it's off to a slow start... hopefully the tortoise still gets there in the end.
 
Old 01-29-2018, 01:07 AM   #7
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-current
Posts: 340

Original Poster
Rep: Reputation: 133Reputation: 133
The dd process still disappears from top, sorted by %cpu, and running on tty1; whereas the dd process remained, using, on tty2
Code:
watch ps u -C dd
. And the watch process still reports >99%cpu, so seems like it should be at the top of top... and I tried sorting by other fields, but can't find dd listed in the same active top session--watch ps is listed--but not dd.
 
Old 01-29-2018, 08:17 AM   #8
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 5,336

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
There was a thread in the Slackware forum recently about the increased performance of /dev/urandom since the change to a ChaCha20 algorithm in newer kernels. (Whoever thought an Intel Core 2 Duo could dust an AMD Ryzen).
There is also a neat suggestion in post #5.

Last edited by allend; 01-29-2018 at 08:42 AM.
 
1 members found this post helpful.
Old 01-29-2018, 12:59 PM   #9
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-current
Posts: 340

Original Poster
Rep: Reputation: 133Reputation: 133
Deciding that three days was more than even morbid curiosity bargained for, I cancelled dd... whew good thing, because it wasn't even half way through yet:
Code:
root@spore:/# dd if=/dev/urandom of=/dev/sda5
^C702131966+0 records in
702131965+0 records out
359491566080 bytes (359 GB, 335 GiB) copied, 250006 s, 1.4 MB/S
maybe something is broke on this new machine: 1.4 MB/S -- really?

here's some info on cpu:
Code:
root@spore:~# cat /proc/cpuinfo | grep "model name"
model name     : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G
model name     : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G
model name     : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G
model name     : AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G
Code:
root@spore:~# cat /proc/cpuinfo | grep MHz
cpu MHz        : 2500.000
cpu MHz        : 1300.000
cpu MHz        : 1300.000
cpu MHz        : 1300.000
Hmm... Why is the first cpu nearly twice as fast as the other three?

And, so I wonder about write speeds on Hard Disk. Besides using dd, what is a good way to benchmark or test if this brand new drive is trash on arrival? I tried smartctl -t short /dev/sda, but two minutes later, smartctl -a /dev/sda tells me that zero errors were reported. I'm currently running the "long" test.

Since this is weirder than I at first thought, here is some more info on my partition layout

/dev/sda1 260M EFI System
/dev/sda2 32G full install of slackware
/dev/sda3 32G mirror of slackware64-14.2 tree
/dev/sda4 512M boot partition (currently mounted at /boot)
/dev/sda5 866G Linux LVM

I am currently booted into the full install on /dev/sda2, though using /dev/sda4 for boot partition, and /dev/sda3 is mounted at /slac

I was proceeding to write random data on /dev/sda5 before running cryptsetup, pvcreate, lvcreate, etc, to create logical volumes for the rest of my installation. However, since the write speed on the disk is 1.4MB/S, I am concerned that the drive is not functioning properly. Even on some smaller 100M dd tests, with different bs= values, I am never getting over 2.5 MB/S.

smartctl -a /dev/sda does report one error, though not from its short test, and the long isn't finished yet; here's an abbreviated output:

Code:
root@spore:/tmp#  smartctl -a /dev/sda

=== START OF INFORMATION SECTION === 
Device Model:     TOSHIBA MQ01ABD100
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches

=== START OF READ SMART DATA SECTION ===
Self-test execution status:                    (249) Self-test routine in progress...

SMART Error Log Version : 1
ATA Error Count: 1
     CR = Command Register
     FR = Features Register
     SC = Sector Count Register
     SN = Sector Number Register
     CL = Cylinder Low Register
     CH = Cylinder High Register
     DH = Device/Head Register
     DC = Device Command Register
     ER = Error Register
     ST = Status register
Powered_Up_Time is measured from power on, and printed as 
DDd+hh:mm:SS.sss

Error 1 occured at disk power-on lifetime: 9 hours (o days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:

ER  ST  SC  SN  CL  CH  DH
--  --  --  --  --  --  --
04  51  ff  00  00  00  e0

Commands leading to the command that caused the error were:

CR  FR  SC  SN  CL  CH  DH  DC     Powered_Up_Time     Command/Feature_Name
--  --  --  --  --  --  --  --     ---------------     --------------------
f2  00  00  00  00  00  e0  00       00:00:57.168       SECURITY UNLOCK
ec  00  00  00  00  00  e0  00       00:00:55.777       IDENTIFY DEVICE
What's that SECURITY UNLOCK error about? Some kind of bios unlock? I disabled "secure boot" in bios before this install.

Any suggestions on how to further check this drive? Or is 2.5 MB/S normal on low-end laptops, even though new. I figured quad core AMD, with 12G ram, should suffice for basic needs. But taking a week to write a TB of random data, regardless of bs= parameter, seems like somethings messed up. I'll report what the "long" test finds when it's complete.

I posted here instead of in slackware, because I thought this was just a post about dd process disappearing into top--question about linux software... but now I'm finding it may be hardware related too: grr...
 
Old 01-29-2018, 02:04 PM   #10
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,340

Rep: Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989Reputation: 1989
If you have a CPU frequency adjusting daemon running, it's normal to see some or all of the CPU cores running at their minimum frequency unless your system is heavily loaded.

If you don't have a recent kernel, the /dev/urandom pseudo-random number generator is quite slow -- not nearly as slow as /dev/random, but still slow. I'm sure someone can chime in here and tell what kernel version introduced the improved urandom generator.

EDIT: I've seen one report that kernel version 4.4.99 is "old" in this context.

Last edited by rknichols; 01-29-2018 at 02:16 PM. Reason: add kernel version info
 
1 members found this post helpful.
Old 01-29-2018, 04:29 PM   #11
jefro
Moderator
 
Registered: Mar 2008
Posts: 19,423

Rep: Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998
Thoughts.

dd shouldn't be taking much cpu. In this case it seems that your math function may be taking a lot of resources for some reason??? As in how linux does that task or how the kernel is using processors both physical and logical. Might even disable HT and try again.

Other thoughts. https://serverfault.com/questions/64...to-dev-urandom
 
1 members found this post helpful.
Old 01-30-2018, 02:16 AM   #12
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-current
Posts: 340

Original Poster
Rep: Reputation: 133Reputation: 133
I never figured out why top loses track of dd, but if I were to guess, I think it is that dd with urandom took so much cpu and ram, that top might have needed swap space, which I hadn't set up yet.

I never got dd with urandom to speeds over 2.5MB/S, but rknichols' trick from the slackware thread brought it up to a whopping 88 MB/S, and it only took a few hours. Jefro's link was to a similar approach, though using openssl instead of cryptsetup.

During the cryptsetup method, dd only took 20% cpu, and the rest of the load seemed divided up among kworkers.

Just out of curiosity, how would one disable HT? I see ht as a cpu flag in /proc/cpuinfo. Could I just prefix my dd command like
Code:
CPU_FLAGS="same list as in /proc/cpuinfo, minus the ht" dd if=/dev/urandom of=/dev/sda5 bs=4096
?

Don't know if this thread will ever get solved, as far as top losing dd... but since my LVM partition has been prepared, I'll have to continue on.

Till next time, LQ! Much Thanks!
 
1 members found this post helpful.
Old 01-30-2018, 06:26 AM   #13
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 5,336

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
Quote:
Till next time, LQ! Much Thanks!
It is always gratifying when people report back that they have had success.
 
Old 01-30-2018, 06:30 PM   #14
jefro
Moderator
 
Registered: Mar 2008
Posts: 19,423

Rep: Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998Reputation: 2998
Disable HT in bios usually if there is an option. Might have different name too.
 
1 members found this post helpful.
  


Reply

Tags
top dd


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Top 5 open source community metrics to track LXer Syndicated Linux News 0 12-05-2015 11:30 AM
[SOLVED] IP banging on my firewall from inside - how to track the process back to src ceyx Linux - Security 8 07-21-2015 06:56 PM
How to track which process is accessing disk? mldqj Linux - Kernel 6 02-12-2009 08:20 AM
Track/Trace which process/command modified some file ? Horos23 Linux - Security 1 06-28-2007 02:17 PM
Keeping track of process CPU % dive Linux - Software 5 03-30-2006 03:45 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 01:42 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration