LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS
User Name
Password
CentOS This forum is for the discussion of CentOS Linux. Note: This forum does not have any official participation.

Notices


Reply
  Search this Thread
Old 09-13-2017, 09:07 AM   #1
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Rep: Reputation: Disabled
progress - to monitor progress of commands


hi all,

i have found a really good command that monitors the progress of a bunch of commands, its here -

https://github.com/Xfennec/progress

once installed you can run this command -

cp -r /mnt/local/data/call_the_midwife_7_1708/ /mnt/local/data/new/

then open a new terminal and run this command -

watch -n 0.5 progress -w

this will give you this -

Every 0.5s: progress -w Wed Sep 13 15:05:16 2017

[12254] cp /mnt/local/data/call_the_midwife_7_1708/Promo/grading_output/for_approval/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or/192
0x1080/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or_V1.mxf
19.6% (2.3 GiB / 11.9 GiB) 27.4 MiB/s remaining 0:05:55

has anyone heard of this

but this gives you details of individual files being copied over, i need something that can give me the ETA and percent of the whole directory copied over and not just individual files?

rob

Last edited by robertkwild; 09-13-2017 at 09:10 AM.
 
Old 09-13-2017, 09:41 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 12,698

Rep: Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980
you can check the command pv
 
Old 09-14-2017, 06:47 AM   #3
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
mmm...

got me thinking can i use the command pv and progress together to get the total ETA/percent of the whole directory instead of an ETA/percent of each individual file in the directory?
 
Old 09-15-2017, 10:38 AM   #4
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
smashed it -

[root@robw-linux data]# tar -c call_the_midwife_7_1708/ | pv -lep -s 32455212 | tar -x -C /mnt/local/data/new/
[=> ] 2% ETA 2:34:31

and to find the dir size i did -

du -s call_the_midwife_7_1708/

but doing it via this method takes ages as its creating the tar and extracting the tar, normally doing a normal copy only takes roughly 18 minutes
 
Old 09-15-2017, 12:02 PM   #5
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 12,698

Rep: Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980
probably cp -v -r is enough for you.
 
Old 09-20-2017, 03:30 AM   #6
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
just thought of another idea -

il get the size of the source path -

du -s /source_path/

then i will start the copy -

cp -r /source_path/ /destination_path/

while im copying i will monitor the progress -

watch -n 0.5 du -s /destination_path/

but i want to do this all in a bash script but my issue is it wont watch the destination path while the copy is going on, how do i do both at the same time

rob
 
Old 09-20-2017, 05:53 AM   #7
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 12,698

Rep: Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980
what do you mean by that? (what information do you need? what do you want to really achieve?)
 
Old 09-20-2017, 06:51 AM   #8
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
im trying to achive so the end user gets a real time monitor/progress of how far the copy command is going and when its going to finish

it would be nice to get an ETA, percent and a progress bar, or just count up how big the directory size gets

Last edited by robertkwild; 09-20-2017 at 06:53 AM.
 
Old 09-20-2017, 09:16 AM   #9
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 12,698

Rep: Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980
probably this helps: https://www.cyberciti.biz/faq/show-p...file-transfer/
 
Old 09-21-2017, 08:06 AM   #10
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
i have seen this -

https://stackoverflow.com/questions/...=votes#tab-top

if you read the answer by mitch he has made a script that does this with echo commands

but when i try it and run it on my linux box i just get 33, 66, 100% with the hashes, how can i get this to count up from 1-100

also how can i implement this with my copy command "cp -r /source /dest"

many thanks,

rob
 
Old 09-22-2017, 06:36 AM   #11
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
sorted it,


yum remove rsync


https://download.samba.org/pub/rsync/rsync-3.1.2.tar.gz


installed it by untarring it cd'd into the dir and running "./configure.sh" "make" "make install"


and now i get the result i wanted -


[root@robw-linux data]# rsync -a --info=progress2 call_the_midwife_7_1708/ new/
14,874,971,690 44% 27.58MB/s 0:10:48 xfr#16, to-chk=2/143)
 
Old 09-23-2017, 04:13 PM   #12
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
now i have the rsync command progress sorted now i want to do the same with the find command

i have done the commands below -

cd /source_dir/

find . -type f -exec md5sum {} \; >> /md5sum/source.txt

this works fine but i want to see the progress of the find command, is there a way i can do this via pv command or something else

many thanks,

rob
 
Old 09-25-2017, 12:56 AM   #13
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 12,698

Rep: Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980
something like this:
Code:
find . -type f -exec <shell_script> {} \; 

# shell script:

#!/bin/bash
echo -n .
md5sum $1 >> /md5sum/source.txt
(if you want to say thanks just click on yes)
 
1 members found this post helpful.
Old 09-25-2017, 04:11 AM   #14
robertkwild
Member
 
Registered: Feb 2015
Posts: 376

Original Poster
Rep: Reputation: Disabled
thanks pan64 but how is this going to show me the real life progress of the find command
 
Old 09-25-2017, 06:54 AM   #15
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 12,698

Rep: Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980Reputation: 3980
find has no any idea about the amount of work/time required by the -exec commands.
find also has no idea about the amount of files to look for/to be processed, so it cannot tell you any kind or percentage/progress (or similar).
If you want to do that you need to collect all the files reported by find into a (status) file and you can use pv or similar on that file. But that will slow down the whole process.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Copying files in dolphin - how to monitor progress and or stop process cilbuper Linux - Software 4 05-26-2018 12:57 AM
LXer: How to monitor progress of Linux commands using PV and Progress utilities LXer Syndicated Linux News 0 11-11-2016 07:01 PM
git checkout: how to monitor progress (node.js) eantoranz Programming 8 11-06-2015 05:18 PM
Clone Win XP and monitor progress -- Knoppix dw85745 Linux - Newbie 5 05-02-2009 07:08 PM
Some progress - but need more help mrh7184 Linux - Wireless Networking 1 10-04-2005 06:58 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS

All times are GMT -5. The time now is 09:55 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration