progress

robertkwild · 09-13-2017, 09:07 AM

hi all,

i have found a really good command that monitors the progress of a bunch of commands, its here -

https://github.com/Xfennec/progress

once installed you can run this command -

cp -r /mnt/local/data/call_the_midwife_7_1708/ /mnt/local/data/new/

then open a new terminal and run this command -

watch -n 0.5 progress -w

this will give you this -

Every 0.5s: progress -w Wed Sep 13 15:05:16 2017

[12254] cp /mnt/local/data/call_the_midwife_7_1708/Promo/grading_output/for_approval/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or/192
0x1080/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or_V1.mxf
19.6% (2.3 GiB / 11.9 GiB) 27.4 MiB/s remaining 0:05:55

has anyone heard of this

but this gives you details of individual files being copied over, i need something that can give me the ETA and percent of the whole directory copied over and not just individual files?

rob

pan64 · 09-13-2017, 09:41 AM

you can check the command pv

robertkwild · 09-14-2017, 06:47 AM

mmm...

got me thinking can i use the command pv and progress together to get the total ETA/percent of the whole directory instead of an ETA/percent of each individual file in the directory?

robertkwild · 09-15-2017, 10:38 AM

smashed it -

[root@robw-linux data]# tar -c call_the_midwife_7_1708/ | pv -lep -s 32455212 | tar -x -C /mnt/local/data/new/
[=> ] 2% ETA 2:34:31

and to find the dir size i did -

du -s call_the_midwife_7_1708/

but doing it via this method takes ages as its creating the tar and extracting the tar, normally doing a normal copy only takes roughly 18 minutes

pan64 · 09-15-2017, 12:02 PM

probably cp -v -r is enough for you.

robertkwild · 09-20-2017, 03:30 AM

just thought of another idea -

il get the size of the source path -

du -s /source_path/

then i will start the copy -

cp -r /source_path/ /destination_path/

while im copying i will monitor the progress -

watch -n 0.5 du -s /destination_path/

but i want to do this all in a bash script but my issue is it wont watch the destination path while the copy is going on, how do i do both at the same time

rob

pan64 · 09-20-2017, 05:53 AM

what do you mean by that? (what information do you need? what do you want to really achieve?)

robertkwild · 09-20-2017, 06:51 AM

im trying to achive so the end user gets a real time monitor/progress of how far the copy command is going and when its going to finish

it would be nice to get an ETA, percent and a progress bar, or just count up how big the directory size gets

pan64 · 09-20-2017, 09:16 AM

probably this helps: https://www.cyberciti.biz/faq/show-p...file-transfer/

robertkwild · 09-21-2017, 08:06 AM

i have seen this -

https://stackoverflow.com/questions/...=votes#tab-top

if you read the answer by mitch he has made a script that does this with echo commands

but when i try it and run it on my linux box i just get 33, 66, 100% with the hashes, how can i get this to count up from 1-100

also how can i implement this with my copy command "cp -r /source /dest"

many thanks,

rob

robertkwild · 09-22-2017, 06:36 AM

sorted it,

yum remove rsync

https://download.samba.org/pub/rsync/rsync-3.1.2.tar.gz

installed it by untarring it cd'd into the dir and running "./configure.sh" "make" "make install"

and now i get the result i wanted -

[root@robw-linux data]# rsync -a --info=progress2 call_the_midwife_7_1708/ new/
14,874,971,690 44% 27.58MB/s 0:10:48 xfr#16, to-chk=2/143)

robertkwild · 09-23-2017, 04:13 PM

now i have the rsync command progress sorted now i want to do the same with the find command

i have done the commands below -

cd /source_dir/

find . -type f -exec md5sum {} \; >> /md5sum/source.txt

this works fine but i want to see the progress of the find command, is there a way i can do this via pv command or something else

many thanks,

rob

pan64 · 09-25-2017, 12:56 AM

something like this:

Code:

find . -type f -exec <shell_script> {} \; 

# shell script:

#!/bin/bash
echo -n .
md5sum $1 >> /md5sum/source.txt

(if you want to say thanks just click on yes)

robertkwild · 09-25-2017, 04:11 AM

thanks pan64 but how is this going to show me the real life progress of the find command

pan64 · 09-25-2017, 06:54 AM

find has no any idea about the amount of work/time required by the -exec commands.
find also has no idea about the amount of files to look for/to be processed, so it cannot tell you any kind or percentage/progress (or similar).
If you want to do that you need to collect all the files reported by find into a (status) file and you can use pv or similar on that file. But that will slow down the whole process.