rsync using the pv command wrong ETA

robertkwild · 09-12-2017, 07:01 AM

hi all,

i am copying a directory from folder a to folder b like so -

time rsync -avh /folder_a/ /folder_b/

this is 30gigs and it takes roughly 18 min to complete, i have done it multiple times

it works but now i want to pipe it with the pv command to get the ETA and the progress ie the progress bar and the percent

i use this command -

rsync -avh /folder_a/ /folder_b/ | pv -lep -s 17 > /dev/null

folder a has 17 files in it, i work this out by doing a "find -type f | wc -l" when i cd in folder a

but when i run the command the ETA just keeps going up up and up, it never stops so it doesnt give you 18 min like with the rsync command did

any help on this would be greatly appreciated

many thanks,

rob

Turbocapitalist · 09-12-2017, 07:52 AM

pv can't really do more than get in the way of the transfer. rsync has it's own built-in option for tracking progress:

Code:

rsync -avh --human-readable --progress /folder_a/ /folder_b/

For details, see "man rsync"

robertkwild · 09-12-2017, 08:35 AM

this is good but this does a progress for each individual file copied, i want a whole progress for the directory including ETA aswell

rknichols · 09-12-2017, 09:50 AM

Quote:

Originally Posted by robertkwild

Code:

rsync -avh /folder_a/ /folder_b/ | pv -lep -s 17 > /dev/null

That "-s 17" is telling the pv command that the expected total size of the transfer is 17 bytes. You're not going to get reasonable progress indications from that. When pv is getting its input from a pipe, that "-s" (--size) argument is the only clue it has about the expected size of the transfer, and you can't, in general, accurately predict how much data rsync will send. If all the files are new at the destination and there is no compression involved, you can estimate by using the total size of all the files plus some overhead. For incremental updates of existing files, it's really difficult to predict.

robertkwild · 09-12-2017, 09:53 AM

ok 17 is the file count and not the size of the directory, so shall i change it to the size, will this give me a better ETA?

rknichols · 09-12-2017, 10:11 AM

Quote:

Originally Posted by robertkwild

ok 17 is the file count and not the size of the directory, so shall i change it to the size, will this give me a better ETA?

That depends on what you mean by, "size of the directory." The directory file itself has a size. That's just for storage of the file names, inode numbers, and some status information. What you need is the total size of the files, like what would be reported by du for that directory. There will be some overhead in addition to that (see how much rsync would send if you use the "-n" option to suppress the actual transfer), but if the files themselves are large it should be a reasonable approximation.

robertkwild · 09-12-2017, 10:18 AM

ok im doing this -

[root@robw-linux ~]# du -s /mnt/local/data/call_the_midwife_7_1708/
32455212 /mnt/local/data/call_the_midwife_7_1708/
[root@robw-linux ~]# rsync -avh /mnt/local/data/call_the_midwife_7_1708/ /mnt/local/data/folder/ | pv -lep -s 32455212 > /dev/null
[> ] 0% ETA 1011:03:44

but the ETA is way way out

michaelk · 09-12-2017, 10:57 AM

From the pv man pages:

Quote:

-l, --line-mode
Instead of counting bytes, count lines (newline characters). The
progress bar will only move when a new line is found, and the
value passed to the -s option will be interpreted as a line
count.

Using the number of files is correct versus byte size when using the -l option. I have not played with rsync/pv in awhile and can not say why the ETA is off at the moment.

https://www.cyberciti.biz/faq/show-p...file-transfer/

robertkwild · 09-12-2017, 11:16 AM

this is very interesting -

https://linoxide.com/linux-command/t...mand-progress/

mmm... seems to give you the status of just files copied and not the whole directory which is a shame