[SOLVED] sed processing wget progress bar

RichardS · 11-02-2010, 10:02 AM

Hi,

I am attempting to "export" the progress bar from wget display using sed. Basically, we have an app that starts wget to download a large file and we want to show a progress bar. Our application has a dbus interface to receive the download progress, so we were think of a command like:

wget [] | sed [] | dbus-send[]

The problem at the moment is, how do you get the matched string out of sed and into dbus-send?

I can get the progress string by:
sed -u 's/[0-9]*%/&/'

This populated '&' with the correct percentage, but I cannot seem to get this out of sed.

Is there anyway of doing this?

Thanks.

David the H. · 11-02-2010, 11:05 PM

What environment are you talking about? You say it's an "app", but you're calling external tools like sed and wget? So is this being done as a shell script or what?

Assuming it's a script, you've got a couple of issues here. First of all, the wget output is being made to stderr instead of stdout, so you have to redirect it before sed can read it. Second, sed needs to have the -n option enabled and use "p" to the end of the substitution command so that it only exports the line you want.

Also, what exactly what text format is the dbus expecting for input? It's not clear to me exactly what you want sed to output. I used the following command and got the output shown:

Code:

wget downloadfile -O/dev/null |& sed -u -n 's/[0-9]*%/&/p'
     0K .......... .......... .......... .......... ..........  0%  694K 8s
    50K .......... .......... .......... .......... ..........  1% 1.09M 7s
   100K .......... .......... .......... .......... ..........  2% 2.23M 5s
   150K .......... .......... .......... .......... ..........  3% 2.41M 4s
   200K .......... .......... .......... .......... ..........  4% 3.51M 4s
   (...etc.)

(|& is a new Bash synonym for 2>&1 | , which redirects stderr to stdout, then into a pipe. Try the latter form if your shell is an older model.)

Is that what you want? For some reason the piped output doesn't match the regular output. It probably has something to do with the way wget dynamically updates the display.

But if the above is ok, grep can do the same thing more efficiently.

Code:

wget downloadfile -O/dev/null |& grep "%"

Or if you really want to grab only the percentage numbers and exclude the rest of the line, then you could use something like this:

Code:

wget downloadfile -O/dev/null |& grep -o "[0-9]\+%"

RichardS · 11-03-2010, 01:35 AM

Hi David,

Thanks for the input. I see now that the question did leave questions about the environment, etc.

To give a more in depth explanation, our application is a C app that calls an init script to launch wget and do a few other thing. The app does this on a host of conditions, etc.

Hence, when wget operates it is in a sh script environment.

Quote:

Originally Posted by David the H.

What environment are you talking about?

wget is operating inside a sh shell script.

Quote:

Originally Posted by David the H.

You say it's an "app", but you're calling external tools like sed and wget? So is this being done as a shell script or what?

The C app calls an init script which has #!/bin/sh operator.

Quote:

Originally Posted by David the H.

Also, what exactly what text format is the dbus expecting for input?

The dbus interface is a simple test string:
dbus-send --dest=[...] --type=method_call [...] [...].progress string:"persentage would go here [0..100]"

Hence in the example you gave, I only want the [0..100] before the percent:

Code:

0K .......... .......... .......... .......... ..........  0%  694K 8s

I want to pass dbus:

Code:

Quote:

Originally Posted by David the H.

(|& is a new Bash synonym for 2>&1 | , which redirects stderr to stdout, then into a pipe. Try the latter form if your shell is an older model.)

Thanks, I will have to use the later as we are running under a busybox environment.

Quote:

Originally Posted by David the H.

But if the above is ok, grep can do the same thing more efficiently.

Code:

wget downloadfile -O/dev/null |& grep "%"

Or if you really want to grab only the percentage numbers and exclude the rest of the line, then you could use something like this:

Code:

wget downloadfile -O/dev/null |& grep -o "[0-9]\+%"

Ok, I see. Grep is working like this using the above:

Code:

0%
1%
3%

This is fine, but now do I get the output to dbus-send exe?

Is there an easier way of doing this?

Thanks.

crts · 11-03-2010, 04:35 AM

Hi,

I am not familiar with dbus-send, but after some small research I think dbus-send is not able to read STDIN. So you might want to try

Code:

wget [] | sed [] | xargs dbus-send []

However, you might have to pipe the complete message to it, i.e. use sed to compose the whole message that you want dbus-send to send to your app. Not 100% sure about that.
Example:

Code:

wget [] | sed -r 's/.*([0-9]{1,3}%).*/Progress: \1/' | xargs dbus-send []

[UNTESTED]

RichardS · 11-05-2010, 03:45 AM

Hi crts,

Thanks for the input. Unfortunately it does not quite work. I have tried a simple:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | sed -r 's/.*([0-9]{1,3}%).*/Progress: \1/'

I get the "Progress 1%", "Progress 3%", etc. However, if I do:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | sed -r 's/.*([0-9]{1,3}%).*/Progress: \1/' | xargs echo

Nothing is printed at all.

Any idea's?

Thanks.

crts · 11-05-2010, 06:15 AM

Quote:

Originally Posted by RichardS

Hi crts,

Thanks for the input. Unfortunately it does not quite work. I have tried a simple:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | sed -r 's/.*([0-9]{1,3}%).*/Progress: \1/'

I get the "Progress 1%", "Progress 3%", etc. However, if I do:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | sed -r 's/.*([0-9]{1,3}%).*/Progress: \1/' | xargs echo

Nothing is printed at all.

Any idea's?

Thanks.

Yeah, that's a tricky one. Try it this way:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | sed -nru '/[0-9]{1,3}%/ {s/.*([0-9]{1,3}%).*/\1/;p}' | xargs -L 1 echo

When I tested it I did not use the -0/dev/null option. Not sure if this will have any impact.

RichardS · 11-05-2010, 06:31 AM

Hi crts,

Thanks for the help. It "sort-of" works now

I have two problems with the code

1. it does not print 100%, it prints only single digits (1,2,..9,0,1,2..)

2. dbus-send gives me:
dbus-send: Data item "0%" is badly formed

Is there any way of finding what the parameters being passed to dbus are?

The line I am using is:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | sed -nru '/[0-9]{1,3}%/ {s/.*([0-9]{1,3}%).*/\1/;p}' | xargs -L 1 dbus-send --system --dest=com.file.progress --type=method_call --print-reply /com/file/progress com.file.progress.percent

Thanks.

crts · 11-05-2010, 07:15 AM

Quote:

Originally Posted by RichardS

Hi crts,

Thanks for the help. It "sort-of" works now

I have two problems with the code

1. it does not print 100%, it prints only single digits (1,2,..9,0,1,2..)

2. dbus-send gives me:
dbus-send: Data item "0%" is badly formed

Is there any way of finding what the parameters being passed to dbus are?

The line I am using is:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | sed -nru '/[0-9]{1,3}%/ {s/.*([0-9]{1,3}%).*/\1/;p}' | xargs -L 1 dbus-send --system --dest=com.file.progress --type=method_call --print-reply /com/file/progress com.file.progress.percent

Thanks.

Ok,
fix the first issue by changing the substitution command from

Code:

s/.*([0-9]{1,3}%).*/\1/

to

Code:

s/.*[^0-9]([0-9]{1,3}%).*/\1/

About dbus-send:
As I said, I am not familiar with dbus-send, so I hope you know where you have to put the piped data. By default xargs simply appends it at the end.
You can change this by

Code:

# example
xargs -I{} echo Progress is {} ....

Now the output would be

Code:

Progress is 0% ....

So the {} is replaced by whatever comes through the pipe.
Maybe you need to only send a number, i.e. omit the '%' or ...
Just post an example of how you would send data to your app without using the pipe, e.g. how would you just send '12%' to your app?

David the H. · 11-05-2010, 07:28 AM

Quote:

1. it does not print 100%, it prints only single digits (1,2,..9,0,1,2..)

That's because of the greediness of .* in regex. In your expression:

Code:

.*([0-9]{1,3}%).*

The first .* consumes the longest possible match, which includes the first one or two digits of the percentage if they exist, before the next section kicks in. So the [0-9] only matches the last digit.

You need some way to stop the greediness. In this case, simply including the space before the number appears to work.

Code:

.* ([0-9]{1,3}%).*

But why are you back to using sed for this? It's only necessary if you need to modify the output string in some way, such as including some additional text. If you really only need the percentage, use the grep version I gave before.

I know nothing about dbus, but "badly formed" sounds like there's something in the input string that it doesn't like. I wonder if the % mark is restricted and needs to be escaped or something. In which case you may be back to using sed to modify the line. Try testing it without the percent sign. Just move it outside the parentheses in sed. Here's my slightly modified version.

Code:

sed -nru '/[0-9]%/ s/.* ([0-9]+)%.*/\1/p'

Also, I only put /dev/null in there as a testing device. It simply dumps the downloaded file into the bit-bucket instead of saving it. You probably don't want to do that in the final version. It shouldn't affect anything else.

RichardS · 11-05-2010, 11:01 AM

Hi David, crts,

Thanks for all the help, it has been really helpful - and it is working

The final line I have used is:

Code:

wget http://.../file.bin -O/dev/null 2>&1 | grep -o "[0-9]\+%" | xargs -I{} dbus-send --system --dest=com.file.progress --type=method_call --print-reply /com/file/progress com.file.progress.percent string:{}

It turned out that I had missed the string: at the end there and that is the reason for my badly formed response from dbus.

But thanks again for all the help, it has been much informative.

Thanks.