[SOLVED] [wget] Display output with "-i" option?

littlebigman · 07-08-2023, 04:26 AM

Hello,

For basic web crawling, I'd like to feed wget a list of URLs, and then massage the data with grep + sed.

It works fine with a single URL, but I get no output on the screen when calling it with a list of URLs (-i switch):

Code:

wget -i list.txt -qO - | grep -Poha "some_pattern" | sed -r "s@some_pattern@\1 \2@"

Is there a work-around?

Thank you.

pan64 · 07-08-2023, 04:35 AM

here you can find some ideas: https://stackoverflow.com/questions/...in-a-text-file

lvm_ · 07-08-2023, 05:12 AM

The correct syntax for redirecting to stdout is "wget -O -", not "wget -"

Turbocapitalist · 07-08-2023, 05:14 AM

Quote:

Originally Posted by littlebigman

It works fine with a single URL, but I get no output on the screen when calling it with a list of URLs (-i switch):

You might also need to use the --unbuffered and --line-buffered options with sed and grep, though if you are dealing with HTML or XML you will need a proper parser instead. For simple stuff there is xmlstarlet and for more complex stuff there is Perl or Python.

littlebigman · 07-08-2023, 05:21 AM

Thanks all.