[wget] Syntax to download single HTML page and its dependencies?

littlebigman · 04-27-2020, 01:25 PM

Hello,

I need to find a way to download a single web page and its dependencies before calling pandoc to turn it into an EPUB file.

The following works… but not when I add "-O local.html" to rename the file:

Code:

wget -E -H -k -K -p -e robots=off -O local.html https://www.acme.com/remote.html

Does someone know?

Thank you.

scasey · 04-27-2020, 02:23 PM

Quote:

Originally Posted by littlebigman

Hello,

I need to find a way to download a single web page and its dependencies before calling pandoc to turn it into an EPUB file.

The following works… but not when I add "-O local.html" to rename the file:

Code:

wget -E -H -k -K -p -e robots=off -O local.html https://www.acme.com/remote.html

Does someone know?

Thank you.

That code snippet appears to have a zero rather than an uppercase. Maybe use the long version of the option to be sure:

Code:

--output-document=file

I also note in the wget man page:

Quote:

Similarly, using -r or -p with -O may not work as you expect: Wget won't just download the first file to file and then download the rest to their normal names: all downloaded content will be placed in file. This was
disabled in version 1.11, but has been reinstated (with a warning) in 1.11.2, as there are some cases where this behavior can actually have some use.

Note that a combination with -k is only permitted when downloading a single document, as in that case it will just convert all relative URIs to external ones; -k makes no sense for multiple URIs when they're all being
downloaded to a single file; -k can be used only when the output is a regular file.

That said, what does (or doesn't) happen for you?

Another option is not use -O and simply redirect the output to a file

Code:

wget -E -H -k -K -p -e robots=off  https://www.acme.com/remote.html > local.html

(also mentioned in the man page)

littlebigman · 04-27-2020, 02:34 PM

Here's the error message:

Code:

wget -E -H -k -K -p -e robots=off --output-document=local.html https://www.acme.com/mypage.html

Cannot specify both -k or --convert-file-only and -O if multiple URLs are given, or in combination with -p or -r. See the manual for details.

The redirection creates an empty file at the root, and I can find no local.html anywhere.

scasey · 04-27-2020, 02:44 PM

Quote:

Originally Posted by littlebigman

Here's the error message:

Code:

wget -E -H -k -K -p -e robots=off --output-document=local.html https://www.acme.com/mypage.html

Cannot specify both -k or --convert-file-only and -O if multiple URLs are given, or in combination with -p or -r. See the manual for details.

The redirection creates an empty file at the root, and I can find no local.html anywhere.

That's a pretty clear error message, and is what the man page section I copied says as well. It simply won't work. -k and -O can't be used in the same command. Try it without the -k, or the -O (which you said worked, yes?)
What happens without the -O? Several files? Can you cat them together?

littlebigman · 04-27-2020, 02:55 PM

Without "k-", I get just the HTML file, and nothing else, meaning I only get text and no pictures.

I'll just rename the file in a second command since wget seems unable to do it.

Thank you.

ondoho · 04-29-2020, 06:55 AM

You do know that most browsers can do this?
Caveat: the "dependencies" are placed in a separate folder that needs to travel with the original downloaded file.
But I believe you will have the same caveat with wget, or curl for that matter.
It might be possible to convert the whole pile into something that is one file only, but that goes beyond mere downloading.