Batch download linked assets from PDF files in website

ericlindellnyc · 10-22-2021, 08:28 PM

I have downloaded a website that has many PDFs on it for download. I used

Code:

wget -r -l 2 -N -t1 --mirror --convert-links --adjust-extension --page-requisites --no-parent --no-check-certificate http://mileswmathis.com/

to get the PDFs.
I thought by adding -l 2 it would also retrieve the media assets linked to from the PDFs, but it did not. It will do this from an HTML file, but not a PDF.
There are great PDFs on this site, but I would like the linked assets also, such as when it links to an image or video file -- or anything else, for that matter.
I don't know how to batch sift through the PDFs to find the links and download linked content.
Ideas?
Thank you.

b1bb2 · 10-23-2021, 01:21 AM

I did a similar project. Here are the project links to get you started. If needed, I can look for more details later. Sorry, I do not remember the commands offhand.
b1bb2.com/b/Downloads/application/makehumancommunity.org/sites/default/files/20210901/assetts/CC0/CC0.htm
http://www.makehumancommunity.org/fo...p?f=20&t=19858

If I were doing your project, first convert pdf to text. Make sure all links are not split between lines. There is a bash command that copies each line that has certain text. A link is easy to detect.

shruggy · 10-23-2021, 08:10 AM

To preserve links, you may convert PDF to HTML first, say, with pdftohtml from poppler-utils.

But AFAICT, many of the links are to his other essays, so if you've downloaded them all in the same directory, you'll just have to adjust them by removing http://milleswmathis.com/ from them.