wget: download if file size does not match any file in folder
I have the download link to pdf files, stored in links.txt
Some of the pdf files have same content (with different names, but same size). I do not want to download same content several times. That means I want to download all the files that have unique file size.
Is it possible to do this with wget?
For example, 1.pdf is downloaded. Next, 2.pdf is checked for the size, if there is any file in the folder which is the same size of 2.pdf, then 2.pdf is not downloaded. Otherwise, it is downloaded to the folder, and 3.pdf is checked and so on.
Please let me know if you have any suggestions.
Thank you for your help.
|