[SOLVED] wget/curl file that has been uploaded within the last 2-3 days?

orangepeel190 · 07-24-2020, 06:13 AM

Hi all,

I am seeking thought what line I could add to a download script. I am seeking to only download $file that was uploaded/modified within the last 2-3 days?

The reason is that every week, the same filename to $file so when my script seeks out this file, it has no idea if the file is the latest version (modified/uploaded within the last few days) or if it is the previous/old version of the file.

I am thinking of using something like:

find . -mtime -2 for two days prior

Then there is ls -ltr but I am struggling to work out what to grep/pipe as output to ensure that it only downloads the latest file (within the last 2-3 days_ when the script runs

the download command is similar to:

Code:

  echo "Getting file from Webserver"
  /usr/bin/curl -SkLo $file http://some.webite/here

I am having difficulties is working out what commands put in place to verify that the file that is about to be downloaded has been uploaded in the past 2-3 days. I am assuming this "magic line" would go after the curl line .....curl -SkLo $file http://some.webite/here | (HERE)

I am reading something about curl --time-cond or -z. Will that check the date details of the file on the server (to be potentially downloaded) or simply compare against an existing file?

Am I on the right path?

Turbocapitalist · 07-24-2020, 06:37 AM

You'd need to try tweaking the HTTP Request headers so that it sends an if-modified-since field along with the actual request. Both wget and curl can do that, but you'll have to dig in the manual pages. However, not all servers will respect that header.

orangepeel190 · 07-24-2020, 06:45 AM

Hi Turbo. Thanks for your reply.

What about something similar to:

Code:

findme="`date --date="-3 days ago"`-restoffilenamehere"
file="alwaysthesamename"

curl -SkLoz $findme $file http://some.webite/here

Would that best the best way to get the file name (to compare to - 3 days earlier) into the download curl scripting or correct syntax? I am unable to get this to work....

Turbocapitalist · 07-24-2020, 06:55 AM

Untested:

Code:

curl --header "If-Modified-Since: $(date -d '-3 days' -u +'%a, %d %b %Y %T GMT')" \
        -SkLoz http://some.webite/here/$file

See "man curl" and mind the quotes and see the link in the previous post.

orangepeel190 · 07-24-2020, 10:40 PM

Thanks Turbo!!!

Ive implemented the correct syntax (thanks to your help) into my script and it adopted it to work as desired!

Appreciate your assistance!!