Automated FTP script to grab ONLY newest dated files

kaiserbeto · 10-19-2006, 06:02 PM

Hi All,

I'm trying to automate the FTP download of logs that are newly generated every day.

I have everything setup as far as .netrc automation goes, but I don't know how to set it up where it will only get the newest daily log.

If it were a bash script i'd do omething like ...
file='ls -latr | tail -1 | grep tar.gz'
get $file

But I'm not sure how to pass these similiar commands through FTP.

The format is log.20061018.tar.gz ... so I just want it to download each day's new file. (20061019, 20061020, etc)

Any ideas how to set this part up?

BrianK · 10-20-2006, 12:55 AM

First off, I recommend using the ncftp suite for all things ftp via a command line or shell script. Almost all distros have an ncftp package in them somewhere & if they don't, it's freely available and very easy to install from their website.

Option 1:
Do you know the date for which you'll be downloading? In other words, will it always be a file called log.`today's date`.tar.gz? If so, then you can always just get that particular file:

ncftpget -u user -p pass remote-host /local/save/folder /path/on/ftp/server/log.`date +%Y``date +%m``date +%d`.tar.gz

Option 2:
use ncftpls to find the most recent file, then ncftpget to get it...

Code:

#!/bin/tcsh -f

set file = `ncftpls -x "latr" ftp://host/path | grep tar.gz | tail -n1`
echo "Found $file"
ncftpget remote-host ./ $file

kaiserbeto · 10-20-2006, 01:09 AM

Thank you for the assistance BrianK ... I'm unfamiliar with tcsh, but i'll play around with it a bit ... either way, it's very helpful. You're right about ncftp ... much more versatile than the standard ftp client.

As for the today's date problem, i forgot to mention there's a 48 hour lag time in when the logs get posted.

So, logs for the 17th won't be posted until some arbitrary time on the 19th ... hence why I couldn't use the exact date/time variables and had to find a way so that only the newest log was downloaded. But either way, every file is the exact same format ... so, on the 19th I will get a log called log.20061017.tar.gz. This is something determined by our CDN, and I have no control over it.

Isn't there a "get" flag you can use to only grab the newest file added?

BrianK · 10-20-2006, 01:30 AM

Quote:

Originally Posted by kaiserbeto

Thank you for the assistance BrianK ... I'm unfamiliar with tcsh, but i'll play around with it a bit ... either way, it's very helpful. You're right about ncftp ... much more versatile than the standard ftp client.

As for the today's date problem, i forgot to mention there's a 48 hour lag time in when the logs get posted.

So, logs for the 17th won't be posted until some arbitrary time on the 19th ... hence why I couldn't use the exact date/time variables and had to find a way so that only the newest log was downloaded. But either way, every file is the exact same format ... so, on the 19th I will get a log called log.20061017.tar.gz. This is something determined by our CDN, and I have no control over it.

Isn't there a "get" flag you can use to only grab the newest file added?

let me rewrite that script in bash....

Code:

#!/bin/bash

file = `ncftpls -x "latr" ftp://host/path | grep tar.gz | tail -n1`
echo "Found $file"
ncftpget remote-host ./ $file

(you may even be able to get away without using the backticks if that makes ya feel better)

AFAIK, There's no flag on the get side... All the non-interactive ncftp tools are pretty dumb (by design). Use them in conjunction with one another to get done what you want to get done.

edited to add:
If you're more comfortable, you can always assign the day of the date to a variable then subtract 2. Just be sure to padzero the date for those days < 10. That said, it sure seems easier to ncftpls & get the most recent file.

kaiserbeto · 10-20-2006, 02:46 AM

sweet ... it worked more or less ... I had to add an awk '{print 9}' to it so it would grab the column with the file instead of the entire output line of ls -latr. thanks again though!