LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
LinkBack Search this Thread
Old 10-19-2006, 06:02 PM   #1
kaiserbeto
LQ Newbie
 
Registered: Oct 2006
Posts: 10

Rep: Reputation: 0
Automated FTP script to grab ONLY newest dated files


Hi All,

I'm trying to automate the FTP download of logs that are newly generated every day.

I have everything setup as far as .netrc automation goes, but I don't know how to set it up where it will only get the newest daily log.

If it were a bash script i'd do omething like ...
file='ls -latr | tail -1 | grep tar.gz'
get $file

But I'm not sure how to pass these similiar commands through FTP.

The format is log.20061018.tar.gz ... so I just want it to download each day's new file. (20061019, 20061020, etc)

Any ideas how to set this part up?

Last edited by kaiserbeto; 10-19-2006 at 06:04 PM.
 
Old 10-20-2006, 12:55 AM   #2
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Rep: Reputation: 51
First off, I recommend using the ncftp suite for all things ftp via a command line or shell script. Almost all distros have an ncftp package in them somewhere & if they don't, it's freely available and very easy to install from their website.

Option 1:
Do you know the date for which you'll be downloading? In other words, will it always be a file called log.`today's date`.tar.gz? If so, then you can always just get that particular file:

ncftpget -u user -p pass remote-host /local/save/folder /path/on/ftp/server/log.`date +%Y``date +%m``date +%d`.tar.gz

Option 2:
use ncftpls to find the most recent file, then ncftpget to get it...

Code:
#!/bin/tcsh -f

set file = `ncftpls -x "latr" ftp://host/path | grep tar.gz | tail -n1`
echo "Found $file"
ncftpget remote-host ./ $file
 
Old 10-20-2006, 01:09 AM   #3
kaiserbeto
LQ Newbie
 
Registered: Oct 2006
Posts: 10

Original Poster
Rep: Reputation: 0
Thank you for the assistance BrianK ... I'm unfamiliar with tcsh, but i'll play around with it a bit ... either way, it's very helpful. You're right about ncftp ... much more versatile than the standard ftp client.

As for the today's date problem, i forgot to mention there's a 48 hour lag time in when the logs get posted.

So, logs for the 17th won't be posted until some arbitrary time on the 19th ... hence why I couldn't use the exact date/time variables and had to find a way so that only the newest log was downloaded. But either way, every file is the exact same format ... so, on the 19th I will get a log called log.20061017.tar.gz. This is something determined by our CDN, and I have no control over it.

Isn't there a "get" flag you can use to only grab the newest file added?
 
Old 10-20-2006, 01:30 AM   #4
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Rep: Reputation: 51
Quote:
Originally Posted by kaiserbeto
Thank you for the assistance BrianK ... I'm unfamiliar with tcsh, but i'll play around with it a bit ... either way, it's very helpful. You're right about ncftp ... much more versatile than the standard ftp client.

As for the today's date problem, i forgot to mention there's a 48 hour lag time in when the logs get posted.

So, logs for the 17th won't be posted until some arbitrary time on the 19th ... hence why I couldn't use the exact date/time variables and had to find a way so that only the newest log was downloaded. But either way, every file is the exact same format ... so, on the 19th I will get a log called log.20061017.tar.gz. This is something determined by our CDN, and I have no control over it.

Isn't there a "get" flag you can use to only grab the newest file added?
let me rewrite that script in bash....

Code:
#!/bin/bash

file = `ncftpls -x "latr" ftp://host/path | grep tar.gz | tail -n1`
echo "Found $file"
ncftpget remote-host ./ $file


(you may even be able to get away without using the backticks if that makes ya feel better)

AFAIK, There's no flag on the get side... All the non-interactive ncftp tools are pretty dumb (by design). Use them in conjunction with one another to get done what you want to get done.



edited to add:
If you're more comfortable, you can always assign the day of the date to a variable then subtract 2. Just be sure to padzero the date for those days < 10. That said, it sure seems easier to ncftpls & get the most recent file.

Last edited by BrianK; 10-20-2006 at 01:33 AM.
 
Old 10-20-2006, 02:46 AM   #5
kaiserbeto
LQ Newbie
 
Registered: Oct 2006
Posts: 10

Original Poster
Rep: Reputation: 0
sweet ... it worked more or less ... I had to add an awk '{print 9}' to it so it would grab the column with the file instead of the entire output line of ls -latr. thanks again though!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
grab FTP output in bash script bokini Linux - General 2 02-03-2006 02:11 PM
script for display a clock from dated images osio Programming 0 01-14-2006 10:38 AM
automated ftp Anthraxnz Linux - Newbie 4 10-15-2005 09:36 PM
download newest files on ftp exodist Linux - Networking 0 03-06-2004 03:14 PM
Cant grab files from my other computer? Lason Linux - Networking 3 11-06-2003 10:42 PM


All times are GMT -5. The time now is 05:13 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration