LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-10-2006, 11:14 PM   #1
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Rep: Reputation: 15
Scripting and FTP


Hi everyone, the learning curve for Linux is steep but I am hopefully learning more each day.

Here's what I would like to accomplish: automatically ftp to a site and retrieve files from directories which change each day to a name derived from the current date, i.e. 20060110 for today, Jan 10th.

In my way of thinking, there should be a way to write a script which does this which can be executed as a cron job.

I've put the necessary login information for the ftp server into .netrc.

I've tried using:

set CURDAY = 'date -u +%Y%m%d'

but I cannot figure out how to insert CURDAY into the sequence of ftp commands.

Is this approach doable? Is there something built-in or easier that I don't know about?
 
Old 01-10-2006, 11:20 PM   #2
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738
Hope I am reading you correctly. You want to look for a file named CURDAY...

ftp <site>
##login junk##
cd <directory of interest>
get $CURDAY
bye

Am I going in the right direction?? What embellishements do we need?---eg might these files be in several different directories?
 
Old 01-10-2006, 11:28 PM   #3
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Original Poster
Rep: Reputation: 15
CURDAY is actually a directory on the remote server.

My first question: where would CURDAY be defined in your example. Also, I attempted:

cd /pub/data/nccf/com/nam/prod/nam.$CURDAY

Once you enter the ftp program, can defined variables like CURDAY be recognized?
 
Old 01-11-2006, 02:06 AM   #4
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738
Sorry, I misread the original post.
Variables can be referenced in any valid scripting command.
A valid scripting command can generate any command required by the ftp program---since the script simply sends stuff to stdout (or wherever you might redirect it.
Another way to say it: If you can type it in a terminal, you can put it in a script.

What happens with your cd command if you just type it in a terminal? And if you run it in a script?
One possibility is you may need to say: "echo "cd /etc/etc/etc""
 
Old 01-11-2006, 03:15 AM   #5
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
Try "wget". It's a handy downloading tool that you can easily integrate into a cron job.
This avoids the whole problem of having to write an FTP script.

For instance, make a script like:
#!/bin/bash
CURDAY=`date -u +%Y%m%d`
cd /path/for/downloaded/files
wget -N -a your_log_file --ftp-user user --ftp-password pwd ftp://ftp/path/${CURDAY}/

And add that script to your cron settings (ie via "crontab -e").
Please note that the actual options of wget depend on the version you're using. Look at "man wget".
 
Old 01-11-2006, 10:05 AM   #6
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Original Poster
Rep: Reputation: 15
Thanks for the input. Much to consider and try. I'm sure I'll get it to work.
 
Old 01-11-2006, 10:09 AM   #7
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Original Poster
Rep: Reputation: 15
timmeke,

What is the significance of the curly brackets around $CURDAY?
 
Old 01-11-2006, 11:23 AM   #8
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
The curly braces tell the shell that the variable's name is CURDAY. This avoids confusion, when the
variable is part of a larger string.

For instance, consider the following silly example:
Time="12:00"
TimeUTC="13:00".
echo "The time is now: $TimeUTC"
Now, which variable should the shell print? $Time or $TimeUTC?
It will most likely use $TimeUTC in this case, but it could also think it's supposed to print "$Time" followed by the literal string "UTC".

To avoid such confusion, put curly braces around the name of your variable:
${Time}UTC
or ${TimeUTC}
In both cases, it's clear to the shell which of the 2 you want to use.

That's the only purpose of the curly braces.
For more info, check out
man bash
(or the man page of your shell, if you're not using the bash shell).
 
Old 01-11-2006, 07:19 PM   #9
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Original Poster
Rep: Reputation: 15
This is what ended up working:

#!/bin/bash
set CURDAY = `date -u +%Y%m%d`
#echo "$CURDAY"
cd data/ldm/gempak/model
wget -N -v -a wx_log --ftp-user anonymous --ftp-password ted@ceaselesswind.com ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.${CURDAY}/nam.t12z.awip1218.tm00

Final question, shouldn't I be able to put wildcards '??' where it says '18' in the filename? The file numbers range from '00' to '84' This kicks back a "no match" message when I try it.

Thanks for the help folks!
 
Old 01-11-2006, 10:33 PM   #10
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Original Poster
Rep: Reputation: 15
Never mind, recursion will set you free!
 
Old 01-12-2006, 03:09 AM   #11
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
I think wget provides you with some filename globbing options.
When not downloading recursively, you may be able to use ?? and * wildcards, but I'm not sure.

The -A option to specify a filter for filenames may be your friend too.
 
Old 01-12-2006, 04:03 PM   #12
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Original Poster
Rep: Reputation: 15
I'm all over the -A option. It works like a charm!
 
Old 01-12-2006, 10:53 PM   #13
stickman
Senior Member
 
Registered: Sep 2002
Location: Nashville, TN
Posts: 1,552

Rep: Reputation: 53
Quote:
Originally Posted by ceaseless
What is the significance of the curly brackets around $CURDAY?
Curly braces are handy if your variable is in a text string. For example, let's say you have filenames with the day embedded in them (ie monfile, tuefile, wedfile) and you set DAY as the variable. You would use ${DAY}file as the pattern. If you didn't use the curly brackets then your script probably would not work since it would be using the variable DAYfile (which is most likely unset).
 
Old 01-13-2006, 12:34 AM   #14
ceaseless
LQ Newbie
 
Registered: Jan 2006
Posts: 24

Original Poster
Rep: Reputation: 15
With reference to my posted script above, a question.

Why would a system reboot cause the:

set CURDAY = `date -u +%Y%m%d`

line to stop returning the proper string?

Only after I ran a script to set environmental variables for the programs I'm using did CURDAY start returning the correct string.

I strongly suspect it has something to do with bash vs csh. That's why I'm posting as a newbie!
 
Old 01-13-2006, 02:45 AM   #15
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
In Bash/sh, you can simply do:
CURDAY=`date -u +%Y%m%d`
Bash's "set" command doesn't assign values to variables, using the "name=value" syntax.
I tried (in Bash):
set a=5
echo $a
returned empty string.

In csh, however, "set" can assign values to variables (using "set name=value" syntax).

So, you need to decide which of the 2 shells you want to use.
If you want to make sure your script runs under a specific shell, you need to use the "shebang syntax".
This means that you need to add:
#!/path/to/shell
at the top.
Examples:
#!/bin/bash
or
#!/bin/csh

Make sure that all the commands in the script use the correct syntax. Command syntax can be significantly different between shells.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
FTP Server Up and running... how do I hide ftp users from local login screen? joe1031 Mandriva 2 03-18-2005 05:24 PM
ftp scripting doctorwebbox Linux - General 7 01-21-2005 01:01 PM
problem with ftp on mandrake 10.1 Official, ftp speeds system wide (anybody noticed?) equinox Mandriva 15 11-10-2004 03:07 PM
Scripting of routine ftp download BioHak Linux - Newbie 0 03-02-2003 09:17 PM
how can I restrict ftp users listing files from a pure-ftp server adrianmak Linux - Networking 2 12-31-2002 09:23 AM


All times are GMT -5. The time now is 06:23 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration