Scripting and FTP
 

Hi everyone, the learning curve for Linux is steep but I am hopefully learning more each day.

Here's what I would like to accomplish: automatically ftp to a site and retrieve files from directories which change each day to a name derived from the current date, i.e. 20060110 for today, Jan 10th.

In my way of thinking, there should be a way to write a script which does this which can be executed as a cron job.

I've put the necessary login information for the ftp server into .netrc.

I've tried using:

set CURDAY = 'date -u +%Y%m%d'

but I cannot figure out how to insert CURDAY into the sequence of ftp commands.

Is this approach doable? Is there something built-in or easier that I don't know about?

Hope I am reading you correctly. You want to look for a file named CURDAY...

ftp <site>
##login junk##
cd <directory of interest>
get $CURDAY
bye

Am I going in the right direction?? What embellishements do we need?---eg might these files be in several different directories?

CURDAY is actually a directory on the remote server.

My first question: where would CURDAY be defined in your example. Also, I attempted:

cd /pub/data/nccf/com/nam/prod/nam.$CURDAY

Once you enter the ftp program, can defined variables like CURDAY be recognized?

Sorry, I misread the original post.
Variables can be referenced in any valid scripting command.
A valid scripting command can generate any command required by the ftp program---since the script simply sends stuff to stdout (or wherever you might redirect it.
Another way to say it: If you can type it in a terminal, you can put it in a script.

What happens with your cd command if you just type it in a terminal? And if you run it in a script?
One possibility is you may need to say: "echo "cd /etc/etc/etc""

Try "wget". It's a handy downloading tool that you can easily integrate into a cron job.
This avoids the whole problem of having to write an FTP script.

For instance, make a script like:
#!/bin/bash
CURDAY=`date -u +%Y%m%d`
cd /path/for/downloaded/files
wget -N -a your_log_file --ftp-user user --ftp-password pwd ftp://ftp/path/${CURDAY}/

And add that script to your cron settings (ie via "crontab -e").
Please note that the actual options of wget depend on the version you're using. Look at "man wget".

Thanks for the input. Much to consider and try. I'm sure I'll get it to work.

timmeke,

What is the significance of the curly brackets around $CURDAY?

The curly braces tell the shell that the variable's name is CURDAY. This avoids confusion, when the
variable is part of a larger string.

For instance, consider the following silly example:
Time="12:00"
TimeUTC="13:00".
echo "The time is now: $TimeUTC"
Now, which variable should the shell print? $Time or $TimeUTC?
It will most likely use $TimeUTC in this case, but it could also think it's supposed to print "$Time" followed by the literal string "UTC".

To avoid such confusion, put curly braces around the name of your variable:
${Time}UTC
or ${TimeUTC}
In both cases, it's clear to the shell which of the 2 you want to use.

That's the only purpose of the curly braces.
For more info, check out
man bash
(or the man page of your shell, if you're not using the bash shell).

This is what ended up working:

#!/bin/bash
set CURDAY = `date -u +%Y%m%d`
#echo "$CURDAY"
cd data/ldm/gempak/model
wget -N -v -a wx_log --ftp-user anonymous --ftp-password ted@ceaselesswind.com ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.${CURDAY}/nam.t12z.awip1218.tm00

Final question, shouldn't I be able to put wildcards '??' where it says '18' in the filename? The file numbers range from '00' to '84' This kicks back a "no match" message when I try it.

Thanks for the help folks!

Never mind, recursion will set you free!

I think wget provides you with some filename globbing options.
When not downloading recursively, you may be able to use ?? and * wildcards, but I'm not sure.

The -A option to specify a filter for filenames may be your friend too.

I'm all over the -A option. It works like a charm!

Curly braces are handy if your variable is in a text string. For example, let's say you have filenames with the day embedded in them (ie monfile, tuefile, wedfile) and you set DAY as the variable. You would use ${DAY}file as the pattern. If you didn't use the curly brackets then your script probably would not work since it would be using the variable DAYfile (which is most likely unset).

With reference to my posted script above, a question.

Why would a system reboot cause the:

set CURDAY = `date -u +%Y%m%d`

line to stop returning the proper string?

Only after I ran a script to set environmental variables for the programs I'm using did CURDAY start returning the correct string.

I strongly suspect it has something to do with bash vs csh. That's why I'm posting as a newbie!

In Bash/sh, you can simply do:
CURDAY=`date -u +%Y%m%d`
Bash's "set" command doesn't assign values to variables, using the "name=value" syntax.
I tried (in Bash):
set a=5
echo $a
returned empty string.

In csh, however, "set" can assign values to variables (using "set name=value" syntax).

So, you need to decide which of the 2 shells you want to use.
If you want to make sure your script runs under a specific shell, you need to use the "shebang syntax".
This means that you need to add:
#!/path/to/shell
at the top.
Examples:
#!/bin/bash
or
#!/bin/csh

Make sure that all the commands in the script use the correct syntax. Command syntax can be significantly different between shells.