[SOLVED] Removing spaces from URL in bash script

PatrickDickey · 12-22-2011, 07:32 PM

Hi everyone,

I'm trying to create a bash script that will take the current date, subtract 350 days from it, and retrieve a table of stock prices for the time between the two dates.

I've got everything working up to the url. Right now, my script just echoes the url back to me. It works, but there are spaces before the dates provided by the variables.

Here is my script so far:

Code:

#!/bin/bash

#Declaration of variables

strtmo=$(date --date='350 days ago' '+ %m')
if [ "$strtmo" -eq 1 ]; then
    startmo=0$(($strtmo - 1))
else
    startmo=$(($strtmo -1))
fi
startday=$(date --date='350 days ago' '+ %d')
startyr=$(date --date='350 days ago' '+ %y')
endm=$(date '+ %m')
if [ "$endm" -eq 1 ]; then
    endmo=0$(($endm - 1))
else
    endmo=$(($endm - 1))
fi
endday=$(date '+ %d')
endyr=$(date '+ %y')

echo the dates will be $startmo $startday $startyr and $endmo $endday $endyr
sturl="http://ichart.finance.yahoo.com/table.csv?s=%5EGSPC&a="$startmo"&b="$startday"&c="$startyr"&d="$endmo"&e="$endday"&f="$endyr"&g=d&ignore=.csv"
echo $sturl

And here are the results of running it.

Quote:

the dates will be 00 06 11 and 11 22 11
http://ichart.finance.yahoo.com/tabl...5EGSPC&a=00&b= 06&c= 11&d=11&e= 22&f= 11&g=d&ignore=.csv[/url]

What it should look like is

Quote:

the dates will be 00 06 11 and 11 22 11
http://ichart.finance.yahoo.com/tabl...=d&ignore=.csv

I'm not sure what to change, or whether it will matter when I change the echo to a wget statement. Ultimately, it will get two files, and rename them (because they will be named table.csv?s=^GSPC and table.csv?s=^IXIC. I've got everything else figured out (I think), but the spaces in the url will cause me problems.

I plan on setting this script up as a cronjob that will run every night at 1:00 am. If this would be easier to do in python or some other language, I would appreciate some assistance in making the script (as I don't know how to program in Python).

Thanks, and have a great day

Patrick.

chrism01 · 12-22-2011, 08:00 PM

You are generating extra spaces in front of those nums; put 'set -xv'

Code:

#!/bin/bash
set -xv

at the top of your prog and you'll see

PatrickDickey · 12-22-2011, 09:47 PM

Quote:

Originally Posted by chrism01

You are generating extra spaces in front of those nums; put 'set -xv'

Code:

#!/bin/bash
set -xv

at the top of your prog and you'll see

So it ended up being the '+ %m' (and %d and %y) portions. They needed to be '+%m' '+%d' and '%y' respectively. Good to know for future references. The script works now.

For anyone interested in this, here is the entire script

Code:

#!/bin/bash
# set -xv

#Declaration of variables

strtmo=$(date --date='350 days ago' '+%m')
if [ "$strtmo" -eq 1 ]; then
    startmo=0$(($strtmo - 1))
else
    startmo=$(($strtmo -1))
fi
startday=$(date --date='350 days ago' '+%d')
startyr=$(date --date='350 days ago' '+%y')
endm=$(date '+ %m')
if [ "$endm" -eq 1 ]; then
    endmo=0$(($endm - 1))
else
    endmo=$(($endm - 1))
fi
endday=$(date '+%d')
endyr=$(date '+%y')

# Remove any csv files prior to running the script.

rm *.csv

# Get the files.

sturl='http://ichart.finance.yahoo.com/table.csv?s=%5EGSPC&a='$startmo'&b='$startday'&c='$startyr'&d='$endmo'&e='$endday'&f='$endyr'&g=d&ignore=.csv'
sturl2="http://ichart.finance.yahoo.com/table.csv?s=%5EIXIC&a="$startmo"&b="$startday"&c="$startyr"&d="$endmo"&e="$endday"&f="$endyr"&g=d&ignore=.csv"
wget $sturl
wget $sturl2

# Rename the csv files so my macro can find them.

mv table.csv?s=^GSPC* GSPC.csv
mv table.csv?s=^IXIC* IXIC.csv

The purpose of the script is to get the historical prices for the S&P 500 and NASDAQ stock exchanges (the previous 200+ days) and save them as a csv file. That way, these can be imported into Calc (or Excel on Windows) and used to calculate the 50 day and 200 day moving averages of the stock prices.

Have a great day

Patrick.

David the H. · 12-23-2011, 09:14 AM

You should quote all your variable expansions, and any other string that can contain whitespace or shell-reserved characters.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

Also, this section (and others like it) has a problem:

Code:

strtmo=$(date --date='350 days ago' '+%m')
if [ "$strtmo" -eq 1 ]; then
    startmo=0$(($strtmo - 1))
else
    startmo=$(($strtmo -1))
fi

"date +%m" outputs the month in the zero-padded format (01-12).

But bash treats numbers with leading zeros as octal, so when your script reaches month 08, you're going to get a syntax error.

http://mywiki.wooledge.org/ArithmeticExpression

An easy fix here is simply to tell date to not zero-pad the output. Just add a "-" in front of the formatting character. Just remember to correctly zero-pad the output yourself later, when and if you need it. You can use printf for zero-padding.

Also, in bash, numerical evaluations are better performed with ((..)), and for strings and complex tests it's recommended to use the new [[..]] extended test keyword.

But actually, I think you can skip the test entirely and simplify the whole section into this:

Code:

strtmo=$( date --date='350 days ago' '+%-m' )
printf -v startmo "%02d" "$(( strtmo - 1 ))"

BTW, those two variable names (strtmo/startmo) are very similar, and probably easy to confuse. It's generally a good idea to use clear and unambiguous names wherever possible.

In fact, in this case, there really isn't even a need for two separate variables. Since you aren't using the initial value anywhere else in the script, why not just use a single name for the whole operation?

Code:

startmo=$( date --date='350 days ago' '+%-m' )
printf -v startmo "%02d" "$(( startmo - 1 ))"