BASH variables, quotation marks and cURL struggles

matthias_k · 03-24-2008, 12:10 AM

Hi,

I am currently running cURL from a BASH script in Cygwin to send HTTP requests to a Web application. I am sending the data as POSTed multipart/form-data using the -F flag.

Now, cURL requires the argument to -F to be surrounded with quotes if the data contains spaces, like this:

Code:

-F "title=Hello World!" -F "body=Content goes here."

That works perfectly fine. However, I have more data to send and I want to aggregate this data first in a BASH variable. I did it like this:

Code:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'

Now comes the weird part:
When I invoke cURL using the data string directly (i.e. without going through that variable), everything works and the data is transmitted correctly. If however I go through that variable as an argument to cURL, although it contains the exact same string, the whole thing will break (no data arrives at the server because cURL interprets part of the string as a server URL, obviously because the double quotation marks disappeared somehow and it interprets everything that is not a whitespace as an argument).

Shouldn't these two approaches result in the exact same thing? ^^ Am I missing something?

Thanks for your attention.

doc.nice · 03-24-2008, 05:29 AM

Well, if you do

Code:

echo some "te xt" here

there will be three params to echo with contents some and te xt and here.
But if you use

Code:

DOCDATA='some "te xt" here'
echo "$DOCDATA"

there will be only one param to echo, with contents some "te xt" here.
If they should stay individual parameters, you must leave out the quotes around $DOCDATA,
so this will again pass three params to the echo command:

Code:

echo $DOCDATA

might this help? Maybe you could post the whole script (or at least the part constructing and using the DOCDATA var)
then I could have a better look at it.

matthias_k · 03-24-2008, 08:20 PM

Hi doc.nice,

yes, I am aware of the difference between single and double quotation marks. I want that argument string to not be broken into several parts, but passed to curl "as-is". The odd thing is, when I echo that variable, it prints exactly what I want. I mean, I can even copy and paste that output to curl as an argument and then it will work. It just doesn't work when I go through that variable.

Here is how I invoke it:

Code:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

curl $SERVER $OPTS $DOCDATA

The last line breaks.

osor · 03-24-2008, 10:16 PM

Quote:

Originally Posted by matthias_k

The last line breaks.

There are a few ways to fix this, the easiest being:

Code:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

eval curl $SERVER $OPTS $DOCDATA

matthias_k · 03-25-2008, 12:03 AM

I just received this answer on the curl mailing list:
http://thread.gmane.org/gmane.comp.w...716/focus=8723

The thread I opened there actually covered a different problem, but along the road I also asked the same question there. I wasn't aware that strings in variables, even when surrounded with single quotation marks, will still be broken into words when passed as arguments. That explains a lot of course :-)

Thanks guys!

colucix · 03-25-2008, 12:23 AM

Code:

# STEP 2: Send document data
eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command

Code:

eval curl "$TNSERVER?op=101

because "&" is interpreted as "run job in background" then the command

Code:

upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

gives the error. The first part is somewhat correct because the shell assigns a value to the variable "upload" then try to execute the "-L" command. Ok... solution: escape the ampersand and strip away the eval command.

Code:

curl "$TNSERVER?op=101\&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

matthias_k · 03-25-2008, 12:28 AM

Quote:

Originally Posted by colucix

Code:

# STEP 2: Send document data
eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command
...

Does this only happen when being invoked using eval? Because, without eval, I'm pretty sure the ampersand wasn't interpreted by the shell (at least it made its way properly to the server!).

colucix · 03-25-2008, 12:42 AM

Could be. Eval strips away the double quotes and the ampersand is not passed inside the string anymore.

sithemac · 07-07-2008, 05:10 PM

Hi, sorry this is a newbie questions I fear:

I am trying to update one of my bash scripts to work with Firefox3's sqlite cookie format.

I can extract the cookie values but am trying to pass them as a string variable to the -b parameter in curl, rather than as a reference to a netscape format cookie.txt file.

The curl MAN page explains that this is possible and it works if I pass the hardcoded string "pass=xyz;userID=123", but when I pass the same data as a variable it breaks. I've tried all manner of escaping, quoting, echoing the variable and using brackets and backticks but am really stuck.

I'm building the string by querying sqlite for the individual values:

Code:

  thepass=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='pass'"`
  theUID=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='uid'"`

then concatenating them into a variable

Code:

  inputcookiefn="pass="$thepass"';'UID="$theUID"
  echo "Input Cookie String:" "$inputcookiefn"

which looks right when I echo it back

Quote:

Input Cookie String: pass=xyz;UID=123

so my curl command looks like:

Code:

curl -s -S -b "$inputcookies"  -c newcookies.txt -A 'Mozilla/4.0' [website URL here] > testresults.htm

The cURL command completes, but the site doesn't accept the cookies when passed as a variable, although it accepts the hardcoded string as shown above, which _appears_ to be identical.

I am trying to avoid the need to generate a cookies.txt file from the sqlite queries and just build a string value for the -b parameter.

Any help or pointers much appreciated.