BASH variables, quotation marks and cURL struggles

Hi,

I am currently running cURL from a BASH script in Cygwin to send HTTP requests to a Web application. I am sending the data as POSTed multipart/form-data using the -F flag.

Now, cURL requires the argument to -F to be surrounded with quotes if the data contains spaces, like this:

-F "title=Hello World!" -F "body=Content goes here."

That works perfectly fine. However, I have more data to send and I want to aggregate this data first in a BASH variable. I did it like this:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'

Now comes the weird part:
When I invoke cURL using the data string directly (i.e. without going through that variable), everything works and the data is transmitted correctly. If however I go through that variable as an argument to cURL, although it contains the exact same string, the whole thing will break (no data arrives at the server because cURL interprets part of the string as a server URL, obviously because the double quotation marks disappeared somehow and it interprets everything that is not a whitespace as an argument).

Shouldn't these two approaches result in the exact same thing? ^^ Am I missing something?

Thanks for your attention.

Well, if you do

echo some "te xt" here

there will be three params to echo with contents some and te xt and here.
But if you use

DOCDATA='some "te xt" here'
echo "$DOCDATA"

there will be only one param to echo, with contents some "te xt" here.
If they should stay individual parameters, you must leave out the quotes around $DOCDATA,
so this will again pass three params to the echo command:

echo $DOCDATA

might this help? Maybe you could post the whole script (or at least the part constructing and using the DOCDATA var)
then I could have a better look at it.

Hi doc.nice,

yes, I am aware of the difference between single and double quotation marks. I want that argument string to not be broken into several parts, but passed to curl "as-is". The odd thing is, when I echo that variable, it prints exactly what I want. I mean, I can even copy and paste that output to curl as an argument and then it will work. It just doesn't work when I go through that variable.

Here is how I invoke it:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

curl $SERVER $OPTS $DOCDATA

The last line breaks.

Originally Posted by matthias_k The last line breaks.

There are a few ways to fix this, the easiest being:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

eval curl $SERVER $OPTS $DOCDATA

I just received this answer on the curl mailing list:
http://thread.gmane.org/gmane.comp.w...716/focus=8723

The thread I opened there actually covered a different problem, but along the road I also asked the same question there. I wasn't aware that strings in variables, even when surrounded with single quotation marks, will still be broken into words when passed as arguments. That explains a lot of course :-)

Thanks guys!

# STEP 2: Send document data
eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command

eval curl "$TNSERVER?op=101

because "&" is interpreted as "run job in background" then the command

upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

gives the error. The first part is somewhat correct because the shell assigns a value to the variable "upload" then try to execute the "-L" command. Ok... solution: escape the ampersand and strip away the eval command.

curl "$TNSERVER?op=101\&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

Originally Posted by colucix # STEP 2: Send document data eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command ...

Does this only happen when being invoked using eval? Because, without eval, I'm pretty sure the ampersand wasn't interpreted by the shell (at least it made its way properly to the server!).

Could be. Eval strips away the double quotes and the ampersand is not passed inside the string anymore.

Hi, sorry this is a newbie questions I fear:

I am trying to update one of my bash scripts to work with Firefox3's sqlite cookie format.

I can extract the cookie values but am trying to pass them as a string variable to the -b parameter in curl, rather than as a reference to a netscape format cookie.txt file.

The curl MAN page explains that this is possible and it works if I pass the hardcoded string "pass=xyz;userID=123", but when I pass the same data as a variable it breaks. I've tried all manner of escaping, quoting, echoing the variable and using brackets and backticks but am really stuck.

I'm building the string by querying sqlite for the individual values:

  thepass=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='pass'"`
  theUID=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='uid'"`

then concatenating them into a variable

  inputcookiefn="pass="$thepass"';'UID="$theUID"
  echo "Input Cookie String:" "$inputcookiefn"

which looks right when I echo it back

Input Cookie String: pass=xyz;UID=123

so my curl command looks like:

curl -s -S -b "$inputcookies"  -c newcookies.txt -A 'Mozilla/4.0' [website URL here] > testresults.htm

The cURL command completes, but the site doesn't accept the cookies when passed as a variable, although it accepts the hardcoded string as shown above, which _appears_ to be identical.

I am trying to avoid the need to generate a cookies.txt file from the sqlite queries and just build a string value for the -b parameter.

Any help or pointers much appreciated.

Hello,

When I execute this command within a terminal it works fine. Bu when I execute it from within a shell script (test.sh) it fails to execute properly.

terminal

curl -i -k  -H "authorization-code: <HASH>" -H Content-Type:application/tst.server.commandParams+xml -X POST https://xxx.xxx.xxx/api/test/test-sadasd-asdewfdsf-dsfsdfsdf-dsf/action/rework -d "@deploy-request"

Shell script (test.sh)

EXEC=`curl -i -k -H "$authHeader" -H "Content-Type:application/tst.server.commandParams+xml" -X POST "$HREF""/action/rework" -d "@deploy-request"`
echo  $EXEC

The latter one does not work when executed from within the shell script.

Can someone please help me with only this problem?

Thank you in advance.

Perhaps you need to place the variable inside double quotes perhaps to prevent expansion:

echo "$EXEC"

Also please start a new thread next time. It's not good to bring back an old thread.

(Edit: Apologies, I just noticed that I mis-identified the backticks for hard-quotes. Still, using a function should make the command more robust. I've added an example capture to variable to the code below. And remember, $(..) is highly recommended over `..`.)

Variables are for storing data, not code. Don't try to put commands in them and expect them to work.

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050

If you want to create a dynamic command, use a function.

curl_command() {
	HREF=$1	authHeader=$2
	curl -i -k -H "$2" -H "Content-Type:application/tst.server.commandParams+xml" -X POST "$2/action/rework" -d "@deploy-request"
}

#to output the results directly
curl_command <href> <authheader>

#to capture the results to a variable for later use
EXEC=$( curl_command <href> <authheader> )
echo "$EXEC"

And to expand on the earlier conversation above, to dynamically concatenate multiple options for a command, don't use a scalar variable, use an array.

DOCDATA=( 
    -F 'title=Hello World!'
    -F 'body=Content goes here.'
)

curl "${DOCDATA[@]}" ....

Hi,

Thank you for your great reply!
When I use the function you described I get the following error:

curl_command() {
  HREF=$1 authHeader=$2

  curl -i -k -H "$2" -H "Content-Type:application/vnd.vmware.vcloud.composeVAppParams+xml" -X POST $1"/action/composeVApp" -d "@deploy-request"
}

EXEC=$(curl_command $DemoVApp "$authHeader" )
echo "$EXEC"

curl: (6) Could not resolve host: x-cloud-authorization; nodename nor servname provided, or not known

When I perform it like this it works:

EXEC=$(curl_command $DemoVApp "x-code-auth: sadsd98762-da98Sds39-sdrgdfhFGd" )

But how is that possible while $authHeader contains x-code-auth: sadsd98762-da98Sds39-sdrgdfhFGd

The problem has something to do with the format of the curl command then. The error message is coming from it, in any case. It's unlikely that the function format itself has anything to do with it.

Make sure your variables are set correctly. Try putting an echo in front of the command so you can see exactly what would be executed. Or use #!/bin/bash -x at the top of the script to get debugging-style output.