LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (http://www.linuxquestions.org/questions/programming-9/)
-   -   BASH variables, quotation marks and cURL struggles (http://www.linuxquestions.org/questions/programming-9/bash-variables-quotation-marks-and-curl-struggles-630151/)

matthias_k 03-24-2008 12:10 AM

BASH variables, quotation marks and cURL struggles
 
Hi,

I am currently running cURL from a BASH script in Cygwin to send HTTP requests to a Web application. I am sending the data as POSTed multipart/form-data using the -F flag.

Now, cURL requires the argument to -F to be surrounded with quotes if the data contains spaces, like this:
Code:

-F "title=Hello World!" -F "body=Content goes here."
That works perfectly fine. However, I have more data to send and I want to aggregate this data first in a BASH variable. I did it like this:
Code:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
Now comes the weird part:
When I invoke cURL using the data string directly (i.e. without going through that variable), everything works and the data is transmitted correctly. If however I go through that variable as an argument to cURL, although it contains the exact same string, the whole thing will break (no data arrives at the server because cURL interprets part of the string as a server URL, obviously because the double quotation marks disappeared somehow and it interprets everything that is not a whitespace as an argument).

Shouldn't these two approaches result in the exact same thing? ^^ Am I missing something?

Thanks for your attention.

doc.nice 03-24-2008 05:29 AM

Well, if you do
Code:

echo some "te xt" here
there will be three params to echo with contents some and te xt and here.
But if you use
Code:

DOCDATA='some "te xt" here'
echo "$DOCDATA"

there will be only one param to echo, with contents some "te xt" here.
If they should stay individual parameters, you must leave out the quotes around $DOCDATA,
so this will again pass three params to the echo command:
Code:

echo $DOCDATA
might this help? Maybe you could post the whole script (or at least the part constructing and using the DOCDATA var)
then I could have a better look at it.

matthias_k 03-24-2008 08:20 PM

Hi doc.nice,

yes, I am aware of the difference between single and double quotation marks. I want that argument string to not be broken into several parts, but passed to curl "as-is". The odd thing is, when I echo that variable, it prints exactly what I want. I mean, I can even copy and paste that output to curl as an argument and then it will work. It just doesn't work when I go through that variable.

Here is how I invoke it:

Code:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

curl $SERVER $OPTS $DOCDATA

The last line breaks.

osor 03-24-2008 10:16 PM

Quote:

Originally Posted by matthias_k (Post 3099313)
The last line breaks.

There are a few ways to fix this, the easiest being:
Code:

DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

eval curl $SERVER $OPTS $DOCDATA


matthias_k 03-25-2008 12:03 AM

I just received this answer on the curl mailing list:
http://thread.gmane.org/gmane.comp.w...716/focus=8723

The thread I opened there actually covered a different problem, but along the road I also asked the same question there. I wasn't aware that strings in variables, even when surrounded with single quotation marks, will still be broken into words when passed as arguments. That explains a lot of course :-)

Thanks guys!

colucix 03-25-2008 12:23 AM

Code:

# STEP 2: Send document data
eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command
Code:

eval curl "$TNSERVER?op=101
because "&" is interpreted as "run job in background" then the command
Code:

upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA
gives the error. The first part is somewhat correct because the shell assigns a value to the variable "upload" then try to execute the "-L" command. Ok... solution: escape the ampersand and strip away the eval command.
Code:

curl "$TNSERVER?op=101\&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

matthias_k 03-25-2008 12:28 AM

Quote:

Originally Posted by colucix (Post 3099456)
Code:

# STEP 2: Send document data
eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA

Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command
...

Does this only happen when being invoked using eval? Because, without eval, I'm pretty sure the ampersand wasn't interpreted by the shell (at least it made its way properly to the server!).

colucix 03-25-2008 12:42 AM

Could be. Eval strips away the double quotes and the ampersand is not passed inside the string anymore.

sithemac 07-07-2008 05:10 PM

Bash problem providing -b string variable to cURL
 
Hi, sorry this is a newbie questions I fear:

I am trying to update one of my bash scripts to work with Firefox3's sqlite cookie format.

I can extract the cookie values but am trying to pass them as a string variable to the -b parameter in curl, rather than as a reference to a netscape format cookie.txt file.

The curl MAN page explains that this is possible and it works if I pass the hardcoded string "pass=xyz;userID=123", but when I pass the same data as a variable it breaks. I've tried all manner of escaping, quoting, echoing the variable and using brackets and backticks but am really stuck.

I'm building the string by querying sqlite for the individual values:
Code:

  thepass=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='pass'"`
  theUID=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='uid'"`

then concatenating them into a variable
Code:

  inputcookiefn="pass="$thepass"';'UID="$theUID"
  echo "Input Cookie String:" "$inputcookiefn"

which looks right when I echo it back
Quote:

Input Cookie String: pass=xyz;UID=123
so my curl command looks like:

Code:

curl -s -S -b "$inputcookies"  -c newcookies.txt -A 'Mozilla/4.0' [website URL here] > testresults.htm
The cURL command completes, but the site doesn't accept the cookies when passed as a variable, although it accepts the hardcoded string as shown above, which _appears_ to be identical.

I am trying to avoid the need to generate a cookies.txt file from the sqlite queries and just build a string value for the -b parameter.

Any help or pointers much appreciated.

redneck26 01-26-2013 06:10 AM

Executing curl from shell script not the same as manually in a terminal
 
Hello,

When I execute this command within a terminal it works fine. Bu when I execute it from within a shell script (test.sh) it fails to execute properly.

terminal
Code:

curl -i -k  -H "authorization-code: <HASH>" -H Content-Type:application/tst.server.commandParams+xml -X POST https://xxx.xxx.xxx/api/test/test-sadasd-asdewfdsf-dsfsdfsdf-dsf/action/rework -d "@deploy-request"
Shell script (test.sh)
Code:

EXEC=`curl -i -k -H "$authHeader" -H "Content-Type:application/tst.server.commandParams+xml" -X POST "$HREF""/action/rework" -d "@deploy-request"`
echo  $EXEC

The latter one does not work when executed from within the shell script.

Can someone please help me with only this problem?:redface:

Thank you in advance.

konsolebox 01-26-2013 09:13 PM

Perhaps you need to place the variable inside double quotes perhaps to prevent expansion:
Code:

echo "$EXEC"
Also please start a new thread next time. It's not good to bring back an old thread.

David the H. 01-27-2013 03:57 PM

(Edit: Apologies, I just noticed that I mis-identified the backticks for hard-quotes. Still, using a function should make the command more robust. I've added an example capture to variable to the code below. And remember, $(..) is highly recommended over `..`.)

Variables are for storing data, not code. Don't try to put commands in them and expect them to work.

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050

If you want to create a dynamic command, use a function.

Code:

curl_command() {
        HREF=$1        authHeader=$2
        curl -i -k -H "$2" -H "Content-Type:application/tst.server.commandParams+xml" -X POST "$2/action/rework" -d "@deploy-request"
}

#to output the results directly
curl_command <href> <authheader>

#to capture the results to a variable for later use
EXEC=$( curl_command <href> <authheader> )
echo "$EXEC"


And to expand on the earlier conversation above, to dynamically concatenate multiple options for a command, don't use a scalar variable, use an array.

Code:

DOCDATA=(
    -F 'title=Hello World!'
    -F 'body=Content goes here.'
)

curl "${DOCDATA[@]}" ....


redneck26 01-27-2013 04:12 PM

Hi,

Thank you for your great reply!
When I use the function you described I get the following error:

Code:


curl_command() {
  HREF=$1 authHeader=$2

  curl -i -k -H "$2" -H "Content-Type:application/vnd.vmware.vcloud.composeVAppParams+xml" -X POST $1"/action/composeVApp" -d "@deploy-request"
}

EXEC=$(curl_command $DemoVApp "$authHeader" )
echo "$EXEC"

Code:

curl: (6) Could not resolve host: x-cloud-authorization; nodename nor servname provided, or not known

When I perform it like this it works:

Code:

EXEC=$(curl_command $DemoVApp "x-code-auth: sadsd98762-da98Sds39-sdrgdfhFGd" )
But how is that possible while $authHeader contains x-code-auth: sadsd98762-da98Sds39-sdrgdfhFGd

David the H. 01-27-2013 04:56 PM

The problem has something to do with the format of the curl command then. The error message is coming from it, in any case. It's unlikely that the function format itself has anything to do with it.

Make sure your variables are set correctly. Try putting an echo in front of the command so you can see exactly what would be executed. Or use #!/bin/bash -x at the top of the script to get debugging-style output.


All times are GMT -5. The time now is 09:02 AM.