LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 03-24-2008, 01:10 AM   #1
matthias_k
LQ Newbie
 
Registered: Nov 2004
Posts: 16

Rep: Reputation: 0
Question BASH variables, quotation marks and cURL struggles


Hi,

I am currently running cURL from a BASH script in Cygwin to send HTTP requests to a Web application. I am sending the data as POSTed multipart/form-data using the -F flag.

Now, cURL requires the argument to -F to be surrounded with quotes if the data contains spaces, like this:
Code:
-F "title=Hello World!" -F "body=Content goes here."
That works perfectly fine. However, I have more data to send and I want to aggregate this data first in a BASH variable. I did it like this:
Code:
DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
Now comes the weird part:
When I invoke cURL using the data string directly (i.e. without going through that variable), everything works and the data is transmitted correctly. If however I go through that variable as an argument to cURL, although it contains the exact same string, the whole thing will break (no data arrives at the server because cURL interprets part of the string as a server URL, obviously because the double quotation marks disappeared somehow and it interprets everything that is not a whitespace as an argument).

Shouldn't these two approaches result in the exact same thing? ^^ Am I missing something?

Thanks for your attention.
 
Old 03-24-2008, 06:29 AM   #2
doc.nice
Member
 
Registered: Oct 2004
Location: Germany
Distribution: Debian
Posts: 274

Rep: Reputation: 34
Well, if you do
Code:
echo some "te xt" here
there will be three params to echo with contents some and te xt and here.
But if you use
Code:
DOCDATA='some "te xt" here'
echo "$DOCDATA"
there will be only one param to echo, with contents some "te xt" here.
If they should stay individual parameters, you must leave out the quotes around $DOCDATA,
so this will again pass three params to the echo command:
Code:
echo $DOCDATA
might this help? Maybe you could post the whole script (or at least the part constructing and using the DOCDATA var)
then I could have a better look at it.
 
Old 03-24-2008, 09:20 PM   #3
matthias_k
LQ Newbie
 
Registered: Nov 2004
Posts: 16

Original Poster
Rep: Reputation: 0
Hi doc.nice,

yes, I am aware of the difference between single and double quotation marks. I want that argument string to not be broken into several parts, but passed to curl "as-is". The odd thing is, when I echo that variable, it prints exactly what I want. I mean, I can even copy and paste that output to curl as an argument and then it will work. It just doesn't work when I go through that variable.

Here is how I invoke it:

Code:
DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

curl $SERVER $OPTS $DOCDATA
The last line breaks.
 
Old 03-24-2008, 11:16 PM   #4
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 70
Quote:
Originally Posted by matthias_k View Post
The last line breaks.
There are a few ways to fix this, the easiest being:
Code:
DOCDATA='-F "title=Hello World!" -F "body=Content goes here."'
SERVER="http://localhost:8080/tnmain/http"
OPTS="-v"

eval curl $SERVER $OPTS $DOCDATA
 
Old 03-25-2008, 01:03 AM   #5
matthias_k
LQ Newbie
 
Registered: Nov 2004
Posts: 16

Original Poster
Rep: Reputation: 0
I just received this answer on the curl mailing list:
http://thread.gmane.org/gmane.comp.w...716/focus=8723

The thread I opened there actually covered a different problem, but along the road I also asked the same question there. I wasn't aware that strings in variables, even when surrounded with single quotation marks, will still be broken into words when passed as arguments. That explains a lot of course :-)

Thanks guys!

Last edited by matthias_k; 03-25-2008 at 01:20 AM.
 
Old 03-25-2008, 01:23 AM   #6
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Code:
# STEP 2: Send document data
eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA
Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command
Code:
eval curl "$TNSERVER?op=101
because "&" is interpreted as "run job in background" then the command
Code:
upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA
gives the error. The first part is somewhat correct because the shell assigns a value to the variable "upload" then try to execute the "-L" command. Ok... solution: escape the ampersand and strip away the eval command.
Code:
curl "$TNSERVER?op=101\&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA
 
Old 03-25-2008, 01:28 AM   #7
matthias_k
LQ Newbie
 
Registered: Nov 2004
Posts: 16

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by colucix View Post
Code:
# STEP 2: Send document data
eval curl "$TNSERVER?op=101&upload=true" $OPTS -c $COOKIEJAR -b $COOKIEJAR $DOCDATA
Here it is the ampersand "&" which triggers the error: in the line above the shell run in background the command
...
Does this only happen when being invoked using eval? Because, without eval, I'm pretty sure the ampersand wasn't interpreted by the shell (at least it made its way properly to the server!).
 
Old 03-25-2008, 01:42 AM   #8
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Could be. Eval strips away the double quotes and the ampersand is not passed inside the string anymore.
 
Old 07-07-2008, 06:10 PM   #9
sithemac
LQ Newbie
 
Registered: Jul 2008
Posts: 5

Rep: Reputation: 0
Bash problem providing -b string variable to cURL

Hi, sorry this is a newbie questions I fear:

I am trying to update one of my bash scripts to work with Firefox3's sqlite cookie format.

I can extract the cookie values but am trying to pass them as a string variable to the -b parameter in curl, rather than as a reference to a netscape format cookie.txt file.

The curl MAN page explains that this is possible and it works if I pass the hardcoded string "pass=xyz;userID=123", but when I pass the same data as a variable it breaks. I've tried all manner of escaping, quoting, echoing the variable and using brackets and backticks but am really stuck.

I'm building the string by querying sqlite for the individual values:
Code:
  thepass=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='pass'"`
  theUID=`sqlite3 "$ffcf" "select value from moz_cookies where host='[host string]' and name='uid'"`
then concatenating them into a variable
Code:
  inputcookiefn="pass="$thepass"';'UID="$theUID"
  echo "Input Cookie String:" "$inputcookiefn"
which looks right when I echo it back
Quote:
Input Cookie String: pass=xyz;UID=123
so my curl command looks like:

Code:
curl -s -S -b "$inputcookies"  -c newcookies.txt -A 'Mozilla/4.0' [website URL here] > testresults.htm
The cURL command completes, but the site doesn't accept the cookies when passed as a variable, although it accepts the hardcoded string as shown above, which _appears_ to be identical.

I am trying to avoid the need to generate a cookies.txt file from the sqlite queries and just build a string value for the -b parameter.

Any help or pointers much appreciated.

Last edited by sithemac; 07-07-2008 at 06:41 PM.
 
Old 01-26-2013, 07:10 AM   #10
redneck26
LQ Newbie
 
Registered: Jan 2013
Posts: 4

Rep: Reputation: Disabled
Executing curl from shell script not the same as manually in a terminal

Hello,

When I execute this command within a terminal it works fine. Bu when I execute it from within a shell script (test.sh) it fails to execute properly.

terminal
Code:
curl -i -k  -H "authorization-code: <HASH>" -H Content-Type:application/tst.server.commandParams+xml -X POST https://xxx.xxx.xxx/api/test/test-sadasd-asdewfdsf-dsfsdfsdf-dsf/action/rework -d "@deploy-request"
Shell script (test.sh)
Code:
EXEC=`curl -i -k -H "$authHeader" -H "Content-Type:application/tst.server.commandParams+xml" -X POST "$HREF""/action/rework" -d "@deploy-request"`
echo  $EXEC
The latter one does not work when executed from within the shell script.

Can someone please help me with only this problem?

Thank you in advance.
 
Old 01-26-2013, 10:13 PM   #11
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
Perhaps you need to place the variable inside double quotes perhaps to prevent expansion:
Code:
echo "$EXEC"
Also please start a new thread next time. It's not good to bring back an old thread.
 
Old 01-27-2013, 04:57 PM   #12
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
(Edit: Apologies, I just noticed that I mis-identified the backticks for hard-quotes. Still, using a function should make the command more robust. I've added an example capture to variable to the code below. And remember, $(..) is highly recommended over `..`.)

Variables are for storing data, not code. Don't try to put commands in them and expect them to work.

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050

If you want to create a dynamic command, use a function.

Code:
curl_command() {
	HREF=$1	authHeader=$2
	curl -i -k -H "$2" -H "Content-Type:application/tst.server.commandParams+xml" -X POST "$2/action/rework" -d "@deploy-request"
}

#to output the results directly
curl_command <href> <authheader>

#to capture the results to a variable for later use
EXEC=$( curl_command <href> <authheader> )
echo "$EXEC"

And to expand on the earlier conversation above, to dynamically concatenate multiple options for a command, don't use a scalar variable, use an array.

Code:
DOCDATA=( 
    -F 'title=Hello World!'
    -F 'body=Content goes here.'
)

curl "${DOCDATA[@]}" ....

Last edited by David the H.; 01-27-2013 at 05:05 PM.
 
Old 01-27-2013, 05:12 PM   #13
redneck26
LQ Newbie
 
Registered: Jan 2013
Posts: 4

Rep: Reputation: Disabled
Hi,

Thank you for your great reply!
When I use the function you described I get the following error:

Code:
curl_command() {
  HREF=$1 authHeader=$2

  curl -i -k -H "$2" -H "Content-Type:application/vnd.vmware.vcloud.composeVAppParams+xml" -X POST $1"/action/composeVApp" -d "@deploy-request"
}

EXEC=$(curl_command $DemoVApp "$authHeader" )
echo "$EXEC"
Code:
curl: (6) Could not resolve host: x-cloud-authorization; nodename nor servname provided, or not known

When I perform it like this it works:

Code:
EXEC=$(curl_command $DemoVApp "x-code-auth: sadsd98762-da98Sds39-sdrgdfhFGd" )
But how is that possible while $authHeader contains x-code-auth: sadsd98762-da98Sds39-sdrgdfhFGd

Last edited by redneck26; 01-27-2013 at 05:51 PM.
 
Old 01-27-2013, 05:56 PM   #14
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
The problem has something to do with the format of the curl command then. The error message is coming from it, in any case. It's unlikely that the function format itself has anything to do with it.

Make sure your variables are set correctly. Try putting an echo in front of the command so you can see exactly what would be executed. Or use #!/bin/bash -x at the top of the script to get debugging-style output.
 
  


Reply

Tags
bash, curl, quotation, variables


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Execvp With Quotation Marks amitbern Programming 10 11-25-2005 11:28 AM
no quotation marks jdoe9898 Linux - Newbie 1 10-12-2005 04:07 AM
Quotation Marks Linux / Firefox jjdoll42 Linux - Software 6 04-19-2005 11:13 AM
Quotation marks and strings - in C lazyuser Programming 5 01-25-2005 09:14 AM
quotation marks donīt display in X Obi-Wan_Kenobi Slackware 1 08-31-2003 11:42 AM


All times are GMT -5. The time now is 05:51 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration