LinuxQuestions.org - [SOLVED] curl parameter problem when invoked in script

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - curl parameter problem when invoked in script (https://www.linuxquestions.org/questions/linux-newbie-8/curl-parameter-problem-when-invoked-in-script-939080/)

curl parameter problem when invoked in script

Hello, i have a problem with curl not executing properly (as in how i'd like it to)
so i use curl to get some files from a portal using cookies and from the command like it's all really simple :

Code:

user@host $ curl -b "cookie1=val1; cookie2=val2" -A "Mozilla/5.0" --location http://mysite.com/member.php?page=1

all good so far. say i put all that on one line in file 'curl'

Code:

user@host $ `cat curl | head -n 1`

curl: (6) Could not resolve host: cookie1=val1;; Name or service not known

curl: (6) Could not resolve host: cookie2=val2; Name or service not known

curl: (6) Could not resolve host: Mozilla;; Name or service not known

curl: (6) Could not resolve host: 5.0; Name or service not known

:( instead of taking those as parameters to -b and -A switches, it inteprets them as addresses to download.

Quote:

say i put all that on one line in file 'curl'

If I am reading this part correctly, you have created a file called curl (of course maybe not the best choice when there is already an application with this name) and
entered the line exactly as the working example (correct so far?). My question then would be, what have cat and head got to do with anything?

Also, as you have not called your script nor entered any parameters, I am a little perplexed by the question topic you have chosen?

I feel I have missed something so maybe you could explain further??

well, the script goes a little something like this :

Code:

#!/bin/bash

conf = $1

max=10

name=`cat $conf | grep -m1 ^name | cut -d: -f2-`

link=`cat $conf | grep -m1 ^link | cut -d: -f2-`

cookies=`cat $conf | grep -m1 ^cookies | cut -d: -f2-`

useragent=`cat $conf | grep -m1 ^useragent | cut -d: -f2-`



for i in `seq 1 $max`

do

        link=`echo $link$i`

        `which curl` -b $cookies -A $useragent --location $link > $name_$i.html

done

and the config file looks like this:

Code:

name:mypage.com

link:http://mypage.com/browse.php?page=

cookies:"cookie1:value; cookie2:value"

useragent:"Mozilla/5.0"

well, the cat and head were supposed to simulate the execution of the catted file's first line. kind of what that for loop does.

Right, so now we see that the switches are being fed information stored in variables, hence really quite different than what is in the command line example.

Try putting set -xv as the second line of the script and you may be surprised as to what you see is stored in the variables :)

On a side note, I will assume the current second line is a typo as of course any spaces around equals sign will cause no end of issues.
The second thing I would point out is you do not need cat in any of the invocations where you are using it as grep, or maybe better with awk or sed to also ignore the cut, can easily
be used to read a file. I may even suggest that using a bash loop to read the config may have been just as simple :)

Let us know how you get on?

What grail said. awk can completely replace sed (and cut); and both in turn completely replace grep.

http://www.grymoire.com/Unix/Awk.html
http://www.gnu.org/software/gawk/man...ode/index.html
http://www.grymoire.com/Unix/Sed.html
http://sed.sourceforge.net/grabbag/

Also have a look at parameter substitution and other built-in shell string manipulations. You'll find quite often that you don't need any external programs at all.

In addition:

QUOTE ALL OF YOUR VARIABLE SUBSTITUTIONS. You should never leave the quotes off a parameter expansion unless you explicitly want the resulting string to be word-split by the shell (globbing patterns are also expanded). This is a vitally important concept in scripting, so train yourself to do it correctly now. You can learn about the exceptions later.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

This is probably one of the places you're encountering problems.

2)
$(..) is highly recommended over `..`

3)
I recommend using a c-style for loop instead of seq.

4)

Code:

link=`echo $link$i`

In addition to points 1 and 2 above, this is a completely useless use of echo. Just set the variable directly ( e.g. var1="$var2$var3" ).

This might also be a good place to consider using an array instead.

thank you alot both of you. in the long run i still have much to go. but the script now works. and i will mark this as solved. thanks.

Don't forget to share your solution so others may learn what you have :)