LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 04-10-2012, 09:36 AM   #1
seccentral
LQ Newbie
 
Registered: Mar 2011
Distribution: Linux mint
Posts: 23

Rep: Reputation: 0
Question curl parameter problem when invoked in script


Hello, i have a problem with curl not executing properly (as in how i'd like it to)
so i use curl to get some files from a portal using cookies and from the command like it's all really simple :

Code:
user@host $ curl -b "cookie1=val1; cookie2=val2" -A "Mozilla/5.0" --location http://mysite.com/member.php?page=1
all good so far. say i put all that on one line in file 'curl'

Code:
user@host $ `cat curl | head -n 1`
curl: (6) Could not resolve host: cookie1=val1;; Name or service not known
curl: (6) Could not resolve host: cookie2=val2; Name or service not known
curl: (6) Could not resolve host: Mozilla;; Name or service not known
curl: (6) Could not resolve host: 5.0; Name or service not known
instead of taking those as parameters to -b and -A switches, it inteprets them as addresses to download.
 
Old 04-10-2012, 11:00 AM   #2
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Quote:
say i put all that on one line in file 'curl'
If I am reading this part correctly, you have created a file called curl (of course maybe not the best choice when there is already an application with this name) and
entered the line exactly as the working example (correct so far?). My question then would be, what have cat and head got to do with anything?

Also, as you have not called your script nor entered any parameters, I am a little perplexed by the question topic you have chosen?

I feel I have missed something so maybe you could explain further??
 
Old 04-10-2012, 01:54 PM   #3
seccentral
LQ Newbie
 
Registered: Mar 2011
Distribution: Linux mint
Posts: 23

Original Poster
Rep: Reputation: 0
well, the script goes a little something like this :
Code:
#!/bin/bash
conf = $1
max=10
name=`cat $conf | grep -m1 ^name | cut -d: -f2-`
link=`cat $conf | grep -m1 ^link | cut -d: -f2-`
cookies=`cat $conf | grep -m1 ^cookies | cut -d: -f2-`
useragent=`cat $conf | grep -m1 ^useragent | cut -d: -f2-`

for i in `seq 1 $max`
do
	link=`echo $link$i`
	`which curl` -b $cookies -A $useragent --location $link > $name_$i.html
done
and the config file looks like this:
Code:
name:mypage.com
link:http://mypage.com/browse.php?page=
cookies:"cookie1:value; cookie2:value"
useragent:"Mozilla/5.0"
well, the cat and head were supposed to simulate the execution of the catted file's first line. kind of what that for loop does.
 
Old 04-10-2012, 02:13 PM   #4
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Right, so now we see that the switches are being fed information stored in variables, hence really quite different than what is in the command line example.

Try putting set -xv as the second line of the script and you may be surprised as to what you see is stored in the variables

On a side note, I will assume the current second line is a typo as of course any spaces around equals sign will cause no end of issues.
The second thing I would point out is you do not need cat in any of the invocations where you are using it as grep, or maybe better with awk or sed to also ignore the cut, can easily
be used to read a file. I may even suggest that using a bash loop to read the config may have been just as simple

Let us know how you get on?
 
Old 04-11-2012, 02:54 AM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
What grail said. awk can completely replace sed (and cut); and both in turn completely replace grep.

http://www.grymoire.com/Unix/Awk.html
http://www.gnu.org/software/gawk/man...ode/index.html
http://www.grymoire.com/Unix/Sed.html
http://sed.sourceforge.net/grabbag/

Also have a look at parameter substitution and other built-in shell string manipulations. You'll find quite often that you don't need any external programs at all.


In addition:

QUOTE ALL OF YOUR VARIABLE SUBSTITUTIONS. You should never leave the quotes off a parameter expansion unless you explicitly want the resulting string to be word-split by the shell (globbing patterns are also expanded). This is a vitally important concept in scripting, so train yourself to do it correctly now. You can learn about the exceptions later.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

This is probably one of the places you're encountering problems.

2)
$(..) is highly recommended over `..`

3)
I recommend using a c-style for loop instead of seq.

4)
Code:
link=`echo $link$i`
In addition to points 1 and 2 above, this is a completely useless use of echo. Just set the variable directly ( e.g. var1="$var2$var3" ).

This might also be a good place to consider using an array instead.

Last edited by David the H.; 04-11-2012 at 02:59 AM. Reason: added a sub-point
 
Old 04-14-2012, 05:19 AM   #6
seccentral
LQ Newbie
 
Registered: Mar 2011
Distribution: Linux mint
Posts: 23

Original Poster
Rep: Reputation: 0
thank you alot both of you. in the long run i still have much to go. but the script now works. and i will mark this as solved. thanks.
 
Old 04-15-2012, 06:39 AM   #7
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Don't forget to share your solution so others may learn what you have
 
  


Reply

Tags
bash, curl, parameters, resolve, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
shell script is defined in someother directory, how it is being invoked with path? vaibhavs17 Linux - Newbie 2 05-30-2011 11:24 PM
cURL: Server has many IPs, how would I make a cURL script use those IPs to send data? guest Programming 0 04-11-2009 12:42 PM
Bash variable problem: cURL -b parameter (string form e.g. "name=value;n2=v2") sithemac Other *NIX 3 07-09-2008 07:15 PM
kill the process invoked from a shell script, when the script is killed kskkumar Linux - Software 8 05-23-2007 12:29 PM
partimage acting strange when invoked from script ffaker Linux - Software 1 08-19-2005 10:55 AM


All times are GMT -5. The time now is 03:00 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration