Cannot get Curl to POST correctly

pobrika · 07-12-2012, 04:34 AM

Hello,

I have been messing with curl recently and am wanting to get results from mxtoolbox.com for blacklisted IP's as part of a bash script.

However I cannot get it to work, I have tried several ways but each fail.

I have recently found out about firebug for firefox using firecurl I am able to get what looks like a good curl command however it fails, If someone has the time to show me the error of my ways I would be grateful.

From firecurl I get two results each fails to return the same page as firefox does.

Code:

curl \     $FIRECURL_OPTIONS \     -X 'POST' \       -H 'Host: mxtoolbox.com' \       -H 'User-Agent: Mozilla/5.0 (X11; Linux i686; rv:13.0) Gecko/20100101 Firefox/13.0' \       -H 'Accept: application/json, text/javascript, */*; q=0.01' \       -H 'Accept-Language: en-gb,en;q=0.5' \       -H 'Accept-Encoding: gzip, deflate' \       -H 'Connection: keep-alive' \       -H 'Content-Type: application/json; charset=utf-8' \       -H 'X-Requested-With: XMLHttpRequest' \       -H 'Referer: http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a109.170.232.44' \       -H 'Content-Length: 40' \       --cookie '__utma=74094414.1029208595.1313509340.1342083630.1342085408.316' \       --cookie '__utmz=74094414.1313509340.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' \       --cookie 'is_returning=1' \       --cookie '__utmv=74094414.|2=OnBlacklist=true=1' \       --cookie 'ismobile=False' \       --cookie 'ASP.NET_SessionId=uipl50dx3ezmrhz4xocpgmqb' \       --cookie '__utmc=74094414' \       --cookie '__utmb=74094414.6.8.1342085409722' \       --form-string '' \     'http://mxtoolbox.com/Public/Lookup.aspx/DoLookup'

and

Code:

curl \     $FIRECURL_OPTIONS \     -X 'POST' \       -H 'Host: mxtoolbox.com' \       -H 'User-Agent: Mozilla/5.0 (X11; Linux i686; rv:13.0) Gecko/20100101 Firefox/13.0' \       -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \       -H 'Accept-Language: en-gb,en;q=0.5' \       -H 'Accept-Encoding: gzip, deflate' \       -H 'Connection: keep-alive' \       -H 'X-Requested-With: XMLHttpRequest' \       -H 'X-MicrosoftAjax: Delta=true' \       -H 'Cache-Control: no-cache' \       -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \       -H 'Referer: http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a109.170.232.44' \       -H 'Content-Length: 2891' \       --cookie '__utma=74094414.1029208595.1313509340.1342083630.1342085408.316' \       --cookie '__utmz=74094414.1313509340.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' \       --cookie 'is_returning=1' \       --cookie '__utmv=74094414.|2=OnBlacklist=true=1' \       --cookie 'ismobile=False' \       --cookie 'ASP.NET_SessionId=uipl50dx3ezmrhz4xocpgmqb' \       --cookie '__utmc=74094414' \       --cookie '__utmb=74094414.6.8.1342085409722' \       --form-string '' \     'http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a109.170.232.44'

The error I get is:

Code:

Warning: Illegally formatted input field!
curl: option --form-string: is badly used here
curl: try 'curl --help' for more information

Can anyone help me out?

rigor · 07-13-2012, 02:49 PM

If I understand the situation correctly, if you were requesting the information
using a browser, you'd normally visit a page, type in values to a form,
then submit the page.

If the values are submitted via POST, I'd usually expect to see the form data
"encoded".

In the second curl command example you posted, one of the headers indicates
that encoded form data will be provided. AFAIK, with curl, form data can
be provided with options such as -F or --form or --form-string.

It appears the --form-string option was used, but it's empty. Curl seems
to be complaining because the string supplied is empty.

I don't know which page on the site you are trying to emulate, but when I visit
a number of the pages on the site, they have actual forms with actual input
fields, to look up information. I would expect that the site would want actual
form data to be submitted.

Maybe there's something I'm missing, or perhaps if I looked at the very latest
RFC's I'd discover that parameters are somehow acceptable on a URL for a POST,
but it's not something I'm used to seeing. I'm currently building a web site,
using POST virtually exclusively, and you won't see any parameters on the URL's
submitted by the browser. Yet, in your second example there's a parameter on a
URL, but curl is seemingly being instructed, via the -X parameter, to use POST.

I use firebug for somethings; I'm not quite sure how you are trying to use it,
or if it may somehow be confusing the situation.

Usually if I'm trying to automate the retrieval of data from a web site,
I'll use something a bit simpler than firebug, such as httpfox, to largely
just watch the HTTP going back and forth, as I'm using the web site manually
to successfully get data. I'll use some of the options on curl or wget
to handle cookies.

I'll also analyze the pages involved. There can be hidden fields, and
I believe when I glanced at some of the pages from the web site involved
in your situation, I saw a variety of hidden fields.

There can also be javascript data, or values modified by javascript.

All those things may need to be sent to the site in one way or another,
for a request to be honored. With javascript, the values it sets may
depend on the data entered into a form. So it may not be possible to
simulate the page in a static/fixed manner. The simulation may need
to be dynamic/changeable.

I don't know how the curl commands are being generated in your situation,
but one way or another, I would expect that you'd have to get the form
data correctly specified to curl.

Hope this helps.

pobrika · 07-16-2012, 04:11 AM

Thanks kakaka for your reply, when I examined the site more closely I saw it uses javascript for the form http://mxtoolbox.com/blacklists.aspx Usually I enter a Server IP and use the submit button which after a few seconds returns the results as a list of blacklist sites and verifies if the IP is on any of them.

Since it is using javascript I assume this is going to be more difficult than I expected, as for the firecurl data, I was more hoping it was mostly good and just needed a tweak to fix the error message, I see now that it probably incomplete and to get the information I require from a script will take a lot more work and understanding of javascript, cookies and curl.