[SOLVED] Click 'submit' in a remote web form from Bash, really
ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I can easily surf along to the URL http://www.mysite.com/update.html and click the submit button which pokes script.php and everyone's happy. Automating this in iMacros or Selenium - no problem.
I want to automate this in a bash cron on the same server - preferably as a one-liner.
The HTTP request must be sent to the PHP URL 'http://www.mysite.com/script.php', not to the web page. The request sent to the web page will simply return the page. Of course, the accordant POST data must be attached to the request, but you seem to have figured this out.
You should be able to easily see the details of what the HTTP request needs to look like by running tcpdump with an appropriate filter to capture and disassemble the HTTP request packet(s) being sent from a real browser.
Are you getting that cookie back from the server? If you do, does the cookie have a correct information (whatever it is)? Maybe the cookie is not enough for this configuration and you need to pass the session id over the GET or POST... It's hard to tell since I don't know how your authentication mechanism works. You're on the right track though. Get Firebug and check out what is sent with your requests when you use browser, then try to replicate with wget.
I think the problem is an authentication routine getting in the way.
All the more reason to snoop online to see how a real browser handles the job. This (truncated) sample shows what a request to this thread on LQ looks like. If there was authentication going on, it should show up:
TcpDump is a bit too low level for analyzing high level protocols like HTTP though still invaluable tool in many other cases
I strongly recommend Firebug and it's Net panel. It shows nicely formatted request headers etc etc...
However, this is not the original site. I've somehow diddled the real site with my experimentation and now I cannot login through a browser. Suspect it's a permissions issue.
As soon as I solve that, I'll try to poke the script and post the results.
If the site is not your own site, perhaps they've banned you due to too many failed login attempts. Sometimes the ban is only temporary, and automatically lifted in a day or so.
--- rod.
When I use --save-cookies cookie.txt the file is empty. Clearly, && /usr/bin/wget http://www.mysite.com/script.php is hitting the authentication form again.
I need to know how I can authenticate to the site then separately access the protected pages; wedge the door open then steal the gold.
I guess that your authentication is based on the PHP session id. If you don't get the cookie that means the server doesn't send it. In PHP when you disable session cookie the session id will be attached to the URL. It may be passed over hidden field. First, you must have a full understanding of how your authentication works. Can you disable cookies in your browser and then see if you can use the page?
You may need to fetch the session id from the response and attach it to the URL request (post or get, whatever is used).
It is based on PHP session ID. It is not sent by the server. I don't notice anything in the URL, but I know what the ID code is from Firebug. How can I use it with wget?
Try adding this to your request: --keep-session-cookies --cookies=on check if you will get anyting saved. I still think you get that session cookie.
If you can see the session id in the Firebug then you can also see where it comes from (cookie? hidden field in HTML?). Also, you are using "spider" option, so if the session id is passed over html hidden field you are unable to fetch it as you simply discard it. You may have to use "-O -" with wget to print it to stdio and then grep it to fetch the session id. Then in your second following wget requests you may want to add something like --post-data="PHPSESSID=<your session id>". PHPSESSID is the default variable name that php uses so you can give it a blind shot.
I had an old version of wget installed 1.9.1, so I upgraded to 1.11.4 which gave me the --keep-session-cookies function. So, I can now do it from the command line successfully:
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.