LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-19-2004, 06:02 PM   #1
yimboli
LQ Newbie
 
Registered: Feb 2004
Distribution: Slackware
Posts: 24

Rep: Reputation: 15
automating lynx in a shell script


Hello all-

I would like to use lynx to access a website, login, perform some actions, and logout. By "perform some actions", I mean something like this: on the autoloaded webpage, move to a set link, say, 21 links from the first one in the document, and then in the next page, typing some text and proceeding to another link, and performing other similar tasks and then logging out. I know programs like this have been written for windows, but I want to set up a cron job to do it in linux. any ideas? I am VERY new to shell script programming, but I get some of the basics from my knowledge base in c++ and the bash syntax from a chapter i just read at {url removed cuz i only have two posts...}

or if this has been addressed just shoot me a link to the thread and i'll stfu

thanks
-tim

//Moderator's Note
Moving to Programming

Last edited by MasterC; 02-20-2004 at 11:35 AM.
 
Old 02-20-2004, 12:45 PM   #2
david_ross
Moderator
 
Registered: Mar 2003
Location: Scotland
Distribution: Slackware, RedHat, Debian
Posts: 12,047

Rep: Reputation: 64
You should be able to get the links from the output of:
lynx --dump http://www.yourhost.com

Then for posting data to a form you can echo data into:
lynx -post_data -dump http://www.yourhost.com

For more information see:
man lynx
 
Old 02-20-2004, 09:03 PM   #3
yimboli
LQ Newbie
 
Registered: Feb 2004
Distribution: Slackware
Posts: 24

Original Poster
Rep: Reputation: 15
Thanks for the reply david

what do you mean by echoing data? I knew about the post_data option and the dump option, and the source option too. I've also seen the man page, but like most man pages it makes no sense to me at all.

Another thing I don't get is that if I can post data to a website, how does lynx know which textboxes to put the data in? And how do I specify the username and password so lynx knows what to post?

And one more thing (for now): assuming i can actually figure out how to get a script to load a webpage and log me in to it with lynx, how do I tell lynx to go to other sites after that? I think what I need to do is use the Visible Links listed onder the References at the end of the dump, but how do I tell lynx to go to one of those links once it's already posted my login informatoin and logged me in?

Thanks again!
-Tim
 
Old 02-20-2004, 09:31 PM   #4
yimboli
LQ Newbie
 
Registered: Feb 2004
Distribution: Slackware
Posts: 24

Original Poster
Rep: Reputation: 15
Okay, I just figured out the cmd_log and cmd_script options, so I'm good for lynx. So I can just put that in a script file and do a cron job, no? I've read that cron lets you automatically do tasks at given times, but I have no idea what it is. Can someone point me in the right direction? A thread, tutorial, something?

THANKS
-tim
 
Old 02-21-2004, 08:23 AM   #5
david_ross
Moderator
 
Registered: Mar 2003
Location: Scotland
Distribution: Slackware, RedHat, Debian
Posts: 12,047

Rep: Reputation: 64
The post data that gets sent will contain the fieldnames too. If you want an example of the post data just run ethereal while you submit the data.
 
Old 02-21-2004, 09:25 AM   #6
yimboli
LQ Newbie
 
Registered: Feb 2004
Distribution: Slackware
Posts: 24

Original Poster
Rep: Reputation: 15
=) this is why i originally posted this in the newbie category... do I specify the post data when I call lynx in the command line?
 
Old 02-21-2004, 09:28 AM   #7
david_ross
Moderator
 
Registered: Mar 2003
Location: Scotland
Distribution: Slackware, RedHat, Debian
Posts: 12,047

Rep: Reputation: 64
Yes. With the -post_data option you can pipe data into lynx - from "man lynx":
Code:
       -post_data
              properly formatted data for a post form are read in
              from stdin and passed to the form.  Input is termi-
              nated by a line that starts with '---'.
So you can create your file with the post data then pipe it into lynx with:
cat datafile | lynx -post_data http://yoursite.com/cgi-bin/script.cgi
 
Old 02-21-2004, 10:17 AM   #8
yimboli
LQ Newbie
 
Registered: Feb 2004
Distribution: Slackware
Posts: 24

Original Poster
Rep: Reputation: 15
Cool. I still don't really understand it; i've never seen "cat" before. and piping is a vague concept to me. i had seen the lynx man page, but it doesn't specify what "proper formattin" would be, and thats one thing that confused me. here's my latest problem, and if i get it fixed i will [most likely not] stop popping questions :

i'm trying to make my script so that you pass it a parameter to specify which account you want to log in. The first parameter is accessed by $1 (i learned that last night!), so here's what my script says:

#!/bin/bash
####used to debug:
#echo "${#} Parameters"
#echo "${1} is first"

if [ $# -eq 0 ]
then
echo "Login parameter not provided!"
else
lynx -cmd_script=$1 -accept_all_cookies http://www.theurl.com
fi

first, i realize that this is a ridiculous reason to write a script; its more of a learning lesson than anything.

so basically i've stored the key strokes for doing the stuff on the website into several different command scripts for lynx, one for each of five accounts. I'd like to be able to specify which command script file to use as a parameter into the script, such that executing ...
./myscript spd
... would run lynx using the command script stored as spd. When I run the script, lynx opens up and loads the correct url, but does NOT load the command script. HOWEVER, when I run lynx outside the script and specify the command script by name, it runs just fine.

Any thoughts?

THANKS
-tim
 
Old 02-22-2004, 07:45 AM   #9
david_ross
Moderator
 
Registered: Mar 2003
Location: Scotland
Distribution: Slackware, RedHat, Debian
Posts: 12,047

Rep: Reputation: 64
Try using the full path to your lynx script within the bash script.
 
Old 02-22-2004, 05:48 PM   #10
yimboli
LQ Newbie
 
Registered: Feb 2004
Distribution: Slackware
Posts: 24

Original Poster
Rep: Reputation: 15
well i'll be damned, that worked!!

thx alot for your help.

also, i googled ethereal cuz i had no idea what it was... it looks way over my head. is the format of the post data particularly difficult? i mean, is it something that lynx is going to interpret and then send in the correct format over the web? or does it already have to be in the format to send over the web? thanks!

-Tim

i affero'd you =)
 
Old 02-23-2004, 02:51 PM   #11
david_ross
Moderator
 
Registered: Mar 2003
Location: Scotland
Distribution: Slackware, RedHat, Debian
Posts: 12,047

Rep: Reputation: 64
It does need to be in the correct format and the easiest way to see what is getting sent will be with a packet sniffer such as etherreal.
 
Old 07-13-2005, 11:22 PM   #12
ash_bk1
LQ Newbie
 
Registered: Jul 2005
Posts: 2

Rep: Reputation: 0
Hi,
I've a requirement very similar to yours. I tried my hand on lynx but cudn't figure out how the cmd_script works. How does lynx know which form the data needs to be fed into. Wud really appreciate you sharing your thoughts (maybe even the script

Regards,
Ash.
 
Old 07-14-2005, 01:21 PM   #13
david_ross
Moderator
 
Registered: Mar 2003
Location: Scotland
Distribution: Slackware, RedHat, Debian
Posts: 12,047

Rep: Reputation: 64
You'll need to tell lynx which url you want to post to when you invoke it.
 
Old 07-15-2005, 08:33 AM   #14
ash_bk1
LQ Newbie
 
Registered: Jul 2005
Posts: 2

Rep: Reputation: 0
Hi........thanks for your help. This is what I did........I first recorded the keys that I hit to do the required actions by using cmd_log. Then I used the same log file as input file for cmd_script. That may sound confusing but it works like charm. Only issue that I now have is that the version of lynx that I've doesn't support https..........but I guess I can take care of that.......your suggestions are most welcome.

Thanks,
Ash.
 
Old 07-16-2005, 10:59 AM   #15
eddiebaby1023
Member
 
Registered: May 2005
Posts: 378

Rep: Reputation: 33
Quote:
Originally posted by yimboli
Okay, I just figured out the cmd_log and cmd_script options, so I'm good for lynx. So I can just put that in a script file and do a cron job, no? I've read that cron lets you automatically do tasks at given times, but I have no idea what it is. Can someone point me in the right direction? A thread, tutorial, something?

THANKS
-tim
man cron and man crontab for more details than you need.

crontab -l will list your crontab file, or complain if you don't have one. crontab -e will drop your crontab in your favourite editor so you can add/delete/change stuff. the crontab entry is of the form
Code:
* * * * * command
where the *s represent the minute, hour, day of month, month number, and day of week that the job will run, so it's pretty flexible. An asterisk will match anything, so the example above will run command every minute.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Automating the "killing" process via a shell script fatrandy13 Linux - Software 5 09-13-2007 11:59 PM
Trouble when automating ftp with shell script hari_s_82 Linux - Newbie 2 09-23-2004 02:17 AM
automating using shell scripting LinToWinWin Linux - Newbie 1 07-23-2004 09:57 PM
Automating a playlist using shell script... DesiLILO Linux - Newbie 4 04-02-2004 11:03 PM
Automating lynx prompt enties cmfarley19 Linux - Software 6 01-23-2004 03:23 PM


All times are GMT -5. The time now is 02:54 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration