LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-13-2005, 10:38 PM   #1
pittopitto
Member
 
Registered: Sep 2005
Location: Australia
Distribution: SUSE Linux Professional 9.3
Posts: 35

Rep: Reputation: 15
Exclamation A PhD student in trouble...


Hello folks,

I am a PhD student in Population Genetics, and I am currently working with a really boring web-based program called Genepop (http://wbiomed.curtin.edu.au/genepop/genepop_op6.html).

I have 10 local files called:
Code:
GLA-1001.gen
GLA-1002.gen
GLA-1003.gen
GLA-1004.gen
GLA-1005.gen
GLA-1006.gen
GLA-1007.gen
GLA-1008.gen
GLA-1009.gen
GLA-1010.gen
So they can be called in Linux (*01.gen, *02.gen etc...)

In the Genepop website, I have to select the option 'Fst for all populations' and the 'HTML' as output format. Then input JUST ONE FILE mentioned above, and then press the submit button. In the next screen I need to save the page as *.txt format, for example 1.txt for the file *01.gen, 2.txt for the file *02.gen etc... then came back to the prevoius window and submit the subsequent file and repeat it again...

The BIG problem is that I have those 10 files saved in 32 folders, and each folder is saved in another 32 folders.... you do the math....

So the continuous 'point and click' with the mouse on Firefox for long time is very stressing... so I was wondering if it possible to build a little program or script that can make all the work automatically in my place, maybe with Firefox or with Lynx...

I hope so, I already worked out almost a thousand of files and my arm hurts a lot.
I don't know any programming language.

Thanks a lot

G
 
Old 10-14-2005, 01:50 AM   #2
puffinman
Member
 
Registered: Jan 2005
Location: Atlanta, GA
Distribution: Gentoo, Slackware
Posts: 217

Rep: Reputation: 30
Here's something that may work for you. I can't seem to feed it data it likes so I couldn't test it thoroughly, but read the comments in the file and maybe you can get it working. Beats all that mousework anyway. Perhaps others can help and add to this program.

Code:
#!/usr/bin/perl
use LWP::UserAgent;
# you will need the LWP::UserAgent module

my @files;

# feed the list of files to stdin
# something like: locate *.gen | thisprogram
# make sure the files are the full path
while (<>) {
  chomp;
  push @files, $_;
}

$ua = LWP::UserAgent->new();

foreach my $fn (@files) {

  $resp = $ua->post("http://wbiomed.curtin.edu.au/cgi-bin/genepop.cgi",
                    # put the options you want here
                    # view the page source to see the tags and values
                    { "option6" => "1",
                      "ln" => "1",
                      "fstats" => "yes",
                      "htmltype" => "plain",
                      "digits" => "2",
                      "file" => $fn,
                    });
  $content = $resp->content;
  open(OF,">$fn.txt") or die "Couldn't open $fn.txt: $!\n";
  print OF $content;
}
 
Old 10-14-2005, 07:52 PM   #3
pittopitto
Member
 
Registered: Sep 2005
Location: Australia
Distribution: SUSE Linux Professional 9.3
Posts: 35

Original Poster
Rep: Reputation: 15
Wow! Thanks a lot!!! How can I compile the code? How can I execute it?
If possible I can send you ten *.gen files so you can test your code...

Let me know and thanks again!!!

G
 
Old 10-14-2005, 08:09 PM   #4
sasho
Member
 
Registered: Jan 2005
Distribution: Arch
Posts: 120

Rep: Reputation: 17
Good call puffinman!!! I was gonna suggest LWP as well, but you beat me to it

Quote:
How can I compile the code? How can I execute it?
1. Save his code in a file
2. Execute
Code:
perl filename
on the command prompt, where filename is the name of the file where you saved the code.
 
Old 10-15-2005, 06:26 AM   #5
pittopitto
Member
 
Registered: Sep 2005
Location: Australia
Distribution: SUSE Linux Professional 9.3
Posts: 35

Original Poster
Rep: Reputation: 15
Hi folks,

I tried to compile the file, but it seems that I haven't the LWP UserAgent module.

I looked in YaST if I have this module and then install it. Well, I haven't it either, so I need to take a look in internet and find it. Where can I look for it?

Thanks

G
 
Old 10-15-2005, 06:38 AM   #6
zeropash
Member
 
Registered: Apr 2003
Location: Bangalore,India
Distribution: FC2, RHES, RH9, FC3, FC1, Slackware 3.0
Posts: 208

Rep: Reputation: 30
you need to install the perl module for that.

option 1)
use the perl CPAN
become root and type
perl -MCPAN -e shell
it will ask for some questions if you are running for the first time. answer them and when it drops you off on a prompt type
install LWP::UserAgent
option 2) search CPAN download, compile and install
let this link help you.
http://perl.about.com/od/perlmodule1/l/aa030500a.htm
 
Old 10-15-2005, 06:59 AM   #7
pittopitto
Member
 
Registered: Sep 2005
Location: Australia
Distribution: SUSE Linux Professional 9.3
Posts: 35

Original Poster
Rep: Reputation: 15
Hi folks,

Wanted to say that I have arranged for myself to install the LWP module...

I have found a very useful website: http://www.cpan.org/ where I have found all the libraries for Perl...

I have installed the Compress::Zlib; HTML::Parser; HTML::Tagset; URI module; MIME-Base 64 and finally the libwww-perl (LWP I presume )

Well as I have zero experience in programming, I did it good don't you????

when I made this command, after installing all the modules:
Code:
perl script.pl
where script.pl is the code puffinman sent to me.

It seems that there are some errors, but I can't correct them.... I am so ignorant

Please let me know.... and Thanks

G
 
Old 10-15-2005, 08:03 AM   #8
zeropash
Member
 
Registered: Apr 2003
Location: Bangalore,India
Distribution: FC2, RHES, RH9, FC3, FC1, Slackware 3.0
Posts: 208

Rep: Reputation: 30
What is the error you are getting?
I dont know much but looking at perldoc LWP::UserAgent
if you are behind a proxy you may need to do the following changes.
add this line
$ua->env_proxy;
after
$ua = LWP::UserAgent->new();

so it should look like
$ua = LWP::UserAgent->new();
$ua->env_proxy;

now set the CGI_HTTP_PROXY
in sh or bash use
export CGI_HTTP_PROXY=http://myproxy:8088 or what ever the proxy is
the 8088 is the port number
and try running it again.
 
Old 10-15-2005, 08:13 PM   #9
pittopitto
Member
 
Registered: Sep 2005
Location: Australia
Distribution: SUSE Linux Professional 9.3
Posts: 35

Original Poster
Rep: Reputation: 15
No, it was a syntax error.... in particular I don't know how can I put the file names in this row:
Code:
my @files;
everything I try there seems a compilation error....

Ciao

Gianlu
 
Old 10-16-2005, 09:48 AM   #10
zeropash
Member
 
Registered: Apr 2003
Location: Bangalore,India
Distribution: FC2, RHES, RH9, FC3, FC1, Slackware 3.0
Posts: 208

Rep: Reputation: 30
you dont need to change the script
read the comments in the script
you can either store the file names in a file and then do a
cat filenames.txt | thisprogram
where thsiprogram is the name of the script
or you can do a locate .gen | thisprogram
or you can run thisprogram and enter the files one by one.
 
Old 10-16-2005, 06:51 PM   #11
pittopitto
Member
 
Registered: Sep 2005
Location: Australia
Distribution: SUSE Linux Professional 9.3
Posts: 35

Original Poster
Rep: Reputation: 15
I have stored my *.gen files on a txt file and called it store.txt
then I made this command:

Code:
cat store.txt | ./script.pl
Where script.pl is the code
and there appears new txt files.
When I visualize them, I have noticed that they are the HTML code of the GENEPOP page, not of the results

It seems that the 'submit' button has not been pressed....

Please let me know.... the importance of this script has been increased because I have lots of work to do now and I have to finish it in a week, a impossible task if you have almost 10000 files to study.

Thanks
G

PS I've bought a Perl manual, I wanna learn it!!!
 
Old 10-16-2005, 11:02 PM   #12
johnMG
Member
 
Registered: Jul 2003
Location: CT, USA
Distribution: Debian Sarge (server), Etch (work/home)
Posts: 601

Rep: Reputation: 32
> PS I've bought a Perl manual, I wanna learn it!!!

Most of the stuff in this thread should make much more sense to you after you've done some more reading. :)

Incidentally, this task is the sort of thing that Perl was made for, so you should be pretty well set.

Many folks learn Perl from the Llama book (O'reilly's "Learning Perl", by Schwartz and Phoenix), but that's by no means the only way to get started.

One change you might make to puffinman's originally posted script is to add

use strict;
use warnings;
use diagnostics;

just under the "#!/usr/bin/perl" line. Those lines aren't strictly required, but if you start asking around on the web for help with a perl script, one of the first things folks will want to see that you've done is that you've included those three lines in your script. :)

Last edited by johnMG; 10-16-2005 at 11:06 PM.
 
Old 10-18-2005, 03:28 AM   #13
zeropash
Member
 
Registered: Apr 2003
Location: Bangalore,India
Distribution: FC2, RHES, RH9, FC3, FC1, Slackware 3.0
Posts: 208

Rep: Reputation: 30
ok try this

Ok try this code.
just some small modifications to puffinmans code.

to run this store all the file names (complete path) and then
cat file_list | ./myprogram.pl
if you are behind a http proxy then you can uncomment the line
$ua->proxy('http', 'http://yourproxy:portnumber/');
(just remove the hash and replace your.proxy and portnumber to correct values)
and yeah you have to fill in the proper values for
ln,fstats,mindist,nperms,n1,n2,n3,n4,digits



Code:
#!/usr/bin/perl -w
use HTTP::Request::Common;
use LWP::UserAgent;
use strict;
use warnings;
use diagnostics;

my @files;

while (<>) {
  chomp;
  push @files, $_;
}

my $ua = LWP::UserAgent->new();
#$ua->proxy('http', 'http://yourproxy:portnumber/');

foreach my $fn (@files) {
  print "processing $fn\n";
  my $resp = $ua->request(POST 'http://wbiomed.curtin.edu.au/cgi-bin/genepop.cgi',
     Content_Type=>'form-data',
     Content => [
       "Option"=>'6',               # This is fixed for this form 
       option6=>'1',                # from 1- 8 options 
       ln=>'l',                          # can be l or d
       fstats=>'y',                    # y or n
       mindist=>'3',                 
       nperms=>'5',                
       n1=>'5',
       n2=>'25',
       n3=>'19',
       n4=>'77',
       htmltype=>'plain',          # should be plain
       digits=>'2',                    
       file=>["$fn"],                 # dont change
     ]);
  my $content = $resp->content;
  open(OF,">$fn.txt") or die "Couldn't open $fn.txt: $!\n";
  print OF $content;
}
 
Old 10-19-2005, 05:07 AM   #14
pittopitto
Member
 
Registered: Sep 2005
Location: Australia
Distribution: SUSE Linux Professional 9.3
Posts: 35

Original Poster
Rep: Reputation: 15
Thank you VERY much!!!

Dear folks!!

I wish to thank you to all, in particular to Zeropash and puffinman for helping me.

The code is finally working!!!! I couldn't believe to my eyes!!!!

You saved my life, my poor arm against the continuous point'n'click, my phd EVERYTHING!!!!

THANK YOU!!!!!! I am really touched

Thanks again!!!!!

G
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
A student that needs help!!! smithers_3 Linux - General 6 05-27-2005 03:00 PM
PhD in network security david4 Linux - Networking 0 09-06-2004 12:35 PM
Little Help with a student... Crowe182 Linux - Newbie 1 08-16-2003 08:58 PM
a student iam3 Linux - Newbie 1 11-01-2001 04:51 PM
a student iam3 Programming 1 11-01-2001 04:43 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration