LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-15-2008, 03:57 AM   #1
wondergirl
LQ Newbie
 
Registered: Feb 2008
Posts: 18

Rep: Reputation: 0
Perl with LWP and NTLM


Hi guys!
Another newbie question...

I'm trying to get a csv file from a link on a website automatically instead of logging in to that website manually and download the file everyday.

I could do this with curl :

---------------------------------------------------
curl -k -A "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)" -u username:******** --ntlm https://egsite/Reports/20080212/Enti...t_20080212.csv

---------------------------------------------------

This works OK on Linux, but curl that is available on my Solaris machine (where I want to do this automatically) does not support ntlm.

So I'm trying to do it with Perl LWP and NTLM modules but since I'm a perl beginner I'm finding it difficult

From documentation I got this :

==========================================================
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(
agent=>'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)',
keep_alive=>'1'
);

my $host = 'serverort';
my $user = 'myDomain\\myUserName';
my $pass = 'password';

# note the empty '' is required.
$ua->credentials($host, '', $user, $pass);

my $req = new HTTP::Request GET => "http://winhost/path/to/file";
my $res = $ua->request( $req );

if ( $res->is_success() ) {
# Success !
}
else {
# Failure
}

=============================================================

But no mention of NTLM here....can anyone help me out? I would really appreciate it. Thanks!
 
Old 02-15-2008, 05:38 AM   #2
wondergirl
LQ Newbie
 
Registered: Feb 2008
Posts: 18

Original Poster
Rep: Reputation: 0
I actually managed to do this, although a bit messy :


----------------------------------------------
#!/opt/perl/5.8.0/bin/perl

$|++;
use strict;

use FindBin;
use lib "$FindBin::Bin/lib";

use File::Basename;
use POSIX qw(strftime);
use LWP::UserAgent;
use HTTP::Headers;
use HTTP::Request::Common;
use Authen::NTLM;
use HTML::TableExtract;
use HTML::Form;
use HTML::Template;
use MIME::Entity;

my $Options = {
user => "me",
password => "xxxx",
domain => "\\",
timeout => 30,
protocol => "https",
AuthMethod => "NTLM",
BrowserAgent => "MSIE 6.0; Windows NT 5.0",
RequestMethod => "GET",
DataDir => "/tmp",
};

my $log = "/var/tmp/get_url.log";
my $DataDir = "/tmp";

my $browser = LWP::UserAgent->new(
agent=>'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)',
keep_alive=>'1'
);

my $header = HTTP::Headers->new(
Content_Type => 'text/html',
'WWW-Authenticate' => $Options->{'AuthMethod'}
);

#
# First stage of NTLM authentication
#

my $url = "https://egserver/Reports/Userlist_20080212.csv";

ntlm_domain($Options->{'domain'});
ntlm_user($Options->{'user'});
ntlm_password($Options->{'password'});

my $Authorization = Authen::NTLM::ntlm();

my $header = HTTP::Headers->new(
Content_Type => 'text/html',
'WWW-Authenticate' => $Options->{'AuthMethod'}
);

$header->header('Authorization' => "NTLM $Authorization");
my $request = HTTP::Request->new($Options->{'RequestMethod'} => $url, $header);
my $res = $browser->request( $request );

#
# Second stage of authentication
#
my $Challenge = $res->header('WWW-Authenticate');
$Challenge =~ s/^NTLM //g;

$Authorization = Authen::NTLM::ntlm($Challenge);
$header->header('Authorization' => "NTLM $Authorization");
$request = HTTP::Request->new($Options->{'RequestMethod'} => $url, $header);

$res = $browser->request($request);
#
# ntlm needs to be resetted after second stage
#
ntlm_reset();

if($res->is_success) {
&dump2file("$Options->{'DataDir'}/test_url", $res->content);
}
else {
&out2logfile($log, "ERROR 1 : Can not dump data from $url\n Returned code: " . $res->code . " (" . $res->status_line . ")\n");
}


sub dump2file {
my ($FileName, $Message) = @_;
my $Flag;

my $LogDir = dirname($FileName);
system("/bin/mkdir -p $LogDir") unless(-d $LogDir);

if(open(FILE, "> $FileName")){
for(my $i=0 ; $i<10 ; $i++){
$Flag = flock(FILE,2);
last if($Flag);
sleep(1);
}

unless($Flag){
return 1;
}

print FILE $Message, "\n";

unless(flock(FILE,8)){
return 1;
}

close(FILE);
}
return 0;
}

sub out2logfile {
my ($FileName, $Message, $PrintError, $Option) = @_;
my $Flag;

my $LogDir = dirname($FileName);
system("/bin/mkdir -p $LogDir") unless(-d $LogDir);

$Option=">>" if(!defined($Option) || $Option ne ">");

print $Message if($PrintError);

if(open(FILE, "$Option $FileName")){
for(my $i=0 ; $i<10 ; $i++){
$Flag = flock(FILE,2);
last if($Flag);
sleep(1);
}

unless($Flag){
return 1;
}

my $time=strftime "%Y-%m-%d %H:%M ", localtime;
print FILE $time, $Message;

unless(flock(FILE,8)){
return 1;
}

close(FILE);
}
return 0;
}

==========================================================

Any comments on this? it could use some improving..

Now I need to figure out how to get the latest report automatically based on the date of the report (the latest is always yesterday's report). Right now I'm hardcoding it :

my $url = "https://egserver/Reports/Userlist_20080212.csv";

Any suggestion how I can make it download the right file automatically by manipulating the _20080212 part? Its a stupid (and lazy) question, I guess I'm gonna look into the books now and see...(maybe use strftime in a way).

Thanks!
 
Old 02-17-2008, 07:25 PM   #3
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,261

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
Well, here's a way I get yesterday:

Code:
$today = get_todays_date();
($curr_yr, $curr_mth, $curr_day) = (substr($today, 0, 4),
                                    substr($today, 4, 2),
                                    substr($today, 6, 2));
print "today: $curr_yr, $curr_mth, $curr_day\n";
# calc prev date
($prev_yr, $prev_mth, $prev_day) = Add_Delta_Days($curr_yr, $curr_mth,
                                                        $curr_day, -1);
print "prev: $prev_yr, $prev_mth, $prev_day\n";

sub get_todays_date
{
    my (
        $year,  # year ( -1900 )
        $mth,   # month ( -1 )
        $mday,  # day-of-mth
        $today  # today string: yyyymmdd
        );


    # Get today
    # NB: Mth array starts at zero
    ($mday, $mth, $year) = (localtime)[3..5];
    $today = sprintf( "%04d%02d%02d", $year+1900, $mth+1, $mday );
    return $today;
}
You might also want to look at http://search.cpan.org/~petdance/WWW...W/Mechanize.pm "WWW::Mechanize, or Mech for short, helps you automate interaction with a website"
I've found it very useful for a very similar job to yours.
 
Old 05-22-2010, 02:32 PM   #4
deisecairo
LQ Newbie
 
Registered: May 2010
Posts: 1

Rep: Reputation: 0
Hi all,

I found an easier way. It uses the simple lib LWP::Authen::Ntlm, but it needs some fixation before.

To use simple NTLM authentication through HTTP in a script, we need to be sure that we have installed the three modules in our system:

LWP::Authen::Ntlm
Authen::NTLM
MIME::Base64

then, we need to edit the file LWP/Authen/Ntlm.pm from module LWP::Authen::Ntlm, wherever it was installed in our system and modify the lines from:

use Authen::NTLM "1.02";
use MIME::Base64 "2.12";

to

use Authen::NTLM;
use MIME::Base64;

and is it.

Inside the file LWP/Authen/Ntlm.pm anyone can follow a complete secuence steps for NTLM authentication through HTTP. Anyway, wondergirl showed us a good secuence to make a NTLM access, and its good for curiosity humans.

Regards.

Last edited by deisecairo; 05-22-2010 at 02:34 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Perl install LWP module ; problems ... danran Programming 5 05-03-2007 01:02 PM
Does anyone use LWP anymore?? Mini__C Programming 1 08-19-2006 12:03 AM
Perl's LWP to Python?? chutsu Programming 3 06-05-2006 12:55 PM
Can't locate LWP/Simple.pm Dr_Death_UAE Linux - General 1 03-21-2006 03:30 PM
using LWP linuxRules Programming 0 06-27-2002 01:20 PM


All times are GMT -5. The time now is 11:39 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration