lwp-request returns 500 from https

roertel · 08-06-2013, 05:48 AM

I've automated fetching electronic bill statements from various web sites via perl & WWW::Mechanize, but after a recent upgrade, everything has stopped working. I've boiled the problem down to a issue making a lwp-request to an https site. From my debian machine, the request fails. From my fedora machine, the request succeeds. I can view the page with lynx on both. What's the difference and how do I make it work on both?

Code:

[user@debian]~% lwp-request -UuSsEed https://www.mnpass.net/customer
GET https://www.mnpass.net/customer
User-Agent: lwp-request/6.03 libwww-perl/6.04

500 Can't connect to www.mnpass.net:443
Content-Type: text/plain
Client-Date: Tue, 06 Aug 2013 10:43:45 GMT
Client-Warning: Internal response
--------------------------------------------------------------------------------
[user@fedora]~% lwp-request -UuSsEed https://www.mnpass.net/customer
GET https://www.mnpass.net/customer
User-Agent: lwp-request/6.03 libwww-perl/6.03

302 Found
Connection: close
Date: Tue, 06 Aug 2013 09:27:49 GMT
Location: https://www.mnpass.net/customer/
Server: IBM_HTTP_Server
Content-Language: en-US
Content-Length: 0
Content-Type: text/plain
Client-Date: Tue, 06 Aug 2013 10:45:19 GMT
Client-Peer: 207.108.84.8:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=GeoTrust Inc./OU=Domain Validated SSL/CN=GeoTrust DV SSL CA
Client-SSL-Cert-Subject: /serialNumber=Ne/esnkU2-rR1Vsu3/82pdon3tAdUa6Q/C=US/O=www.mnpass.net/OU=GT85693240/OU=See www.geotrust.com/resources/cps (c)11/OU=IT/CN=www.mnpass.net
Client-SSL-Cipher: AES256-SHA
Client-SSL-Socket-Class: IO::Socket::SSL

GET https://www.mnpass.net/customer/
User-Agent: lwp-request/6.03 libwww-perl/6.03

200 OK
Cache-Control: no-cache="set-cookie, set-cookie2"
Connection: close
Date: Tue, 06 Aug 2013 09:27:49 GMT
Server: IBM_HTTP_Server
Content-Language: en-US
Content-Length: 13731
Content-Type: text/html;charset=ISO-8859-1
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Client-Date: Tue, 06 Aug 2013 10:45:20 GMT
Client-Peer: 207.108.84.8:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=GeoTrust Inc./OU=Domain Validated SSL/CN=GeoTrust DV SSL CA
Client-SSL-Cert-Subject: /serialNumber=Ne/esnkU2-rR1Vsu3/82pdon3tAdUa6Q/C=US/O=www.mnpass.net/OU=GT85693240/OU=See www.geotrust.com/resources/cps (c)11/OU=IT/CN=www.mnpass.net
Client-SSL-Cipher: AES256-SHA
Client-SSL-Socket-Class: IO::Socket::SSL
Content-Style-Type: text/css
Link: </customer/theme/Master.css>; rel="stylesheet"; type="text/css"
Set-Cookie: JSESSIONID=0000N3kUwHDf8e-20UyWWNQNTzy:-1; Path=/
Title: Login
X-Meta-GENERATOR: IBM WebSphere Studio

angryfirelord · 09-06-2013, 10:00 AM

How did you install WWW::Mechanize on Debian and Fedora? I suspect that the WWW::Mechanize module (if it came from the Debian repository) is older than the Fedora one, which might have some outstanding bugs that aren't working with the secure page or the site's firewall.

roertel · 09-14-2013, 09:13 PM

Quote:

Originally Posted by angryfirelord

How did you install WWW::Mechanize on Debian and Fedora? I suspect that the WWW::Mechanize module (if it came from the Debian repository) is older than the Fedora one, which might have some outstanding bugs that aren't working with the secure page or the site's firewall.

Both came from the distro's packaging system. I just now installed LWP from CPAN and ran it with the same results. I've had the debian system for a long time and saw many, many old versions of perl modules in the /usr/share directory and others. I'm starting to wonder if some clean-up is in order. Although I only have 1 LWP.pm on the system, maybe there's something else that's getting picked up.