Please help me debug an SSL issue using WWW::Mechanize (or LWP::UserAgent for that matter)

Question

I'm using WWW::Mechanize to load the catalog from our product provider into our database. I run this script every 2 hours everyday and it completes in arround 12 minutes by using around 50 simultaneous threads.

Everything was working perfectly, until this weekend. They put their website offline for a scheduled maintenance and, once they where online again, my script no longer worked. After analyzing things, it comes down to the following code failing:

use strict;
use warnings;

use WWW::Mechanize;

my $mec = WWW::Mechanize->new;
$mec->get('https://www.imstores.com/Ingrammicromx/login/login.aspx');

print $mec->content;

The code dies (after about 60 seconds) with the following message:

Error GETing https://www.imstores.com/Ingrammicromx/login/login.aspx:
Can't connect to www.imstores.com:443 at test.pl line 7.

Now, these are the points that are making me difficult to find the problem:

1. It's not network-related - if I visit the same URL from any of my browsers, I get the page.

2. If I try the same code on a remote machine that contains an exact copy of my Perl installation, it works.

3. If I use Net::SSL before WWW::Mechanize, it takes a very LONG time, but finally gets the page.

4. If I try any other SSL page, like 'https://www.paypal.com', it works and very fast.

5. Then again, it was working before their scheduled maintenance.

I'm not sure what else to try. If I switch to the non-SSL version, it works, but I don't want to do that since we automate purchasing operations.

Along with many things that have crossed my mind, thinking about why it works on the remote machine and why I can open the page in my browsers in the local one:

Is it possible to get blocked with my SSL public key? Is that possible? If so, what public key is LWP/Mechanize using for SSL sessions and how can I use a different one?

Some data on my current setup:

• OS: Windows 7 Ultimate x64
• Perl version: 5.16.3 x64
• LWP::UserAgent version: 6.05
• WWW::Mechanize version: 1.72
• IO::Socket version: 1.34
• IO::Socket::SSL version: 1.85
• Net::SSL version: 2.85
• Crypt::SSLeay version: 0.64

Thanks in advance for any helpful comment.

Answer 1

Here's the actual reason for the problem: You need to use SSLv3 or TLS1 instead of TLS1.2 to connect to that server. This is probably why it worked when you used Net::SSL first; I believe it tries different ciphers in a way that WWW:Mechanize doesn't.

This is how I found it:

I tried connecting from several different servers, and I find that the ones that worked have an older SSL version. I then checked the difference between what ciphers are used in the versions, and tried connecting with different ciphers.

When I connect using TLS1.2, I get:

$ openssl s_client -connect www.imstores.com:443 -tls1_2
CONNECTED(00000003)
write:errno=54
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 322 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

But when I connect with SSLv3 or TLS1, I get:

$ openssl s_client -connect www.imstores.com:443 -tls1
CONNECTED(00000003)
depth=0 /serialNumber=O3gPUAuGGROuHEhlyLaeJfj7SOn6tFTx/C=US/O=www.imstores.com/OU=GT29846307/OU=See www.geotrust.com/resources/cps (c)11/OU=Domain Control Validated - QuickSSL(R) Premium/CN=www.imstores.com
verify error:num=20:unable to get local issuer certificate
[...and so on, including server certificate...]

Exactly how to make WWW:Mechanize use TLS1 or SSLv3 is left as an exercise to the student.