I cannot Connect to any HTTPS site using LWP::UserAgent

Question

I am trying to create a script that simply just connect to a website. However, for some reason it will not connect to anything that is using HTTPS.

We have a proxy enabled here. However, I believe the proxy is not the problem, because if I were to connect to an HTTPS inside the network that does not tunnel through a proxy it still fails.

If I were to run this program on any site that is not using HTTPS, I can get through and the script works as intended.

What I'm wondering is what could possible by blocking the script from connecting to any SSL secured sites.

Here is the code that I wrote:

$ENV{HTTPS_DEBUG} = 1;

my $ua = LWP::UserAgent->new( keep_alive => 1);
$ua->agent('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36');
my $pac = HTTP::ProxyPAC->new( URI->new("http://pacfilelocation:8080/pac_file.pac") );
my $res = $pac->find_proxy("https://www.google.com");


if ($res->direct) {
    print "No Proxy Needed\n";
} elsif ($res->proxy) {
    print "Proxy: " . $res->proxy . "\n";
    $ENV{HTTPS_PROXY} = $res->proxy;
    $ENV{HTTP_PROXY} = $res->proxy;
    $ua->env_proxy;
}

my $req = new HTTP::Request('GET', 'https://www.google.com/');
$req->header('Accept' => 'text/html');
$req->header('Host', 'www.google.com');

my $res2 = $ua->request($req);

if ( $res2->is_success ) {
    print $res2->decoded_content;
} else {
    print "Error: " . $res2->status_line . "\n";
}

The HTTPS_DEBUG feature for some reason does not output the debug, which makes this all the more hard to solve.

When running the script I get a generic error:

 Error: 500 Can't connect to www.google.com:443

Any help would be great!

Answer 1

Please make sure, that you use at least version 6.06 of LWP::UserAgent and version 6.06 of LWP::Protocol::https. Any versions before do not have proper support for https proxy, at least not when using IO::Socket::SSL as the underlying SSL library.

To get the versions you are running:

use LWP::UserAgent;
use LWP::Protocol::https;
print "LWP::UserAgent: ".LWP::UserAgent->VERSION,"\n";
print "LWP::Protocol::https: ".LWP::Protocol::https->VERSION,"\n";

If you use anything less than the required versions upgrade. While this is easy for LWP::UserAgent, cpan might not install the newest version of LWP::Protocol::https by default, see http://www.nntp.perl.org/group/perl.libwww/2014/05/msg7718.html. In this case you have to get it explicitly from http://search.cpan.org/~mschilli/LWP-Protocol-https-6.06/.

If you are on a recent Debian system or Ubuntu >=14.04 you will still have version 6.04 of LWP::Protocol::https, but this includes already the necessary patches for proper https proxy support.

The other alternative would be to use the old Net::SSL/Crypt::SSLeay backend for LWP, but I advice against it, because it does not implement all necessary certificate checks and thus mounting a man-in-the-middle attack against it is easy.