Perl LWP:UserAgent how to I add headers?

Question

I'm a new perl programmer trying to convert a curl request into a Perl script get using LWP:UserAgent.

The curl request example is:

curl -X GET -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Cache-Control: no-cache" -H "Postman-Token: eb3955f1-a7b5-65d7-f5c0-808c7aba6cef" "https://10.51.10.26/10/download?startTime=1461698250&endTime=1461698252&cNat=True&cNatShowDst=True&tuplesFile=True&summarizeTuples=False"

And my PERL equivalent:

use LWP::UserAgent;
my $browser = LWP::UserAgent->new;
my $url = 'https://10.51.10.26/10/download';
my @headers = (
   "startTime" => $queryStart, 
   "endTime" => $queryEnd, 
   "cNat" => "True", 
   "cNatShowDst" => "False", 
   "tuplesFile" => "False", 
   "summarizeTuples" => "False",
   "Authorization" => "Basic YWRtaW46YWRtaW4",
   "Cache-Control" => "no-cache", 
   "Postman-Token" => "eb3955f1-a7b5-65d7-f5c0-808c7aba6cef", 
);

Results in - HTTP::Response=HASH(0x27884bc)

Is this the correct way of adding headers?

Answer 1

If you want to do a GET request with custom headers with LWP::UserAgent, you can put them into the $ua->get() call in the way the documentation describes.

This method will dispatch a GET request on the given $url. Further arguments can be given to initialize the headers of the request. These are given as separate name/value pairs. The return value is a response object. See HTTP::Response for a description of the interface it provides.

Your example is missing the part where you are sending the request, so it's hard to tell what you are doing.

Your @headers array contains both headers and URL params. That's not going to do what you expect. If you want to construct the URL and the headers like this, you need a different approach.

Use the URI module to create the URI programmatically, then use LWP::UA's get to send it including the headers.

use strict;
use warnings;
use LWP::UserAgent;
use URI;

my $uri = URI->new('https://10.51.10.26/10/download');
$uri->query_form(
    "startTime"       => $queryStart, # these two need 
    "endTime"         => $queryEnd,   # to be set above
    "cNat"            => "True", 
    "cNatShowDst"     => "False", 
    "tuplesFile"      => "False", 
    "summarizeTuples" => "False",   
);

my $ua = LWP::UserAgent->new;
my $res = $ua->get(
    $uri,
    "Authorization" => "Basic YWRtaW46YWRtaW4",
    "Cache-Control" => "no-cache", 
    "Postman-Token" => "eb3955f1-a7b5-65d7-f5c0-808c7aba6cef", 
);

if ($res->is_success) {
    # do stuff with content
} else {
    # request failed
}

To output the full HTTP::Response object, use Data::Dumper.

use Data::Dumper;
print Dumper $res;

Answer 2

Your Perl code doesn't result in the HTTP::Response object that you show. It can't possibly do that as your code doesn't actually make a request.

Putting new headers in an array called @headers isn't going to achieve anything useful either. You need to attach those headers to the request in some way.

LWP includes a useful tutorial. It would be a good idea to read that before trying to do too much with the tools. In particular, it includes a section entitled Adding Other HTTP Request Headers which says:

The most commonly used syntax for requests is $response = $browser->get($url), but in truth, you can add extra HTTP header lines to the request by adding a list of key-value pairs after the URL, like so:

$response = $browser->get( $url, $key1, $value1, $key2, $value2, ... );

For example, here's how to send some more Netscape-like headers, in case you're dealing with a site that would otherwise reject your request:

my @ns_headers = (
  'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)', 
  'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,image/png, */*',
  'Accept-Charset' => 'iso-8859-1,*,utf-8', 
  'Accept-Language' => 'en-US', );

...

$response = $browser->get($url, @ns_headers);

If you weren't reusing that array, you could just go ahead and do this:

$response = $browser->get($url,
  'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
  'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*',
  'Accept-Charset' => 'iso-8859-1,*,utf-8',
  'Accept-Language' => 'en-US',
);

If you were only ever changing the 'User-Agent' line, you could just change the $browser object's default line from "libwww-perl/5.65" (or the like) to whatever you like, using the LWP::UserAgent agent method:

$browser->agent('Mozilla/4.76 [en] (Win98; U)');

It's worth pointing out that LWP::UserAgent also has a default_headers() method which allows you to define headers which are added to every request made by that useragent.

People have put a lot of effort into creating a lot of useful documentation for Perl tools. That effort is rather wasted if people don't read it.