What is the easiest way in pure Perl to stream from another HTTP resource?

Question

What is the easiest way (without opening a shell to curl and reading from stdin) in Perl to stream from another HTTP resource? I'm assuming here that the HTTP resource I'm reading from is a potentially infinite stream (or just really, really long)

Answer 1

Good old LWP allows you to process the result as a stream.

E.g., here's a callback to yourFunc, reading/passing byte_count bytes to each call to yourFunc (you can drop that param if you don't care how large the data is to each call, and just want to process the stream as fast as possible):

use LWP;
...
$browser = LWP::UserAgent->new();
$response = $browser->get($url, 
                          ':content_cb' => \&yourFunc, 
                          ':read_size_hint' => byte_count,);
...
sub yourFunc {
   my($data, $response) = @_;
   # do your magic with $data
   # $respose will be a response object created once/if get() returns
}

Answer 2

HTTP::Lite's request method allows you to specify a callback.

The $data_callback parameter, if used, is a way to filter the data as it is received or to handle large transfers. It must be a function reference, and will be passed: a reference to the instance of the http request making the callback, a reference to the current block of data about to be added to the body, and the $cbargs parameter (which may be anything). It must return either a reference to the data to add to the body of the document, or undef.

However, looking at the source, there seems to be a bug in sub request in that it seems to ignore the passed callback. It seems safer to use set_callback:

#!/usr/bin/perl

use strict;
use warnings;

use HTTP::Lite;

my $http = HTTP::Lite->new;
$http->set_callback(\&process_http_stream);
$http->http11_mode(1);

$http->request('http://www.example.com/');

sub process_http_stream {
    my ($self, $phase, $dataref, $cbargs) = @_;
    warn $phase, "\n";
    return;
}

Output:

C:\Temp> ht
connect
content-length
done-headers
content
content-done
data
done

It looks like a callback passed to the request method is treated differently:

#!/usr/bin/perl

use strict;
use warnings;

use HTTP::Lite;

my $http = HTTP::Lite->new;
$http->http11_mode(1);

my $count = 0;
$http->request('http://www.example.com/',
    \&process_http_stream,
    \$count,
);

sub process_http_stream {
    my ($self, $data, $times) = @_;
    ++$$times;
    print "$$times====\n$$data\n===\n";
}