Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why can't I fetch wikipedia pages with LWP::Simple?

I'm trying to fetch Wikipedia pages using LWP::Simple, but they're not coming back. This code:

#!/usr/bin/perl
use strict;
use LWP::Simple;

print get("http://en.wikipedia.org/wiki/Stack_overflow");

doesn't print anything. But if I use some other webpage, say http://www.google.com, it works fine.

Is there some other name that I should be using to refer to Wikipedia pages?

What could be going on here?

like image 932
Jesse Beder Avatar asked Aug 23 '08 19:08

Jesse Beder


1 Answers

Apparently Wikipedia blocks LWP::Simple requests: http://www.perlmonks.org/?node_id=695886

The following works instead:

#!/usr/bin/perl
use strict;
use LWP::UserAgent;

my $url = "http://en.wikipedia.org/wiki/Stack_overflow";

my $ua = LWP::UserAgent->new();
my $res = $ua->get($url);

print $res->content;
like image 57
Jesse Beder Avatar answered Sep 28 '22 16:09

Jesse Beder