How to parse between <div class ="foo"> and </div> easily in Perl

Question

I want to parse a Website into a Perl data structure. First I load the page with

use LWP::Simple;
my $html = get("http://f.oo");

Now I know two ways to deal with it. First are the regular expressions and secound the modules.

I started with reading about HTML::Parser and found some examples. But I'm not that sure about by Perl knowledge.

My code example goes on

my @links;

my $p = HTML::Parser->new();
$p->handler(start => \&start_handler,"tagname,attr,self");
$p->parse($html);

foreach my $link(@links){
  print "Linktext: ",$link->[1],"\tURL: ",$link->[0],"\n";
}

sub start_handler{
  return if(shift ne 'a');
  my ($class) = shift->{href};
  my $self = shift;
  my $text;
  $self->handler(text => sub{$text = shift;},"dtext");
  $self->handler(end => sub{push(@links,[$class,$text]) if(shift eq 'a')},"tagname");
}

I don't understand why there is two times a shift. The secound should be the self pointer. But the first makes me think that the self reference is allready shiftet, used as a Hash and the Value for href is stored in $class. Could someone Explain this line (my ($class) = shift->{href};)?

Beside this lack, I do not want to parse all the URLs, I want to put all the code between <div class ="foo"> and </div> into a string, where lots of code is between, specially other <div></div> tags. So I or a module has to find the right end. After that I planed to scan the string again, to find special classes, like <h1>,<h2>, <p class ="foo2"></p>, etc.

I hope this informations helps you to give me some usefull advices, and please have in mind that first of all I want an easy understanding way, which has not to be a great performance in the first level!

Answer 1

HTML::Parser is more of a tokenizer than a parser. It leaves a lot of hard work up to you. Have you considered using HTML::TreeBuilder (which uses HTML::Parser) or XML::LibXML (a great library which has support for HTML)?

Answer 2

Use HTML::TokeParser::Simple.

Untested code based on your description:

#!/usr/bin/env perl

use strict; use warnings;

use HTML::TokeParser::Simple;

my $p = HTML::TokeParser::Simple->new(url => 'http://example.com/example.html');

my $level;

while (my $tag = $p->get_tag('div')) {
    my $class = $tag->get_attr('class');
    next unless defined($class) and $class eq 'foo';

    $level += 1;

    while (my $token = $p->get_token) {
        $level += 1 if $token->is_start_tag('div');
        $level -= 1 if $token->is_end_tag('div');
        print $token->as_is;
        unless ($level) {
            last;
        }
    }
}

Answer 3

No need to get so complicated. You can retrieve and find elements in the DOM using CSS selectors with Mojo::UserAgent:

say Mojo::UserAgent->new->get('http://f.oo')->res->dom->find('div.foo');

or, loop through the elements found:

say $_ for Mojo::UserAgent->new->get('http://f.oo')->res->dom
    ->find('div.foo')->each;

or, loop using a callback:

Mojo::UserAgent->new->get('http://f.oo')->res->dom->find('div.foo')->each(sub {
  my ($count, $el) = @_;
  say "$count: $el";
});

Answer 4

According to the docs, the handler's signature is (\%attr, \@attr_seq, $text). There are three shifts, one for each argument.

my ($class) = shift->{href};

is equivalent to:

my $class;
my %attr_seq;
my $attr_seq_ref;

$attr_seq_ref = shift;
%attr_seq = %$attr_seq_ref;
$class = $attr_seq{'href'};