perl - how to get all similar matched substrings using RegEx

Question

I want to parse a webpage (from lynda.com) and get all titles and links of courses. So I used LWP::UserAgent to get the url and then I tried to get titles and links using the following codes:

#!/usr/bin/perl

use LWP::UserAgent;
use strict;
use warnings;

my $ua = new LWP::UserAgent;
my $response = $ua->get('http://www.lynda.com/search?q=android');
unless ($response->is_success) {
    die $response->status_line;
}
my $content = $response->decoded_content();
if (utf8::is_utf8($content)) {
    binmode STDOUT,':utf8';
} else {
    binmode STDOUT,':raw';
}

$content =~ s/[\h\v]+/ /g;
$content =~ s/\r|\n//g; 
$content =~ s/<\/span>|<span>//g;

# Links
my @links = ($content =~ m/<a id="course-result-info.*href="(.*)\?.*class="title">/g);

# titles
my @titles = ($content =~ m/<a id="course-result-info.*class="title"> (.*)<\/a> <span class="author">/g);

print join(", ", @titles);
print "\n---------------------------\n";
print join(", ", @links);

But I only got the last matched one (i.e. Developing Applications for Amazon Kindle Devices).

Answer 1

Your regex parsing was broken because you were using greedy matching .* instead of non-greedy .*?.

However, the much better solution is not to use regular expressions at all, but to instead use an actual HTML Parser such as Mojo::DOM with the full power of css selectors.

The following uses Mojo::UserAgent which is installed with Mojolicious to download the webpage and then uses Mojo::DOM to parse the returned results. There is a nice 8 minute tutorial on this framework at Mojocast Episode 5.

#!/usr/bin/perl

use strict;
use warnings;

use Mojo::UserAgent;

my $url = 'http://www.lynda.com/search?q=android';

my $dom = Mojo::UserAgent->new->get($url)->res->dom;

# Process all links
for my $link ($dom->find('a[id^="course-result-info"]')->each) {
    printf "%s\n    %s\n", $link->all_text(), $link->{href};
}

Outputs:

Android Studio First Look
    http://www.lynda.com/Android-tutorials/Android-Studio-First-Look/143103-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Android SDK Essential Training
    http://www.lynda.com/Android-tutorials/Android-SDK-Essential-Training/143102-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Distributing Android Apps
    http://www.lynda.com/Android-tutorials/Distributing-Android-Apps/143101-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Building and Monetizing Game Apps for Android
    http://www.lynda.com/Android-tutorials/Building-Monetizing-Game-Apps-Android/107169-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Building a Note-Taking App for Android
    http://www.lynda.com/Android-tutorials/Building-Note-Taking-App-Android/122466-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Android SDK: Local Data Storage
    http://www.lynda.com/Android-tutorials/Android-SDK-Local-Data-Storage/112584-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Android 4.1 SDK Jelly Bean New Features
    http://www.lynda.com/Android-tutorials/Android-SDK-41-Jelly-Bean-New-Features/107922-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Building Mobile Apps with Google Maps Android API v2
    http://www.lynda.com/Android-tutorials/Building-Mobile-Apps-Google-Maps-Android-API-v2/133347-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Building Android and iOS Applications with Flex
    http://www.lynda.com/Flash-Builder-4-5-tutorials/Building-Android-and-iOS-Applications-with-Flex/80254-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Flash Professional CS5: Creating a Simple Game for Android Devices
    http://www.lynda.com/Flash-CS5-tutorials/flash-professional-cs5-creating-a-simple-game-for-android-devices/74928-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Java Essential Training
    http://www.lynda.com/Java-tutorials/Essential-Training/86005-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Up and Running with Java Applications
    http://www.lynda.com/Android-tutorials/Up-Running-Java-Applications/94344-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Building Mobile Apps for Multiple Devices with Flash Professional
    http://www.lynda.com/Flash-Professional-CS5-5-tutorials/Building-Mobile-Apps-for-Multiple-Devices-with-Flash-Professional/89049-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2
Developing Applications for Amazon Kindle Devices
    http://www.lynda.com/Android-tutorials/Developing-Applications-Amazon-Kindle-Devices/117102-2.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aandroid%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2