Can Perl substitution operator match an element in an array?

Question

I have an array like this

my @stopWords = ("and","this",....)

My text is in this variable

my $wholeText = "....and so this is...."

I want to match every occurrence of every element of my stopWords array in the scalar wholeText and replace it with spaces.

One way of doing this is as follows :

foreach my $stopW (@stopWords)
{
   $wholeText =~ s/$stopW/ /;
}

This works and replaces every occurrence of all the stop words. I was just wondering, if there is a shorter way of doing it.

Like this:

$wholeText =~ s/@stopWords/ /;

The above does not seem to work though.

Answer 1

While the various map/for-based solutions will work, they'll also do regex processing of your string separately for each and every stopword. While this is no big deal in the example given, it can cause major performance issues as the target text and stopword list grow.

Jonathan Leffler and Robert P are on the right track with their suggestions of mashing all the stopwords together into a single regex, but a simple join of all the stopwords into a single alternation is a crude approach and, again, becomes inefficient if the stopword list is long.

Enter Regexp::Assemble, which will build you a much 'smarter' regex to handle all the matches at once - I've used it to good effect with lists of up to 1700 or so words to be checked against:

#!/usr/bin/env perl

use strict;
use warnings;
use 5.010;

use Regexp::Assemble;

my @stopwords = qw( and the this that a an in to );

my $whole_text = <<EOT;
Fourscore and seven years ago our fathers brought forth
on this continent a new nation, conceived in liberty, and
dedicated to the proposition that all men are created equal.
EOT

my $ra = Regexp::Assemble->new(anchor_word_begin => 1, anchor_word_end => 1);
$ra->add(@stopwords);
say $ra->as_string;

say '---';

my $re = $ra->re;
$whole_text =~ s/$re//g;
say $whole_text;

Which outputs:

\b(?:t(?:h(?:at|is|e)|o)|a(?:nd?)?|in)\b
---
Fourscore  seven years ago our fathers brought forth
on  continent  new nation, conceived  liberty, 
dedicated   proposition  all men are created equal.

Answer 2

My best solution:

$wholeText =~ s/$_//g for @stopWords;

You might want to sharpen the regexp using some \b and whitespace.

Answer 3

What about:

my $qrstring = '\b(' . (join '|', @stopWords) . ')\b';
my $qr = qr/$qrstring/;
$wholeText =~ s/$qr/ /g;

Concatenate all the words to form '\b(and|the|it|...)\b'; the parentheses around the join are necessary to give it a list context; without them, you end up with the count of the number of words). The '\b' metacharacters mark word boundaries, and therefore prevent you changing 'thousand' into 'thous'. Convert that into a quoted regular expression; apply it globally to your subject string (so that all occurrences of all stop words are removed in a single operation).

You can also do without the variable '$qr':

my $qrstring = '\b(' . (join '|', @stopWords) . ')\b';
$wholeText =~ s/$qrstring/ /g;

I don't think I'd care to maintain the code of anyone who managed to do without the variable '$qrstring'; it probably can be done, but I don't think it would be very readable.

Answer 4

My paranoid version:

$wholeText =~ s/\b\Q$_\E\b/ /gi for @stopWords;

Use \b to match word boundaries, and \Q..\E just in case any of your stopwords contains characters which may be interpreted as "special" by the regex engine.

Answer 5

You could consider using a regex join to create a single regex.

my $regex_str = join '|', map { quotemeta } @stopwords;
$string =~ /$regex_str/ /g;

Note that the quotemeta part just makes sure that any regex characters are properly escaped.