Python's "re" module not working?

Question

I'm using Python's "re" module as follows:

request = get("http://www.allmusic.com/album/warning-mw0000106792")
print re.findall('<hgroup>(.*?)</hgroup>', request)

All I'm doing is getting the HTML of this site, and looking for this particular snippet of code:

<hgroup>
    <h3 class="album-artist">
        <a href="http://www.allmusic.com/artist/green-day-mn0000154544">Green Day</a>        </h3>

    <h2 class="album-title">
        Warning        </h2>
</hgroup>

However, it continues to print an empty array. Why is this? Why can't re.findall find this snippet?

Answer 1

The HTML you are parsing is on multiple lines. You need to pass the re.DOTALL flag to findall like this:

print re.findall('<hgroup>(.*?)</hgroup>', request, re.DOTALL)

This allows . to match newlines, and returns the correct output.

@jsalonen is right, of course, that parsing HTML with regex is a tricky problem. However, in small cases like this especially for a one-off script I'd say it's acceptable.

Answer 2

re module is not broken. What you are likely encountering is the fact that not all HTML cannot be easily matched with simple regexps.

Instead, try parsing your HTML with an actual HTML parser like BeautifulSoup:

from BeautifulSoup import BeautifulSoup
from requests import get

request = get("http://www.allmusic.com/album/warning-mw0000106792")
soup = BeautifulSoup(request.content)
print soup.findAll('hgroup')

Or alternatively, with pyquery:

from pyquery import PyQuery as pq

d = pq(url='http://www.allmusic.com/album/warning-mw0000106792')
print d('hgroup')