Python Regex with IMDB Top 250 list

Question

I'm kind of just getting started with Python, and I'm trying to match the top 250 movies on IMDB with this malfunctioning code:

import urllib2
import re

def main():
    response = urllib2.urlopen('http://www.imdb.com/chart/top')
    html = response.read()
    entries = re.findall("/title/.*</font>", html) #Wrong regex
    print entries

if __name__ == "__main__":
    main()

My rationale is that I want to match everything between /title/ and </font>, hence the .* in between, but obviously this isn't the right way to go here since it simply matches the entire list instead of each individual entry. I'm pretty confused by regex tutorials I've been perusing online.... Help?

Answer 1

So, trying to parse HTML using regex is a bad practice to handle these kind of things html parsers are built. There are many option available in python like Beautiful Soup, lxml etc.

I am going to show how to use lxml with XPath expressions to fetch all the top 250 titles

import lxml
from lxml import etree
import urllib2

response = urllib2.urlopen('http://www.imdb.com/chart/top')
html = response.read()
imdb = etree.HTML(html)
titles = imdb.xpath('//div[@id="main"]/table//tr//a/text()')

if you do print titles[0] it will give 'The Shawshank Redemption' as output. For, XPath use firefox's firebug extension or install firepath

Answer 2

try this

def main(s):
    response = urllib2.urlopen('http://www.imdb.com/chart/top')
    html = response.read()
    entries = re.findall("<a.*?/title/(.*?)/\">(.*?)</a>", html) #Wrong regex
    return entries

it uses groups for the imdb id and the title. entries will be a list of tuples