Beautiful Soup to parse url to get another urls data

Question

I need to parse a url to get a list of urls that link to a detail page. Then from that page I need to get all the details from that page. I need to do it this way because the detail page url is not regularly incremented and changes, but the event list page stays the same.

Basically:

example.com/events/     <a href="http://example.com/events/1">Event 1</a>     <a href="http://example.com/events/2">Event 2</a>  example.com/events/1     ...some detail stuff I need  example.com/events/2     ...some detail stuff I need 

Answer 1

import urllib2 from BeautifulSoup import BeautifulSoup  page = urllib2.urlopen('http://yahoo.com').read() soup = BeautifulSoup(page) soup.prettify() for anchor in soup.findAll('a', href=True):     print anchor['href'] 

It will give you the list of urls. Now You can iterate over those urls and parse the data.

• inner_div = soup.findAll("div", {"id": "y-shade"}) This is an example. You can go through the BeautifulSoup tutorials.