Parse HTML using Python and Beautiful Soup

Question

<div class="profile-row clearfix"><div class="profile-row-header">Member Since</div><div class="profile-information">January 2010</div></div>
<div class="profile-row clearfix"><div class="profile-row-header">AIGA Chapter</div><div class="profile-information">Alaska</div></div>
<div class="profile-row clearfix"><div class="profile-row-header">Title</div><div class="profile-information">Owner</div></div>
<div class="profile-row clearfix"><div class="profile-row-header">Company</div><div class="profile-information">Mad Dog Graphx</div></div>

I'm using Beautiful Soup to get to this point in HTML code. I now want to search through the code, and pull the data like January 2010, Alaska, Owner, and Mad Dog Graph. All this data has the same class but they have different variables like "Member Since", "AIGA Chapter," etc. before hand. How can I search for Member Since and then get January 2010. And do the same for the other 3 fields?

Answer 1

>>> from BeautifulSoup import BeautifulSoup
>>> soup = BeautifulSoup('''<div class="profile-row clearfix"><div class="profile-row-header">Member Since</div><div class="profile-information">January 2010</div></div>
... <div class="profile-row clearfix"><div class="profile-row-header">AIGA Chapter</div><div class="profile-information">Alaska</div></div>
... <div class="profile-row clearfix"><div class="profile-row-header">Title</div><div class="profile-information">Owner</div></div>
... <div class="profile-row clearfix"><div class="profile-row-header">Company</div><div class="profile-information">Mad Dog Graphx</div></div>
... ''')
>>> for row in soup.findAll('div', {'class':'profile-row clearfix'}):
...  field, value = row.findAll(text = True)
...  print field, value
... 
Member Since January 2010
AIGA Chapter Alaska
Title Owner
Company Mad Dog Graphx

You can of course do anything you want with field and value, like create a dict with them or store them in a database.

If there are other divs or other text nodes within the "profile-row clearfix" div, you'll need to do something like field = row.find('div', {'class':'profile-row-header'}).findAll(text=True), etc.