Web Scraping Python Access Table data

Question

So I am trying to use Beautiful Soup to do some web scraping of this website http://www.killedbypolice.net/kbp2013.html and access the data in the table. My current code is:

url = "http://www.killedbypolice.net/kbp2013.html"
page = urllib.request.urlopen(url)
soup = BeautifulSoup(page, "html.parser")

data = soup.find_all('table')
data[0]

But... I am getting a maximum recursion depth runtime error. I'm not sure how to access the 'td' fields inside of table which hold the data. Thanks

Answer 1

The error is because the html is very badly formatted, you get RuntimeError: maximum recursion depth exceeded creating the soup object with both lxml and html.parser, the only parser that works at all is html5lib:

html = requests.get("http://www.killedbypolice.net/kbp2013.html").content
soup = BeautifulSoup(html, "html5lib")

# get all the table rows
table = soup.find("table")

That gets the table:

    <table background="/kbp/bg1.jpg" border="1" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td><img src="http://www.killedbypolice.net/size.jpg" width="185"/><br/><b><center># since Jan 1st '14</center></b></td>
<td><b><center>St.</center></b></td>
<td><b><center>g/r</center></b></td>
<td><img src="http://www.killedbypolice.net/size.jpg" width="200"/><b><center>Name, Age</center></b></td>
<td></td>
<td><img src="http://www.killedbypolice.net/size.jpg" width="297"/><b><center>KBP link <font color="red">(plus extensive follow-ups)</font></center></b></td>
<td><b><center>News link</center></b></td>
......................................................................
</font></a></td></tr><tr><td><center>(2) May 2, 2013        </center>
</td><td>CA</td><td>M/B</td><td>Kenneth Bernard Williams, 55   </td><td><font size="2">G</font></td><td><a href="http://facebook.com/KilledByPolice/posts/622539181107556" target="new"><font size="2"></font></a><font size="2"><center><a href="http://facebook.com/KilledByPolice/posts/622539181107556" target="new">facebook.com/KilledByPolice/posts/622539181107556  </a></center></font></td><td><a href="http://www.nbclosangeles.com/news/local/Police-Shoot-Kill-Suspect-in-Skid-Row-Prompting-Angry-Crowd-to-Gather-205646861.html" target="new"><font size="2">http://www.nbclosangeles.com/news/local/Police-Shoot-Kill-Suspect-in-Skid-Row-Prompting-Angry-Crowd-to-Gather-205646861.html
</font></a></td></tr><tr><td><center>(1) May 1, 2013        </center></td><td>MI</td><td>M/B</td><td><a href="http://www.killedbypolice.net/victims/2261.jpg" target="new">Jordan West-Morson, 26   </a></td><td><font size="2">G</font></td><td><a href="http://facebook.com/KilledByPolice/posts/1033800406648096" target="new"></a><center><a href="http://facebook.com/KilledByPolice/posts/1033800406648096" target="new"><font size="2">facebook.com/KilledByPolice/posts/1033800406648096    </font></a></center></td><td><a href="http://www.mlive.com/news/detroit/index.ssf/2013/09/detroit_transit_officer_charge.html" target="new"><font size="2">http://www.mlive.com/news/detroit/index.ssf/2013/09/detroit_transit_officer_charge.html</font></a><font size="2"><br/><i>Detroit transit officer not guilty in fatal shooting: </i><a href="http://www.clickondetroit.com/news/detroit-transit-officer-not-guilty-in-fatal-shooting/32405878" target="new">http://www.clickondetroit.com/news/detroit-transit-officer-not-guilty-in-fatal-shooting/32405878

</a></font></td></tr></tbody></table>

But then a simple call to find_all:

print(table.find_all("tr"))

Gives you:

 AttributeError: 'NoneType' object has no attribute 'next_element'

The html is just a complete mess, unfortunately I cannot see a simple way to parse it with bs4, this may be one of the rare occasions you need to resort to some regex.