Python BeautifulSoup - Using find_next while parsing invalid HTML

Question

I am using Beautifulsoup in python to parse this webpage.

My goal is to get the table after the headline "Autre compétitions":

page = BeautifulSoup(requests.get(website_link,proxies=proxy).text,'html.parser')
page.find("h3",text=u'Autres comp\xe9titions').find_next("table")

I manage to get the page, but the problem is the result I get with find is only the first cell of the header of the table:

<table class="gradient" id="tosort">
<tr>
<th class="gradient">Type</th></tr></table>

I thought that the problem comes from page being in unicode, so:

page = BeautifulSoup(requests.get(path,proxies=proxy).text,'html.parser')
page.find("h3",text=u'Autres comp\xe9titions'.encode('utf-8')).find_next("table")

but I get the same result.

Thanks.

Answer 1

Change your parser from html.parser to lxml or html5lib.

If you don't have either of those parser installed, here is the relevant Beautiful soup documentation on how to install different parsers.

BeautifulSoup(requests.get(website_link,proxies=proxy).text, 'lxml')

The HTML source code of the page that you are trying to scrape is invalid.

I just tested this out locally and the lxml and html5lib parsers are much more lenient and they were able to properly select the table element despite the invalid HTML. According to the documentation, the html.parser is not very lenient depending on your Python version.

More specifically, the invalid HTML that was causing the problem for the html.parser were the stray div tags in the th elements:

<th class='gradient'>Type</div></td>

As a result, the html.parser wasn't able to parse past the stray div tags, which resulted in it being unable to select the entire table element.