Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Beautifulsoup sibling structure with br tags

I'm trying to parse a HTML document using the BeautifulSoup Python library, but the structure is getting distorted by <br> tags. Let me just give you an example.

Input HTML:

<div>
  some text <br>
  <span> some more text </span> <br>
  <span> and more text </span>
</div>

HTML that BeautifulSoup interprets:

<div>
  some text
  <br>
    <span> some more text </span>
    <br>
      <span> and more text </span>
    </br>
  </br>
</div>

In the source, the spans could be considered siblings. After parsing (using the default parser), the spans are suddenly no longer siblings, as the br tags became part of the structure.

The solution I can think of to solve this is to strip the <br> tags altogether, before pouring the html into Beautifulsoup, but that doesn't seem very elegant, as it requires me to change the input. What's a better way to solve this?

like image 711
Joost Avatar asked Jul 14 '13 11:07

Joost


2 Answers

Your best bet is to extract() the line breaks. It's easier than you think :).

>>> from bs4 import BeautifulSoup as BS
>>> html = """<div>
...   some text <br>
...   <span> some more text </span> <br>
...   <span> and more text </span>
... </div>"""
>>> soup = BS(html)
>>> for linebreak in soup.find_all('br'):
...     linebreak.extract()
... 
<br/>
<br/>
>>> print soup.prettify()
<html>
 <body>
  <div>
   some text
   <span>
    some more text
   </span>
   <span>
    and more text
   </span>
  </div>
 </body>
</html>
like image 129
TerryA Avatar answered Sep 20 '22 08:09

TerryA


You could also do something like that:

str(soup).replace("</br>", "")
like image 28
jns Avatar answered Sep 20 '22 08:09

jns