lxml / BeautifulSoup parser warning

Question

Using Python 3, I'm trying to parse ugly HTML (which is not under my control) by using lxml with BeautifulSoup as explained here: http://lxml.de/elementsoup.html

Specifically, I want to use lxml, but I'd like to use BeautifulSoup because like I said, it's ugly HTML and lxml will reject it on its own.

The link above says: "All you need to do is pass it to the fromstring() function:"

from lxml.html.soupparser import fromstring
root = fromstring(tag_soup)

So that's what I'm doing:

URL = 'http://some-place-on-the-internet.com'
html_goo = requests.get(URL).text
root = fromstring(html_goo)

It works in the sense that I can manipulate the HTML just fine after that. My problem is that every time I run the script, I receive this annoying warning:

/usr/lib/python3/dist-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

To get rid of this warning, change this:

 BeautifulSoup([your markup])

to this:

 BeautifulSoup([your markup], "html.parser")

  markup_type=markup_type))

My problem is perhaps obvious: I'm not instantiating BeautifulSoup myself. I've tried adding the proposed parameter to the fromstring function, but that just gives me the error: TypeError: 'str' object is not callable. Searches online have proven fruitless so far.

I'd like to get rid of that warning message. Help appreciated, thanks in advance.

Answer 1

For others init like:

soup = BeautifulSoup(html_doc)

Use

soup = BeautifulSoup(html_doc, 'html.parser')

instead

Answer 2

While using the BeautifulSoup, we always do the things like below:

[variable] = BeautifulSoup([contents you want to analyze])

Here is the problem:

If you have installed "lxml" before, BeautifulSoup will automatically notice that it used it as the praser. It's not the error, just a notification.

So how to remove it?

Just do this like below:

[variable] = BeautifulSoup([contents you want to analyze], features = "lxml")

"Based on the latest version of BeautifulSoup, 4.6.3"

Notice that different versions of BeautifulSoup have different ways, or the grammar, to add this pattern, just look at the notice message carefully.

Good luck!