BeautifulSoup' has no attribute 'HTML_ENTITIES

Question

I have recently upgrade BeautifulSoup from version 3.0 to version 4.1 on a Windows machine.

I am now getting a strange error:

File "C:\path\to\myscript.py", line 23
0, in soupify
    return BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)
AttributeError: type object 'BeautifulSoup' has no attribute 'HTML_ENTITIES'

Here is the snippet of code that causes the exception to be thrown:

def soupify(html):
    return BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)

The doc for BS does not mention how the constructor signature has changed fro v3 to v4. How may I fix this?

Answer 1

An incoming HTML or XML entity is always converted into the corresponding Unicode character. Beautiful Soup 3 had a number of overlapping ways of dealing with entities, which have been removed. The BeautifulSoup constructor no longer recognizes the smartQuotesTo or convertEntities arguments. (Unicode, Dammit still has smart_quotes_to, but its default is now to turn smart quotes into Unicode.)

If you want to turn those Unicode characters back into HTML entities on output, rather than turning them into UTF-8 characters, you need to use an output formatter.

Source: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#entities