How do I remove html entities (and more) using lxml?

Question

I've got an html file that has some text that looks like this (after running it through lxml.html parse, lxml.html clean, and this is the result of etree.tostring(table, pretty_print=True))

 <tr><td>&#13;
224&#13;
9:00 am&#13;
-3:00 pm&#13;
NPHC Leadership</td>&#13;
<td>&#13;
<font>ALSO IN 223; WALL OPEN</font></td>&#13;

The documentation that I've found on lxml has been somewhat spotty. I've been able to do quite a bit to get to this point, but what I would like to do is strip out all the tags except <table>, <td>, and <tr>. I would also like to strip all the attributes from those tags, and I would also like to get rid of the entities, such as 
.

To strip the attributes currently I use:

    etree.strip_attributes(tree, 'width', 'href', 'style', 'onchange',
                           'ondblclick', 'class', 'colspan', 'cols',
                           'border', 'align', 'color', 'value',
                           'cellpadding', 'nowrap', 'selected',
                           'cellspacing')

which works fine, but it seems like there should be a better way. It seems like there should be some fairly simple methods to do what I want, but I haven't been able to find any examples that worked right for me.

I tried using Cleaner, but when I passed it allow_tags, like this:

error: Cleaner(allow_tags=['table', 'td', 'tr']).clean_html(tree) it gave me this error:

ValueError: It does not make sense to pass in both allow_tags and remove_unknown_tags. Also, when I add remove_unkown_tags=False I get this error:

Traceback (most recent call last):
  File "parse.py", line 73, in <module>
    SParser('schedule.html').test()
  File "parse.py", line 38, in __init__
    self.clean()
  File "parse.py", line 42, in clean
    Cleaner(allow_tags=['table', 'td', 'tr'], remove_unknown_tags=False).clean_html(tree)
  File "/usr/lib/python2.6/dist-packages/lxml/html/clean.py", line 488, in clean_html
    self(doc)
  File "/usr/lib/python2.6/dist-packages/lxml/html/clean.py", line 390, in __call__
    el.drop_tag()
  File "/usr/lib/python2.6/dist-packages/lxml/html/__init__.py", line 191, in drop_tag
    assert parent is not None
AssertionError

So, to sum up:

1. I want to remove HTML entities, such as 

2. I want to remove all tags except <table>, <tr>, and <td>
3. I want to remove all the attributes from the remaining tags.

Any help would be greatly appreciated!

Answer 1

Here is an example of stripping out all attributes and allowing only tags in [table, tr, td]. I've added a few Unicode entities for sake of illustration.

DATA = '''<table border="1"><tr colspan="4"><td rowspan="2">\r
224&#13;
&#8220;hi there&#8221;
9:00 am\r
-3:00 pm&#13;
NPHC Leadership</td>\r
<td rowspan="2">\r
<font>ALSO IN 223; WALL OPEN</font></td>\r
</table>'''

import lxml.html
from lxml.html import clean

def _clean_attrib(node):
    for n in node:
        _clean_attrib(n)
    node.attrib.clear()

tree = lxml.html.fromstring(DATA)
cleaner = clean.Cleaner(allow_tags=['table','tr','td'],
                        remove_unknown_tags=False)
cleaner.clean_html(tree)
_clean_attrib(tree)

print lxml.html.tostring(tree, encoding='utf-8', pretty_print=True, 
                         method='html')

Result:

<table><tr>
<td>
224
“hi there”
9:00 am
-3:00 pm
NPHC Leadership</td>
<td>
<font>ALSO IN 223; WALL OPEN</font>
</td>
</tr></table>

Are you sure you want to strip out all entities? The 
 corresponds to a carriage return, and when lxml parses the document it converts all entities to their corresponding Unicode characters.

Whether entities show up is also dependent on the output method and encoding. For example, if you use lxml.html.tostring(encoding='ascii', method='xml') the '\r' and Unicode characters will be output as entities:

<table>
  <tr><td>&#13;
  &#8220;hi there&#8221;
...

Answer 2

For me, I find writing it based on the basic elements of text, tag and tail makes it much easier to specialize the behaviour to what you want and include error checking (eg to ensure there are no unexpected tags in the incoming data).

The if statements on the text and tail are because they return None rather than "" when zero length.

def ctext(el):
    result = [ ]
    if el.text:
        result.append(el.text)
    for sel in el:
        if sel.tag in ["tr", "td", "table"]:
            result.append("<%s>" % sel.tag)
            result.append(ctext(sel))
            result.append("</%s>" % sel.tag)
        else:
            result.append(ctext(sel))
        if sel.tail:
            result.append(sel.tail)
    return "".join(result)

html = """your input string"""
el = lxml.html.fromstring(html)
print ctext(el)

Remember the relationship is:

  <b>text of the bold <i>text of the italic</i> tail of the italic</b>