Why html2text module throws UnicodeDecodeError?

Question

I have problem with html2text module...shows me UnicodeDecodeError:

UnicodeDecodeError: 'ascii' codec can't decode byte 
0xbe in position 6: ordinal not in range(128)

Example :

#!/usr/bin/python
# -*- coding: utf-8 -*-
import html2text
import urllib

h = html2text.HTML2Text()
h.ignore_links = True

html = urllib.urlopen( "http://google.com" ).read()

print h.handle( html )

...also have tried h.handle( unicode( html, "utf-8" ) with no success. Any help. EDIT :

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    print h.handle(html)
  File "/home/alex/Desktop/html2text-master/html2text.py", line 254, in handle
    return self.optwrap(self.close())
  File "/home/alex/Desktop/html2text-master/html2text.py", line 266, in close
    self.outtext = self.outtext.join(self.outtextlist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xbe in position 6: ordinal not in range(128)

Answer 1

The issue is easily reproducable when not decoding, but works just fine when you decode your source correctly. You also get the error if you reuse the parser!

You can try this out with a known good Unicode source, such as http://www.ltg.ed.ac.uk/~richard/unicode-sample.html.

If you don't decode the response to unicode, the library fails:

>>> h = html2text.HTML2Text()
>>> h.handle(html)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mj/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/html2text.py", line 240, in handle
    return self.optwrap(self.close())
  File "/Users/mj/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/html2text.py", line 252, in close
    self.outtext = self.outtext.join(self.outtextlist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

Now, if you reuse the HTML2Text object, its state is not cleared up, it still holds the incorrect data, so even passing in Unicode will now fail:

>>> h.handle(html.decode('utf8'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mj/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/html2text.py", line 240, in handle
    return self.optwrap(self.close())
  File "/Users/mj/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/html2text.py", line 252, in close
    self.outtext = self.outtext.join(self.outtextlist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

You need to use a new object and it'll work just fine:

>>> h = html2text.HTML2Text()
>>> result = h.handle(html.decode('utf8'))
>>> len(result)
12750
>>> type(result)
<type 'unicode'>