Python NLTK snowball stemmer UnicodeDecodeError in terminal but not Eclipse PyDev

Question

I'm using the snowball stemmer to stem words in documents as shown in below code snippet.

    stemmer = EnglishStemmer()
    # Stem, lowercase, substitute all punctuations, remove stopwords.
    attribute_names = [stemmer.stem(token.lower()) for token in wordpunct_tokenize(re.sub('[%s]' % re.escape(string.punctuation), '', doc)) if token.lower() not in stopwords.words('english')]

When I run this on documents using PyDev in Eclipse, I receive no errors. When I run it in the Terminal (Mac OSX) I receive below error. Can someone please help?

File "data_processing.py", line 171, in __filter__
attribute_names = [stemmer.stem(token.lower()) for token in   wordpunct_tokenize(re.sub('[%s]' % re.escape(string.punctuation), '', doc)) if token.lower()     not in stopwords.words('english')]

File "7.3/lib/python2.7/site-packages/nltk-2.0.4-py2.7.egg/nltk/stem/snowball.py", line   694, in stem
word = (word.replace(u"\u2019", u"\x27")

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 7: ordinal not in range(128)

Answer 1

This works in PyDev because it configures Python itself to work in the encoding of the console (which is usually UTF-8).

You can reproduce the same error in PyDev if you go to the run configuration (run > run configurations) then on the 'common' tab say that you want the encoding to be ascii.

This happens because your word is a string and you're replacing with unicode chars.

I hope the code below sheds some light for you:

This is all considering ascii as the default encoding:

>>> 'íã'.replace(u"\u2019", u"\x27")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa1 in position 0: ordinal not in range(128)

But if you do it all in unicode, it works (you may need to encode it back afterwards to the encoding you expect if you expect to deal with strings and not unicode).

>>> u'íã'.replace(u"\u2019", u"\x27")
u'\xed\xe3'

So, you can make your string unicode before the replace

>>> 'íã'.decode('cp850').replace(u"\u2019", u"\x27")
u'\xed\xe3'

Or you can encode the replace chars

>>> 'íã'.replace(u"\u2019".encode('utf-8'), u"\x27".encode('utf-8'))
'\xa1\xc6'

Note however that you must know what's the actual encoding you're working on in any place (so, although I'm using cp850 or utf-8 in the examples, it may be different from the encodings you have to use)