Python: UnicodeEncodeError when reading from stdin

Question

When running a Python program that reads from stdin, I get the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 320: ordinal not in range(128)

How can I fix it?

Note: The error occurs internal to antlr and the line looks like that:

        self.strdata = unicode(data)

Since I don't want to modify the source code, I'd like to pass in something that is acceptable.

The input code looks like that:

#!/usr/bin/python
import sys
import codecs
import antlr3
import antlr3.tree
from LatexLexer import LatexLexer
from LatexParser import LatexParser


char_stream = antlr3.ANTLRInputStream(codecs.getreader("utf8")(sys.stdin))
lexer = LatexLexer(char_stream)
tokens = antlr3.CommonTokenStream(lexer)
parser = LatexParser(tokens)
r = parser.document()

Answer 1

The problem is, that when reading from stdin, python decodes it using the system default encoding:

>>> import sys
>>> sys.getdefaultencoding()
'ascii'

The input is very likely UTF-8 or Windows-CP-1252, so the program chokes on non-ASCII-characters.

To convert sys.stdin to a stream with the proper decoder, I used:

import codecs
char_stream = codecs.getreader("utf-8")(sys.stdin)

That fixed the problem.

BTW, this is the method ANTLRs FileStream uses to open a file with given filename (instead of a given stream):

    fp = codecs.open(fileName, 'rb', encoding)
    try:
        data = fp.read()
    finally:
        fp.close()

BTW #2: For strings I found

a_string.encode(encoding) 

useful.

Answer 2

You're not getting this error on input, you're getting this error when trying to output the read data. You should be decoding data you read, and throwing the unicodes around instead of dealing with bytestrings the whole time.