How can I open UTF-16 files on Python 2.x?

Question

I'm working on a Python tool that must be able to open files of UTF-8 and UTF-16 encoding. In Python 3.2, I use the following code to try opening the file using UTF-8, then try it with UTF-16 if there's a unicode error:

def readGridFromPath(self, filepath):
    try:
        self.readGridFromFile(open(filepath,'r',encoding='utf-8'))
    except UnicodeDecodeError:
            self.readGridFromFile(open(filepath,'r',encoding='utf-16'))

(readGridFromFile will either run through to completion, or raise a UnicodeDecodeError. )

However, when I run this code in Python 2.x, I get:

TypeError: 'encoding' is an invalid keyword argument for this function

I see in the docs that Python 2.x's open() doesn't have an encoding keyword. Is there any way around this that will allow me to make my code Python 2.x compatible?

Answer 1

io.open is drop-in replacement for your needs, so code sample you've provided will look as follows in Python 2.x:

import io

def readGridFromPath(self, filepath):
    try:
        self.readGridFromFile(io.open(filepath, 'r', encoding='utf-8'))
    except UnicodeDecodeError:
        self.readGridFromFile(io.open(filepath, 'r', encoding='utf-16'))

io.open is described here in detail. Its prototype is:

io.open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True)

io module itself was designed as compatibility layer between Python 2.x and Python 3.x, to ease transition to Py3k and simplify back-porting and maintenance of existing Python 2.x code.

Also, please note that there can be a caveat using codecs.open, as it works in binary mode only:

Note: Files are always opened in binary mode, even if no binary mode was specified. This is done to avoid data loss due to encodings using 8-bit values. This means that no automatic conversion of '\n'` is done on reading and writing.

Also you may run into issues of manually detecting and stripping out UTF8 BOM — codecs.open leaves UTF8 BOM inline as u'\ufeff' character.