Reading utf-8 characters from a gzip file in python

Question

I am trying to read a gunzipped file (.gz) in python and am having some trouble.

I used the gzip module to read it but the file is encoded as a utf-8 text file so eventually it reads an invalid character and crashes.

Does anyone know how to read gzip files encoded as utf-8 files? I know that there's a codecs module that can help but I can't understand how to use it.

Thanks!

import string import gzip import codecs  f = gzip.open('file.gz','r')  engines = {} line = f.readline() while line:     parsed = string.split(line, u'\u0001')      #do some things...      line = f.readline() for en in engines:   print(en) 

Answer 1

This is possible since Python 3.3:

import gzip gzip.open('file.gz', 'rt', encoding='utf-8') 

Notice that gzip.open() requires you to explicitly specify text mode ('t').

Answer 2

I don't see why this should be so hard.

What are you doing exactly? Please explain "eventually it reads an invalid character".

It should be as simple as:

import gzip fp = gzip.open('foo.gz') contents = fp.read() # contents now has the uncompressed bytes of foo.gz fp.close() u_str = contents.decode('utf-8') # u_str is now a unicode string 

---

EDITED

This answer works for Python2 in Python3, please see @SeppoEnarvi 's answer at https://stackoverflow.com/a/19794943/610569 (it uses the rt mode for gzip.open.