Python gzip refuses to read uncompressed file

Question

I seem to remember that the Python gzip module previously allowed you to read non-gzipped files transparently. This was really useful, as it allowed to read an input file whether or not it was gzipped. You simply didn't have to worry about it.

Now,I get an IOError exception (in Python 2.7.5):

   Traceback (most recent call last):
  File "tst.py", line 14, in <module>
    rec = fd.readline()
  File "/sw/lib/python2.7/gzip.py", line 455, in readline
    c = self.read(readsize)
  File "/sw/lib/python2.7/gzip.py", line 261, in read
    self._read(readsize)
  File "/sw/lib/python2.7/gzip.py", line 296, in _read
    self._read_gzip_header()
  File "/sw/lib/python2.7/gzip.py", line 190, in _read_gzip_header
    raise IOError, 'Not a gzipped file'
IOError: Not a gzipped file

If anyone has a neat trick, I'd like to hear about it. Yes, I know how to catch the exception, but I find it rather clunky to first read a line, then close the file and open it again.

Answer 1

The best solution for this would be to use something like https://github.com/ahupp/python-magic with libmagic. You simply cannot avoid at least reading a header to identify a file (unless you implicitly trust file extensions)

If you're feeling spartan the magic number for identifying gzip(1) files is the first two bytes being 0x1f 0x8b.

In [1]: f = open('foo.html.gz')
In [2]: print `f.read(2)`
'\x1f\x8b'

gzip.open is just a wrapper around GzipFile, you could have a function like this that just returns the correct type of object depending on what the source is without having to open the file twice:

#!/usr/bin/python

import gzip

def opener(filename):
    f = open(filename,'rb')
    if (f.read(2) == '\x1f\x8b'):
        f.seek(0)
        return gzip.GzipFile(fileobj=f)
    else:
        f.seek(0)
        return f

Answer 2

Maybe you're thinking of zless or zgrep, which will open compressed or uncompressed files without complaining.

Can you trust that the file name ends in .gz?

if file_name.endswith('.gz'):
    opener = gzip.open
else:
    opener = open

with opener(file_name, 'r') as f:
    ...