Python - mechanism to identify compressed file type and uncompress

Question

A compressed file can be classified into below logical groups
a. The operating system which you are working on (*ix, Win) etc.
b. Different types of compression algorithm (i.e .zip,.Z,.bz2,.rar,.gzip). Atleast from a standard list of mostly used compressed files.
c. Then we have tar ball mechanism - where I suppose there are no compression. But it acts more like a concatenation.

Now, if we start addressing the above set of compressed files,
a. Option (a) would be taken care by python since it is platform independent language.
b. Option (b) and (c) seems to have a problem.

What do I need
How do I identify the file type (compression type) and then UN-compress them?

---

Like:

fileType = getFileType(fileName)  
switch(fileType):  
case .rar:  unrar....
case .zip:  unzip....

etc  

---

So the fundamental question is how do we identify the compression algorithm based on the file (assuming the extension is not provided or incorrect)? Is there any specific way to do it in python?

Answer 1

This page has a list of "magic" file signatures. Grab the ones you need and put them in a dict like below. Then we need a function that matches the dict keys with the start of the file. I've written a suggestion, though it can be optimized by preprocessing the magic_dict into e.g. one giant compiled regexp.

magic_dict = {
    "\x1f\x8b\x08": "gz",
    "\x42\x5a\x68": "bz2",
    "\x50\x4b\x03\x04": "zip"
    }

max_len = max(len(x) for x in magic_dict)

def file_type(filename):
    with open(filename) as f:
        file_start = f.read(max_len)
    for magic, filetype in magic_dict.items():
        if file_start.startswith(magic):
            return filetype
    return "no match"

This solution should be cross-plattform and is of course not dependent on file name extension, but it may give false positives for files with random content that just happen to start with some specific magic bytes.

Answer 2

Based on lazyr's answer and my comment, here is what I mean:

class CompressedFile (object):
    magic = None
    file_type = None
    mime_type = None
    proper_extension = None

    def __init__(self, f):
        # f is an open file or file like object
        self.f = f
        self.accessor = self.open()

    @classmethod
    def is_magic(self, data):
        return data.startswith(self.magic)

    def open(self):
        return None

import zipfile

class ZIPFile (CompressedFile):
    magic = '\x50\x4b\x03\x04'
    file_type = 'zip'
    mime_type = 'compressed/zip'

    def open(self):
        return zipfile.ZipFile(self.f)

import bz2

class BZ2File (CompressedFile):
    magic = '\x42\x5a\x68'
    file_type = 'bz2'
    mime_type = 'compressed/bz2'

    def open(self):
        return bz2.BZ2File(self.f)

import gzip

class GZFile (CompressedFile):
    magic = '\x1f\x8b\x08'
    file_type = 'gz'
    mime_type = 'compressed/gz'

    def open(self):
        return gzip.GzipFile(self.f)


# factory function to create a suitable instance for accessing files
def get_compressed_file(filename):
    with file(filename, 'rb') as f:
        start_of_file = f.read(1024)
        f.seek(0)
        for cls in (ZIPFile, BZ2File, GZFile):
            if cls.is_magic(start_of_file):
                return cls(f)

        return None

filename='test.zip'
cf = get_compressed_file(filename)
if cf is not None:
    print filename, 'is a', cf.mime_type, 'file'
    print cf.accessor

Can now access the compressed data using cf.accessor. All the modules provide similar methods like 'read()', 'write()', etc. to do this.