Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can mmap and gzip collaborate?

Tags:

python

gzip

mmap

I'm trying to figure how to use mmap with a gzip compressed file. Is that even possible ?

import mmap
import os
import gzip

filename = r'C:\temp\data.gz'

file = gzip.open(filename, "rb+")
size = os.path.getsize(filename)

file = mmap.mmap(file.fileno(), size)

print file.read(8)

The output data is compressed.

like image 510
mab Avatar asked Feb 26 '11 15:02

mab


2 Answers

You can do easilly. Indeed the gzip module gets as optional argument a file-like object.

import mmap
import gzip

filename = "a.gz"
handle = open(filename, "rb")
mapped = mmap.mmap(handle.fileno(), 0, access=mmap.ACCESS_READ)
gzfile = gzip.GzipFile(mode="r", fileobj=mapped)

print gzfile.read()

The same applies to tarfile module:

import sys
import mmap
import tarfile

f = open(sys.argv[1], 'rb')
fo = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
tf = tarfile.open(mode='r:gz', fileobj=fo)

print tf.getnames()
like image 105
nopper Avatar answered Nov 11 '22 10:11

nopper


Well, not the way you want.

mmap() can be used to access the gzipped file if the compressed data is what you want.

mmap() is a system call for mapping disk blocks into RAM almost as if you were adding swap.

You can't map the uncompressed data into RAM with mmap() as it is not on the disk.

like image 21
Joshua Avatar answered Nov 11 '22 10:11

Joshua