Python ungzipping stream of bytes?

Question

Here is the situation:

• I get gzipped xml documents from Amazon S3

    import boto   from boto.s3.connection import S3Connection   from boto.s3.key import Key   conn = S3Connection('access Id', 'secret access key')   b = conn.get_bucket('mydev.myorg')   k = Key(b)   k.key('documents/document.xml.gz') 

• I read them in file as

    import gzip   f = open('/tmp/p', 'w')   k.get_file(f)   f.close()   r = gzip.open('/tmp/p', 'rb')   file_content = r.read()   r.close() 

Question

How can I ungzip the streams directly and read the contents?

I do not want to create temp files, they don't look good.

Answer 1

Yes, you can use the zlib module to decompress byte streams:

import zlib  def stream_gzip_decompress(stream):     dec = zlib.decompressobj(32 + zlib.MAX_WBITS)  # offset 32 to skip the header     for chunk in stream:         rv = dec.decompress(chunk)         if rv:             yield rv 

The offset of 32 signals to the zlib header that the gzip header is expected but skipped.

The S3 key object is an iterator, so you can do:

for data in stream_gzip_decompress(k):     # do something with the decompressed data 

Answer 2

I had to do the same thing and this is how I did it:

import gzip f = StringIO.StringIO() k.get_file(f) f.seek(0) #This is crucial gzf = gzip.GzipFile(fileobj=f) file_content = gzf.read()