Compress Python Object in Memory

Question

Most tutorials on compressing a file in Python involve immediately writing that file to disk with no intervening compressed python object. I want to know how to pickle and then compress a python object in memory without ever writing to or reading from disk.

Answer 1

I use this to save memory in one place:

import cPickle
import zlib

# Compress:
compressed = zlib.compress(cPickle.dumps(obj))

# Get it back:
obj = cPickle.loads(zlib.decompress(compressed))

If obj has references to a number of small objects, this can reduce the amount of memory used by a lot. A lot of small objects in Python add up because of per-object memory overhead as well as memory fragmentation.

Answer 2

Batteries included.

>>> zlib.compress(pickle.dumps([42]))
'x\x9c\xd3\xc8)0\xe0\xf241\xe2J\xd4\x03\x00\x10A\x02\x87'

Answer 3

bz2.compress(pickle.dumps(some_object))