How to save a pandas dataframe in gzipped format directly? [duplicate]

Question

I have a pandas data frame, called df.

I want to save this in a gzipped format. One way to do this is the following:

import gzip
import pandas

df.save('filename.pickle')
f_in = open('filename.pickle', 'rb')
f_out = gzip.open('filename.pickle.gz', 'wb')
f_out.writelines(f_in)
f_in.close()
f_out.close()

However, this requires me to first create a file called filename.pickle. Is there a way to do this more directly, i.e., without creating the filename.pickle?

When I want to load the dataframe that has been gzipped I have to go through the same step of creating filename.pickle. For example, to read a file filename2.pickle.gzip, which is a gzipped pandas dataframe, I know of the following method:

f_in = gzip.open('filename2.pickle.gz', 'rb')
f_out = gzip.open('filename2.pickle', 'wb')
f_out.writelines(f_in)
f_in.close()
f_out.close()

df2 = pandas.load('filename2.pickle')

Can this be done without creating filename2.pickle first?

Answer 1

Better serialization with compression has recently been added to Pandas. (Starting in pandas 0.20.0.) Here is an example of how it can be used:

df.to_csv("my_file.gz", compression="gzip")

For more information, such as different forms of compression available, check out the docs.

Answer 2

For some reason, the Python zlib module has the ability to decompress gzip data, but it does not have the ability to directly compress to that format. At least as far as what is documented. This is despite the remarkably misleading documentation page header "Compression compatible with gzip".

You can compress to the zlib format instead using zlib.compress or zlib.compressobj, and then strip the zlib header and trailer and add a gzip header and trailer, since both the zlib and gzip formats use the same compressed data format. This will give you data in the gzip format. The zlib header is fixed at two bytes and the trailer at four bytes, so those are easy to strip. Then you can prepend a basic gzip header of ten bytes: "\x1f\x8b\x08\0\0\0\0\0\0\xff" (C string format) and append a four-byte CRC in little-endian order. The CRC can be computed using zlib.crc32.