cPickle:SystemError: error return without exception set

Question

While dumping large matrix (170000*20000) as follows

 cPickle.dump(train_set,gzip.open('train.pickle.gz','wb'), cPickle.HIGHEST_PROTOCOL)

I get following error:

SystemError: error return without exception set

How would I deal in this case?

Answer 1

cPickle can't be used for storing very large objects ( see http://bugs.python.org/issue11564 ).

You have several options:

• split the data into chunks and store it in multiple files
• numpy.save
• h5py <- my favorite because of its convenient numpy-like interface, plus the data are stored in hdf5 which enables interopability with other software/languages.
• pytables