python pickle size limit

Question

i want to pickle a large (1810392*255) numpy array. However when pickling i get an error:

[...]error: 'i' format requires -2147483648 <= number <= 2147483647

Code:

import numpy
import pickle
l=numpy.zeros((1810392,255))
f=open('file.pkl','wb')
pickle.dump(l,f,2)

Is there a size limit? Is there a workaround? If not necessary I do not want to use hdf5 or something not build into python.

I also tried numpy.savez and numpy.savez_compressed. Code:

import numpy
l=numpy.zeros((1810392,255))
numpy.savez_compressed('file.npz',l)

Saving works but when i try to load the data I get an error. Code:

import numpy
l=numpy.load('file.npz')
l['arr_0']

I need to use numpy.savez instead of numpy.save because I want to store additional data.

Answer 1

I have got the similar problem. The maximum pickle file I get is 155Mb. If you're planning to add more data, I suggest you to use a database like sqlite3 or firebase.