ValueError: insecure string pickle

Question

When I am trying to load something I dumped using cPickle, I get the error message:

ValueError: insecure string pickle

Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.

How could I solve this problem?

Answer 1

"are much more likely than a never-observed bug in Python itself in a functionality that's used billions of times a day all over the world": it always amazes me how cross people get in these forums.

One easy way to get this problem is by forgetting to close the stream that you're using for dumping the data structure. I just did

>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: insecure string pickle

Which is why I came here in the first place, because I couldn't see what I'd done wrong.
And then I actually thought about it, rather than just coming here, and realized that I should have done:

>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> out.close() # close it to make sure it's all been written
>>> k = cPickle.load(open('xxx.dmp', 'r'))

Easy to forget. Didn't need people being told that they are idiots.

Answer 2

I've get this error in Python 2.7 because of open mode 'rb':

    with open(path_to_file, 'rb') as pickle_file:
        obj = pickle.load(pickle_file)

So, for Python 2 'mode' should be 'r'

Also, I've wondered that Python 3 doesn't support pickle format of Python 2, and in case when you'll try to load pickle file created in Python 2 you'll get:

pickle.unpicklingerror: the string opcode argument must be quoted

Answer 3

Check this thread. Peter Otten says:

A corrupted pickle. The error is raised if a string in the dump does not both start and end with " or '.

and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).

Answer 4

What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.