pickle.load fails on (protocol=4) objects from python 3.7 when using python 3.8+ (with default protocol=5)

Question

Python changed its pickle protocol to 4 in python 3.4 to 3.7 and again changed it to protocol=5 in python 3.8. How do I open older pickled files in python 3.8?

I tried:

>>> with open('data_frame_111.pkl','rb') as pfile:
...     x1 = pickle.load(pfile)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
AttributeError: Can't get attribute 'new_block' on <module 
'pandas.core.internals.blocks' from '/opt/anaconda3/lib/python3.8/site- 
packages/pandas/core/internals/blocks.py'>

and

>>> with open('data_frame_111.pkl','rb') as pfile:
...     x1 = unpkl.load(pfile, protocol=4)

but whereas protocol is a keyword in pickle.dump it is not part of pickle.load. Instantiating pickle.Unpickler() also doesn't work. But obviously there should be a way.

In python 3.7, I would import pickle5 and use that to open newer pickles, but can't find documentation on doing the reverse in python 3.8.

Answer 1

You need to upgrade to the latest version (1.3.1 worked for me) of pandas. Or, to be more precise, the pandas version when you did pickle.dump(some_path) should be the same pandas version as when you will do pickle.load(some_path).