If pickling was interrupted, will unpickling necessarily always fail? - Python

Question

Suppose my attempt to write a pickle object out to disk is incomplete due to a crash. Will an attempt to unpickle the object always lead to an exception or is it possible that the fragment that was written out may be interpreted as valid pickle and the error go unnoticed?

Answer 1

Contra the other answers offered, I believe that we can make a strong argument about the recoverability of a pickle. That answer is: "Yes, an incomplete pickle always leads to an exception."

Why are we able to do this? Because the "pickle" format is in fact a small stack-based language. In a stack-based language you write code that pushes item after item on a stack, then invoke an operator that does something with the data you've accumulated. And it just so happens that a pickle has to end with the command ".", which says: "take the item now at the bottom of the stack and return it as the value of this pickle." If your pickle is chopped off early, it will not end with this command, and you will get an EOF error.

If you want to try recovering some of the data, you might have to write your own interpreter, or call into pickle.py somewhere that gets around its wanting to raise EOFError when done interpreting the stack without finding a ".". The main thing to keep in mind is that, as in most stack-based languages, big data structures are built "backwards": first you put lots of little strings or numbers on the stack, then you invoke an operation that says "put those together into a list" or "grab pairs of items on the stack and make a dictionary". So, if a pickle is interrupted, you'll find the stack full of pieces of the object that was going to be built, but you'll be missing that final code that tells you what was going to be built from the pieces.