How __reduce__ function exactly works in case of pickle module?

Question

I don't understand how __reduce__ function exactly works in case of pickle module in Python.

Suppose that I have the following class:

class Foo(object):
  def __init__(self, file_name = 'file.txt'):
    self.file_name = file_name
    self.f = open(self.file_name, 'w')

It can't be pickled because pickle module doesn't know how to encode file handle:

foo = Foo()
print(pickle.dumps(foo))

Output:

TypeError: can't pickle file objects

But if I add the __reduce__ function it successfully encodes:

import pickle

class Foo(object):
  def __init__(self, file_name = 'file.txt'):
    self.file_name = file_name
    self.f = open(self.file_name, 'w')

  def __reduce__(self):
    return (self.__class__, (self.file_name, ))

foo = Foo()
print(pickle.dumps(foo))

Output:

c__main__
Foo
p0
(S'file.txt'
p1
tp2
Rp3
.

Am I right that the __reduce__ function simply returns "instructions" for the deconstructor to re-create the original object if the pickle.dumps call failed?

It is unclear for me from the documentation.

Answer 1

You're right. The __reduce__ method should return hints how to reconstruct (unpickle) the object in case it cannot be pickled automatically. It may contain an object reference and parameters with which it will be called to create an initial version of the object, object's state, etc.

From the documentation:

If a string is returned, the string should be interpreted as the name of a global variable. It should be the object’s local name relative to its module; the pickle module searches the module namespace to determine the object’s module. This behaviour is typically useful for singletons.

When a tuple is returned, it must be between two and five items long. Optional items can either be omitted, or None can be provided as their value. The semantics of each item are in order:

• A callable object that will be called to create the initial version of the object.
• A tuple of arguments for the callable object. An empty tuple must be given if the callable does not accept any argument.
• Optionally, the object’s state, which will be passed to the object’s __setstate__() method as previously described. If the object has no such method then, the value must be a dictionary and it will be added to the object’s __dict__ attribute.
• Optionally, an iterator (and not a sequence) yielding successive items. These items will be appended to the object either using obj.append(item) or, in batch, using obj.extend(list_of_items). This is primarily used for list subclasses, but may be used by other classes as long as they have append() and extend() methods with the appropriate signature. (Whether append() or extend() is used depends on which pickle protocol version is used as well as the number of items to append, so both must be supported.)
• Optionally, an iterator (not a sequence) yielding successive key-value pairs. These items will be stored to the object using obj[key] = value. This is primarily used for dictionary subclasses, but may be used by other classes as long as they implement __setitem__().