Create object that cannot be pickled

Question

How can I easily create an object that cannot be pickled for testing edge cases in my rpc code?

It needs to be:

1. Simple
2. Reliable (not expected to break in future versions of python or pickle)
3. Cross platform

Edit: The intended use looks something like this:

class TestRPCServer:
    def foo(self):
        return MagicalUnpicklableObject()

def test():
    with run_rpc_server_and_connect_to_it() as proxy:
        with nose.assert_raises(pickle.PickleError):
            proxy.foo()

Answer 1

If all you need is an object that will throw an exception when you pickle it, for the puposes of testing, you can blow up the __getstate__ method.

>>> class C:
...     def __getstate__(self):
...         raise Exception
... 
>>> pickle.dumps(C())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 723, in save_inst
    stuff = getstate()
  File "<stdin>", line 3, in __getstate__
Exception

---

If you want a less artificial scenario, think about objects that use OS resources like file handles, or sockets, or threads, etc.

>>> with open('spam.txt', 'w') as f:
...     pickle.dumps(f)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle file objects

Answer 2

If you want an explicit list of objects that can be pickled versus objects that can't be pickled, both using pickle, and more advanced serializers like dill, this file contains a somewhat comprehensive list for standard library objects. It gives a simple way to build each object (typically, a one-liner), and shows variants for different versions of python, if applicable.

https://github.com/uqfoundation/dill/blob/cccbea9b715e16b742288e1e5a21a687a4d4081b/dill/_objects.py#L255

For example, pickle will fail on the following object, while advanced serializers like dill will not:

>>> import dill
>>> dill.dumps(Ellipsis)
b'\x80\x03cdill.dill\n_eval_repr\nq\x00X\x08\x00\x00\x00Ellipsisq\x01\x85q\x02Rq\x03.'

Advanced serializers can also work on file objects and the like, btw:

>>> dill.dumps(open('foo.pkl', 'w'))
b'\x80\x03cdill.dill\n_create_filehandle\nq\x00(X\x07\x00\x00\x00foo.pklq\x01X\x01\x00\x00\x00wq\x02K\x00\x89cdill.dill\n_get_attr\nq\x03cdill.dill\n_import_module\nq\x04X\x02\x00\x00\x00ioq\x05\x85q\x06Rq\x07X\x04\x00\x00\x00openq\x08\x86q\tRq\n\x89K\x00X\x00\x00\x00\x00q\x0btq\x0cRq\r.'

However, pickle and dill (and other advanced serializers) will fail on any type that is directly tied to a python FrameType, like a generator:

>>> dill.dumps((i for i in []))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 243, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 236, in dump
    pik.dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 412, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 499, in save
    rv = reduce(self.proto)
TypeError: can't pickle generator objects