Python 3.5 dill pickling/unpickling on different servers: "KeyError: 'ClassType'"

Question

See updates at the bottom

--

A similar question was asked here, but never resolved: pickling and unpickling user-defined class

I'm working on a project which necessitates pickling user defined classes, and sending them to a remote server where they are unpickled and called. We use the Dill library to accomplish this, and have had a lot of success.

Unfortunately, I've run into an issue I'm having a hard time debugging. I create and pickle a class as follows:

import dill, base64
import time, random

class periodicSource(object):
    def __call__(self):
        while True:
            time.sleep(0.1)
            yield random.uniform(20,100)

periodic_src = periodicSource()
a = base64.b64encode(dill.dumps(periodic_src)).decode("ascii")
print(a)

It creates an ascii representation of the dilled class.

gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg==

When I go to deserialize it on the other server:

a = 'gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg=='
a = dill.loads(base64.b64decode(a.encode()))
print(a)

I get the following error:

/home/streamsadmin/anaconda3/bin/python /home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py

Traceback (most recent call last):
 File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
   a = dill.loads(base64.b64decode(a.encode()))
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
   return load(file)
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
   obj = pik.load()
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
   return _reverse_typemap[name]
KeyError: 'ClassType'

I would expect this if I were using different version of Python on the remote system, but they're the same:

Server 1:

>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'

Server 2:

>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'

Additionally, both versions of Dill are 0.2.6. Any ideas how I could debug this?

EDIT: I've think it might be something with my environment. I'm using Python 3.5, but list the builtin types:

>>> import types
>>> dir(types)
  ['BuiltinFunctionType',
   'BuiltinMethodType',
   'ClassType',
   'CodeType',
   ...
  ]

It seems ClassType is in the output which should NOT be the case since ClassType was removed in Python 3.5. This is exceedingly strange.

I'm running on a system that has both Python 2.7 and Python 3.5 installed. Could the 2.7 installation somehow be polluting the 3.5 installation?

Answer 1

The culprit is cloudpickle. By default in Python 3.5, types.ClassType is left unset.

>>> import types
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'CodeType', ...]

When cloudpickle is imported, suddenly, types.ClassType becomes defined.

>>> import cloudpickle
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'ClassType', 'CodeType', ...]

Server A uses dill to serialize objects, and also imports cloudpickle. Therefore it includes a reference to ClassType during serialization.

Server B does NOT import cloudpickle, and then tries to find a reference to ClassType during deserialization and fails. Raising the error:

Traceback (most recent call last):
 File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
   a = dill.loads(base64.b64decode(a.encode()))
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
   return load(file)
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
   obj = pik.load()
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
   return _reverse_typemap[name]
KeyError: 'ClassType'

On our system, we can't remove cloudpickle from our environment, so we had to do the following workaround.

On server B, right after we import dill and sometime before the first call to dill.loads, we invoke the following line of code:

dill._dill._reverse_typemap['ClassType'] = type

This defines ClassType appropriately. And causes dill deserialization to work as expected.

Answer 2

I am sure cloudpickle is causing the problem. You can debug it step by step.

1. First Check if classType Exists in your builtin types

   import types dir(types)

if it exist than it should have worked for you, if not than move to next steps.

2. import cloudpickle and now check again. You will have classType in buildin types

3. excute below code

   dill.dill._reverse_typemap['ClassType'] = type

it should work for you :)

But if you are still getting error AttributeError: module 'dill' has no attribute 'dill'

than use this one dill._dill._reverse_typemap['ClassType'] = type because dill.dill is moved to dill._dill