Pickling dynamically generated classes?

Question

I'm using type() to dynamically generate classes that will ultimately be pickled. The problem is that the un-pickling process needs the definition of the class in order to re-construct the object that has been pickled.

This is where I'm stuck. I don't know how to somehow provide the unpickler a way to generate an instance from a class that was dynamically generated.

Any hints appreciated.

Thanks!

Here's an example of the problem:

    >>> class Foo(object):     ...     pass     >>> g=type('Goo',(Foo,),{'run':lambda self,x: 2*x } )()     >>> cPickle.dumps(g)      PicklingError: Can't pickle <class '__main__.Goo'>: attribute lookup __main__.Goo failed 

This evidently works, but only from dynamic classes created from a pickle-able base class (with find-able module definition):

import cPickle  class Foo(object): pass  def dynamic(): return type('Goo',(Foo,),{'run':lambda self,x: 2*x } )()  g=type('Goo',(Foo,),{'run':lambda self,x: 2*x , '__reduce__': lambda self: (dynamic,tuple()) } )()  gg=cPickle.loads ( cPickle.dumps(g) ) print gg.run(10) 

Answer 1

When the Pickler encounters an object of a type it knows nothing about, it looks for a reduce method. Defining this method when you build your custom class using type should solve the problem of pickling.

If you provide initial args then in addition you might need to define a getnewargs method

Answer 2

You can assign a global name to your dynamically generated class to make it picklable.

>>> class Foo(object): ...     pass >>> class_name = 'Goo' >>> my_class = type(class_name, (Foo, ), {'run': lambda self, x: 2*x }) >>> globals()[class_name] = my_class >>> g = my_class() >>> pickle.dumps(g) 

Of course, you need to make sure that the names of your classes are unique.