Unpickling python objects with a changed module path

Question

I'm trying to integrate a project Project A built by a colleague into another python project. Now this colleague has not used relative imports in his code but instead done

from packageA.moduleA import ClassA from packageA.moduleA import ClassB 

and consequently pickled the classes with cPickle. For neatness I'd like to hide the package that his (Project A) built inside my project. This however changes the path of the classes defined in packageA. No problem, I'll just redefine the import using

from ..packageA.moduleA import ClassA from ..packageA.moduleA import ClassB 

but now the un pickling the classes fails with the following message

    with open(fname) as infile: self.clzA = cPickle.load(infile) ImportError: No module named packageA.moduleA 

So why doesn't cPickle apparently see the module defs. Do I need to add the root of packageA to system path? Is this the correct way to solve the problem?

The cPickled file looks something like

ccopy_reg _reconstructor p1 (cpackageA.moduleA ClassA p2 c__builtin__ object p3 NtRp4 

The old project hierarchy is of the sort

packageA/     __init__.py     moduleA.py     moduleB.py packageB/     __init__.py     moduleC.py     moduleD.py 

I'd like to put all of that into a WrapperPackage

MyPackage/ .. __init__.py .. myModuleX.py .. myModuleY.py WrapperPackage/ .. __init__.py .. packageA/    .. __init__.py    .. moduleA.py    .. moduleB.py .. packageB/    .. __init__.py    .. moduleC.py    .. moduleD.py 

Answer 1

You'll need to create an alias for the pickle import to work; the following to the __init__.py file of the WrapperPackage package:

from .packageA import * # Ensures that all the modules have been loaded in their new locations *first*. from . import packageA  # imports WrapperPackage/packageA import sys sys.modules['packageA'] = packageA  # creates a packageA entry in sys.modules 

It may be that you'll need to create additional entries though:

sys.modules['packageA.moduleA'] = moduleA # etc. 

Now cPickle will find packageA.moduleA and packageA.moduleB again at their old locations.

You may want to re-write the pickle file afterwards, the new module location will be used at that time. The additional aliases created above should ensure that the modules in question have the new location name for cPickle to pick up when writing the classes again.

Answer 2

In addition to @MartinPieters answer the other way of doing this is to define the find_global method of the cPickle.Unpickler class, or extend the pickle.Unpickler class.

def map_path(mod_name, kls_name):     if mod_name.startswith('packageA'): # catch all old module names         mod = __import__('WrapperPackage.%s'%mod_name, fromlist=[mod_name])         return getattr(mod, kls_name)     else:         mod = __import__(mod_name)         return getattr(mod, kls_name)  import cPickle as pickle with open('dump.pickle','r') as fh:     unpickler = pickle.Unpickler(fh)     unpickler.find_global = map_path     obj = unpickler.load() # object will now contain the new class path reference  with open('dump-new.pickle','w') as fh:     pickle.dump(obj, fh) # ClassA will now have a new path in 'dump-new' 

A more detailed explanation of the process for both pickle and cPickle can be found here.