Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pickling error: Can't pickle <type 'function'>

I am wondering what this error might mean:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I understand that it has something to do with using multiple cores. I am running my program on a cluster and using 15 threads in this line of my code:

gauss2 = PTSampler(ntemps, renwalkers, rendim, lnlike, lnprior, threads=15)

The sampler in question is the one documented for the Parallel Tempering sampler at http://dan.iel.fm/emcee/current/user/pt/

Any idea what this error might mean?

like image 544
naomig Avatar asked Jul 24 '14 00:07

naomig


People also ask

What Cannot be pickled in Python?

With pickle protocol v1, you cannot pickle open file objects, network connections, or database connections.

What is pickling Python multiprocessing?

Pickle is able to serialize and deserialize Python objects into bytestream. It does work well on most cases — with reservations. When multiprocessing spawns a process, Pickle is called by default to handle data transfer. A simple example of how this is done follows: import pickleclass Foo(object):

Why can t pickle local object?

To solve this type of error we have to declare that variable as global variables. We can't pickle local objects so that we are declaring that variable result as global. Here we have given only one print statement. If it is declared then we can say that the program has run successfully.

What is Python multiprocessing?

multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.


1 Answers

The error means you are trying to pickle a builtin FunctionType… not the function itself. It's likely do to a coding error somewhere picking up the class of the function instead of the function itself.

>>> import sys
>>> import pickle
>>> import types
>>> types.FunctionType
<type 'function'>
>>> try:
...     pickle.dumps(types.FunctionType)
... except:
...     print sys.exc_info()[1]
... 
Can't pickle <type 'function'>: it's not found as __builtin__.function
>>> def foo(x):
...   return x
... 
>>> try:
...     pickle.dumps(type(foo))
... except:
...     print sys.exc_info()[1]
... 
Can't pickle <type 'function'>: it's not found as __builtin__.function
>>> try:
...     pickle.dumps(foo.__class__)
... except:
...     print sys.exc_info()[1]
... 
Can't pickle <type 'function'>: it's not found as __builtin__.function
>>> pickle.dumps(foo)
'c__main__\nfoo\np0\n.'
>>> pickle.dumps(foo, -1)
'\x80\x02c__main__\nfoo\nq\x00.'

If you have a FunctionType object, then all you need to do is get one of the instances of that class -- i.e. a function like foo.

like image 61
Mike McKerns Avatar answered Oct 29 '22 18:10

Mike McKerns