Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can't pickle defaultdict

I have a defaultdict that looks like this:

dict1 = defaultdict(lambda: defaultdict(int)) 

The problem is, I can't pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is, what is module-level function? How can I use the dictionary with cPickle?

like image 534
Fynn Mahoney Avatar asked May 08 '13 11:05

Fynn Mahoney


People also ask

How do I get rid of Defaultdict?

Just use the same syntax as you would on a regular dictionary. del mydict['stock2'] for example, or mydict. pop('stock2') if you want the value returned at the same time. Note that you don't need to be using a defaultdict at all in your example.

What does the Defaultdict () function do?

A defaultdict works exactly like a normal dict, but it is initialized with a function (“default factory”) that takes no arguments and provides the default value for a nonexistent key. A defaultdict will never raise a KeyError. Any key that does not exist gets the value returned by the default factory.

How do you declare a Defaultdict?

A defaultdict can be created by giving its declaration an argument that can have three values; list, set or int. According to the specified data type, the dictionary is created and when any key, that does not exist in the defaultdict is added or accessed, it is assigned a default value as opposed to giving a KeyError .


2 Answers

In addition to Martijn's explanation:

A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it's not nested within another function, and it is a "real" function with a name, not a lambda function.

So, to pickle your defaultdict, create it with module-level function instead of a lambda function:

def dd():     return defaultdict(int)  dict1 = defaultdict(dd) # dd is a module-level function 

than you can pickle it

tmp = pickle.dumps(dict1) # no exception new = pickle.loads(tmp) 
like image 128
sloth Avatar answered Sep 29 '22 11:09

sloth


Pickle wants to store all the instance attributes, and defaultdict instances store a reference to the default callable. Pickle recurses over each instance attribute.

Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full 'path' of the function to be imported and referenced when unpickling again.

like image 22
Martijn Pieters Avatar answered Sep 29 '22 13:09

Martijn Pieters