Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pickle Queue objects in python

I have a class that uses a list of Queue objects. I need to pickle this class including the information saved in the queue objects. For example:

import Queue
import pickle

class QueueTest(object):
    def __init__(self):
        self.queueList = []
    def addQueue(self):
        q = Queue.Queue()
        q.put('test')
        self.queueList.append(q)


obj = QueueTest()
obj.addQueue()

with open('pickelTest.dat','w') as outf:
    pickle.dump(obj,outf)

returns the error

raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle lock objects

Is there a work around to pickle Queue objects?

like image 538
Joel Green Avatar asked May 25 '13 21:05

Joel Green


1 Answers

I suggest replacing your uses of Queue.Queue with collections.deque. The Queue class is intended to be used for synchronized communication between threads, so it will have some unnecessary overhead when used as a regular data structure. collections.deque is a faster alternative. (The name "deque" is pronounced "deck" and means "double-ended queue".)

The deque class does have a different API than the Queue type, but it should be pretty easy to translate between them. Use deque.append in place of Queue.put and deque.popleft in place of q.get() (or appendleft and pop, if you feel like going the other direction). Rather than calling Queue.empty, just use a deque instance as a Boolean value (like you do to test for an empty list).

deque instances are picklable:

>>> import collections, pickle
>>> q = collections.deque(["test"])
>>> pickle.dumps(q)
b'\x80\x03ccollections\ndeque\nq\x00]q\x01X\x04\x00\x00\x00testq\x02a\x85q\x03Rq\x04.'
like image 66
Blckknght Avatar answered Oct 08 '22 16:10

Blckknght