Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is python's multiprocessing Queue "infinite" by default?

The documentation for python's multiprocesing queue: https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Queue

Isn't as clear as the one for queue.Queue: https://docs.python.org/3/library/queue.html

As to whether or not the size of the queue is "infinite" (as in, within the possible bounds of whatever the program can manage to allocate memory wise) when the maxsize argument isn't given to the constructor.

Is this the case ?

like image 826
George Avatar asked May 10 '18 09:05

George


1 Answers

multiprocessing.Queue mimics queue.Queue completly with all features (except .task_done() and .join())

Queue implements all the methods of Queue.Queue except for task_done() and join().

So without arguments (or a negative number) it can take infinite elements

(as an side-note since Queues are internally list like structures (dequeue, heapq, list) it is harder to have an limit, then to not have an limit.)

Edit:

Ok as it turns out after looking through the source code, it turns out that multiprocessing.Queue does have a standard upper bound if no value is specified: 2**31-1

# file multiprocessing/queues.py
class Queue(object):
    def __init__(self, maxsize=0, *, ctx):
        if maxsize <= 0:
            from .synchronize import SEM_VALUE_MAX as maxsize # -> 2**31-1

So it is not infinte, but praticly infinte

like image 108
MegaIng Avatar answered Oct 08 '22 09:10

MegaIng