Are Python Multiprocessing Queues Ordered?

Question

Forgive me if this is obvious, but I can't tell from the documentation whether the output when using a queue is ordered. That is, if the inputs are [1, 2, 3] will the outputs be f(1), f(2), f(3), or can they be out of order?

Edit: I understand that the inputs are handled in FIFO. But does FIFO retrieval necessarily guarantee that the results are also returned in order?

Answer 1

No, they will enter the queue in the order they finish, not in the order they were started. This means that if you start your processes with a certain order in mind, they may or may not finish in that order. If order is important to you, you should pass back the starting order with your results in the queue, perhaps as a tuple queue.put((order, result)). Here's an example showing the random order you get out of the queue. (Which is still FIFO)

import multiprocessing
import random
import time

def square(x, queue):
    # Sleep for a value between 0 and 1. 
    time.sleep(random.random())
    queue.put(x**2)

queue = multiprocessing.Queue()

for i in range(10):
    process = multiprocessing.Process(target = square, args = (i, queue))
    process.start()

results = []
for i in range(10):
    results.append(queue.get())

print (results)

This gave me [25, 16, 9, 0, 36, 64, 81, 1, 49, 4].

Assuming your task is reasonably straight forward, I always prefer using Python's Pool functionality and its corresponding pool.map() function which does guarantee that output order is preserved according to input order.

Answer 2

I have a case where I have to replace all multiprocessing.Queue into m.Queue where m = multiprocess.Manager() so that all Queues are ordered. However, I can't reproduce it with a simple example.

I don't really know why but if anyone comes here because their Queue is out of order, I recommend trying this.