Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using threads and processes together with shared queues in Python

I'm trying to use a multithreading queue and a multiprocessing queue at the same time. The threading queue will be used by 20 threads to retrieve many web pages. I then want to put the pages into a multiprocess queue so that 4 process workers can crunch the data. Below is my basic structure. My issue is that, the work queue, gives an error saying Queue is not iterable. I think the multithreading queue is overwriting the multiprocess queue but I really don't know what's wrong.

ticker_queue = Queue()

work_queue = Queue()
tickers = get_tickers()


for i in tickers:
    ticker_queue.put(i)

for i in range(20):
    t = Thread(target=network_worker, args = (ticker_queue, work_queue)).start()

for i in range(4):
    p = Process(target = worker, args = (work_queue)).start()

Here is the traceback

Traceback (most recent call last):
  File "OneDrive\Python\andys.py", line 108, in <module>
    p = Process(target = worker, args = (work_queue)).start()
  File "C:\Python27\lib\multiprocessing\process.py", line 104, in __init__
    self._args = tuple(args)
TypeError: 'Queue' object is not iterable
like image 850
nicholas.reichel Avatar asked Jan 24 '15 18:01

nicholas.reichel


People also ask

Can you use multithreading and multiprocessing together?

Simultaneously executing programming tasks before now had always been daunting. But today, many programming languages support Multithreading and Multiprocessing, which aids concurrent execution of code.

Can Python can execute multiple instructions simultaneously by using threads?

To recap, threading in Python allows multiple threads to be created within a single process, but due to GIL, none of them will ever run at the exact same time. Threading is still a very good option when it comes to running multiple I/O bound tasks concurrently.

How do I share data between two processes in Python?

Passing Messages to Processes A simple way to communicate between process with multiprocessing is to use a Queue to pass messages back and forth. Any pickle-able object can pass through a Queue. This short example only passes a single message to a single worker, then the main process waits for the worker to finish.

How do you queue and thread in Python?

You can make a queue or line of tasks or objects by using the queue library in Python. Simply you can add a task to the queue (using put() method) or get a task out of the line and processes it (using get() method). Threading package in Python let you run multiple tasks at the same time.


1 Answers

There is a comma missing in

p = Process(target = worker, args = (work_queue)).start()

since

(work_queue)

is just an expression, while what you want is a 1-element tuple:

(work_queue,)

notice the additional comma.

like image 104
thebjorn Avatar answered Sep 26 '22 03:09

thebjorn