I'm trying to use a multithreading queue and a multiprocessing queue at the same time. The threading queue will be used by 20 threads to retrieve many web pages. I then want to put the pages into a multiprocess queue so that 4 process workers can crunch the data. Below is my basic structure. My issue is that, the work queue, gives an error saying Queue is not iterable. I think the multithreading queue is overwriting the multiprocess queue but I really don't know what's wrong.
ticker_queue = Queue()
work_queue = Queue()
tickers = get_tickers()
for i in tickers:
ticker_queue.put(i)
for i in range(20):
t = Thread(target=network_worker, args = (ticker_queue, work_queue)).start()
for i in range(4):
p = Process(target = worker, args = (work_queue)).start()
Here is the traceback
Traceback (most recent call last):
File "OneDrive\Python\andys.py", line 108, in <module>
p = Process(target = worker, args = (work_queue)).start()
File "C:\Python27\lib\multiprocessing\process.py", line 104, in __init__
self._args = tuple(args)
TypeError: 'Queue' object is not iterable
Simultaneously executing programming tasks before now had always been daunting. But today, many programming languages support Multithreading and Multiprocessing, which aids concurrent execution of code.
To recap, threading in Python allows multiple threads to be created within a single process, but due to GIL, none of them will ever run at the exact same time. Threading is still a very good option when it comes to running multiple I/O bound tasks concurrently.
Passing Messages to Processes A simple way to communicate between process with multiprocessing is to use a Queue to pass messages back and forth. Any pickle-able object can pass through a Queue. This short example only passes a single message to a single worker, then the main process waits for the worker to finish.
You can make a queue or line of tasks or objects by using the queue library in Python. Simply you can add a task to the queue (using put() method) or get a task out of the line and processes it (using get() method). Threading package in Python let you run multiple tasks at the same time.
There is a comma missing in
p = Process(target = worker, args = (work_queue)).start()
since
(work_queue)
is just an expression, while what you want is a 1-element tuple:
(work_queue,)
notice the additional comma.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With