Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to multi-thread with "for" loop?

Similar questions may have been asked a couple of times before, but none of them seem to have my case/scenario or it does not work.

I am trying to multithread a for loop as showed in an example. This for loop will do a function as it loops through an array. I would like to multithread it.

Example:

array = ["a", "b", "c", "d", "e"]
def dosomething(var):
    #dosomething this is just an example as my actual code is not relevant to this question

for arrayval in array:
    dosomething(arrayval)

This should loop through the array and do the function dosomething with the variables a, then b, c, etc.

Any idea on how I can do that?

like image 501
Gem Clash Avatar asked Mar 02 '19 00:03

Gem Clash


People also ask

How do you do a multiprocess loop in Python?

To parallelize the loop, we can use the multiprocessing package in Python as it supports creating a child process by the request of another ongoing process. The multiprocessing module could be used instead of the for loop to execute operations on every element of the iterable. It's multiprocessing.

Can multiple threads run in parallel?

On a system with more than one processor or CPU cores (as is common with modern processors), multiple processes or threads can be executed in parallel.

Can one CPU run multiple threads?

Yes you can do multithreading on a single processor system. In multi-processor system , multiple threads execute , simultaneously on different cores. Eg- If there are two threads and two cores , then each thread would run on individual core.


1 Answers

You can use threading.Thread:

from threading import Thread
from time import sleep
from random import randint

def dosomething(var):
    sleep(randint(1,5))
    print(var)

array = ["a", "b", "c", "d", "e"]
threads = []
for arrayval in array:
    threads.append(Thread(target=dosomething, args=(arrayval,)))
    threads[-1].start()
for thread in threads:
    thread.join()

This outputs in random order within 5 seconds:

e
b
c
a
d

If you want to limit the number of threads you can use multiprocessing.pool.ThreadPool instead. The following example limits the number of worker threads to 2 so it can possibly take as long as 15 seconds to complete (if the workers all happen to take 5 seconds):

from multiprocessing.pool import ThreadPool
from time import sleep
from random import randint

def dosomething(var):
    sleep(randint(1,5))
    print(var)

array = ["a", "b", "c", "d", "e"]
with ThreadPool(processes=2) as pool:
    pool.map(dosomething, array)
like image 110
blhsing Avatar answered Oct 10 '22 12:10

blhsing