Python threading in a loop but with max threads

Question

Right now I have a for loop that loops through a list, usually this list is 100-500 items long. In the for loop, a new thread is opened per item. So right now my code looks like this:

    threads = []
    for item in items:
        t = threading.Thread(target=myfunction, args=(item,))
        threads.append(t)
        t.start()

But I don't want to start a new thread per, seeing it only takes a few seconds MAX per thread to execute myfunction. I want to do my loop still, calling upon myfunction with each item in argument. But to close the thread once it's done, and allow another one to take over. The max number of threads I want to open is no less than 3, no more than 20. Although if it's easier, that range can vary. I just don't want to open up a new thread each item in the loop.

For those that are curious, and if it matters. myfunction is a function that I defined that uses urllib to send a post request to a site.

I'm new to python, but I'm not new to coding all together. Sorry for the noob question.

Answer 1

I think you are looking for a thread pool to solve your problem.

The answers to this question detail some possible solutions.

One of the simplest (assuming python3 or the backport in pypi) is:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=10)
futures = []
for item in items:
    a = executor.submit(myfunction, item)
    futures.append(a)

This will execute myfunction for all items using 10 threads. You can later wait for completion of the calls using the futures list.

Answer 2

modify your code slightly to include the check on the number of active threads at any given time:

threads = []
consumed_by_threads = 0
consumed_by_main = 0
for item in items:
    at = threading.activeCount()
    if at <= 20:
        t = threading.Thread(target=myfunction, args=(item,))
        threads.append(t)
        consumed_by_threads += 1
        t.start()
    else:
        print "active threads:", at
        consumed_by_main += 1
        myfunction(item)

print "consumed_by_threads: ", consumed_by_threads
print "consumed_by_main: ", consumed_by_main

# here the rest of your code, thread join, etc

Note: I am only checking for a max number of threads. BTW: it should be 21, since the main thread is included in the count (see here and follow the link to enumerate)

Nota Bene: as usual, double check the benefit of multithreading on your specific application, depending on which python implementation you use and whether the threads are cpu-bound or I/O bound.