Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I limit the number of active threads in python?

Am new to python and making some headway with threading - am doing some music file conversion and want to be able to utilize the multiple cores on my machine (one active conversion thread per core).

class EncodeThread(threading.Thread):
    # this is hacked together a bit, but should give you an idea
    def run(self):
        decode = subprocess.Popen(["flac","--decode","--stdout",self.src],
                            stdout=subprocess.PIPE)
        encode = subprocess.Popen(["lame","--quiet","-",self.dest],
                                stdin=decode.stdout)
        encode.communicate()

# some other code puts these threads with various src/dest pairs in a list

for proc in threads: # `threads` is my list of `threading.Thread` objects
    proc.start()

Everything works, all the files get encoded, bravo! ... however, all the processes spawn immediately, yet I only want to run two at a time (one for each core). As soon as one is finished, I want it to move on to the next on the list until it is finished, then continue with the program.

How do I do this?

(I've looked at the thread pool and queue functions but I can't find a simple answer.)

Edit: maybe I should add that each of my threads is using subprocess.Popen to run a separate command line decoder (flac) piped to stdout which is fed into a command line encoder (lame/mp3).

like image 642
thornomad Avatar asked Nov 24 '09 02:11

thornomad


People also ask

What is thread limit in Python?

Use Python's ThreadPoolExecutor with max_workers argument set to 10. The pool will automatically allocate threads as needed, limiting maximum number of allocation to 10.

How do I find the number of active threads in Python?

In Python, the method threading. active_co unt() from the threading module is used to count the currently active or running threads.

How many threads can you run at once Python?

Generally, Python only uses one thread to execute the set of written statements. This means that in python only one thread will be executed at a time.


1 Answers

If you want to limit the number of parallel threads, use a semaphore:

threadLimiter = threading.BoundedSemaphore(maximumNumberOfThreads)

class EncodeThread(threading.Thread):

    def run(self):
        threadLimiter.acquire()
        try:
            <your code here>
        finally:
            threadLimiter.release()

Start all threads at once. All but maximumNumberOfThreads will wait in threadLimiter.acquire() and a waiting thread will only continue once another thread goes through threadLimiter.release().

like image 95
Andre Holzner Avatar answered Oct 29 '22 13:10

Andre Holzner