Threading subprocesses in Python

Question

If I use Python's Threading library can I accomplish a batch of subprocess processes faster? Let's say for instance I need to convert 100 .wav files to .mp3 files. If I wrap 'ffmpeg' in a Python script that used threading could I accomplish the task much faster? Does threading enable me to actually use all 8 threads available in my i7?

I recently stumbled across a Python script that pings a list of hosts but utilizes threading to make things faster. The script seemed succinct and was easy enough for a beginner such as myself to read and understand. So this is where my other question arises: What is all the talk about running parallel tasks being so complex? Maybe I don't understand parallel and parallel != threading. If it is that simple then why wouldn't people use threading for any and all batch conversions when running on a modern processor?

Answer 1

Here is a simple and working solution for your case. I use it often and it proves to be very useful! This code creates as many threads as cores and lets them execute a (large) number of tasks (in this case, calling a shell program):

import Queue
import threading
import multiprocessing
import subprocess

q = Queue.Queue()
for i in range(30): #put 30 tasks in the queue
    q.put(i)

def worker():
    while True:
        item = q.get()
        #execute a task: call a shell program and wait until it completes
        subprocess.call("echo "+str(item), shell=True) 
        q.task_done()

cpus=multiprocessing.cpu_count() #detect number of cores
print("Creating %d threads" % cpus)
for i in range(cpus):
     t = threading.Thread(target=worker)
     t.daemon = True
     t.start()

q.join() #block until all tasks are done