Run os.system() multiple times simultaneously in Python?

Question

I've written the following short python script to download flv videos using cclive on a Fedora 17 system.

urls = [line.strip() for line in open("urls.txt")]
for url in urlstoget:
    os.system('cclive %s' % url)

It works fine but the videos are limited to about 80kbps. I have a 39 to download and would like to download 2-4 simultaneously.

How can I run the os.system() command multiple times simultaneously?

Answer 1

use either threading or multiprocessing.

Here's an example using multiprocessing:

def retrieve_url(url):
    os.system('cclive %s' % url)

pool = multiprocessing.Pool(4)
pool.map(retrieve_url, list_of_urls)

And a link to another SO question: Python - parallel commands

Answer 2

Look at the subprocess module, the Popen() method in particular. You could also use os.fork()