Parallelize my python program

Question

I have a python program that reads a line from a input file, does some manipulation and writes it to output file. I have a quadcore machine, and I want to utilize all of them. I think there are two alternatives to do this,

1. Creating n multiple python processes each handling a total number of records/n
2. Creating n threads in a single python process for every input record and each thread processing a record.
3. Creating a pool of n threads in a single python process, each executing a input record.

I have never used python mutliprocessing capabilities, can the hackers please tell which method is best option?

Answer 1

The reference implementation of the Python interpreter (CPython) holds the infamous "Global Interpreter Lock" (GIL), effectively allowing only one thread to execute Python code at a time. As a result, multithreading is very limited in Python -- unless your heavy lifting gets done in C extensions that release the GIL.

The simplest way to overcome this limitation is to use the multiprocessing module instead. It has a similar API to threading and is pretty straight-forward to use. In your case, you could use it like this (assuming that the manipulation is the hard part):

import multiprocessing

def process_line(line):
    # This function is executed in your worker processes.  Manipulate the
    # line and return the results.
    return manipulate(line)

if __name__ == '__main__':
    with open('input.txt') as fin, open('output.txt', 'w') as fout:
        # This creates a pool of N worker processes, where N is the number
        # of CPUs in your machine.
        pool = multiprocessing.Pool()

        # Let the workers do the manipulation and write the results to
        # the output file:
        for manipulated_line in pool.imap(process_line, fin):
            fout.write(manipulated_line)