Python Multiprocessing combined with Multithreading

Question

I am not sure if what i am trying to do is a valid practice but here it goes: I need my program to be highly parallelized so i thought i could make 2-3 processes and each process can have 2-3 threads.

1) Is this possible? 2) Is there any point into it? 3) This is my code but it hangs when i try to join the processes.

PQ = multiprocessing.Queue()

[...]

def node(self, files, PQ):

        l1, l2 = self.splitList(files)
        p1 = multiprocessing.Process(target=self.filePro, args=(l1,PQ,))
        p2 = multiprocessing.Process(target=self.filePro, args=(l2,PQ,))
        p1.daemon = True
        p2.daemon = True
        p1.start()
        p2.start()

        p1.join() # HANGS HERE
        p2.join()
        while 1:
            if PQ.empty():
                break
            else:
                print(PQ.get())
        PQ.join()

    def filePro(self,lst,PQ):
        TQ = queue.Queue()
        l1, l2 = self.splitList(lst)
        t1 = threading.Thread(target=self.fileThr, args=('a',l1,TQ,))
        t2 = threading.Thread(target=self.fileThr, args=('b',l2,TQ,))
        t1.daemon = True
        t2.daemon = True
        t1.start()
        t2.start()

        t1.join()
        t2.join()
        while 1:
            if TQ.empty():
                break
            else:
                PQ.put(TQ.get())
                TQ.task_done()
        TQ.join()

def fileThr(self,id,lst,TQ):
        while lst:
            tmp_path = lst.pop()
            if (not tmp_path[1]):
                continue
            for item in tmp_path[1]:
                TQ.put(1)
        TQ.join()

Answer 1

1) Is this possible?

Yes.

---

2) Is there any point into it?

Yes. But generally not the point you're looking for.

First, just about every modern operating system uses a "flat" scheduler; there's no difference between 8 threads scattered across 3 programs or 8 thread across 8 programs.*

_{* Some programs can get a significant benefit by carefully using intraprocess-only locks or other synchronization primitives in some places where you know you're only sharing with threads from the same program—and, of course, by avoiding shared memory in those places—but you're not going to get that benefit by spreading your jobs across threads and your threads across processes evenly.}

Second, even if you were using, say, old SunOS, in the default CPython interpreter, the Global Interpreter Lock (GIL) ensures that only one thread can be running Python code at a time. If you're spending your time running code from a C extension library that explicitly releases the GIL (like some NumPy functions), threads can help, but otherwise, they all just end up serialized anyway.

The main case where threads and processes are useful together is where you have both CPU-bound and I/O-bound work. In that case, usually one is feeding the other. If the I/O feeds the CPU, use a single thread pool in the main process to handle the I/O, then use a pool of worker processes to do the CPU work on the results. If it's the other way around, use a pool of worker processes to do the CPU work, then have each worker process use a thread pool to do the I/O.

---

3) This is my code but it hangs when i try to join the processes.

It's very hard to debug code when you don't give a minimal, complete, verifiable example.

However, I can see one obvious problem.

You're trying to use TQ as a producer-consumer queue, with t1 and t2 as producers and the filePro parent as the consumer. Your consumer doesn't call TQ.task_done() until after t1.join() and t2.join() return, which doesn't happen until those threads are done. But those producers won't finish because they're waiting for you to call TQ.task_done(). So, you've got a deadlock.

And, because each of your child processes' main threads are deadlocked, they're never finish, so the p1.join() will block forever.

If you really want the main thread to wait until the other threads are done before doing any work, you don't need the producer-consumer idiom; just let the children do their work and exit without calling TQ.join(), and don't bother with TQ.task_done() in the parent. (Note that you're already doing this correctly with PQ.)

If, on the other hand, you want them to work in parallel, don't try to join the child threads until you've finished your loop.