Multiprocessing with threading?

Question

when I trying to make my script multi-threading,

I've found out multiprocessing,

I wonder if there is a way to make multiprocessing work with threading?

• cpu 1 -> 3 threads(worker A,B,C)
• cpu 2 -> 3 threads(worker D,E,F)
• ...

Im trying to do it myself but I hit so much problems.

is there a way to make those two work together?

Answer 1

You can generate a number of Processes, and then spawn Threads from inside them. Each Process can handle almost anything the standard interpreter thread can handle, so there's nothing stopping you from creating new Threads or even new Processes within each Process. As a minimal example:

def foo():
    print("Thread Executing!")

def bar():
    threads = []
    for _ in range(3): # each Process creates a number of new Threads
        thread = threading.Thread(target=foo) 
        threads.append(thread)
        thread.start()
    for thread in threads:
        thread.join()

if __name__ == "__main__": 
    processes = []
    for _ in range(3):
        p = multiprocessing.Process(target=bar) # create a new Process
        p.start()
        processes.append(p)
    for process in processes:
        process.join()

Communication between threads can be handled within each Process, and communication between the Processes can be handled at the root interpreter level using Queues or Manager objects.