Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

The relationship between thread and process in multi-process program

OS: debian9.
A simple multi-processes program named mprocesses.py.

import os
import multiprocessing

def run_task(name):
    print("task %s (pid = %s) is running"  %(name,os.getpid()))
    while True:
        pass

if __name__ == "__main__":
    print("current process %s ." %os.getpid())
    pool = multiprocessing.Pool(processes = 2)
    for i in range(2):
        pool.apply_async(run_task,args=(i,))
    pool.close()
    pool.join()

Run python3 mprocesses.py and get below output.

python3 mprocesses.py
current process 6145 .
task 0 (pid = 6146) is running
task 1 (pid = 6147) is running

Get processes info.

ps lax |grep 'python3 mprocesses.py' |grep -v grep 
0  1000  6145  5615  20   0 275428 14600 -      Sl+  pts/1      0:00 python3 mprocesses.py
1  1000  6146  6145  20   0  54232 10340 -      R+   pts/1      1:01 python3 mprocesses.py
1  1000  6147  6145  20   0  54232 10348 -      R+   pts/1      1:01 python3 mprocesses.py

Check processes tree view.

pstree -p 5615
bash(5615)───python3(6145)─┬─python3(6146)
                           ├─python3(6147)
                           ├─{python3}(6148)
                           ├─{python3}(6149)
                           └─{python3}(6150)

What confused me is the three threads 6148,6149,6150.
Does that mean every process contain one process ? Maybe my logical graph is better to express relationships between processes and threads here.

bash(5615)───python3(6145)─┬─────────────────python3(6146)
                           |                    └─{python3}(6149)
                           |             
                           ├──────────────────python3(6147)
                           ├─{python3}(6148)     └─{python3}(6150)

1.bash(5615) is the python3 mprocesses.py(6145) 's father process.
2.python3 mprocesses.py(6145) contains two processes 6146 and 6147 created by pool = multiprocessing.Pool(processes = 2).
3.Process(6145) contain thread(6148),Process(6146) contain thread(6149),Process(6147) contain thread(6150).
It does no matter which exact process id contain which thread id.
Is my understanding right?

like image 218
showkey Avatar asked Jan 25 '18 03:01

showkey


1 Answers

You have:

  • 3 processes (1 parent process and 2 children to match your processes = 2 argument)
  • 2 threads in each process (1 main thread, and 1 communication and management thread)

The extra communication and management thread per process is an implementation detail of the multiprocessing module; if you are sharing resources between processes more threads may be used. You can see hints that threads are used for these tasks in the documentation

For example, under Pipes and Queues:

Note: When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe.

[...]

class multiprocessing.Queue([maxsize])
Returns a process shared queue implemented using a pipe and a few locks/semaphores. When a process first puts an item on the queue a feeder thread is started which transfers objects from a buffer into the pipe.

(italic emphasis mine)

You don't need to worry about these threads; they are there to implement the multiprocessing functionality and make it all run smoothly.

like image 59
Martijn Pieters Avatar answered Oct 09 '22 17:10

Martijn Pieters