Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python multiprocessing - How to release memory when a process is done?

I encountered a weird problem while using python multiprocessing library.

My code is sketched below: I spawn a process for each "symbol, date" tuple. I combine the results afterwards.

I expect that when a process has done computing for a "symbol, date" tuple, it should release its memory? apparently that's not the case. I see dozens of processes (though I set the process pool to have size 7) that are suspended¹ in the machine. They consume no CPU, and they don't release the memory.

How do I let a process release its memory, after it has done its computation?

Thanks!

¹ by "suspended" I mean their status in ps command is shown as "S+"

def do_one_symbol( symbol, all_date_strings ):     pool = Pool(processes=7)     results = [];     for date in all_date_strings:         res = pool.apply_async(work, [symbol, date])         results.append(res);      gg = mm = ss = 0;     for res in results:         g, m, s = res.get()         gg += g;          mm += m;          ss += s; 
like image 651
CuriousMind Avatar asked Oct 04 '11 13:10

CuriousMind


People also ask

Do Python processes share memory?

Shared memory can be a very efficient way of handling data in a program that uses concurrency. Python's mmap uses shared memory to efficiently share large amounts of data between multiple Python processes, threads, and tasks that are happening concurrently.

How do you end a process in multiprocessing?

We can kill or terminate a process immediately by using the terminate() method. We will use this method to terminate the child process, which has been created with the help of function, immediately before completing its execution.

How lock in multiprocessing Python?

Python provides a mutual exclusion lock for use with processes via the multiprocessing. Lock class. An instance of the lock can be created and then acquired by processes before accessing a critical section, and released after the critical section. Only one process can have the lock at any time.

Can Python multiprocessing access global variables?

You can inherit global variables between forked processes, but not spawned processes. In this tutorial you will discover how to inherit global variables between processes in Python. Learn multiprocessing systematically with this 7-day jump-start.


1 Answers

Did you try to close pool by using pool.close and then wait for process to finish by pool.join, because if parent process keeps on running and does not wait for child processes they will become zombies

like image 176
Anurag Uniyal Avatar answered Sep 28 '22 15:09

Anurag Uniyal