Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python Multiprocessing: billiard vs multiprocessing

Original problem: using celery task queue, I want the processes in the process pool to use shared CUDA arrays ( i.e. instead of each process having its unique array, I want one array to be accessed by all the processes. This is safe because only reads are performed ). Pytorch's torch.multiprocessing library allows this and according to the doc, it is a simple drop-in replacement for multiprocessing.

billiard and multiprocessing seem to be two viable options for creating process pools. Currently, celery, the python task queue library, uses billiard over multiprocessing due to some feature improvements. Someone asked a question here, but the answer is not really specific.

It backports changes from the Python 2.7 and 3.x. 
The current version is compatible with Py2.4 - 2.7 and falls back to multiprocessing for 3.x, 
the next version will only support 2.6, 2.7 and 3.x. 

I need to replace billiard with multiprocessing in celery's source code ( in order to use pytorch's multiprocessing library torch.multiprocessing), but would this be okay? What are the differences between multiprocessing and billiard?

like image 848
AgnosticCucumber Avatar asked Sep 14 '25 01:09

AgnosticCucumber


1 Answers

To those who are looking to replace celery's billiard with multiprocessingit is not possible according to this reply from celery's dev

like image 115
AgnosticCucumber Avatar answered Sep 16 '25 16:09

AgnosticCucumber