Python get the active process number of multiprocessing pool

Question

I create a process pool with multiprocessing pool. I have many task to deal with, but it is not easy to get the qps of the task. So I want to get the active process number of the pool so i can set a appropriate pool size. This is the whole code:

import time
from multiprocessing import Pool

def do_work(msg):
    # do some work


if __name__ == '__main__':
    consumer = KafkaConsumer(
    group_id=worker_config.kafka_group_id,
    bootstrap_servers=kafka_url,
    auto_offset_reset=worker_config.kafka_reset,
    enable_auto_commit=True)
    consumer.subscribe(topics=worker_config.kafka_topics)

    for message in consumer:
        logging.info('topic=%s, partition=%d, msg=%s' % (message.topic, message.partition, msg))
        pool.apply_async(do_work, (message,))
        process_count = number_of_active_process_of_pool
        logging.info("number_of_active_process_number is %d", process_count)


    pool.close()
    pool.join()

Answer 1

apply_async gives you an AsyncResult: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.AsyncResult

Which you can use .ready() on to find out if it's done. This way you get the amount of tasks done and by extension the amount of tasks left to be done. As long as this number exceeds the poolsize, you can assume that poolsize many processes are running, if it does not, then the remaining amount of tasks would be the amount of running processes.

Alternatives:

If you don't use apply_async but instead a Queue, such as this one you can then get the approximate queue size with .qsize()

there is also multiprocessing.active_children, but that only works if those processes end, a pool however does not; unless you order it to .join() So in your case it would work.