Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

ThreadPoolExecutor logging? (python)

I have some code that looks like

with futures.ThreadPoolExecutor(max_workers=2) as executor:
    for function in functions:
        executor.submit(function)

How would I log which function is currently being handled by the executor? I may or may not have the capability to log from within the functions - would want the executor itself to log something like

print "handling process {i}".format(i=current_process)

Any thoughts on how to approach this?

like image 566
Andrew Avatar asked Oct 18 '22 13:10

Andrew


People also ask

What is ThreadPoolExecutor in Python?

The Python ThreadPoolExecutor allows you to create and manage thread pools in Python. Although the ThreadPoolExecutor has been available since Python 3.2, it is not widely used, perhaps because of misunderstandings of the capabilities and limitations of Threads in Python.

Is ThreadPoolExecutor thread safe Python?

ThreadPoolExecutor Thread-Safety Although the ThreadPoolExecutor uses threads internally, you do not need to work with threads directly in order to execute tasks and get results. Nevertheless, when accessing resources or critical sections, thread-safety may be a concern.

Is ThreadPoolExecutor faster?

The ThreadPoolExecutor is designed to speed-up your program by executing tasks concurrently. Nevertheless, in some use cases, using the ThreadPoolExecutor can make your program slower. Sometimes dramatically slower than performing the same task in a for loop.

How do you create a ThreadPool in Python?

How to create a ThreadPoolExecutor? With the help of concurrent. futures module and its concrete subclass Executor, we can easily create a pool of threads. For this, we need to construct a ThreadPoolExecutor with the number of threads we want in the pool.


1 Answers

I guess this is a little old but I stumbled across the questions and thought I would put an answer in. I just used a wrapper that can reference an instance of a logger prior to calling the function:

import logging
import os
import concurrent.futures


logging.basicConfig(filename=os.path.expanduser('~/Desktop/log.txt'), level=logging.INFO)
logger = logging.getLogger("MyLogger")


def logging_wrapper(func):
    def wrapped(*args, **kwargs):
        logger.info("Func name: {0}".format(func.__name__))
        func(*args, **kwargs)
    return wrapped


def a():
    print('a ran')


def b():
    print('b ran')


with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    for func in [a, b]:
        executor.submit(logging_wrapper(func))
like image 111
Lucas Currah Avatar answered Nov 04 '22 18:11

Lucas Currah