Timeout function using threading in python does not work

Question

I have found a code creating a timeout function here, which does not seem to work. The complete test code is below:

def timeout(func, args=(), kwargs={}, timeout_duration=1, default=None):
    import threading
    class InterruptableThread(threading.Thread):
        def __init__(self):
            threading.Thread.__init__(self)
            self.result = None

        def run(self):
            try:
                self.result = func(*args, **kwargs)
            except:
                self.result = default

    it = InterruptableThread()
    it.start()
    it.join(timeout_duration)
    if it.isAlive():
        return default
    else:
        return it.result


def foo():
    while True:
        pass

timeout(foo,timeout_duration=3)

Expected behavior: code ends within 3 seconds. Where is the problem?

Answer 1

A thread can not gracefully kill another thread, so with your current code, foo never terminates. (With thread.daemon = True the Python program will exit when only daemon threads are left, but that does not allow you to terminate foo without also terminating the main thread.)

Some people have tried to use signals to halt execution, but this may be unsafe in some cases.

If you can modify foo, there are many solutions possible. For instance, you could check for a threading.Event to break out of the while-loop.

But if you can not modify foo, you could run it in a subprocess using the multiprocessing module since unlike threads, subprocesses can be terminated. Here is an example of how that might look:

import time
import multiprocessing as mp

def foo(x = 1):
    cnt = 1
    while True:
        time.sleep(1)
        print(x, cnt)
        cnt += 1

def timeout(func, args = (), kwds = {}, timeout = 1, default = None):
    pool = mp.Pool(processes = 1)
    result = pool.apply_async(func, args = args, kwds = kwds)
    try:
        val = result.get(timeout = timeout)
    except mp.TimeoutError:
        pool.terminate()
        return default
    else:
        pool.close()
        pool.join()
        return val


if __name__ == '__main__':
    print(timeout(foo, kwds = {'x': 'Hi'}, timeout = 3, default = 'Bye'))
    print(timeout(foo, args = (2,), timeout = 2, default = 'Sayonara'))

yields

('Hi', 1)
('Hi', 2)
('Hi', 3)
Bye
(2, 1)
(2, 2)
Sayonara

Note that this has some limitations too.

• subprocesses receive a copy of the parent processes' variables. If you modify a variable in a subprocess, it will NOT affect the parent process. If your function func needs to modify variables, you will need to use a shared variable.

• arguments (passed through args) and keywords (kwds) must be picklable.

• processes are more resource-heavy than threads. Usually, you only want to create a multiprocessing Pool once at the beginning of a program. This timeout function creates a Pool every time you call it. This was necessary since we needed pool.terminate() to terminate foo. There might be a better way, but I haven't thought of it.

Answer 2

You need to turn it into a daemon thread:

it = ...
it.daemon = True
it.start()

Otherwise it's created as a user thread, and the process won't get stopped until all the user threads have finished.

Note that with your implementation the thread will continue to run and consume resources even after you've timed out waiting for it. CPython's Global Interpreter Lock could exacerbate the issue further.