Allowing Ctrl-C to interrupt a python C-extension

Question

I'm running some computationally heavy simulation in (home-made) C-based python extensions. Occasionally I get stuff wrong and would like to terminate a simulation. However, Ctrl-C doesn't seem to have any effect (other than printing ^C to the screen) so I have to kill the process using kill or the system monitor.

As far as I can see python just waits for the C extension to finish and doesn't really communicate with it during this time.

Is there a way to make this work?

Update: The main answers (for my specific problem) turned out to be: 1. rewrite the code to regularly pass control back to the caller (answer Allowing Ctrl-C to interrupt a python C-extension below), or 2. Use PyErr_CheckSignals() (answer https://stackoverflow.com/a/33652496/423420 below)

Answer 1

However, Ctrl-C doesn't seem to have any effect

Ctrl-C in the shell sends SIGINT to the foreground process group. python on receiving the signal sets a flag in C code. If your C extension runs in the main thread then no Python signal handler will be run (and therefore you won't see KeyboardInterrupt exception on Ctrl-C) unless you call PyErr_CheckSignals() that checks the flag (it means: it shouldn't slow you down) and runs Python signal handlers if necessary or if your simulation allows Python code to execute (e.g., if the simulation uses Python callbacks). Here's a code example of an extension module for CPython created using pybind11 suggested by @Matt:

PYBIND11_MODULE(example, m)
{
    m.def("long running_func", []()
    {
        for (;;) {
            if (PyErr_CheckSignals() != 0)
                throw py::error_already_set();
            // Long running iteration
        }
    });
}

If the extension runs in a background thread then it is enough to release GIL (to allow Python code to run in the main thread that enables the signal handlers to run). PyErr_CheckSignals() always returns 0 in a background thread.

Related: Cython, Python and KeybordInterrupt ingored

Answer 2

Python has a signal handler installed on SIGINT which simply sets a flag that is checked by the main interpreter loop. For this handler to work properly, the Python interpreter has to be running Python code.

You have a couple of options available to you:

1. Use Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release the GIL around your C extension code. You cannot use any Python functions when not holding the GIL, but Python code (and other C code) may run concurrently with your C thread (true multithreading). A separate Python thread can execute alongside the C extension and catch Ctrl+C signals.
2. Set up your own SIGINT handler and call the original (Python) signal handler. Your SIGINT handler can then do whatever it needs to do to cancel the C extension code and return control to the Python interpreter.