try-except-finally code not working as expected in threaded application

Question

Execution abruptly halting if the thread / process is killed makes sense

Why it won't execute cleanup code when I exit the main program normally by clicking the [X] on my terminal window?

---

I'm still learning the ins-and-outs of multithreaded applications, and I assume my problems come from not understanding how Python handles killing background threads.

Questions:

1. Why won't my finally: block execute all the time?
2. When else won't a finally: block execute?
3. What happens to code execution inside a thread when the thread is killed?
4. What happens to daemon/non-daemon threads when you exit the main process?

Details:

I'm trying to write a multithreaded program using ZMQ sockets that (among other things) writes stuff to a log file. I want the logging thread to unconditionally perform some messaging and clean-up right before it dies, but it won't most of the time.

The function below starts an infinite loop in a background thread and returns a zmq.PAIR socket for communication. The loop it starts listens to a socket, and anything written to that socket gets written to the file. The loop also (should) transmit back diagnostic messages like "I'm starting to log now!","Oops, there's been an error!" a "I'm exiting now". so the main program can keep tabs on it.

The main program generates a few threads using this pattern to monitor/control different bits and pieces. It polls several ZMQ sockets (connected to STDIN and a serial port) for messages, and forwards some of them to the socket connected to the file.

But now I'm stuck. The main program's routing & control logic works fine. get_logfile_sock's file writing works fine, and normal exception handling works as expected. But the "I'm exiting now" code doesn't execute when the thread is killed from the main program, or when I stop the main program altogether.

Example:

def get_logfile_sock(context, file_name):
    """
    Returns a ZMQ socket. Anything written to the socket gets appended to the a specified file. The socket will send diagnostic messages about file opening/closing and any exceptions encountered. 

    """

    def log_file_loop(socket):
        """
        Read characters from `socket` and write them to a file. Send back diagnostic and exception information.
        """
        try:
            socket.send("Starting Log File {}".format(file_name))
            with open(file_name, "a+") as fh:
                # File must start with a timestamp of when it was opened
                fh.write('[{}]'.format(get_timestamp()))
                # Write all strings/bytes to the file
                while True:
                    message = socket.recv()

                    fh.write(message)
                    fh.flush()

                    # Un-comment this line to demonstrate that the except: and finally: blocks both get executed when there's an error in the loop
                    # raise SystemExit

        except Exception as e:
            # This works fine when/if there's an exception in the loop
            socket.send("::".join(['FATALERROR', e.__class__.__name__, e.message]))
        finally:
            # This works fine if there's an exception raised in the loop
            # Why doesn't this get executed when my program exits? Isn't that just the main program raising SystemExit? 

            # Additional cleanup code goes here
            socket.send("Closing socket to log file {}".format(file_name))
            socket.close()


    # Make a socket pair for communication with the loop thread
    basename = os.path.basename(file_name).replace(":", "").replace(" ", "_").replace(".", "")
    SOCKNAME = 'inproc://logfile-{}'.format(basename)
    writer = context.socket(zmq.PAIR)
    reader = context.socket(zmq.PAIR)
    writer.bind(SOCKNAME)
    reader.connect(SOCKNAME)

    # Start the loop function in a separate thread
    thread = threading.Thread(target=log_file_loop, args=[writer])
    thread.daemon = True  # is this the right thing to do?
    thread.start()

    # Return a socket endpoint to the thread
    return reader

Answer 1

doesn't execute when the thread is killed

Don't kill threads. Ask them nicely to exit and then join on them. Consider passing in a Condition for them to check.

Long answer: executing a kill will cause the thread to exit without guaranteeing that it complete any particular block and you should not expect good behavior of your system afterwards. It's probably a little safer to do this when using multiprocessing though.