Obscure, repeatable crashes in multi-threaded Python console application using tk

Question

Using tk in my multi-threaded(*) console application causes it to crash without stacktrace, giving the message "Tcl_WaitForEvent: Notifier not initialized Abort trap".

The symptoms were that all my program's functions worked fine, until I brought up the tk window - then the very next operation would cause the crash.

Immediate searching found that Tkinter is not safe with respect to Python threads, so I made sure that I was not calling any Tk functions anywhere other than my main thread. The crashes continued.

I lost several hours because I believed that it was the specific command I was using that crashed the program - but eventually I realized that any keyboard input would crash the program.

After a lot of debugging, I finally boiled it down to a small program that demonstrates the issue, exposing what I believe is a bug or certainly a feature that needs documentation in the Tkinter library.

I was working on this posting I was debugging. I'm going to post it and answer my own question in the hopes that it will prevent the next person from wasting a day on it.

--

(* - Yes, it certainly needs to be multi-threaded. I have a thread for my socket connection, a thread that listens to a mic and finds levels, a thread to drive my serial port and more. In each case, the thing I'm reading on the thread naturally blocks most of the time.)

Answer 1

The solution!

The issue is that tk crashes if you read from Python's raw_input in a different thread from the tk thread!

A small program demonstrating the issue is here. If you run it, it gets keyboard input perfectly happily from the second thread - until you enter the command "tk" when it brings up an empty tk window. You can do whatever you like with that window - until you type in the console window and press return, when the whole program crashes.

Why am I reading from raw_input in a thread that isn't the main thread?

My program is a console application, but I'm controlling a lot of different parts, one of which is the pi3d OpenGL ES 2.0 graphics library which must be updated at or near the frame rate from the main Python thread.

How to work around it?

"Simple" enough - register for tk menu events and just get the keys directly! Except that's a crappy solution, because you have to emulate all those things that the console does for you already - deleting, left and right arrows and that sort of thing. But that's what I'll have to do.

Should the program become a fully-fledged tk application?

But I can't do that - the whole point of this program is that you can run it through a terminal window - often sshing into headless machines. I'm frankly more likely to make it a curses program!

The tk window is a minor part of the whole thing - just a window to show emulated lights when developing if you don't have the hardware hooked up (or don't want to keep flashing yourself in the face). I won't try to bring it up on headless machines, and that's fine.

Is this a bug?

I'm always loathe to attach such a label to software not my own, but I'm hard-pressed to come up with any other description. It causes a crash, and that crash produces no useful information of any type. I consider that Tkinter is somewhat lame for simply crashing when called from different threads, but at least this behavior is documented (if you dig down a little) - in this case, I'm calling a Python built-in, so I have no basis to expect that it will interact with this library at all, and there's no documentation of this problem.

Could there be a workaround?

I'm sort of hoping there will be a work-around - this one-page program was a single item on a long list of features has turned into a full-day head-scratching debugging session and I don't want to have to throw another day at least after this when none of this time is actually producing new features.

The best thing is if the tk team admitted this was a bug and came up with a fix. But I wouldn't expect that at my desktop before a year from now...

So perhaps the real best thing would be if there were some way to get tk to simply ignore the keyboard, and not crash. I did a tiny experiment with tk's "busy" but that didn't work and just doesn't seem to be the right sort of thing.

A little later, I'm now thinking about running the lighting as an independent program, a separate subprocess using Python's Subprocess library, and sending it text commands through stdin. This would be overkill if this were the only problem that was being solved, but in fact

Got it.

Replacing raw_input() with sys.stdin.readline() did the trick - at least in the demo (which I updated). Feel free to download this and experiment with it yourself!

I hope this saves someone else the time.

Answer 2

In my case (as mentioned in the comments under @Tom Swirly's answer) the solution was to switch to a non-interactive backend:

import matplotlib
matplotlib.use('Agg')