Monitoring gevent exceptions in jobs

Question

I'm building an application using gevent. My app is getting rather big now as there are a lot of jobs being spawned and destroyed. Now I've noticed that when one of these jobs crashes my entire application just keeps running (if the exception came from a non main greenlet) which is fine. But the problem is that I have to look at my console to see the error. So some part of my application can "die" and I'm not instantly aware of that and the app keeps running.

Jittering my app with try catch stuff does not seem to be a clean solution. Maybe a custom spawn function which does some error reporting?

What is the proper way to monitor gevent jobs/greenlets? catch exceptions?

In my case I listen for events of a few different sources and I should deal with each different. There are like 5 jobs extremely important. The webserver greenlet, websocket greenlet, database greenlet, alarms greenlet, and zmq greenlet. If any of those 'dies' my application should completely die. Other jobs which die are not that important. For example, It is possible that websocket greenlet dies due to some exception raised and the rest of the applications keeps running fine like nothing happened. It is completely useless and dangerous now and should just crash hard.

Answer 1

I think the cleanest way would be to catch the exception you consider fatal and do sys.exit() (you'll need gevent 1.0 since before that SystemExit did not exit the process).

Another way is to use link_exception, which would be called if the greenlet died with an exception.

spawn(important_greenlet).link_exception(lambda *args: sys.exit("important_greenlet died"))

Note, that you also need gevent 1.0 for this to work.

If on 0.13.6, do something like this to kill the process:

gevent.get_hub().parent.throw(SystemExit())