How to debug silent crashes? Multiprocessing python

Question

I have a program that spawns a bunch of worker daemons using the multiprocessing library.

Sometimes these daemons will sporadically fall over and die. This isn't caused by a python-level Exception. The crashes seem to be totally silent...

Can anyone provide tips on how I may go about debugging this issue?

Answer 1

The faulthandler library might help debug such behavior. Run this code in every process run method to save tracebacks into separate files:

f = open('fault_%s.log' % multiprocessing.current_process().name, 'w')
faulthandler.enable(file=f, all_threads=True)