Collate output in Python logging MemoryHandler with SMTPHandler

Question

I have the logging module MemoryHandler set up to queue debug and error messages for the SMTPHandler target. What I want is for an email to be sent when the process errors that contains all debug statements up to that point (one per line). What I get instead is a separate email for every debug message.

This seems like it should be trivial, and part of the logging package, but I can't find anything about it, no examples, nothing on Google.

log = logging.getLogger() log.setLevel(logging.DEBUG) debug_format = logging.Formatter("%(levelname)s at %(asctime)s in %(filename)s (line %(lineno)d):: %(message)s")  # write errors to email error_mail_subject = "ERROR: Script error in %s on %s" % (sys.argv[0], os.uname()[1]) error_mail_handler = logging.handlers.SMTPHandler(SMTP_HOST, 'errors@'+os.uname()[1], [LOG_EMAIL], error_mail_subject) error_mail_handler.setLevel(logging.ERROR) #error_mail_handler.setLevel(logging.DEBUG) error_mail_handler.setFormatter(debug_format)  # buffer debug messages so they can be sent with error emails memory_handler = logging.handlers.MemoryHandler(1024*10, logging.ERROR, error_mail_handler) memory_handler.setLevel(logging.DEBUG)  # attach handlers log.addHandler(memory_handler) log.addHandler(error_mail_handler) 

Related to this:

Do I need to add the error_mail_handler to the logger explicitly if it is a target of memory_handler anyway? Should error_mail_handler be set to DEBUG or ERROR target? Does it even need a target when it is being fed from memory_handler?

Would love to see some working code from anyone who has solved this problem.

Answer 1

You might want to use or adapt the BufferingSMTPHandler which is in this test script.

In general, you don't need to add a handler to a logger if it's the target of a MemoryHandler handler which has been added to a logger. If you set the level of a handler, that will affect what the handler actually processes - it won't process anything which is less severe than its level setting.

Answer 2

Instead of buffering for email, consider posting unbuffered to a message stream on a messaging app, e.g. on Matrix, Discord, Slack, etc. Having said that, I wrote my own beastly thread-safe implementation of BufferingSMTPHandler (backup link) which sends emails from a separate thread. The primary goal is to not block the main thread.

As written, it uses two queues - this seemed necessary in order to implement some useful class-level parameters that are defined in the "Configurable parameters" section of the code. Although you can use the code as-is, it's probably better if you study and use it to write your own class.

Issues:

• Some class-level parameters can perhaps be instance-level instead.
• Either threading.Timer or the signal module could perhaps be used to avoid loops that run forever.