Most pythonic way to collect warnings from a function

Question

Consider a very simple function:

def generate_something(data):
    if data is None:
        raise Exception('No data!')

    return MyObject(data)

Its output is basically an instance of an object I want to create or an exception if the function cannot create the object. We can say that the output is binary since it either succeeds (and gives back an object) or not (and gives back an Exception).

What is the most pythonic way to handle a third state, that is "success but with some warnings"?

def generate_something(data):
    warnings = []

    if data is None:
        raise Exception("No data!")

    if data.value_1 == 2:
        warnings.append('Hmm, value_1 is 2')    

    if data.value_2 == 1:
        warnings.append('Hmm, value_2 is 1')    

    return MyObject(data), warnings

Is returning a tuple the only way to handle this, or it is possible to broadcast or yield warnings from within the functions and catch them from the caller?

Answer 1

The built-on option: warnings

Python has a built-in warning mechanism implemented in the warnings module. The problem with this is that warnings maintains a global warnings filter, which might unintenionally cause the warnings your function throws to be suppressed. Here's a demonstration of the problem:

import warnings

def my_func():
    warnings.warn('warning!')

my_func()  # prints "warning!"

warnings.simplefilter("ignore")
my_func()  # prints nothing

If you want to use warnings regardless of this, you can use warnings.catch_warnings(record=True) to collect all thrown warnings in a list:

with warnings.catch_warnings(record=True) as warning_list:
    warnings.warn('warning 3')

print(warning_list)  # output: [<warnings.WarningMessage object at 0x7fd5f2f484e0>]

The self-made option

For the reason explained above, I recommend rolling your own warning mechanism instead. There are various ways to implement this:

• Just return a list of warnings

  The easiest solution with the least overhead: Just return the warnings.

  def example_func():
      warnings = []
  
      if ...:
          warnings.append('warning!')
  
      return result, warnings
  
  result, warnings = example_func()
  for warning in warnings:
      ...  # handle warnings
  

• Pass a warning handler to the function

  If you want to handle the warnings immediately when they're generated, you can rewrite your function to accept a warning handler as argument:

  def example_func(warning_handler=lambda w: None):
      if ...:
          warning_handler('warning!')
  
      return result
  
  
  def my_handler(w):
      print('warning', repr(w), 'was produced')
  
  result = example_func(my_handler)
  

• contextvars (python 3.7+)

  With python 3.7 we got the contextvars module, which lets us implement a higher-level warning mechanism based on context managers:

  import contextlib
  import contextvars
  import warnings
  
  
  def default_handler(warning):
      warnings.warn(warning, stacklevel=3)
  
  _warning_handler = contextvars.ContextVar('warning_handler', default=default_handler)
  
  
  def warn(msg):
      _warning_handler.get()(msg)
  
  
  @contextlib.contextmanager
  def warning_handler(handler):
      token = _warning_handler.set(handler)
      yield
      _warning_handler.reset(token)
  

  Usage example:

  def my_warning_handler(w):
      print('warning', repr(w), 'was produced')
  
  with warning_handler(my_warning_handler):
      warn('some problem idk')  # prints "warning 'some problem idk' was produced"
  
  warn(Warning('another problem'))  # prints "Warning: another problem"
  

  Caveats: As of now, contextvars doesn't support generators. (Relevant PEP.) Things like the following example won't work correctly:

  def gen(x):
      with warning_handler(x):
          for _ in range(2):
              warn('warning!')
              yield
  
  g1 = gen(lambda w: print('handler 1'))
  g2 = gen(lambda w: print('handler 2'))
  
  next(g1)  # prints "handler 1"
  next(g2)  # prints "handler 2"
  next(g1)  # prints "handler 2"
  

• without contextvars (for python <3.7)

  If you don't have contextvars, you can use this async-unsafe implementation instead:

  import contextlib
  import threading
  import warnings
  
  
  def default_handler(warning):
      warnings.warn(warning, stacklevel=3)
  
  _local_storage = threading.local()
  _local_storage.warning_handler = default_handler
  
  
  def _get_handler():
      try:
          return _local_storage.warning_handler
      except AttributeError:
          return default_handler
  
  
  def warn(msg):
      handler = _get_handler()
      handler(msg)
  
  
  @contextlib.contextmanager
  def warning_handler(handler):
      previous_handler = _get_handler()
      _local_storage.warning_handler = handler
  
      yield
  
      _local_storage.warning_handler = previous_handler