Is there a pythonic way to decouple optional functionality from a function's main purpose?

Question

Context

Suppose I have the following Python code:

def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        for _ in range(n_iters):
            number = halve(number)
        sum_all += number
    return sum_all


ns = [1, 3, 12]
print(example_function(ns, 3))

example_function here is simply going through each of the elements in the ns list and halving them 3 times, while accumulating the results. The output of running this script is simply:

2.0

Since 1/(2^3)*(1+3+12) = 2.

Now, let's say that (for any reason, perhaps debugging, or logging), I would like to display some type of information about the intermediate steps that the example_function is taking. Maybe I would then rewrite this function into something like this:

def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        print('Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            print(number)
        sum_all += number
        print('sum_all:', sum_all)
    return sum_all

which now, when called with the same arguments as before, outputs the following:

Processing number 1
0.5
0.25
0.125
sum_all: 0.125
Processing number 3
1.5
0.75
0.375
sum_all: 0.5
Processing number 12
6.0
3.0
1.5
sum_all: 2.0

This achieves exactly what I intended. However, this goes a bit against the principle that a function should only do one thing, and now the code for example_function is sligthly longer and more complex. For such a simple function this is not a problem, but in my context I have quite complicated functions calling each other, and the printing statements often involve more complicated steps than shown here, resulting in a substantial increase in complexity of my code (for one of my functions there were more lines of code related to logging than there were lines related to its actual purpose!).

Furthermore, if I later decide that I don't want any printing statements in my function anymore, I would have to go through example_function and delete all of the print statements manually, along with any variables related this functionality, a process which is both tedious and error-prone.

The situation gets even worse if I would like to always have the possibility of printing or not printing during the function execution, leading me to either declaring two extremely similar functions (one with the print statements, one without), which is terrible for maintaining, or to define something like:

def example_function(numbers, n_iters, debug_mode=False):
    sum_all = 0
    for number in numbers:
        if debug_mode:
            print('Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            if debug_mode:
                print(number)
        sum_all += number
        if debug_mode:
            print('sum_all:', sum_all)
    return sum_all

which results in a bloated and (hopefully) unnecessarily complicated function, even in the simple case of our example_function.

---

Question

Is there a pythonic way to "decouple" the printing functionality from the original functionality of the example_function?

More generally, is there a pythonic way to decouple optional functionality from a function's main purpose?

---

What I have tried so far:

The solution I have found at the moment is using callbacks for the decoupling. For instance, one can rewrite the example_function like this:

def example_function(numbers, n_iters, callback=None):
    sum_all = 0
    for number in numbers:
        for i_iter in range(n_iters):
            number = number/2

            if callback is not None:
                callback(locals())
        sum_all += number
    return sum_all

and then defining a callback function that performs whichever printing functionality I want:

def print_callback(locals):
    print(locals['number'])

and calling example_function like this:

ns = [1, 3, 12]
example_function(ns, 3, callback=print_callback)

which then outputs:

0.5
0.25
0.125
1.5
0.75
0.375
6.0
3.0
1.5
2.0

This successfully decouples the printing functionality from the base functionality of example_function. However, the main problem with this approach is that the callback function can only be run at a specific part of the example_function (in this case right after halving the current number), and all of the printing has to happen exactly there. This sometimes forces the design of the callback function to be quite complicated (and makes some behaviors impossible to achieve).

For instance, if one would like to achieve exactly the same type of printing as I did in a previous part of the question (showing which number is being processed, along with its corresponding halvings) the resulting callback would be:

def complicated_callback(locals):
    i_iter = locals['i_iter']
    number = locals['number']
    if i_iter == 0:
        print('Processing number', number*2)
    print(number)
    if i_iter == locals['n_iters']-1:
        print('sum_all:', locals['sum_all']+number)

which results in exactly the same output as before:

Processing number 1.0
0.5
0.25
0.125
sum_all: 0.125
Processing number 3.0
1.5
0.75
0.375
sum_all: 0.5
Processing number 12.0
6.0
3.0
1.5
sum_all: 2.0

but is a pain to write, read, and debug.

Answer 1

If you need functionality outside the function to use data from inside the function, then there needs to be some messaging system inside the function to support this. There is no way around this. Local variables in functions are totally isolated from the outside.

The logging module is quite good at setting up a message system. It is not only restricted to printing out the log messages - using custom handlers, you can do anything.

Adding a message system is similar to your callback example, except that the places where the 'callbacks' (logging handlers) are handled can be specified anywhere inside the example_function (by sending the messages to the logger). Any variables that are needed by the logging handlers can be specified when you send the message (you can still use locals(), but it is best to explicitly declare the variables you need).

A new example_function might look like:

import logging

# Helper function
def send_message(logger, level=logging.DEBUG, **kwargs):
  logger.log(level, "", extra=kwargs)

# Your example function with logging information
def example_function(numbers, n_iters):
    logger = logging.getLogger("example_function")
    # If you have a logging system set up, then we don't want the messages sent here to propagate to the root logger
    logger.propagate = False
    sum_all = 0
    for number in numbers:
        send_message(logger, action="processing", number=number)
        for i_iter in range(n_iters):
            number = number/2
            send_message(logger, action="division", i_iter=i_iter, number=number)
        sum_all += number
        send_message(logger, action="sum", sum=sum_all)
    return sum_all

This specifies three locations where the messages could be handled. On its own, this example_function will not do anything other than the functionality of the example_function itself. It will not print out anything, or do any other functionality.

To add extra functionality to the example_function, then you will need to add handlers to the logger.

For example, if you want to do some printing out of the sent variables (similar to your debugging example), then you define the custom handler, and add it to the example_function logger:

class ExampleFunctionPrinter(logging.Handler):
    def emit(self, record):
        if record.action == "processing":
          print("Processing number {}".format(record.number))
        elif record.action == "division":
          print(record.number)
        elif record.action == "sum":
          print("sum_all: {}".format(record.sum))

example_function_logger = logging.getLogger("example_function")
example_function_logger.setLevel(logging.DEBUG)
example_function_logger.addHandler(ExampleFunctionPrinter())

If you want to plot the results on a graph, then just define another handler:

class ExampleFunctionDivisionGrapher(logging.Handler):
    def __init__(self, grapher):
      self.grapher = grapher

    def emit(self, record):
      if record.action == "division":
        self.grapher.plot_point(x=record.i_iter, y=record.number)

example_function_logger = logging.getLogger("example_function")
example_function_logger.setLevel(logging.DEBUG)
example_function_logger.addHandler(
    ExampleFunctionDivisionGrapher(MyFancyGrapherClass())
)

You can define and add in whatever handlers you want. They will be totally separate from the functionality of the example_function, and can only use the variables that the example_function gives them.

Although logging can be used as a messaging system, it might be better to move to a fully fledged messaging system, such as PyPubSub, so that it doesn't interfere with any actual logging that you might be doing:

from pubsub import pub

# Your example function
def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        pub.sendMessage("example_function.processing", number=number)
        for i_iter in range(n_iters):
            number = number/2
            pub.sendMessage("example_function.division", i_iter=i_iter, number=number)
        sum_all += number
        pub.sendMessage("example_function.sum", sum=sum_all)
    return sum_all

# If you need extra functionality added in, then subscribe to the messages.
# Otherwise nothing will happen, other than the normal example_function functionality.
def handle_example_function_processing(number):
    print("Processing number {}".format(number))

def handle_example_function_division(i_iter, number):
    print(number)

def handle_example_function_sum(sum):
    print("sum_all: {}".format(sum))

pub.subscribe(
    "example_function.processing",
    handle_example_function_processing
)
pub.subscribe(
    "example_function.division",
    handle_example_function_division
)
pub.subscribe(
    "example_function.sum",
    handle_example_function_sum
)

Answer 2

You can define a function encapsulating the debug_mode condition and pass the desired optional function and its arguments to that function (as suggested here):

def DEBUG(function, *args):
    if debug_mode:
        function(*args)

def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        DEBUG(print, 'Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            DEBUG(print, number)
        sum_all += number
        DEBUG(print, 'sum_all:', sum_all)
    return sum_all

ns = [1, 3, 12]
debug_mode = True
print(example_function(ns, 3))

Note that debug_mode must obviously have been assigned a value before calling DEBUG.

It is of course possible to invoke functions other than print.

You could also extend this concept to several debug levels by using a numeric value for debug_mode.

Answer 3

If you want to stick with just print statements, you can use a decorator that adds an argument which turns on/off the printing to console.

Here is a decorator that add the keyword-only argument and default value of verbose=False to any function, updates the docstring and signature. Calling the function as-is returns the expected output. Calling the function with verbose=True will turn on the print statements and return the expected output. This has the added benefit of not having to preface every print with a if debug: block.

from functools import wraps
from inspect import cleandoc, signature, Parameter
import sys
import os

def verbosify(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        def toggle(*args, verbose=False, **kwargs):
            if verbose:
                _stdout = sys.stdout
            else:
                _stdout = open(os.devnull, 'w')
            with redirect_stdout(_stdout):
                return func(*args, **kwargs)
        return toggle(*args, **kwargs)
    # update the docstring
    doc = '\n\nOption:\n-------\nverbose : bool\n    '
    doc += 'Turns on/off print lines in the function.\n '
    wrapper.__doc__ = cleandoc(wrapper.__doc__ or '\n') + doc
    # update the function signature to include the verbose keyword
    sig = signature(func)
    param_verbose = Parameter('verbose', Parameter.KEYWORD_ONLY, default=False)
    sig_params = tuple(sig.parameters.values()) + (param_verbose,)
    sig = sig.replace(parameters=sig_params)
    wrapper.__signature__ = sig
    return wrapper

Wrapping your function now allows you to turn on/off print functions using verbose.

@verbosify
def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        print('Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            print(number)
        sum_all += number
        print('sum_all:', sum_all)
    return sum_all

Examples:

example_function([1,3,12], 3)
# returns:
2.0

example_function([1,3,12], 3, verbose=True)
# returns/prints:
Processing number 1
0.5
0.25
0.125
sum_all: 0.125
Processing number 3
1.5
0.75
0.375
sum_all: 0.5
Processing number 12
6.0
3.0
1.5
sum_all: 2.0
2.0

When you inspect example_function, you will see the updated documentation as well. Since your function doesn't have a docstring, it is just what is in the decorator.

help(example_function)
# prints:
Help on function example_function in module __main__:

example_function(numbers, n_iters, *, verbose=False)
    Option:
    -------
    verbose : bool
        Turns on/off print lines in the function.

In terms of coding philosophy. Having function that incur not side-effects is a functional programming paradigm. Python can be a functional language, but it is not designed to be exclusively that way. I always design my code with the user in mind.

If adding the option to print the calculation steps is a benefit to the user, then there is NOTHING wrong with do that. From a design standpoint, you are going to be stuck with adding in the print/logging commands somewhere.

Answer 4

I have updated my answer with a simplification: function example_function is passed a single callback or hook with a default value such that example_function no longer needs to test to see whether it was passed or not:

hook=lambda *args, **kwargs: None

The above is a lambda expression that returns None and example_function can call this default value for hook with any combination of positional and keyword parameters at various places within the function.

In the example below, I am only interested in the "end_iteration" and "result" events.

def example_function(numbers, n_iters, hook=lambda *args, **kwargs: None):
    hook("init")
    sum_all = 0
    for number in numbers:
        for i_iter in range(n_iters):
            hook("start_iteration", number)
            number = number/2
            hook("end_iteration", number)
        sum_all += number
    hook("result", sum_all)
    return sum_all

if __name__ == '__main__':
    def my_hook(event_type, *args):
        if event_type in ["end_iteration", "result"]:
            print(args[0])

    print('sum = ', example_function([1, 3, 12], 3))
    print('sum = ', example_function([1, 3, 12], 3, my_hook))

Prints:

sum =  2.0
0.5
0.25
0.125
1.5
0.75
0.375
6.0
3.0
1.5
2.0
sum =  2.0

The hook function can be as simple or as elaborate as you want. Here it is doing a check of the event type and doing a simple print. But it could obtain a logger instance and log the message. You can have all the richness of logging if you need it but simplicity if you don't.