Supressing warnings during parallel execution with joblib in Python

Question

I am using a function that generates a warning that I really don't need to read. The problem is that I want to run the function in parallel and when doing so, it seems I can not suppress warnings anymore. Consider this example:

import warnings
import numpy as np
from joblib import Parallel, delayed

def test(a, b):
    if a * b > 10:
        warnings.warn("You are being warned!!")
    return(a*b)

ii = np.arange(5)
jj = ii + 1

with warnings.catch_warnings():
    warnings.filterwarnings("ignore")
    with Parallel(n_jobs=4) as parallel:
        result = parallel(delayed(test)(i, j) for i, j in zip(ii, jj))

This still produces the warning message... Please also be aware that in my case I can not rewrite the function test as it is imported from another package. Is there any way to not get the warning message?

Answer 1

Yes, this is very annoying!

The default joblib backend spawns additional processes, which do not seem to inherit the warning filters applied using warnings.filterwarnings. However, you can use the PYTHONWARNINGS environment variable to set warning filters; this will affect all newly-spawned processes, which inherit their environment variables from those in the main process.

From the relevant documentation page:

The warnings filter is initialized by -W options passed to the Python interpreter command line and the PYTHONWARNINGS environment variable. The interpreter saves the arguments for all supplied entries without interpretation in sys.warnoptions; the warnings module parses these when it is first imported (invalid options are ignored, after printing a message to sys.stderr).

Individual warnings filters are specified as a sequence of fields separated by colons:

action:message:category:module:line

A separate page describes in more detail what each of these fields means, but basically:

• action describes what to do with the warning; for you, you want ignore to suppress the message
• message can be a string that must match the beginning of the warning message in order for it to be filtered
• category is the warning class, e.g. FutureWarning, DeprecationWarning
• module and line refer to where the warning is raised

Any of these fields can be empty, and you can leave off the trailing semicolons.

So, to ignore all FutureWarnings:

Within a Jupyter notebook, you can do something like

%env PYTHONWARNINGS=ignore::FutureWarning

Or in a script, add an entry to os.environ:

import os
os.environ['PYTHONWARNINGS']='ignore::FutureWarning'

It seems there should probably be a way to set environment variables only for the spawned process, but I can't figure out if joblib exposes an API for this.