What is the best way to suppress the spark output in the Jupyter notebook?

Question

I recently changed my os from ubuntu to mac.(after 4+ years)

one thing that annoys me a lot is. The spark stdout/stderr now prints on the notebook, rather than staying only inside the console (as on my ubuntu).

I have tried to disable warnings on ipython startup but it didn't work. ~/.ipython/profile_default/startup/disable-warning.py

import warnings
warnings.filterwarnings('ignore')

on the other hand, I am not entirely sure if that is mac problem. Is something went stupidly wrong during my dev env setup?

Thank you!

Answer 1

The modification of the notebook output comes from one of the changes made in the release 6.0.0 of ipykernel.

All outputs to stdout/stderr should now be captured, including subprocesses and output of compiled libraries (blas, lapack....). In notebook server, some outputs that would previously go to the notebooks logs will now both head to notebook logs and in notebooks outputs.

A subsequent fix provides a way to restore the previous behavior. The fix consists in disabling the capture new behavior by turning it off through the capture_fd_output flag, see this comment for more detail.

You can configure it by turning it off in your ipython profile.

# create a default profile
ipython profile create
# turn off the capture flag
echo "c.IPKernelApp.capture_fd_output = False" >> \
     "~/.ipython/profile_default/ipython_kernel_config.py"

That's it ! All the outputs from Java, Spark and Ivy will no more be displayed in the notebook but only in the logs.

Notes

• It can also be done system-wide by putting ipython_kernel_config.py in the /etc/ipython/ directory, see the doc for more information.
• For the record, here is the PR implementing this fix on Jupyter Docker Stacks.