How can I combine stdlib logging with py.test

Question

I am using py.test to test some modules of mine that contains quite a bit of stdlib logging. I would of course like for the logging to log to stdout, which is captured by py.test, so that I will get all relevant logging messages if a test fails.

The problem with this is that the logging module ends up trying to log messages to the 'stdout'-object provided by py.test after this object has been discarded by py.test. That is, I get:

Traceback (most recent call last):
  File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.6/logging/__init__.py", line 1508, in shutdown
    h.flush()
  File "/usr/lib/python2.6/logging/__init__.py", line 754, in flush
    self.stream.flush()
ValueError: I/O operation on closed file

If I turn off capturing with -s, I don't have any problems, but of course that makes the test output unreadable with irrelevant logging.

Can anyone tell me the proper way to integrate stdlib logging with py.test?

(I tried looking at this, where it looks like it should just work without issues, so it didn't help me much)

Answer 1

Logging/capturing interactions are to better work with the upcoming 2.0.1 release, which you can install already as a development snapshot by:

pip install -i http://pypi.testrun.org pytest 

You should get at least "2.0.1.dev9" when you type "py.test --version" afterwards. And the problem/error your posted should now be gone.

A bit of background: the logging package insists on "owning" the streams it uses, and by default it grabs sys.stderr and insists to close it at process exit, registered via the atexit module. py.test substitutes sys.stdout with a temporary file in order to snapshot output (including output at file-descriptor level in order to also catch subprocess output). So py.test evolved to be very careful to always use the same temporary file as to not have logging's atexit-code complain.

Not that you can also install the [pytest-capturelog][1] plugin which will help some more with dealing with logging output.

[1] http://pypi.python.org/pypi/pytest-capturelog/0.7

Answer 2

If you don't mind the logging going nowhere after the objects have been discarded by py.test you could set the module-level variable logging.raiseExceptions to False somewhere early in your modules or tests.

This will make the logging module swallow exceptions that occur in the logging subsystem. (Also a good practice for production systems where you don't want errors in logging crash your system)

You could also use logging.basicConfig() to set logging output to a separate file. But this is probably not what you want.