How can I include the relative path to a module in a Python logging statement?

Question

My project has a subpackage nested under the root package like so:

• mypackage/
  • __init__.py
  • topmodule.py
  • subpackage/
    • __init__.py
    • nested.py

My goal is to get logging records formatted like:

mypackage/topmodule.py:123: First log message
mypackage/subpackage/nested.py:456: Second log message

so that the paths become clickable in my terminal.

---

I've tried the following formats.

• '%(modulename).pys:%(lineno): %(message)s' isn't clickable (the dots need to be slashes):

  mypackage.topmodule.py:123: First log message
  mypackage.subpackage.nested.py:456: Second log message
  

• 'mypackage/%(filename)s:%(lineno): %(message)s' doesn't work for subpackages:

  mypackage/topmodule.py:123: First log message
  mypackage/nested.py:456: Second log message
  

• '%(pathname)s:%(lineno): %(message)s' produces clickable paths, but they're so long that they cut off the rest of my logging :

  /Users/jacebrowning/Documents/mypackage/topmodule.py:123: First log message
  /Users/jacebrowning/Documents/mypackage/subpackage/nested.py:456: Second log message
  

---

Is there a logging pattern I can pass to logging.basicConfig(format='???') that wil produce the logging records I desire?

Answer 1

You'd have to do additional processing to get the path that you want here.

You can do such processing and add additional information to log records, including the 'local' path for your own package, by creating a custom filter.

Filters don't actually have to do filtering, but they do get access to all log records, so they are a great way of updating records with missing information. Just make sure you return True when done:

import logging
import os
import sys


class PackagePathFilter(logging.Filter):
    def filter(self, record):
        pathname = record.pathname
        record.relativepath = None
        abs_sys_paths = map(os.path.abspath, sys.path)
        for path in sorted(abs_sys_paths, key=len, reverse=True):  # longer paths first
            if not path.endswith(os.sep):
                path += os.sep
            if pathname.startswith(path):
                record.relativepath = os.path.relpath(pathname, path)
                break
        return True

This finds the sys.path entry that's the parent dir for the pathname on the logrecord and adds a new relativepath entry on the log record. You can then use %(relativepath)s to include it in your log.

Add the filter to any handler that you have configured with your custom formatter:

handler.addFilter(PackagePathFilter())

and together with '%(relativepath)s:%(lineno)s: %(message)s' as the format your log messages will come out like:

mypackage/topmodule.py:123: First log message
mypackage/subpackage/nested.py:456: Second log message

(actual output, except I altered the line numbers on that).