Mask out sensitive information in python log

Question

Consider the following code

try:
    r = requests.get('https://sensitive:[email protected]/')
    r.raise_for_status()
except requests.HTTPError:
    logging.exception("Failed to what.ever")

Here, if the endpoint returns non-successful http status code, the following will be logged

Traceback (most recent call last):
  File "a.py", line 5, in <module>
    r.raise_for_status()
  File "venv/lib/python3.5/site-packages/requests/models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://sensitive:[email protected]/

The problem is that the password is logged. I could create a logging filter to filter out this line completely. However, it would be more convenient if the password was just masked out somehow. As no string is passed to logging.exception filtering on the app side is tricky. Where in the logging framwork can I transform a log record?

Answer 1

Apparently, this is done with a Formatter. Example below

import logging
import re


class SensitiveFormatter(logging.Formatter):
    """Formatter that removes sensitive information in urls."""
    @staticmethod
    def _filter(s):
        return re.sub(r':\/\/(.*?)\@', r'://', s)

    def format(self, record):
        original = logging.Formatter.format(self, record)
        return self._filter(original)

Use like so

import logging
import requests

from sensitive_formatter import SensitiveFormatter

LOG_FORMAT = \
    '%(asctime)s [%(threadName)-16s] %(filename)27s:%(lineno)-4d %(levelname)7s| %(message)s'
logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger(__name__)

# Don't actually configure your logging like this, just to showcase
# the above answer. :)
for handler in logging.root.handlers:
   handler.setFormatter(SensitiveFormatter(LOG_FORMAT))

log.warning('https://not:[email protected]/basic-auth/expected-user/expected-pass')
try:
    r = requests.get('https://not:[email protected]/basic-auth/expected-user/expected-pass')
    r.raise_for_status()
except requests.exceptions.RequestException as e:
    log.exception('boom!')

The user/password will be masked out. See example log below

$ python log_example.py 
2018-05-18 11:59:22,703 [MainThread      ]                      log.py:14   WARNING| https://httpbin.org/basic-auth/user/secret
2018-05-18 11:59:22,747 [MainThread      ]           connectionpool.py:824    DEBUG| Starting new HTTPS connection (1): httpbin.org
2018-05-18 11:59:23,908 [MainThread      ]           connectionpool.py:396    DEBUG| https://httpbin.org:443 "DELETE /basic-auth/user/secret HTTP/1.1" 405 178
2018-05-18 11:59:23,913 [MainThread      ]                      log.py:19     ERROR| boom!
Traceback (most recent call last):
  File "log.py", line 17, in <module>
    r.raise_for_status()
  File "/Users/vidstige/src/so/venv/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 405 Client Error: METHOD NOT ALLOWED for url: https://httpbin.org/basic-auth/user/secret