Python Requests library throws exceptions in logging

Question

The Python requests library appears to have some rather strange quirks when it comes to its logging behaviour. Using the latest Python 2.7.8, I have the following code:

import requests
import logging

logging.basicConfig(
    filename='mylog.txt',
    format='%(asctime)-19.19s|%(task)-36s|%(levelname)s:%(name)s: %(message)s',
    level=eval('logging.%s' % 'DEBUG'))

logger = logging.getLogger(__name__)

logger.info('myprogram starting up...', extra={'task': ''})     # so far so good
...
(ommited code)
...
payload = {'id': 'abc', 'status': 'ok'}

# At this point the program continues but throws an exception.
requests.get('http://localhost:9100/notify', params=payload) 

print 'Task is complete! NotifyURL was hit! - Exiting'

My program seems to exit normally, however inside the log file it creates (mylog.txt) I always find the following exception:

KeyError: 'task'
Logged from file connectionpool.py, line 362

If I remove this: requests.get('http://localhost:9100/notify', params=payload) then the exception is gone.

What exactly am I doing wrong here and how may I fix this? I am using requests v2.4.3.

Answer 1

The problem is your custom logging format, where you expect a %(task). Requests (or rather the bundled urllib3) does not include the task parameter when logging, as it has no way of knowing, that you expect this.