How to set up HTTPHandler for python logging

Question

I'm trying to use HTTPHandler class of standard python logging library to send logs. I need to make a https post request with basic credentials(username and password). This is how i'm setting up the HTTPHandler-

    host = 'example.com'
    url = '/path'
    handler = logging.handlers.HTTPHandler(host, url, method='POST', secure=True, credentials=('username','password'), context=None)
    logger.addHandler(handler)

But the problem is, I'm not getting anylogs in my remote server.I'm not even seeing any exception from the handler. Am I setting up the handler arguments incorrectly? I can send similar logs using simple pythong http request-

url = 'https://username:[email protected]/path'
headers = {'content-type': 'application/json'}
jsonLog = { 'id': '4444','level': 'info', 'message': 'python log' };

r = requests.post(url, data = json.dumps(jsonLog), headers=headers)

Do i need to setup header somehow because of json content-type? If yes than how do i set that up in the httphandler?

Update

I thought I should update what I ended up doing. After numerous search i found i can create a custom handler by overriding emit() of logging.Handler.

class CustomHandler(logging.Handler):
def emit(self, record):
    log_entry = self.format(record)
    # some code....
    url = 'url'
    # some code....
    return requests.post(url, log_entry, headers={"Content-type": "application/json"}).content

Feel free to post if any has any better suggestions.

Answer 1

Expanding on the solution saz gave, here's how add a custom HTTP handler that will forward the logs emitted to the specified URL using a bearer token.

It uses a requests session instead of having to establish a new session every log event.

Furthermore, if the request fails it attempts to resend the logs for a given number of retries.

Note: make sure your logging handler is as simple as possible to prevent the application from halting because of a log event.

I tested it with a simple localhost echo server and it works.

Feel free to suggest any changes.

import json
import logging
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

class CustomHttpHandler(logging.Handler):
    def __init__(self, url: str, token: str, silent: bool = True):
        '''
        Initializes the custom http handler
        Parameters:
            url (str): The URL that the logs will be sent to
            token (str): The Authorization token being used
            silent (bool): If False the http response and logs will be sent 
                           to STDOUT for debug
        '''
        self.url = url
        self.token = token
        self.silent = silent

        # sets up a session with the server
        self.MAX_POOLSIZE = 100
        self.session = session = requests.Session()
        session.headers.update({
            'Content-Type': 'application/json',
            'Authorization': 'Bearer %s' % (self.token)
        })
        self.session.mount('https://', HTTPAdapter(
            max_retries=Retry(
                total=5,
                backoff_factor=0.5,
                status_forcelist=[403, 500]
            ),
            pool_connections=self.MAX_POOLSIZE,
            pool_maxsize=self.MAX_POOLSIZE
        ))

        super().__init__()

    def emit(self, record):
        '''
        This function gets called when a log event gets emitted. It recieves a
        record, formats it and sends it to the url
        Parameters:
            record: a log record
        '''
        logEntry = self.format(record)
        response = self.session.post(self.url, data=logEntry)

        if not self.silent:
            print(logEntry)
            print(response.content)

# create logger
log = logging.getLogger('')
log.setLevel(logging.INFO)

# create formatter - this formats the log messages accordingly
formatter = logging.Formatter(json.dumps({
    'time': '%(asctime)s',
    'pathname': '%(pathname)s',
    'line': '%(lineno)d',
    'logLevel': '%(levelname)s',
    'message': '%(message)s'
}))

# create a custom http logger handler
httpHandler = CustomHttpHandler(
    url='<YOUR_URL>',
    token='<YOUR_TOKEN>',
    silent=False
)

httpHandler.setLevel(logging.INFO)

# add formatter to custom http handler
httpHandler.setFormatter(formatter)

# add handler to logger
log.addHandler(httpHandler)

log.info('Hello world!')

Answer 2

You will need to subclass HTTPHandler and override the emit() method to do what you need. You can use the current implementation of HTTPHandler.emit() as a guide.