Python logging module encrypt

Question

I've got a python script with logging. Now I want to encrypt log with AES using pycrypto.

import logging
import base64
from Crypto.Cipher import AES
aes = AES.new(cryptoKey)
logging.basicConfig(filename='example.log',level=logging.DEBUG) #  file name, not custom file
logging.info('text')

I want to use base64.b64encode(aes.encrypt('#logging text#')) before write it to log . What is a most estate way to do it?

Answer 1

There's a bit more to encryption than mere forwarding of data. I would suggest writing your own log formatter and setting it as a root formatter - that way no matter where you log from in your app, even parts not controlled by your code, it will always go through a layer of encryption. So, something like:

import base64
import logging
from Crypto.Cipher import AES
from Crypto.Hash import SHA256
from Crypto import Random

class EncryptedLogFormatter(logging.Formatter):

    # make sure that the `key` is a byte stream on Python 3.x
    def __init__(self, key, fmt=None, datefmt=None):
        self._key = SHA256.new(key).digest()  # use SHA-256 for a proper-sized AES key
        super(EncryptedLogFormatter, self).__init__(fmt=fmt, datefmt=datefmt)

    def format(self, record):
        message = record.msg  # log message to encrypt, if any
        if message:  # no sense to encrypt empty log messages
            # on Python 3.x encode first: message = message.encode("utf-8")
            iv = Random.new().read(AES.block_size)  # we'll be using CBC so generate an IV
            cipher = AES.new(self._key, AES.MODE_CBC, iv)
            # AES demands all blocks to be of `AES.block_size` so we have to pad the message
            # you can use any padding you prefer, I think PKCS#7 is the best option
            padding = AES.block_size - len(message) % AES.block_size
            # pad the message...
            message += chr(padding) * padding # Python 3.x: bytes([padding]) * padding
            message_enc = iv + cipher.encrypt(message)  # add iv and encrypt
            # finally, replace our plain-text message with base64 encoded encrypted one
            record.msg = base64.b64encode(message_enc).decode("latin-1")
        # you can do more here, even print out your own string but we'll just
        # pass it to the default formatter now that the message is encrypted
        # so that it can respect other formatting options.
        return super(EncryptedLogFormatter, self).format(record)

Then you can use it wherever you can change the logging formatter, i.e.:

import sys
import logging

# lets get the root logger
root = logging.getLogger()
root.handlers = []  # blank out the existing handlers

# create a new handler, file handler instead of stdout is perfectly fine
handler = logging.StreamHandler(stream=sys.stdout)
# now lets get to business
handler.setFormatter(EncryptedLogFormatter("Whatever key/pass you'd like to use",
                                           "[%(levelname)s] %(message)s"))
# lets add it to the root logger so it gets called by the rest of the app automatically
root.addHandler(handler)

# And lets see what happens:
logging.warn("Sensitive stuff, hide me!")
# [WARNING] NDKeIav5G5DtbaSPB4Y/DR3+GZ9IwmXKzVTua1tTuDZ7uMwxBAKTXgIi0lam2dOQ
# YMMV, the IV is random so every block will be different every time

You can of course encrypt levels, timestamps, pretty much anything from the logging.LogRecord, and you can output whatever format you prefer. When the time comes to read your logs, you just need to do the reverse - see an example in this answer.

UPDATE: As per request, here's how to do the 'reverse' (i.e. decrypt the encrypted logs). First, lets create a few log entries for testing (continuing with the previous):

root.setLevel(logging.DEBUG)  # let's make sure we support all levels

logging.warn("Lorem ipsum dolor sit amet.")
logging.info("Consectetur adipiscing elit.")
logging.debug("Sed do eiusmod tempor.")

Provided that the format remained the same ([%(levelname)s] %(message)s), this will result in a log like (of course, it will always be different due to the random IV):

[WARNING] LQMLkbx3YF7ra3e5ZLRj3p1mi2dwCOJe/GMfo2Xg8BBSZMDmZO75rrgoiy/6kqjf
[INFO] D+ehnsq1kWQi61AsLOBkqglXla7jgc2myPFaCGcfCRe6drk9ZmNl+M3UkKPWkDiU
[DEBUG] +rHEHkM2YHJCkIL+YwWI4FNqg6AOXfaBLRyhZpk8/fQxrXLWxcGoGxh9A2vO+7bq

To create a reader for such a log (file) we need to be aware of the format so we can differentiate encrypted from non-encrypted data. In this case, separating the parts is easy - each log entry is on a new line, the levels are not encrypted and the actual encrypted data is always separated by a whitespace from the actual log level. So, to put all that together we might construct something like:

import base64
from Crypto.Cipher import AES
from Crypto.Hash import SHA256

# make sure that the `key` is a byte stream on Python 3.x
def log_decryptor(key, stream):  # assume the stream can be iterated line-by-line
    key = SHA256.new(key).digest()  # same derivation as in the EncryptedLogFormatter
    for line in stream:
        if not line.strip():  # empty line...
            continue  # ignore it!
        level, stream = line.split(None, 1)  # split on log level and log data
        message_enc = base64.b64decode(stream.encode("latin-1"))  # decode the stream
        iv = message_enc[:AES.block_size]  # grab the IV from the beginning
        # decrypt the stream
        message = AES.new(key, AES.MODE_CBC, iv).decrypt(message_enc[AES.block_size:])
        padding = ord(message[-1])  # get the padding value; Python 3.x: message[-1]
        if message[-padding:] != chr(padding) * padding:  # verify the padding
            # on Python 3.x:     bytes([padding]) * padding
            raise ValueError("Invalid padding encountered.")
        # Python 3.x: decode the message: message[:-padding].decode("utf-8")
        yield "{} {}".format(level, message[:-padding])   # yield the decrypted value

And then you can use it as a regular generator to decrypt your logs, e.g.:

logs = ["[WARNING] LQMLkbx3YF7ra3e5ZLRj3p1mi2dwCOJe/GMfo2Xg8BBSZMDmZO75rrgoiy/6kqjf",
        "[INFO] D+ehnsq1kWQi61AsLOBkqglXla7jgc2myPFaCGcfCRe6drk9ZmNl+M3UkKPWkDiU",
        "[DEBUG] +rHEHkM2YHJCkIL+YwWI4FNqg6AOXfaBLRyhZpk8/fQxrXLWxcGoGxh9A2vO+7bq"]

for line in log_decryptor("Whatever key/pass you'd like to use", logs):
    print(line)

# [WARNING] Lorem ipsum dolor sit amet.
# [INFO] Consectetur adipiscing elit.
# [DEBUG] Sed do eiusmod tempor.

Or if you've set your log to stream to a file, you can directly decrypt such file as:

with open("path/to/encrypted.log", "r") as f:
    for line in log_decryptor("Whatever key/pass you'd like to use", f):
        print(line)  # or write to a 'decrypted.log' for a more persistent solution