Python: How to create log file everyday using logging module?

Question

I'm new to logging module of python. I want to create a new log file everyday while my application is in running condition.

log file name - my_app_20170622.log
log file entries within time - 00:00:01 to 23:59:59

On next day I want to create a new log file with next day's date -

log file name - my_app_20170623.log
log file entries within time - 00:00:01 to 23:59:59

I'm using logging module of python.

I'm using like below -

log_level = int(log_level)
logger = logging.getLogger('simple')
logger.setLevel(log_level)
fh = logging.FileHandler(log_file_name)
fh.setLevel(log_level)
formatter = logging.Formatter(log_format)
fh.setFormatter(formatter)
logger.addHandler(fh)

Is their any configurations in logging module of python to create a log on daily basis?

Answer 1

You have to create a TimedRotatingFileHandler:

from logging.handlers import TimedRotatingFileHandler
logname = "my_app.log"
handler = TimedRotatingFileHandler(logname, when="midnight", interval=1)
handler.suffix = "%Y%m%d"
logger.addHandler(handler)

This piece of code will create a my_app.log but the log will be moved to a new log file named my_app.log.20170623 when the current day ends at midnight.

I hope this helps.

Answer 2

Finally, I got the correct answer and I want to share this answer.

Basically, need to create a TimedRotatingFileHandler like below -

log_format = "%(asctime)s - %(levelname)s - %(message)s"
log_level = 10
handler = TimedRotatingFileHandler("my_app.log", when="midnight", interval=1)
handler.setLevel(log_level)
formatter = logging.Formatter(log_format)
handler.setFormatter(formatter)

# add a suffix which you want
handler.suffix = "%Y%m%d"

#need to change the extMatch variable to match the suffix for it
handler.extMatch = re.compile(r"^\d{8}$") 

# finally add handler to logger    
logger.addHandler(handler)

This above code will generate file like my_app.log for current day and my_app.log.20170704 for previous day.

Hope it helps.

Answer 3

RotatingFileHandler

class RotatingFileHandler(  filename[, mode[, maxBytes[, backupCount]]])

Returns a new instance of the RotatingFileHandler class. The specified file is opened and used as the stream for logging. If mode is not specified, a is used. By default, the file grows indefinitely.

A RotatingFileHandler allows us to rotate our log statements into a new file every time the current log file reaches a certain size.

In this example we’ll set it up so that when it reaches 500 bytes we’ll rotate into a new file up to a maximum number of 2 backups.

import logging
import logging.handlers as handlers
import time

logger = logging.getLogger('my_app')
logger.setLevel(logging.INFO)

logHandler = handlers.RotatingFileHandler('app.log', maxBytes=500, backupCount=2)
logHandler.setLevel(logging.INFO)
logger.addHandler(logHandler)

def main():
    while True:
        time.sleep(1)
        logger.info("A Sample Log Statement")

main()

Upon execution of this you should notice that every time app.log exceeds 500 bytes, it is then closed and renamed app.log.x where the value of x increments till it reaches whatever we have set backupCount to.

TimedRotatingFileHandler

class TimedRotatingFileHandler( filename [,when [,interval [,backupCount]]])

Returns a new instance of the TimedRotatingFileHandler class. The specified file is opened and used as the stream for logging. On rotating it also sets the filename suffix. Rotating happens based on the product of when and interval.

You can use the when to specify the type of interval. The list of possible values is, note that they are not case sensitive:

|   Value  |    Type of interval   |
|:--------:|:---------------------:|
|     S    |        Seconds        |
|     M    |        Minutes        |
|     H    |         Hours         |
|     D    |          Days         |
|     W    |  Week day (0=Monday)  |
| midnight | Roll over at midnight |

TimedRotatingFileHandler allows us to capture log files by a time slice.

import logging
import logging.handlers as handlers
import time

logger = logging.getLogger('my_app')
logger.setLevel(logging.INFO)

logHandler = handlers.TimedRotatingFileHandler('timed_app.log', when='M', interval=1)
logHandler.setLevel(logging.INFO)
logger.addHandler(logHandler)

def main():
    while True:
        time.sleep(1)
        logger.info("A Sample Log Statement")

main()

Running this code will then create new log files every minute indefinitely. We can set the backupCount parameter on our logHandler instance and it will cap the number of log files we create.

Use of Appropriate Log Levels

With the TimedRotatingFileHandler and the RotatingFileHandler it is possible to do the following things such as log all error messages to a rotating file, but all normal log files to a TimedRotatingFileHandler as we hope that we can expect far more of them than error messages.

Two levels of records are split out two different log levels: INFO and ERROR to two distinct places.

import logging
import logging.handlers as handlers
import time

logger = logging.getLogger('my_app')
logger.setLevel(logging.INFO)

## Here we define our formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logHandler = handlers.TimedRotatingFileHandler('normal.log', when='M', interval=1, backupCount=0)
logHandler.setLevel(logging.INFO)
logHandler.setFormatter(formatter)

errorLogHandler = handlers.RotatingFileHandler('error.log', maxBytes=5000, backupCount=0)
errorLogHandler.setLevel(logging.ERROR)
errorLogHandler.setFormatter(formatter)

logger.addHandler(logHandler)
logger.addHandler(errorLogHandler)

def main():
    while True:
        time.sleep(1)
        logger.info("A Sample Log Statement")
        logger.error("An error log statement")

main()

You should notice that 3 log files are created when you run this. The error.log will contain only logs that are of level ERROR or higher. The normal.log will contain a combination of all log messages logged out of our application.

These are just a few things I feel are important when implementing your own logging system.