what is the proper way to do logging in csv file?

Question

i want to log some information of every single request send to a busy http server in a formatted form,use log module would create some thing i don't want to:

[I 131104 15:31:29 Sys:34]

i think of csv format but i don't know how to customize it,and python got csv module,but read the manual

import csv
with open('some.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    writer.writerows(someiterable)

since it would open and close a file each time, i am afraid in this way would slow down the whole server performance, what could i do?

Answer 1

Just use python's logging module.

You can adjust the output the way you want; take a look at Changing the format of displayed messages:

To change the format which is used to display messages, you need to specify the format you want to use:

import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should appear on the console')
logging.info('So should this')
logging.warning('And this, too')

and Formatters:

Formatter objects configure the final order, structure, and contents of the log message.

You'll find a list of the attribtus you can use here: LogRecord attributes.

---

If you want to produce a valid csv-file, use python's csv module, too.

Here's a simple example:

import logging
import csv
import io

class CsvFormatter(logging.Formatter):
    def __init__(self):
        super().__init__()
        self.output = io.StringIO()
        self.writer = csv.writer(self.output, quoting=csv.QUOTE_ALL)

    def format(self, record):
        self.writer.writerow([record.levelname, record.msg])
        data = self.output.getvalue()
        self.output.truncate(0)
        self.output.seek(0)
        return data.strip()

logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger(__name__)
logging.root.handlers[0].setFormatter(CsvFormatter())

logger.debug('This message should appear on the console')
logger.info('So should "this", and it\'s using quoting...')
logger.warning('And this, too')

Output:

"DEBUG","This message should appear on the console"
"INFO","So should ""this"", and it's using quoting..."
"WARNING","And this, too"

Answer 2

As sloth suggests, you can easily edit the delimiter of the log to a comma, thus producing a CSV file.

Working example:

import logging

# create logger
lgr = logging.getLogger('logger name')
lgr.setLevel(logging.DEBUG) # log all escalated at and above DEBUG
# add a file handler
fh = logging.FileHandler('path_of_your_log.csv')
fh.setLevel(logging.DEBUG) # ensure all messages are logged to file

# create a formatter and set the formatter for the handler.
frmt = logging.Formatter('%(asctime)s,%(name)s,%(levelname)s,%(message)s')
fh.setFormatter(frmt)

# add the Handler to the logger
lgr.addHandler(fh)

# You can now start issuing logging statements in your code
lgr.debug('a debug message')
lgr.info('an info message')
lgr.warn('A Checkout this warning.')
lgr.error('An error writen here.')
lgr.critical('Something very critical happened.')