Log Analysis in Python

Question

For our internal monitoring process, I want to find out how many exceptions have taken place on a particular day. We want to extract the information from the log file of our application (Pylons project).

I want to do this in Python itself. I am aware that I can write a script which will do the offline processing on the log for counting the number of exceptions (and possibly other information related to the exception as well).

I want to ask whether there is already some library which I can use to do log file analysis in Python or what is the best way to do this?

Answer 1

I just had a similar situation and found the logtools Python package for the job. I used it for analyzing a Tomcat6/Solr log file.

Copy log from server and install logtools in a virtualenv:

mkdir /tmp/logwtf
cd /tmp/logwtf
scp server:/var/log/tomcat6/catalina.2012-02-03.log ./catalina.log
virtualenv --system-site-packages --distribute .
. bin/activate
pip install -e 'git+https://github.com/adamhadani/logtools.git#egg=logtools'

Summarize search request traffic:

qps -r'^(.*?) org\.apache\.solr\.core\.SolrCore execute' \
    -F '%b %d, %Y %I:%M:%S %p' \
    -W900 \
    --ignore \
    <catalina.log

All server activity between 1:10 and 1:20 PM:

qps -r'^(.*? 1:1.:.. PM) ' \
    -F '%b %d, %Y %I:%M:%S %p' \
    -W15 \
    --ignore \
    <catalina.log

logtools includes additional scripts for filtering bots, tagging log lines by country, log parsing, merging, joining, sampling and filtering, aggregation and plotting, URL parsing, summary statistics and computing percentiles. See the the package's GitHub page for more information.