How to do HTTP request logging as a means of troubleshooting login errors

Question

I develop and maintain a paywalled publication with 2000+ users. The most common support request relates to log in. Most times these can be solved with a couple of support emails. Every once in a while though, there's that odd user that just can't log in. As a last resort the support person resets the users password, verify that they can log in themselves and send the new credentials of to the user. Every now and then we get at user that still can not log in. At that point I'm out of troubleshooting tools.

So I'd like to have a tool that:

1. Logs all HTTP requests in full (except for users passwords).

2. Let's me search the log for a POST request to my login page containing the users name.

3. Let me look at all requests from the IP-address that I found in step 2 within a certain timeframe, and then analyse those requests very closely.

And I need to be able to do smart log rotation, like: "Hang on to everything you can fit into 30 GB, then start throwing out the old stuff".

Our publication happens to be built with Django and nginx, but I don't think that the tool I'm looking for will be specific to those tools. And I definitely don't want to throw all the request data in the same SQL database as my Django app.

So far I've found Logstash, but I haven't look at it closely enough to know if it's right for me. The important thing to me isn't to get nice graph of all usage, user trends, conversion funnels etc. What I need is better ways to troubleshoot a problem that's affecting a single user.

Answer 1

I think the best option is to use a suite of Logstash (event collecting) + Elasticsearch (event storage) + Kibana (analytics). All three are really good opensource projects with a lot of documentation and very active communities.

And if you need commercial support for any you can request help from: http://www.elasticsearch.org/

Logstash its flexible enough to allow you parse many log file formats out of the box. Moreover, storing all your logs on elastic search will allow you to create custom queries, reports and stuff.

You can check a kibana demo on: http://demo.kibana.org/

Links: http://www.elasticsearch.org/overview/kibana/ http://logstash.net/

Answer 2

As a temporary thing this probably does not require any sophisticated solution.

I successfully used this quick and dirty Django middleware for quite similar purpose - https://gist.github.com/Suor/7870909