Logging/Security considerations and sensitive data

Question

I am working on general "architecture" related to logging and security of web applications. As far as I know, it is considered the best practice to log all request/response data (not only access logs, but the body of the requests/responses). It's good for security analysis, debugging purposes, audit and many more things.

There is an issue, that sensitive information is transferred in some requests, for example, passwords and/or credit card data.

(Please note: of course, I am using HTTPS, but passwords and/or credit card data will appear as plain text in logging or log files. And by the way, I do not store credit card data, because I am not PCI DSS compliant, we transfer this data to our partner, who is PCI DSS compliant).

Currently, I log and store offsite access logs (so logs without request/response bodies, but with GET parameters data) and I log request/response body data in application code (so I am able do decide what kind of data goes to log, and erase sensitive data before writing it to log).

However, I am thinking to implement the logging (of request/response bodies) outside the application, lets say on the server level via some module (for example, mod_dumpio or something similar), but in this case, logging the sensitive information might be a big issue.

How should I implement/configure it?

Answer 1

There are several best practices that you should consider when dealing with sensitive data.

First of all, minimize all of the sensitive data that you transfer at all. By hashing the password as early as possible in the architecture, you can ensure that any downstream components never even run the risk of exposing sensitive data.

Second, which is associated to the prior point, you should constrain/isolate the components that ever receive this sensitive information. For a password, only the authentication server would ever need this information - the rest of the components should really only deal with an authorization token (or something of the same ilk.) With regards to a credit card, this should likely only ever be associated with those services that would ever need it - for example the checkout/billing component of your system.

Third, for those systems that actually do need to get the actual value of the sensitive information for logic (e.g. credit card), you should consider creating/using a token-like system. This would basically be a place where you can store the sensitive information and provide an ID reference that replaces the original sensitive data element (e.g. exchange a credit card number for a credit card number id).

To close out the solution, exchange the sensitive data as close to the edge as possible (e.g. where you captured the credit card) in exchange for this token. Exchange this token for the sensitive information value (with the right authorization) to get the value back and then use it at the very latest possible moment. Finally, audit the logging that is done at those sensitive points to make sure that it's not logged in the clear.

This design pattern how some of the biggest e-commerce sites in the world do it. (Sorry, I am not allowed to disclose whom.) Hope it helps!

Answer 2

You shouldn't store sensitive information such as user passwords or credit card details in log files, doing this with credit card details will mean you fall foul of PCI compliance (as you've noted). For user passwords, there should be no need to log them - you should be checking the user's input password, hashed using whatever method you use, with the password you store in the database.

Without knowing what technology you're using, best practice would suggest filtering out these parameters. For example, the Ruby on Rails framework has (had?) a filter_parameter_logging method which you can pass inputs you want to filter to it - these inputs will then be masked in your log files. It might be worth implementing something similar in your application, so that any field called "password", "card_number" or whatever you define your blacklist to be isn't logged.