Which NoSQL database best for append only audit logging use case?

Question

My use case is audit logging for resources. For discussion consider a very simple schema: a resource name, access time stamp, and accessing user name. With all the NoSQL options out there, I'm wondering which solution is best for my use case?

The resource names are being held in a graph database (Neo4j) and while we could add vertices and edges to an audit vertex connected to the resource vertex, the audit info could be large and I fear pollute a relatively simple graph.

I'm currently leaning towards a document database such as MongoDB or Couchbase in which each resource has it's own document, and the audit log is a simple array within this document that gets appended to. I fear that I/O could become a problem as audit logs get long and the entire document must be exchanged between app server and database. One soultion I see to minimize this is to make each audit entry its own document and append its ID to the parent resource document array.

Search audit logs is not a requirement at this time, but with a document database I feel there is a nice path to integrate Elastic Search at a later time.

It seems Redis may be a bit more optimal to my use case, but the data persistence does not appear as rigorous as other solutions.

Conceptually, I guess I am looking for any NoSQL solutions that support an "append" API method call without needing to exchange much information. Ironically this is basically a SQL INSERT statement, but I fear a traditional RDBMS will not meet my scale requirements. The audit table would get huge fast and I would rather leverage the latest and greatest NoSQL approaches to partitioning / sharding.

Any insights into log append use cases are appreciated!

These seem to be similar SO questions:

Database design for audit logging

NoSQL or RDBMS for audit data

Thanks, Jared

Answer 1

Apache Kafka is log-oriented, highly scalable and allows for subscribing to messages in multiple different ways.