Should I use Prometheus as a log aggregator?

Question

Can/should prometheus be used as a log aggregator? We are deploying apps into a kubernetes cluster. All containers already log to stdout/err and we want all devs to instrument their code with logs to stdout/err. Fluentd will then collate all logs across the whole cluster and send to an aggregator. We have thought about using Elasticsearch/kibana however we will already have Prometheus for node metric gathering so if we can have fluentd send all logs to Prometheus it keeps everything in one place.

So, can/should Prometheus be used as a logging aggregator? Would it still have to poll the fluentd server? Really, it would be great to be able to use the alerting features of Prometheus so that if a certain log is made it (for instance) dumps the log message into a slack channel etc.

Appreciate some pointers on this one, thanks.

Answer 1

Prometheus is a metrics system rather than a logs system. There's the mtail and grok exporters to process logs, but really that's only for cases where instrumenting your code with metrics is not possible.

For logs something like Elasticsearch is far more appropriate.

Answer 2

Update: Loki is a new project that claims "like Prometheus, but for logs."