Is it a good idea to use serilog to write logs directly to the elasticsearch

Question

I'm evaluating different options about the distributed log server.

In the Java world, as I can see, the most popular solution is filebeat + kafka + logstash + elasticsearch + kibana.

However, in .NET world, there's a serilog which can send structure logs directly to the elasticsearch. So the only required components are elasticsearch + kibana.

I searched a lot, but there's not much information about this solution in production. I've no idea whether it's enough to handle large volumes of logs.

Can anyone give me some suggestions? Thanks.

Answer 1

I had the same issue exactly. Our system worked with the "classic" elk-stack architecture i.e. FileBeat -> LogStash -> Elastic ( ->Kibana). but as we found out in big projects with a lot of logs Serilog is much better solution for the following reasons:

1. CI\CD - when you have different types of logs with different structure which you want to have different types, Serilog power comes in handy. in LogStash you need to create a different filter to break down a message according to the pattern. which implies that there is big coupling in the log structure aspect and the LogStash aspect - very bug prone.
2. maintenance - Because of the easy CI\CD and the one point of change, it is easier to maintain a large amount of logs.
3. Scalability - FileBeat has a problem to handle big chunks of data because of the registry file which have a tend to "explode" - reference from personal experience stack overflow flow question ; elastic-forum question
4. Less failure points - with serilog the log send directly to elastic when with Filebeat you have to path through LogStash. one more place to fail.

Hope it helps you with your evaluation.