Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to have different log types using Serilog and ElasticSearch

I am currently trying to change our system configuration to work with Serilog (instead of working with FileBeat as a shipper to LogStash)

We are also working with the log type field (which is easy to configure in the FileBeat config file) in our various queries and for indexing out logs on Elastic.

The problem is that when using Serilog we get the default type logevent, and I didn't find where I can configure it. I want to have an option to determine a specific log type per Serilog instance. At the moment I have the default type for all of my logs.

my Serilop Config is:

        var path = GetLogPath();
        var logger = new LoggerConfiguration()
            .MinimumLevel.Information()
            .Enrich.WithMachineName()
            .Enrich.WithProperty("RequestId", Guid.NewGuid())
            .WriteTo.RollingFile(
                pathFormat: path,
                outputTemplate: "{Timestamp:yyyy-MM-dd HH:mm:ss} [{Level:u4}] [{RequestId}] {Message}{NewLine}{Exception}", buffered: false, shared: true);
        logger.WriteTo.Elasticsearch(
                new ElasticsearchSinkOptions(new Uri(this.configurationService.ElasticSearchUrl())));

How can I change the log type?

EDIT

After some investigation, I found out I suppose to change the typeName filed in the LoggerConfiguration and seem that I can do so only through the AppConfig file, which again event if I will change it, the change will affect all the logger instances.

Do I miss something?

like image 752
Green Avatar asked May 08 '18 13:05

Green


People also ask

Where does serilog store its logs?

For example serilog console sink sends the logs to good old terminal. ElasticSearch sink will store logs in an ElasticSearch index ( In lingo of ElasticSearch, database is called index). A MongoDB sink will write logs into MongoDB and so on. You can check the list of sinks here.

Is it possible to use serilog with Elasticsearch?

Elasticsearch has an adapter for Serilog and is easy to use from .NET. Even with Elasticsearch, the developer is still responsible of properly setting the correlation IDs and the cross-context properties.

What is the difference between serilog console sink and Elasticsearch sink?

For example serilog console sink sends the logs to good old terminal. ElasticSearch sink will store logs in an ElasticSearch index ( In lingo of ElasticSearch, database is called index). A MongoDB sink will write logs into MongoDB and so on.

How do I start with serilog?

Starting with Serilog is as easy as installing a Serilog Nuget package. You will also want to pick some logging sinks to direct where your log messages should go, including the console and a text file.


2 Answers

Just use another overload for the Elasticsearch sink:

var path = GetLogPath();
var logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .Enrich.WithMachineName()
    .Enrich.WithProperty("RequestId", Guid.NewGuid())
    .WriteTo.RollingFile(
        pathFormat: path,
        outputTemplate: "{Timestamp:yyyy-MM-dd HH:mm:ss} [{Level:u4}] [{RequestId}] {Message}{NewLine}{Exception}", buffered: false, shared: true);
logger.WriteTo.Elasticsearch(
        this.configurationService.ElasticSearchUrl(), typeName: "type");

so you don't have to specify the typeName in the appsettings and it also won't affect all instances.

like image 118
Manuel Allenspach Avatar answered Sep 21 '22 06:09

Manuel Allenspach


Updated Answer

To add Properties and values to your logger, you can use Contextual logging and Enrichment

Contextual Logger

The simplest and most direct way to attach contextual properties to a log event

First initialized your logger:

Log.Logger = new LoggerConfiguration().ReadFrom.AppSettings().CreateLogger();

Then you can create your contextual logger:

// adding Log Context
var StudentLogger = Log.Logger.ForContext<Student>();

StudentLogger.Error(/* log message */);

Or you can use correlation log entries:

// correlation Log Entries
var orderId = "some value";
var corrLog = Log.Logger.ForContext("orderId", orderId)

corrLog.Error(/* log message */);

Enrichment

In some cases we would like every event created by a logger to carry the same, fixed, property value. The Application example is one of these.

Serilog provides Enrich.WithProperty() at the level of a LoggerConfiguration for this:

Log.Logger = new LoggerConfiguration()
    .Enrich.WithProperty("Application", "e-Commerce")
    .Enrich.WithProperty("Environment", ConfigurationManager.AppSettings["Environment"])
    // Other logger configuration

Original Answer

There are two ways to configure Serilog:

Using API (need serilog.sinks.elasticsearch package):

var loggerConfig = new LoggerConfiguration()
    .MinimumLevel.Debug()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200") ){
         AutoRegisterTemplate = true,
 });
var logger = loggerConfig.CreateLogger();

Serilog Documentation

Using configuration from AppSettings (need Serilog.Settings.AppSettings in addition to serilog.sinks.elasticsearch)

This way, you put all of your setting in AppSetting file, e.g.

<appSettings>
    <add key="serilog:using" value="Serilog.Sinks.Elasticsearch"/>
    <add key="serilog:write-to:Elasticsearch.nodeUris" value="http://localhost:9200;http://remotehost:9200"/>
    <add key="serilog:write-to:Elasticsearch.indexFormat" value="custom-index-{0:yyyy.MM}"/>
    <add key="serilog:write-to:Elasticsearch.templateName" value="myCustomTemplate"/>
  </appSettings>

And tell serilog to read the config from appSettigns

Log.Logger = new LoggerConfiguration()
  .ReadFrom.AppSettings()
  ... // Other configuration here, then
  .CreateLogger()

See: AppSetting and ElasticSearch Configure Sink

I am not sure which log event type you are referring to? In my case, I pass the object type while logging the errors:

catch (Exception ex)
{
    Logger.Error(ex, string.Format("Exception occured in Controller: {0}, Action: Post.", this.GetType()), this.GetType());
like image 38
Hooman Bahreini Avatar answered Sep 22 '22 06:09

Hooman Bahreini