How to correlate log events in distributed Vertx system

Question

while doing logs in the multiple module of vertx, it is a basic requirement that we should be able to correlate all the logs for a single request.

as vertx being asynchronous what will be the best place to keep logid, conversationid, eventid.

any solution or patterns we can implement?

Answer 1

In a thread based system, you current context is held by the current thread, thus MDC or any ThreadLocal would do.

In an actor based system such as Vertx, your context is the message, thus you have to add a correlation ID to every message you send.

For any handler/callback you have to pass it as method argument or reference a final method variable.

For sending messages over the event bus, you could either wrap your payload in a JsonObject and add the correlation id to the wrapper object

vertx.eventBus().send("someAddr", 
  new JsonObject().put("correlationId", "someId")
                  .put("payload", yourPayload));

or you could add the correlation id as a header using the DeliveryOption

//send
vertx.eventBus().send("someAddr", "someMsg", 
            new DeliveryOptions().addHeader("correlationId", "someId"));

//receive    
vertx.eventBus().consumer("someAddr", msg -> {
        String correlationId = msg.headers().get("correlationId");
        ...
    });

There are also more sophisticated options possible, such as using an Interceptor on the eventbus, which Emanuel Idi used to implement Zipkin support for Vert.x, https://github.com/emmanuelidi/vertx-zipkin, but I'm not sure about the current status of this integration.

Answer 2

There's a surprising lack of good answers published about this, which is odd, given how easy it is.

Assuming you set the correlationId in your MDC context on receipt of a request or message, the simplest way I've found to propagate it is to use interceptors to pass the value between contexts:

vertx.eventBus()
        .addInboundInterceptor(deliveryContext -> {
            MultiMap headers = deliveryContext.message().headers();
            if (headers.contains("correlationId")) {
                MDC.put("correlationId", headers.get("correlationId"));
                deliveryContext.next();
            }
        })
        .addOutboundInterceptor(deliveryContext -> {
            deliveryContext.message().headers().add("correlationId", MDC.get("correlationId"));
            deliveryContext.next();
        });

Answer 3

If by multiple module you mean multiple verticles running on the same Vertx instance, you should be able to use a normal logging library such as SLF4J, Log4J, JUL, etc. You can then keep the logs in a directory of your choice, e.g. /var/logs/appName.

If, however, you mean how do you correlate logs between multiple instances of Vertx, then I'd suggest looking into GrayLog or similar applications for distributed/centralised logging. If you use a unique ID per request, you can pass that around and use it in the logs. Or depending on your authorization system, if you use unique tokens per request you can log those. The centralised logging system can be used to aggregate and filter logs based on that information.