Send TraceId across Threads

Question

We have a distributed application following microservice Architecture. In one of our microservice we are following producer-consumer pattern.

The producer receives requests, persists it to database, pushes the request into a BlockingQueue and sends the response back to the client. The consumer running on a separate thread is listening to the blocking queue. The moment it gets the request object it performs specific operations on it.

The request received by the producer is persisted to the database asynchronously using CompleteableFutures.

The problem here is how to forward TraceId to the methods processing the requestObject inside consumer thread. Since the consumer thread might process these objects much later after the response is sent to the consumer.

Also how to forward the traceId across Asynchronous calls?

Thanks

Answer 1

That's an interesting question. I think that what you can do is to persist the request together with its headers. Then on the consumer side you can use the SpanExtractor interface in a similar way as we do here - https://github.com/spring-cloud/spring-cloud-sleuth/blob/v1.3.0.RELEASE/spring-cloud-sleuth-core/src/main/java/org/springframework/cloud/sleuth/instrument/web/TraceFilter.java#L351 (Span parent = spanExtractor().joinTrace(new HttpServletRequestTextMap(request));). That means that from the HttpServletRequest we're extracting values to build a span. Then, once you've retrieved the Span, you can just use Tracer#continueSpan(Span) method before processing, and then Tracer#detach(Span) in the finally block. E.g.

Span parent = spanExtractor().joinTrace(new HttpServletRequestTextMap(request));
try {
   tracer.continueSpan(parent);
   // do whatever you need
} catch(Exception e) {
  tracer.addTag("error", doSthWithTheExceptionMsg(e));
} finally {
  tracer.detach(parent); 
}