How to query in a Event Driven Microservice architecture?

Question

Let suppose the following simple UC based on a CQRS architecture:

We have a backend managing a Business Object, let says a Movie.

• This backend is composed of 2 Microservices: a CommandManager (Create/Update/Delete Movie) and a QueryManager (Query Movie)
• We have a frontend that offer a web page for creating a new Movie and this action lead automatically to another web page describing the Movie.

A simple way to do that is:

• A web page collect movie information using a form and send them to the frontend.
• The frontend make a POST request to the CommandManager
• The CommandManager write the new movies to the datastore and return the movie key
• The frontend make a GET using this key to the QueryManager
• The QueryManager looks for the Movie in the Datastore using the key and return it.
• The frontend deliver the page with the Movie Information.

Ok, now I want to transform this UC in a more Event Driven way. Here is the new flow:

• A web page collect movie information using a form and send them to the frontend.
• The frontend write a Message in the BUS with the new movie information
• The CommandManager listen the BUS and create the new movies in the datastore. Eventually, it publish a new message in the BUS specifying that a new Movie has been created.

At this point, the frontend is no more waiting for a response due to the fact that this kind of flow is asynchronous. How could we complete this flow in order to forward the user to the Movie Information Web page? We should wait that the creation process is done before querying the QueryManager.

In a more general term, in a asynchronous architecture based on bus/event, how to execute Query used to provide information in a web page?

Answer 1

In addition to @VoiceOfUnreason's answer,

If the two microservices are RESTFul, the CommandManager could return a 202 Accepted with a link pointing to the resource that will be created in the future. The client could then poll that resource until the server responds with a 200 OK.

Another solution would be that the CommandManager would return a 202 Accepted with a link pointing to a command/status endpoint. The client would poll that endpoint until the status is command-processed (including the URL to the the actual resource) or command-failed (including a descriptive message for the failure).

These solutions could be augmented by sending the status of all processed commands using Server Sent Events. In this way, the client gets notified without polling.

If the client is not aware that the architecture is asynchronous, a solution is to use an API gateway that blocks the client's request until the upstream microservice processes the command and then to respond with the complete resource's data.

Answer 2

At this point, the frontend is no more waiting for a response due to the fact that this kind of flow is asynchronous. How could we complete this flow in order to forward the user to the Movie Information Web page? We should wait that the creation process is done before querying the QueryManager.

Short answer: make the protocol explicit.

Longer answer: a good place to look for inspiration here is HTTP.

The front end makes a POST to the origin server; as a result the origin server places a message on the queue and sends a response back.

The representation sent with this response ought to describe the request's current status and point to (or embed) a status monitor that can provide the user with an estimate of when the request will be fulfilled.

The client can then poll the endpoint to find out what progress has been made.

For instance, the endpoint might be a query into the data store, that looks for evidence that the command manager has processed the original command; or it might be an endpoint that is watching the bus for the MovieCreated message, and changes its answer based on whether or not it has seen that.

It may help clarify things to look into idempotent request handling; when the Command Manager pulls a message off of its queue, how does it know if it has previously processed a copy of that message? Your polling endpoint should be able to use the same information to let the consumer know that the message has been successfully processed.

Answer 3

In addition to @Constantin Galbenu's answer, I would like to put in my two cents.

I would strongly advise you to look at a microservices pattern called "BFF" (Backend-For-Frontend) pattern. Instead of having a thick API gateway doing all the work, you can have an API per use-case. For Example: In your case, you can an API called "CreateMovieBFFHandler" which would receive the POST request from front-end and then this guy would coordinate with other things in the system like message queues, events etc. to track the status of the submitted request. UI might have a protocol with this BFFhandler that if the response doesn't come back in X seconds, then the front-end would consider it as failure and if this handler is able to get a successfully processed messaged from message queue or "MovieCreated" event for this key, then it could send a 200 OK back and then you can redirect the page to call write side and then populate the UI.

Useful Link: https://samnewman.io/patterns/architectural/bff/