How to handle HTTP requests in a Microservice / Event Driven Architecture?

Question

Background:

I am building an application and the proposed architecture is Event/Message Driven on a microservice architecture.

The monolithic way of doing thing is that I've a User/HTTP request and that actions some commands that have a direct synchronous response. Thus, to respond to the same User/HTTP request is 'hassle free'.

The problem:

The user sends an HTTP request to the UI Service (there are multiple UI Services) that fires some events to a queue (Kafka/RabbitMQ/any). a N of services picks up that Event/Message do some magic along the way and then at some point that same UI Service should pick that up a response and give that back to the user that originated HTTP request. Request processing is ASYNC but the User/HTTP REQUEST->RESPONSE is SYNC as per your typical HTTP interaction.

Question: How do I send a response to the same UI Service that originated the action (The service thats interacting with the user over HTTP) in this Agnostic/Event driven world?

My research so far I've been looking around and it seems that some people are solving that problem using WebSockets.

But the layer of complexity is that there needs to be some table that maps (RequestId->Websocket(Client-Server)) which is used to ‘discover’ which node in the gateway has the websocket connection for some particular response. But even if I understand the problem and complexity I'm stuck that I can't find any articles that would give me info on how to solve this problem at the implementation layer. AND this still is not a viable option because of 3rd party integrations such as payments providers(WorldPay) that expect REQUEST->RESPONSE - specially on the 3DS validation.

So I am somehow reluctant to think that WebSockets is an option. But even if WebSockets are ok for Webfacing apps, for API that connects to external systems is not a great architecture.

** ** ** Update: ** ** **

Even if long polling is an possible solution for a WebService API with a 202 Accepted a Location header and a retry-after header it wouldn't be performant for a high concurrency & high ability website. Imagine a huge number of people trying to get the transaction status update on EVERY request they make and you have to invalidate CDN cache (go and play with that problem now! ha).

But most important and relatable to my case I've 3rd party APIs such as payment systems where the 3DS systems have automatic redirects that are handled by the payment provider system and they expect a typical REQUEST/RESPONSE flow, thus this model would not work for me nor the sockets model would work.

Because of this use-case the HTTP REQUEST/RESPONSE should be handled in the typical fashion where i have a dumb client that expect that the complexity of the precessing is handled in back-end.

So i am looking for a solution where externally I have a typical Request->Response(SYNC) and the complexity of the status(ASYNCrony of the system) is handled internally

An example of the long polling, but this model wouldn't work for 3rd party API such as payments provider on 3DS Redirects that are not within my control.

 POST /user     Payload {userdata}     RETURNs:          HTTP/1.1 202 Accepted         Content-Type: application/json; charset=utf-8         Date: Mon, 27 Nov 2018 17:25:55 GMT         Location: https://mydomain/user/transaction/status/:transaction_id         Retry-After: 10  GET     https://mydomain/user/transaction/status/:transaction_id 

Answer 1

As I was expecting - people try to fit everything into a concept even if it does not fit there. This is not a criticism, this is an observation from my experience and after reading your question and other answers.

Yes, you are right that microservices architecture is based on asynchronous messaging patterns. However, when we talk about UI, there are 2 possible cases in my mind:

1. UI needs a response immediately (e.g. read operations or those commands on which user expects answer right away). These don't have to be asynchronous. Why would you add an overhead of messaging and asynchrony if the response is required on the screen right away? Does not make sense. Microservice architecture is supposed to solve problems rather than create new ones by adding an overhead.

2. UI can be restructured to tolerate delayed response (e.g. instead of waiting for the result, UI can just submit command, receive acknowledgement, and let the user do something else while response is being prepared). In this case, you can introduce asynchrony. The gateway service (with which UI interacts directly) can orchestrate the asynchronous processing (waits for complete events and so on), and when ready, it can communicate back to the UI. I have seen UI using SignalR in such cases, and the gateway service was an API which accepted socket connections. If the browser does not support sockets, it should fallback to the polling ideally. Anyway, important point is, this can only work with a contingency: UI can tolerate delayed answers.

If Microservices are indeed relevant in your situation (case 2), then structure UI flow accordingly, and there should not be a challenge in microservices on the back-end. In that case, your question comes down to applying event-driven architecture to the set of services (edge being the gateway microservice which connects the event-driven and UI interactions). This problem (event driven services) is solvable and you know that. You just need to decide if you can rethink how your UI works.