Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Micro Services and noSQL - Best practice to enrich data in micro service architecture

I want to plan a solution that manages enriched data in my architecture.
To be more clear, I have dozens of micro services.
let's say - Country, Building, Floor, Worker.
All running over a separate NoSql data store.

When I get the data from the worker service I want to present also the floor name (the worker is working on), the building name and country name.

Solution1.
Client will query all microservices.
Problem - multiple requests and making the client be aware of the structure.
I know multiple requests shouldn't bother me but I believe that returning a json describing the entity in one single call is better.

Solution 2.
Create an orchestration that retrieves the data from multiple services.
Problem - if the data (entity names, for example) is not stored in the same document in the DB it is very hard to sort and filter by these fields.

Solution 3.
Before saving the entity, e.g. worker, call all the other services and fill the relative data (Building Name, Country name).
Problem - when the building name is changed, it doesn't reflect in the worker service.

solution 4.
(This is the best one I can come up with).
Create a process that subscribes to a broker and receives all entities change.
For each entity it updates all the relavent entities.
When an entity changes, let's say building name changes, it updates all the documents that hold the building name.
Problem: Each service has to know what can be updated. When a trailing update happens it shouldnt update the broker again (recursive update), so this can complicate to the microservices.

solution 5.
Keeping everything normalized. Fileter and sort in ElasticSearch. Problem: keeping normalized data in ES is too expensive performance-wise

like image 531
Bick Avatar asked May 17 '15 16:05

Bick


1 Answers

One thing I saw Netflix do (which i like) is create intermediary services for stuff like this. So maybe a new intermediary service that can call the other services to gather all the data then create the unified output with the Country, Building, Floor, Worker.

You can even go one step further and try to come up with a scheme for providing as input which resources you want to include in the output.

So I guess this closely matches your solution 2. I notice that you mention for solution 2 that there are concerns with sorting/filtering in the DB's. I think that if you are using NoSQL then it has to be for a reason, and more often then not the reason is for performance. I think if this was done wrong then yeah you will have problems but if all the appropriate fields that are searchable are properly keyed and indexed (as @Roman Susi mentioned in his bullet points 1 and 2) then I don't see this as being a problem. Yeah this service will only be as fast as the culmination of your other services and data stores, so they have to be fast.

Now you keep your individual microservices as they are, keep the client calling one service, and encapsulate the complexity of merging the data into this new service.

This is the video that I saw this in (https://www.youtube.com/watch?v=StCrm572aEs)... its a long video but very informative.

like image 117
Jose Martinez Avatar answered Oct 10 '22 15:10

Jose Martinez