Data Consistency Across Microservices

Question

While each microservice generally will have its own data - certain entities are required to be consistent across multiple services.

For such data consistency requirement in a highly distributed landscape such as microservices architecture, what are the choices for design? Of course, I do not want shared database architecture, where a single DB manages the state across all the services. That violates isolation and shared-nothing principles.

I do understand that, a microservice can publish an event when an entity is created, updated or deleted. All other microservices which are interested in this event can accordingly update the linked entities in their respective databases.

This is workable, however it leads to a lot of careful and coordinated programming effort across the services.

Can Akka or any other framework solve this use case? How?

EDIT1:
Adding the below diagram for clarity.
Basically, I am trying to understand, if there are available frameworks today that can solve this data consistency problem.

For the queue I can use any AMQP software such as RabbitMQ or Qpid etc. For the data consistency framework, I am not sure if presently Akka or any other software can help. Or is this scenario so uncommon, and such an anti-pattern that no framework should be ever needed?

Answer 1

The Microservices architectural style tries to allow organizations to have small teams own services independent in development and at runtime. See this read. And the hardest part is to define the service boundaries in a useful way. When you discover that the way you split up your application results in requirements impacting multiple services frequently that would tell you to rethink the service boundaries. The same is true for when you feel a strong need to share entities between the services.

So the general advice would be to try very hard to avoid such scenarios. However there may be cases where you cannot avoid this. Since a good architecture is often about making the right trade-offs, here some ideas.

1. Consider expressing the dependency using service interfaces (API) instead of a direct DB dependency. That would allow each service team to change their internal data schema as much as required and only worry about the interface design when it comes to dependencies. This is helpful because it is easier to add additional APIs and slowly deprecate older APIs instead of changing a DB design along with all dependent Microservices (potentially at the same time). In other words you are still able to deploy new Microservice versions independently, as long as the old APIs are still supported. This is the approach recommended by the Amazon CTO, who was pioneering a lot of the Microservices approach. Here is a recommended read of an interview in 2006 with him.

2. Whenever you really really cannot avoid using the same DBs and you are splitting your service boundaries in a way that multiple teams/services require the same entities, you introduce two dependencies between the Microservice team and the team that is responsible for the data scheme: a) Data Format, b) Actual Data. This is not impossible to solve, but only with some overhead in organization. And if you introduce too many of such dependencies your organization will likely be crippled and slowed down in development.

a) Dependency on the data scheme. The entities data format cannot be modified without requiring changes in the Microservices. To decouple this you will have to version the entities data scheme strictly and in the database support all versions of the data that the Microservices are currently using. This would allow the Microservices teams to decide for themselves when to update their service to support the new version of the data scheme. This is not feasible with all use cases, but it works with many.

b) Dependency on the actual collected data. The data that has been collected and is of a known version for a Microservice is OK to use, but the issue occurs when you have some services producing a newer version of the data and another service depends on it - But was not yet upgraded to being able to read the latest version. This problem is hard to solve and in many cases suggests you did not chose the service boundaries correctly. Typically you have no choice but to roll out all services that depend on the data at the same time as upgrading the data in the database. A more wacky approach is to write different versions of the data concurrently (which works mostly when the data is not mutable).

To solve both a) and b) in some other cases the dependency can be reduced by hidden data duplication and eventual consistency. Meaning each service stores its own version of the data and only modifies it whenever the requirements for that service change. The services can do so by listening to a public data flow. In such scenarios you would be using an event based architecture where you define a set of public events that can be queued up and consumed by listeners from the different services that will process the event and store whatever data out of it that is relevant for it (potentially creating data duplication). Now some other events may indicate that internally stored data has to be updated and it is each services responsibility to do so with its own copy of the data. A technology to maintain such a public event queue is Kafka.

Answer 2

Theoretical Limitations

One important caveat to remember is the CAP theorem:

In the presence of a partition, one is then left with two options: consistency or availability. When choosing consistency over availability, the system will return an error or a time-out if particular information cannot be guaranteed to be up to date due to network partitioning.

So by "requiring" that certain entities are consistent across multiple services you increase the probability that you will have to deal with timeout issues.

Akka Distributed Data

Akka has a distributed data module to share information within a cluster:

All data entries are spread to all nodes, or nodes with a certain role, in the cluster via direct replication and gossip based dissemination. You have fine grained control of the consistency level for reads and writes.

Answer 3

Same problem here. We have data in different microservices and there are cases where one service needs to know if there is a specific entity in another microservice. We do not want the services to call each others to complete a request because this adds response time and multipies downtimes. Also it adds a nightmare of coupling depth. The client should not decide about business logic and data validation/consistency either. We also do not want central services like "Saga Controllers" to provide for consistency between services.

So we use a Kafka message bus to inform observing services of state changes in "upstream" services. We try very hard not to miss or ignore any messages even in error conditions and we use Martin Fowler's "tolerant reader" pattern to couple as loosely as possible. Still sometimes services are changed and after the change they might need information from other services that they might have emitted on the bus before but they are now gone (even Kafka cannot store forever).

We decided for now that each Service be split in a pure and decoupled web service (RESTful) that does the actual work and a separate Connector-Service which listens to the Bus and may also call other services. This Connector runs in the background. It is only triggered by bus messages. It then will try to add data to the main service via REST calls. If the service responds with a consistency error, the connector will try to repair this by fetching the needed data from the upstream service and inject it as needed. (We cannot afford batch-jobs to "synchronize" data en block, so we just fetch what we need). If there are better ideas, we are always open, but "pull" or "just change data model" is not what we consider feasible...