An event store could become a single point of failure?

Question

Since a couple of days I've been trying to figure it out how to inform to the rest of the microservices that a new entity was created in a microservice A that store that entity in a MongoDB.

I want to:

• Have low coupling between the microservices

• Avoid distributed transactions between microservices like Two Phase Commit (2PC)

At first a message broker like RabbitMQ seems to be a good tool for the job but then I see the problem of commit the new document in MongoDB and publish the message in the broker not being atomic.

Why event sourcing? by eventuate.io:

One way of solving this issue implies make the schema of the documents a bit dirtier by adding a mark that says if the document have been published in the broker and having a scheduled background process that search unpublished documents in MongoDB and publishes those to the broker using confirmations, when the confirmation arrives the document will be marked as published (using at-least-once and idempotency semantics). This solutions is proposed in this and this answers.

Reading an Introduction to Microservices by Chris Richardson I ended up in this great presentation of Developing functional domain models with event sourcing where one of the slides asked:

How to atomically update the database and publish events and publish events without 2PC? (dual write problem).

The answer is simple (on the next slide)

Update the database and publish events

This is a different approach to this one that is based on CQRS a la Greg Young.

The domain repository is responsible for publishing the events, this would normally be inside a single transaction together with storing the events in the event store.

I think that delegate the responsabilities of storing and publishing the events to the event store is a good thing because avoids the need of 2PC or a background process.

However, in a certain way it's true that:

If you rely on the event store to publish the events you'd have a tight coupling to the storage mechanism.

But we could say the same if we adopt a message broker for intecommunicate the microservices.

The thing that worries me more is that the Event Store seems to become a Single Point of Failure.

If we look this example from eventuate.io

we can see that if the event store is down, we can't create accounts or money transfers, losing one of the advantages of microservices. (although the system will continue responding querys).

So, it's correct to affirmate that the Event Store as used in the eventuate example is a Single Point of Failure?

Answer 1

We handle this with the Outbox approach in NServiceBus:

http://docs.particular.net/nservicebus/outbox/

This approach requires that the initial trigger for the whole operation came in as a message on the queue but works very well.

Answer 2

What you are facing is an instance of the Two General's Problem. Basically, you want to have two entities on a network agreeing on something but the network is not fail safe. Leslie Lamport proved that this is impossible.

So no matter how much you add new entities to your network, the message queue being one, you will never have 100% certainty that agreement will be reached. In fact, the opposite takes place: the more entities you add to your distributed system, the less you can be certain that an agreement will eventually be reached.

A practical answer to your case is that 2PC is not that bad if you consider adding even more complexity and single points of failures. If you absolutely do not want a single point of failure and wants to assume that the network is reliable (in other words, that the network itself cannot be a single point of failure), you can try a P2P algorithm such as DHT, but for two peers I bet it reduces to simple 2PC.