How to handle split-brain?

Question

I have read in Orleans FAQ when split-brain could happen but I don't understand what bad can happen and how to handle it properly.

FAQ says something vague like:

You just need to consider the rare possibility of having two instances of an actor while writing your application.

But how actually should I consider this and what can happen if I won't?

Orleans Paper (http://research.microsoft.com/pubs/210931/Orleans-MSR-TR-2014-41.pdf) says this:

application can rely on external persistent storage to provide stronger data consistency

But I don't understand what this means.

Suppose split brain happened. Now I have two instances of one grain. When I'll send a few messages they could be received by these two (or there can be even more?) different instances. Suppose each instance prior to receiving these messages had same state. Now, after processing these messages they have different states.

How they should persist their states? There could be a conflict.

When another instances will be destroyed and only one will remain what will happen to the states of destroyed instances? It'll be like messages processed by them has never been processed? Then client state and server state could be desyncronized IIUC.

I see this (split-brain) as a big problem and I don't understand why there is so little attention to it.

Answer 1

Orleans leverages the consistency guarantees of the storage provider. When you call this.WriteStateAsync() from a grain, the storage provider ensures that the grain has seen all previous writes. If it has not, an exception is thrown. You can catch that exception and call DeactivateOnIdle() and rethrow the exception or call ReadStateAsync() and retry. So if you have 2 grains during a split-brain scenario, which ever one calls WriteStateAsync() first prevents the other one from writing state without first having read the most up-to-date state.

Update: Starting in Orleans v1.5.0, a grain which allows an InconsistentStateException to be thrown back to the caller will automatically be deactivated when the currently executing calls complete. A grain can catch and handle the exception to avoid automatic deactivation.