Partition is in quorum loss

Question

I have a Service Fabric application that has a stateless web api and a stateful service with two partitions. The stateless web api defines a web api controller and uses ServiceProxy.Create to get a remoting proxy for the stateful service. The remoting call puts a message into a reliable queue.

The stateful service will dequeue the messages from the queue every X minutes.

I am looking at the Service Fabric explorer and my application has been in an error state for the past few days. When I drill down into the details the stateful service has the following error:

Error event: SourceId='System.FM', Property='State'. Partition is in quorum loss.

Looking at the explorer I see that I have my primary replica up and running and it seems like a single ActiveSecondary, but the other two replicas show IdleSecondary and they keep going into a Standby / In Build state. I cannot figure out why this is happening.

What are some of the reasons my other secondaries keep failing to get to an ActiveSecondary state / causing this quorum loss?

Answer 1

Try to reset the cluster. I was facing the same issue having 1 partition for my service. The error was fixed with resetting the cluster