Removing service fabric application fails

Question

I have deployed an application to a 5 node standalone cluster. Deployment succeeded successful. But the application did not start because of some bug in the application. I tried removing the application from the cluster using the Service Fabric Explorer but this fails.

The health State of the application is “Error” and the status is “Deleting” The application has 9 services. 6 services show a Health State “Unknown” with a question mark and a Status “Unknown”. 3 services show a health state “Ok” but with a Status “Deleting”.

I have also tried to remove it using powershell:

Remove-ServiceFabricApplication -ApplicationName fabric:/appname -Force -ForceRemove

The result was an Operation timed out.

I also tried the script below that I found in some other post.

Connect-ServiceFabricCluster -ConnectionEndpoint localhost:19000

$nodes = Get-ServiceFabricNode

foreach($node in $nodes)
{
    $replicas = Get-ServiceFabricDeployedReplica -NodeName $node.NodeName -  ApplicationName "fabric:/MyApp"

    foreach ($replica in $replicas)
    {
        Remove-ServiceFabricReplica -ForceRemove -NodeName $node.NodeName -PartitionId $replica.Partitionid -ReplicaOrInstanceId $replica.ReplicaOrInstanceId
    }
}

Also no result, the script did not find any replica to remove.

At the same time we started removing the application one of the system services also changed state. The fabric:/System/NamingService service shows a “Warning” health state. This is on partition 00000000-0000-0000-0000-000000001002. The primary replica shows:
Unhealthy event: SourceId='System.NamingService', Property='Duration_PrimaryRecovery', HealthState='Warning', ConsiderWarningAsError=false. The PrimaryRecovery started at 2016-10-06 07:55:21.252 is taking longer than 30:00.000.

I also restarted every node (1 at the time) with no result.

How to force to remove the application without recreating the cluster because that is not a option for a production environment.

Answer 1

Yeah this can happen if you don't allow your code to exit RunAsync or Open/Close of your ICommunicationListener.

Some background:

Your service has a lifecycle that is driven by Service Fabric. A small component in your service - you know it as FabricRuntime - drives this. For stateless service instances, it's a simple open/close lifecycle. For stateful services, it's a bit more complex. A stateful service replica opens and closes, but also changes role, between primary, secondary, and none. Lifecycle changes are initiated by Service Fabric and show up as a method call or cancellation token trigger in your code. For example, when a replica is switch to primary, we call your RunAsync method. When it switches from primary to something else, or needs to shut down, the cancellation token is triggered. Either way, the system waits for you to finish your work.

When you go delete a service, we tell your service to change role and close. If your code doesn't respond, then it will get stuck in that state.

To get out of that state, you can run Remove-ServiceFabricReplica -ForceRemove. This essentially drops the replica from the system - as far Service Fabric is concerned, the replica is gone. But your process is still running. So you have to go in and kill the process too.