Replica had multiple failures in API call: IStatelessServiceInstance.Open()

Question

I'm stuck with the following exception from the Service Fabric Explorer:

Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
Replica had multiple failures in API call: IStatelessServiceInstance.Open(); Error = System.Reflection.TargetInvocationException (-2146232828)
Exception has been thrown by the target of an invocation.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.Internal.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at FRSecure.AssessmentDesigner.Web.WebHostBuilderHelper.GetWebHost(IWebHostBuilder webHostBuilder, String protocol, String port, String environment) in D:\a\1\s\web_assessmentdesigner\WebHostBuilderHelper.cs:line 16
   at web_assessmentdesigner.WebHostingService.Microsoft.ServiceFabric.Services.Communication.Runtime.ICommunicationListener.OpenAsync(CancellationToken cancellationToken) in D:\a\1\s\web_assessmentdesigner\WebHostingService.cs:line 75
   at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.d__0.MoveNext()

Here is what I know:

1. App installs and runs fine on a local cluster, but not on Azure.
2. The App consists of a stateless api and a stateful service. The stateful service starts up fine. The stateless goes into a failed state after about 2 minutes.

My question is this: How, or where can I obtain more information about the underlying exception? The service is failing before it has started so my logging is not yet available. I see nothing in Diagnostics on the portal. What is the best way to troubleshoot something like this when it cannot be replicated locally?

Answer 1

If you have enabled the Event provider Microsoft-ServiceFabric (Guid = "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8") to send events to Diagnostics then you should be able to see some Events written relating to this, the class StatelessServiceInstanceAdapter which is calling your WebHostBuilderHelper should emit an event that says something like Got exception when opening communication listeners - XXX when it runs into the exception above, this could potentially hold more information on why your service fails.

You could also simply wrap your WebHostBuilderHelper.GetWebHost(...) implementation in a try/catch and logging any exceptions caught to your ServiceEventSource. The exception happens when the service tries to open the listener, but it is unclear from the above why.