WCF Service returning "requested service '...' could not be activated" the first time it's accessed from an MVC site

Question

We have a WCF service (with no security) that is being accessed by an MVC3 website.

On the developer machines we have no problems with it but when our TeamCity setup builds the central version and deploys it under IIS7 the first time we load the MVC3 site and it accesses the WCF service - the service call returns the message:

System.ServiceModel.ServiceActivationException: The requested service, 'http://localhost:83/ABCStaticData/StaticDataService.svc' could not be activated. See the server's diagnostic trace logs for more information.**

If we immediately hit the same site again, everything works fine.

The server stack track when it throws the error is:

Server stack trace:

at System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(HttpWebRequest request, HttpWebResponse response, HttpChannelFactory1 factory, WebException responseException, ChannelBinding channelBinding)
at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) 
at ABC.StaticData.Editor.StaticDataServiceProxy.IStaticDataService.GetAllMarkets()
at ABC.StaticData.Editor.StaticDataServiceProxy.StaticDataServiceClient.GetAllMarkets() in c:\BuildAgent\work\55a3bcc468adcccb\StaticData\ABC.StaticData.Editor\Service References\StaticDataServiceProxy\Reference.cs:line 3195
at ABC.StaticData.Editor.Controllers.MarketsController.<>c__DisplayClass22.<.ctor>b__13() in c:\BuildAgent\work\55a3bcc468adcccb\StaticData\ABC.StaticData.Editor\Controllers\MarketsController.cs:line 263
at ABC.StaticData.Editor.Controllers.MarketsController.Index() in c:\BuildAgent\work\55a3bcc468adcccb\StaticData\ABC.StaticData.Editor\Controllers\MarketsController.cs:line 306

Does anyone have any ideas on what could be causing the problem or where to start looking?

UPDATE: I forgot to mention the first that if we then don't access the site for a while and then go back, it gives the same error - then works again the second time.

UPDATE2: This is the event log entry text:

WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/3043589
 Exception: System.ServiceModel.ServiceActivationException: The service '/ABCStaticData/StaticDataService.svc' cannot be activated due to an exception during compilation.  The exception message is: Could not load type 'System.Runtime.CompilerServices.AsyncStateMachineAttribute' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.. ---> System.TypeLoadException: Could not load type 'System.Runtime.CompilerServices.AsyncStateMachineAttribute' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
   at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.IsCustomAttributeDefined(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable)
   at System.Reflection.CustomAttribute.IsDefined(RuntimeMethodInfo method, RuntimeType caType, Boolean inherit)
   at System.Reflection.RuntimeMethodInfo.IsDefined(Type attributeType, Boolean inherit)
   at Ninject.Infrastructure.Language.ExtensionsForMemberInfo.HasAttribute(MemberInfo member, Type type)
   at Ninject.Selection.Heuristics.StandardInjectionHeuristic.ShouldInject(MemberInfo member)
   at Ninject.Selection.Selector.<>c__DisplayClassa.<SelectMethodsForInjection>b__9(IInjectionHeuristic h)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at Ninject.Selection.Selector.<SelectMethodsForInjection>b__8(MethodInfo m)
   at System.Linq.Enumerable.WhereArrayIterator`1.MoveNext()
   at Ninject.Planning.Strategies.MethodReflectionStrategy.Execute(IPlan plan)
   at Ninject.Planning.Planner.<>c__DisplayClass1.<CreateNewPlan>b__0(IPlanningStrategy s)
   at Ninject.Infrastructure.Language.ExtensionsForIEnumerableOfT.Map[T](IEnumerable`1 series, Action`1 action)
   at Ninject.Planning.Planner.CreateNewPlan(Type type)
   at Ninject.Planning.Planner.GetPlan(Type type)
   at Ninject.Activation.Providers.StandardProvider.Create(IContext context)
   at Ninject.Activation.Context.Resolve()
   at Ninject.KernelBase.<>c__DisplayClass10.<Resolve>b__c(IBinding binding)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)
   at Ninject.Planning.Targets.Target`1.GetValue(Type service, IContext parent)
   at Ninject.Planning.Targets.Target`1.ResolveWithin(IContext parent)
   at Ninject.Activation.Providers.StandardProvider.GetValue(IContext context, ITarget target)
   at Ninject.Activation.Providers.StandardProvider.<>c__DisplayClass4.<Create>b__2(ITarget target)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Ninject.Activation.Providers.StandardProvider.Create(IContext context)
   at Ninject.Activation.Context.Resolve()
   at Ninject.KernelBase.<>c__DisplayClass10.<Resolve>b__c(IBinding binding)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)
   at Ninject.Planning.Targets.Target`1.GetValue(Type service, IContext parent)
   at Ninject.Planning.Targets.Target`1.ResolveWithin(IContext parent)
   at Ninject.Activation.Providers.StandardProvider.GetValue(IContext context, ITarget target)
   at Ninject.Activation.Providers.StandardProvider.<>c__DisplayClass4.<Create>b__2(ITarget target)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Ninject.Activation.Providers.StandardProvider.Create(IContext context)
   at Ninject.Activation.Context.Resolve()
   at Ninject.KernelBase.<>c__DisplayClass10.<Resolve>b__c(IBinding binding)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at Ninject.Extensions.Wcf.BaseNinjectServiceHostFactory.CreateServiceHost(Type serviceType, Uri[] baseAddresses)
   at System.ServiceModel.Activation.ServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(ServiceActivationInfo serviceActivationInfo, EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   --- End of inner exception stack trace ---
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath, EventTraceActivity eventTraceActivity)
 Process Name: w3wp
 Process ID: 3248

Just another point that we're developing using .NET 4.5 and Visual Studio 2012 RC.

Answer 1

I've just manage to solve similar problem of random 'System.ServiceModel.ServiceActivationException' (thrown exception, then run on second hit.). The reason for the exception in my case was lack of memory on the server. To confirm the memory problem, you can do following:

Application calling web service:

System.ServiceModel.ServiceActivationException: The requested service, 'XXX' could not be activated. See the server's diagnostic trace logs for more information.

Server stack trace: 
   at System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(HttpWebRequest request, HttpWebResponse response, HttpChannelFactory`1 factory, WebException responseException, ChannelBinding channelBinding)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

WCF trace log: nothing logged.

On server hosting wcf service (the same as the calling application btw) in event logs except lot of crap info I've found:

Custom event details:

WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/58366981
 Exception: System.ServiceModel.ServiceActivationException: The service 'xxx' cannot be activated due to an exception during compilation.  The exception message is: Memory gates checking failed because the free memory (xxx bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.. ---> System.InsufficientMemoryException: Memory gates checking failed because the free memory (xxx bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.

So solution for me was: 1. add more ram 2. release some of ram (temporary solution) 3. update web config of the web service:

<configuration>
  <system.serviceModel>
    <serviceHostingEnvironment ... minFreeMemoryPercentageToActivateService="0" />
  </system.serviceModel>
</configuration>

Answer 2

I had this problem too and just simply put

<serviceHostingEnvironment  minFreeMemoryPercentageToActivateService="1">

under

<system.serviceModel>

And everything works fine.

Answer 3

I've hit the same "Could not load type 'System.Runtime.CompilerServices.AsyncStateMachineAttribute'", which I found out was because I had built my application using VS 2012 RTM, but then tried to deploy it on a machine that only had 4.5 Beta. The AsyncStateMachineAttribute is a new type added after Beta.

Answer 4

I had that problem too, what I did is:

• Go to the directory in the file system where the service resides
• you will have 2 files for the service
  • yourService.svc
  • yourService.svc.cs

• Open the file yourService.svc and make sure that your service is configured, and the code behind looks like this:

  <%@ ServiceHost Language="C#" Debug="true" Service="YourNamespace.yourService.svc" CodeBehind="yourService.svc.cs" %>
  

I fixed my issue that way. I hope it helps!