Sporadic WCF Service activation error with message - An error occurred while accessing the IIS Metabase

Question

I was haivng a project in which earlier I was hosting only single WCF service. Everything used to work fine. Later, as part of enhancements we have added two more WCF services to the same project with different Interfaces and different SVC files. All three services share same web.config which define the three end points (corresponding to each service).

WCF services for my project are hosted as separate website with its own App Pool and port number. All three of my services share the same App pool.

With this setup when I am deploying application to test servers many times I am getting sporadic error as below and service stops working. Of the three service, one or two at a time gives this error other keeps on working.

    System.ServiceModel.ServiceHostingEnvironment+HostingManager/4032828
    System.ServiceModel.ServiceActivationException: The service '...svc' cannot be activated due to an exception during compilation.  The exception message is: An error occurred while accessing the IIS Metabase.. ---> 

    System.Runtime.InteropServices.COMException: An error occurred while accessing the IIS Metabase.
   at System.ServiceModel.Activation.MetabaseReader..ctor

I enabled svclogs for the webservice and there I am seeing things like

......
AppDomain unloading
To: construct ServiceHost 'myservice1'
From: construct ServiceHost 'myservice1'
To: Open ServiceHost 'myservice1'
From: ServiceHost 'myservice1'
ASP.NET hosted service activated
**Wrote To Eventlog**     << Exception at this point for myservice2.

I have tried This options but it doesn't help. I have also searched on net but not finding any other solution that can help.

I am having IIS6 on the test servers.

Any help would be greatly appreciated.

UPDATE:

---

I have observed a pattern. After Idle time, whichever service is hit first gets activated properly, other ones fails. Also, to add to Port part, we are specifically mentioning port on which this service would be running. for my application say the port number is 25000, then no other application on this server is sharing this port number, only my application. So if there are multiple Services, then they are sharing the port, but again the same setup is there for other projects having multiple SVC services and none has ever experienced this issue (as far as I know).

UPDATE 2: Below is the config file. I have typed in the config file but have tried to keep this as accurate as possible. (please ignore case-sensitive things)

<system.serviceModel>
    <bindings>
        <basicHttpBinding>
            <binding name="MyBinding">
                <security mode="TransportCredentialOnly">
                    <transport clientCredentialType="Basic" proxyCredentialType="Basic" realm="prod.xxx.net" />
                    <message clientCredentialType="UserName" algorithmSuite="Default"/>
                </security>
            </binding>
        </basicHttpBinding>
    </bindings>

    <behaviours>
        <serviceBehaviours>
            <behaviour name="firstServiceBehaviour">
                <serviceMetadata httpGetEnabled="true" />
                <serviceDebug includeExceptionDetailInFaults="true" />
                <serviceCredentials>
                    <clientCertificate>
                        <authentication mapClientCertificateToWindowsAccount="true" />
                    </clientCertificate>
                </serviceCredentials>
                <dataContractSerializer maxItemsInObjectGraph="2147483646" />
            </behaviour>

<behaviours>
    <serviceBehaviours>
        <behaviour name="secondServiceBehaviour">
            <serviceMetadata httpGetEnabled="true" />
            <serviceDebug includeExceptionDetailInFaults="true" />
            <serviceCredentials>
                <clientCertificate>
                    <authentication mapClientCertificateToWindowsAccount="true" />
                </clientCertificate>
            </serviceCredentials>
        </behaviour>

<behaviours>
    <serviceBehaviours>
        <behaviour name="thirdServiceBehaviour">
            <serviceMetadata httpGetEnabled="true" />
            <serviceDebug includeExceptionDetailInFaults="true" />
            <serviceCredentials>
                <clientCertificate>
                    <authentication mapClientCertificateToWindowsAccount="true" />
                </clientCertificate>
            </serviceCredentials>
            <dataContractSerializer maxItemsInObjectGraph="2147483646" />
        </behaviour>

    <services>
        <service behaviourConfiguration="firstServiceBehaviour" name="...">
            <endpoint address="" binding="basicHttpBinding" bindingConfiguration="MyBinding" name="firstServiceEndPoint" contract="IfirstServiceContract" />
        </service>

        <service behaviourConfiguration="secondServiceBehaviour" name="...">
            <endpoint address="" binding="basicHttpBinding" bindingConfiguration="MyBinding" name="secondServiceEndPoint" contract="IsecondServiceContract" />
        </service>

        <service behaviourConfiguration="thirdServiceBehaviour" name="...">
            <endpoint address="" binding="basicHttpBinding" bindingConfiguration="MyBinding"
name="thirdServiceEndPoint" contract="IthirdServiceContract" />
        </service>
    </services>
    <serviceHostingEnvironment multipleSiteBindingsEnabled="true" />
</System.ServiceModel>

Answer 1

I won't say this as complete answer, but this somewhat helped solve our problem. I still want to know reason / solution to actual cause of the problem. I am mentioning this work around as this might help somebody Temporarily resolve their issues as well who might face similar problem.

As I mentioned the issue was coming only when application was idle for some time. In that case IIS was shutting down (unloading) the application's AppDomain (This is from SVC Logs).

So we created a simple console app that was hitting all the services of our application every 5-10 minutes and was not letting AppDomain to shut down. There is an alternative way for achiving this - Setting IIS configuration to not to unload AppDomain (this was less feisible for us provided the shared infrastructure). This helped us to complete testing.

Then as we moved to load-balanced environments (Testing environments close to production), we suddenly stopped getting the issue and with some analysis we found that the Load-Balancer itself was pinging these services so as to make sure that they are up, and because of this App Domain of these services was never unloading.

So, for now we can say that we are not getting this issue in a load balanced environment, but the question still remains why was it even happening (for non load Balanced environments).