What caused this InvalidOperationException using LINQ to SQL?

Question

We experienced a number of errors on our live application a week or two ago that have so far escaped explanation. We saw these errors internally and they were also experienced by clients as it manifested in a set of web services.

I have included the inner exception below, the project uses the CSLA framework and the error occured when retrieving an object from the database.

No known changes were made to the system at the time we started experiencing the errors, the infrastructure consists of a number of load balances web servers.

The errors seemed to be isolated to one of our servers, we experienced them using a console application connecting to the web services. The server in question was using a local DMZ IP to resolve the web services in its hosts file and by forcing this to go externally it seemed to resolve the issues.

It seems to be a very fine line between application and infrastructure to isolate this, so I'm wondering if anyone has any ideas or theories that could possibly explain this?

<InnerException>
      <ExceptionType>System.InvalidOperationException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
      <Message>Exception of type 'System.InvalidOperationException' was thrown.</Message>
      <Source>mscorlib</Source>
      <HelpLink />
      <Property name="Data">System.Collections.ListDictionaryInternal</Property>
      <Property name="TargetSite">Void VerifyIntegrity()</Property>
      <StackTrace>   at System.Runtime.CompilerServices.ConditionalWeakTable`2.VerifyIntegrity()
   at System.Runtime.CompilerServices.ConditionalWeakTable`2.Add(TKey key, TValue value)
   at System.Linq.Expressions.Expression..ctor(ExpressionType nodeType, Type type)
   at System.Data.Linq.SqlClient.Translator.TranslateLink(SqlLink link, List`1 keyExpressions, Boolean asExpression)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.ConvertToFetchedExpression(SqlNode node)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.ConvertLinks(SqlExpression node)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.FetchExpression(SqlExpression expr)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitMember(SqlMember m)
   at System.Data.Linq.SqlClient.SqlVisitor.Visit(SqlNode node)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitExpression(SqlExpression expr)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitNew(SqlNew sox)
   at System.Data.Linq.SqlClient.SqlVisitor.Visit(SqlNode node)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitExpression(SqlExpression expr)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitSelect(SqlSelect select)
   at System.Data.Linq.SqlClient.SqlVisitor.Visit(SqlNode node)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitAlias(SqlAlias a)
   at System.Data.Linq.SqlClient.SqlVisitor.Visit(SqlNode node)
   at System.Data.Linq.SqlClient.SqlVisitor.VisitSource(SqlSource source)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitSelect(SqlSelect select)
   at System.Data.Linq.SqlClient.SqlVisitor.Visit(SqlNode node)
   at System.Data.Linq.SqlClient.SqlBinder.Visitor.VisitIncludeScope(SqlIncludeScope scope)
   at System.Data.Linq.SqlClient.SqlVisitor.Visit(SqlNode node)
   at System.Data.Linq.SqlClient.SqlBinder.Bind(SqlNode node)
   at System.Data.Linq.SqlClient.SqlProvider.BuildQuery(ResultShape resultShape, Type resultType, SqlNode node, ReadOnlyCollection`1 parentParameters, SqlNodeAnnotations annotations)
   at System.Data.Linq.SqlClient.SqlProvider.BuildQuery(Expression query, SqlNodeAnnotations annotations)
   at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
   at System.Data.Linq.DataQuery`1.System.Linq.IQueryProvider.Execute[S](Expression expression)
   at System.Linq.Queryable.SingleOrDefault[TSource](IQueryable`1 source)
   at NamespaceA.DaSql.NamespaceB.NamespaceBContext.NamespaceA.Da.NamespaceB.INamespaceBContext.GetClassA(Int32 objectId)
   at NamespaceA.NamespaceB.ClassA.DataPortal_Fetch(SingleCriteria`2 criteria)
   at dm(Object , Object[] )
   at Csla.Reflection.MethodCaller.CallMethod(Object obj, DynamicMethodHandle methodHandle, Object[] parameters)</StackTrace>
    </InnerException>

Thank you in advance for any help or theories.

Edit :

Full exception is here

LINQ to SQL below, nothing to it and ObjectA is simply a wrapper class with a properties. Nothing more than a simple select and populate of one object bases on an ID. ctx is a CSLA ContextManager.

var data = from d in ctx.DataContext.ObjectAs
                           where d.ObjectId == objectId
                           select new ObjectA
                            {
                                Id = d.DispatchId,
                                ClientId = d.ClientId,
                                DateCreated = d.DateCreated
                            };

                return data.SingleOrDefault();

Answer 1

I see the inner exception you have attached and as per exception, may be there is a problem with your collection which is created when the SingleOrDefault() method gets executed (LINQ is deffered execution).

I am posting the 2 links which might help you to analyze the problem and solution better if you dig a little deep into the collection creation. Hope these links help you solve your problem:

Link1: http://typedescriptor.net/browse/members/289638-System.Runtime.CompilerServices.ConditionalWeakTable%602%5BTKey,TValue%5D.VerifyIntegrity()

Link 2: http://code.google.com/p/bclcontrib-abstract/source/browse/%2BFromCoreEx/%2BKludge/Runtime/CompilerServices/ConditionalWeakTable.cs?spec=svnd7b68d68e34be8e5db308feaccf935afd2c1b8d9&name=d7b68d68e3&r=d7b68d68e34be8e5db308feaccf935afd2c1b8d9

The culprit methods might be ConditionalWeakTable.Add() and ConditionalWeakTable.VerifyIntegrity()

Above pointers should help you reach to a solution. Thanks