EsentVersionStoreOutOfMemoryAndCleanupTimedOutException when building on new TFS 2013 build machine

Question

I'm getting this exception consistently when I build my project. I've just installed the OS, build agent, dependencies, and SQL Server 2008 R2 on the machine over the last couple of days. I copied a backup (.bak) of all the databases from an older build machine which is working fine and restored them all to the new hardware. All of my other projects that I've ported over work fine but this is the last one and I can't figure out what to do about this error.

From my MSBuild log file I can see that the thing compiles the FooBar.sqlproj file and moves on to creating the .dacpac where it blows up. There are a number of subsequent .sqlproj projects that it doesn't get to because of this exception.

What can I do to resolve the error and build the project?

PrepareForBuild:
     Creating directory ".\sql\".
   GenerateSqlTargetFrameworkMoniker:
   Skipping target "GenerateSqlTargetFrameworkMoniker" because all output files are up-to-date with respect to the input files.
   CoreCompile:
     C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Csc.exe /noconfig /nowarn:1701,1702 /nostdlib+ /highentropyva- /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\mscorlib.dll" /out:obj\Default\FooBar.dll /target:library /utf8output "C:\Users\User_Foo\AppData\Local\Temp\.NETFramework,Version=v4.0.SqlClrAttributes.cs"
   SqlBuild:
     Creating a model to represent the project...
     Loading project references...
     Loading project files...
     Building the project model and resolving object interdependencies...
     Validating the project model...
     Writing model to D:\Builds\16\Foo\Sources\Database\FooBar\obj\Default\Model.xml...
   CopyFilesToOutputDirectory:
     Copying file from "obj\Default\FooBar.dll" to ".\sql\FooBar.dll".
     FooBar -> D:\Builds\16\Foo\Sources\Database\FooBar\sql\FooBar.dll
   SqlPrepareForRun:
     FooBar -> D:\Builds\16\Foo\Sources\Database\FooBar\sql\FooBar.dacpac
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\SSDT\Microsoft.Data.Tools.Schema.SqlTasks.targets (550): The "SqlBuildTask" task failed unexpectedly.
Microsoft.Isam.Esent.Interop.EsentVersionStoreOutOfMemoryAndCleanupTimedOutException: Version store out of memory (and cleanup attempt failed to complete)
   at Microsoft.Isam.Esent.Interop.Update.Save(Byte[] bookmark, Int32 bookmarkSize, Int32& actualBookmarkSize)
   at Microsoft.Data.Tools.Schema.SchemaModel.ModelStore.EseCommand.InsertRow(Object[] values)
   at Microsoft.Data.Tools.Schema.SchemaModel.ModelStore.RelationshipEntry.AddAnnotation(IModelAnnotation annotation)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlModelBuilderSchemaAnalyzer.Annotate(IModelAnnotationHolder annotationHolder, ModelRelationshipClass relationshipClass, SqlElementDescriptor rightHandSide, TSqlFragment fragment)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlModelBuilderSchemaAnalyzer.CreateRelationshipEntry(IModelRelationship relationship, ModelRelationshipClass relationshipClass, SqlElementDescriptor rightHandSide, IModelElement lhs, TSqlFragment fragment)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlModelBuilderSchemaAnalyzer.IdentifiedRelationshipRegularAction(TSqlFragment fragment, SqlElementDescriptor leftHandSide, ModelRelationshipClass relationshipClass, SqlElementDescriptor rightHandSide, IEnumerable`1 annotationDescriptors)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SqlInterpretationVisitor.IdentifyRelationshipToResolvedDescriptor(SqlSchemaAnalyzer analyzer, SqlElementDescriptor leftHandSide, ModelRelationshipClass relationship, ResolvedDescriptor descriptor)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SqlInterpretationVisitor.VisitResolvedDescriptor(SqlSchemaAnalyzer analyzer, ResolvedDescriptor descriptor, Boolean createRelationship)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SetClauseInterpretationVisitor90.ExplicitVisit(AssignmentSetClause node)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlInterpretationUtils.InterpretSetClause(SetClause node, SqlSchemaAnalyzer analyzer, SqlInterpretationContext context, SqlColumnResolver columnResolver, SqlColumnResolver dmlTargetColumnResolver, List`1 errors)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.UpdateSpecificationInterpretationVisitor90.InterpretUpdateSpecification(UpdateSpecification node)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlInterpretationUtils.InterpretDmlSpecification(DataModificationSpecification node, SqlSchemaAnalyzer analyzer, SqlInterpretationContext context, List`1 errors)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.UpdateStatementInterpretationVisitor.InterpretUpdateStatement(UpdateStatement node)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SqlInterpretationVisitor.InterpretDmlWithNewVisitor(TSqlFragment node, SqlInterpretationVisitor visitor)
   at Microsoft.SqlServer.TransactSql.ScriptDom.StatementList.AcceptChildren(TSqlFragmentVisitor visitor)
   at Microsoft.SqlServer.TransactSql.ScriptDom.BeginEndBlockStatement.AcceptChildren(TSqlFragmentVisitor visitor)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SqlInterpretationVisitor90.ExplicitVisit(IfStatement node)
   at Microsoft.SqlServer.TransactSql.ScriptDom.StatementList.AcceptChildren(TSqlFragmentVisitor visitor)
   at Microsoft.SqlServer.TransactSql.ScriptDom.BeginEndBlockStatement.AcceptChildren(TSqlFragmentVisitor visitor)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SqlInterpretationVisitor90.ExplicitVisit(WhileStatement node)
   at Microsoft.SqlServer.TransactSql.ScriptDom.StatementList.AcceptChildren(TSqlFragmentVisitor visitor)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SubroutineInterpretationVisitor.ExplicitVisit(StatementList node)
   at Microsoft.SqlServer.TransactSql.ScriptDom.CreateProcedureStatement.AcceptChildren(TSqlFragmentVisitor visitor)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.ProcedureStatementInterpretationVisitor.ExplicitVisit(CreateProcedureStatement node)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.InterpretationVisitors.SqlInterpretationVisitor.InterpretDdlWithNewVisitor(TSqlFragment node, SqlInterpretationVisitor visitor)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.Sql100Interpreter.InterpretImpl(TSqlFragment fragment, SqlSchemaAnalyzer analyzer, IList`1& errors)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlModelBuilder.InterpretTSqlScript(String filename, TSqlScript script, NecessaryOptions parsedMetadata, IList`1 parseErrors)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlModelBuilder.ParseAndInterpret(String filename, TextReader objectsSource, IDictionary`2 metadata)
   at Microsoft.Data.Tools.Schema.Sql.SchemaModel.SqlModelBuilder.AddObjects(String cacheIdentifier, TextReader script, IDictionary`2 metadata)
   at Microsoft.Data.Tools.Schema.SchemaModel.ModelBuilder.AddOrUpdateObjects(ScriptHandle script)
   at Microsoft.Data.Tools.Schema.Utilities.Sql.Jobs.ModelBuilderJob.BatchLoadOrUpdate(ModelBuilder mb, IList`1 scripts, Int32 index, Int32 count, List`1& processedCacheIds, Exception& handledEx)
   at Microsoft.Data.Tools.Schema.Sql.Build.SqlTaskHost.Add(IList`1 sourceScripts)
   at Microsoft.Data.Tools.Schema.Tasks.Sql.TaskHostLoader.LoadSourceFiles(TaskLoggingHelper providedLogger, SqlTaskHost host, ErrorManager errors)
   at Microsoft.Data.Tools.Schema.Tasks.Sql.TaskHostLoader.LoadImpl(ITaskHost providedHost, TaskLoggingHelper providedLogger)
   at Microsoft.Data.Tools.Schema.Tasks.Sql.TaskHostLoader.Load(ITaskHost providedHost, TaskLoggingHelper providedLogger)
   at Microsoft.Data.Tools.Schema.Tasks.Sql.SqlBuildTask.ExecuteLoadTaskHostStep()
   at Microsoft.Data.Tools.Schema.Tasks.Sql.SqlBuildTask.ExecuteStep(Func`1 step)
   at Microsoft.Data.Tools.Schema.Tasks.Sql.SqlBuildTask.Execute()
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__20.MoveNext()

Answer 1

Have you tried forcing an in-memory model as per:

https://connect.microsoft.com/SQLServer/feedback/details/749108/msbuild-fails-randomly-with-version-store-out-of-memory-cleanup-already-attempted-errors-msb4018

This looks like a problem with our file-backed storage. We're looking at the core problem, but I've added a command-line switch to our next SSDT release (post-December 2013) to force the msbuild commandline utilities to use an in-memory model. With the next release define <CmdLineInMemoryStorage>True</CmdLineInMemoryStorage> in the first PropertyGroup in your .sqlproj file.