Publish website, aspnet_compiler null reference exception

Question

I'm having trouble publishing a solution through the command line. It intermittently fails on various projects, but every so often succeeds. I get both the null reference error and "The application domain in which the thread was running has been unloaded" which may just be the null reference with a race condition. It seems to only fail on projects that have web services (original .asmx and WCF with .svc or file-less activation). The fact that it sometimes succeeds makes me think it's not an issue with our code, though there may be some changes we can do to mitigate the underlying cause (whatever that may be).

What I tried/checked:

• I ran tfpt scorch to clean up any local changes including build output without success, so I know it's not an issue with the source directories.
• I cleared the temp files at C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files (we're on 32-bit) and seemed to get further but still had issues. After another publish the directory was still empty so probably unrelated.
• With Process Monitor I found aspnet_compiler was writing to %LocalAppData%\Temp\Temporary ASP.NET Files. Clearing these files seems to sometimes work or maybe I got lucky, but then it starts failing again. I didn't spot anything else that may be relevant.
• Make sure I have enough RAM, didn't seem to make a difference.
• Run msbuild single-threaded, no difference
• Run the aspnet_compiler from the command line (get command from the MSBuild log output), always succeeds
• There are no compressed folders
• I'm on the latest stable Visual Studio (VS 2013 update 2), but that shouldn't matter since I'm running from the command line and the executables used are in the framework directory.
• There is no App_Code folder in some of the failing projects

The build output doesn't give any good information so I set up DebugDiag to output on crash. Here are the results:

DetailID = 1
    Count:    1
    Type:     System.NullReferenceException
    Message:  
    Stack:    
        System.Web.Compilation.DiskBuildResultCache.CacheBuildResult(System.String, System.Web.Compilation.BuildResult, Int64, System.DateTime)
        System.Web.Compilation.BuildManager.CacheBuildResultInternal(System.String, System.Web.Compilation.BuildResult, Int64, System.DateTime)
        System.Web.Compilation.WebDirectoryBatchCompiler.CacheAssemblyResults(System.Web.Compilation.AssemblyBuilder, System.CodeDom.Compiler.CompilerResults)
        System.Web.Compilation.WebDirectoryBatchCompiler.CompileAssemblyBuilder(System.Web.Compilation.AssemblyBuilder)
        System.Web.Compilation.WebDirectoryBatchCompiler.CompileAssemblyBuilderParallel(System.Collections.ICollection)
        System.Web.Compilation.WebDirectoryBatchCompiler.CompileNonDependentBuildProviders(System.Collections.ICollection)
        System.Web.Compilation.WebDirectoryBatchCompiler.Process()
        System.Web.Compilation.BuildManager.BatchCompileWebDirectoryInternal(System.Web.Hosting.VirtualDirectory, Boolean)
        System.Web.Compilation.BuildManager.BatchCompileWebDirectory(System.Web.Hosting.VirtualDirectory, System.Web.VirtualPath, Boolean)
        System.Web.Compilation.BuildManager.PrecompileWebDirectoriesRecursive(System.Web.Hosting.VirtualDirectory, Boolean)
        System.Web.Compilation.BuildManager.PrecompileAppInternal(System.Web.VirtualPath, System.Collections.Generic.IEnumerable`1<System.String>)
        System.Web.Compilation.BuildManager.PrecompileApp(System.Web.VirtualPath, System.Collections.Generic.IEnumerable`1<System.String>)
        System.Web.Compilation.BuildManager.PrecompileApp(System.Web.Compilation.ClientBuildManagerCallback, System.Collections.Generic.IEnumerable`1<System.String>)
        System.Web.Compilation.BuildManagerHost.PrecompileApp(System.Web.Compilation.ClientBuildManagerCallback, System.Collections.Generic.List`1<System.String>)
        [GCFrame]
        [GCFrame]
        [ContextTransitionFrame]
        [GCFrame]
        [GCFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [TPMethodFrame]
        *** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_32\System.Web\3d247ccfb800c38a29cf91c27a6339da\System.Web.ni.dll
        Module load completed but symbols could not be loaded for C:\Windows\assembly\NativeImages_v4.0.30319_32\System.Web\3d247ccfb800c38a29cf91c27a6339da\System.Web.ni.dll
        System.Web.Compilation.ClientBuildManager.PrecompileApplication(System.Web.Compilation.ClientBuildManagerCallback)
        *** ERROR: Module load completed but symbols could not be loaded for C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_compiler.exe
        [GCFrame]


DetailID = 2
    Count:    1
    Type:     System.NullReferenceException
    Message:  Object reference not set to an instance of an object.
    Stack:    
        [GCFrame]
        [GCFrame]
        [GCFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [TPMethodFrame]
        System.Web.Compilation.ClientBuildManager.PrecompileApplication(System.Web.Compilation.ClientBuildManagerCallback, Boolean)
        System.Web.Compilation.ClientBuildManager.PrecompileApplication(System.Web.Compilation.ClientBuildManagerCallback)
        System.Web.Compilation.Precompiler.Main(System.String[])
        [GCFrame]

I have found others with similar issues but they were either unanswered or the solutions didn't work for me. Any more ideas?

Edit: This is the command line I'm using:

"C:\Program Files (x86)\MSBuild\12.0\bin\MSBuild.exe" Solution.sln /t:Build /m:1 /p:Configuration=Prod /p:Platform="Mixed Platforms" /p:DeployOnBuild=true /p:PublishProfile=Prod /p:AutoParameterizationWebConfigConnectionStrings=false /v:diag > out.txt

Here is some of the console output (redirected to the file):

Target "AspNetPreCompile: (TargetId:3773)" in file "C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\Web\Transform\Microsoft.Web.Publishing.AspNetCompileMerge.targets" from project "C:\Source\Project\Project.csproj" (target "PipelineAspNetCompileMergePhase" depends on it):
Task "AspNetCompiler" (TaskId:2474)
  Task Parameter:PhysicalPath=C:\Source\Project\obj\x86\Release\AspnetCompileMerge\Source (TaskId:2474)
  Task Parameter:TargetPath=C:\Source\Project\obj\x86\Release\AspnetCompileMerge\TempBuildDir (TaskId:2474)
  Task Parameter:VirtualPath=/ (TaskId:2474)
  Task Parameter:Debug=False (TaskId:2474)
  Task Parameter:Updateable=False (TaskId:2474)
  Task Parameter:ToolPath=C:\Windows\Microsoft.NET\Framework\v4.0.30319 (TaskId:2474)

  C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_compiler.exe -v / -p C:\Source\Project\obj\x86\Release\AspnetCompileMerge\Source C:\Source\Project\obj\x86\Release\AspnetCompileMerge\TempBuildDir  (TaskId:2474)
  Microsoft (R) ASP.NET Compilation Tool version 4.0.30319.34209 (TaskId:2474)
  Utility to precompile an ASP.NET application (TaskId:2474)
  Copyright (C) Microsoft Corporation. All rights reserved. (TaskId:2474)
   (TaskId:2474)
ASPNETCOMPILER : error ASPRUNTIME: Object reference not set to an instance of an object. [C:\Source\Project\Project.csproj]
  The command exited with code 1. (TaskId:2474)

Done executing task "AspNetCompiler" -- FAILED. (TaskId:2474)
Done building target "AspNetPreCompile" in project "Project.csproj" -- FAILED.: (TargetId:3773)

So as you can see, the msbuild diagnostic output isn't very helpful.

Edit 2: Trying to see if I can debug aspnet_compiler to shed more light on this. I kicked off gflags.exe to enable auto-attaching to the debugger. I was able to break on the NullReferenceException but I'm having trouble loading the symbols for System.Web. It's coming from C:\Windows\Microsoft.Net\assembly\GAC_32\System.Web\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Web.dll.

From the stack trace, it's failing in this method:

internal override void CacheBuildResult(string cacheKey, BuildResult result, long hashCode, DateTime utcStart)
{
  if (!result.CacheToDisk)
    return;
  if (HostingEnvironment.ShutdownInitiated)
  {
    BuildResultCompiledAssemblyBase compiledAssemblyBase = result as BuildResultCompiledAssemblyBase;
    if (compiledAssemblyBase == null)
      return;
    this.MarkAssemblyAndRelatedFilesForDeletion(compiledAssemblyBase.ResultAssembly.GetName().Name);
  }
  else
    new PreservationFileWriter(this.PrecompilationMode).SaveBuildResultToFile(this.GetPreservedDataFileName(cacheKey), result, hashCode);
}

So any of these could be null and causing the exception: result, HostingEnvironment, compiledAssemblyBase.

Answer 1

I know this is an old post, but just in case someone will find this usefull:

The cause of this problem for me was the McAfee anti-virus

ASPNETCOMPILER(0,0): Error ASPRUNTIME: Object reference not set to an instance of an object

Answer 2

If I was to fix such issue, I would try to isolate it as much as possible, so here is the plan.

I see the following possible reasons:

1. Building environment

2. Building procedure

3. Compiler bug

4. Too complex source code / project structure

In most complex cases like yours all factors plays together, so trying to isolate causes one by one might help, or at least might show additional / new errors which will be easier to fix and proceed.

Building environment

You can try to build the same project on some other machine, or others machines (saying build I mean solving your task - publish website from command line). If you will have the same issues - it shows nothing, but if other environments are OK (or at least one of them) - then you definitely have an issue with environment.

It can be anything - .Net version and / or fixes, OS version, PATH variables for command line use incorrect framework, spaces in path for your local project - whatever, but if you can make it work on other machine, when you will know that you need to look into building process and not into other stuff.

You can try building any sample project just to ensure you can publish simple project.

Building procedure

Looks like you already isolated that issue is in publish step, but you can try to play with it a little - do only compilation without publishing, or run publish target over existing binaries / publish folder etc - you need to try to isolate particular step (if any) which fails, excluding dependencies on other build steps.

Most probably this won't help, as I expect that process fails as a whole - but who knows.

Compiler bug

This one is closely related to the next one. You can try, if it is possible in your case, to build on VS2012 instead of VS2013.

Too complex source code / project structure

Pre-build all the projects in solution, and use reference to resulted DLL instead of reference to projects. If everything OK - return project sources one by one, so that you can understand at least which project causes compiler to fail.

When/if you have isolated a project, then you can try to compile parts of it as separate DLL, and in some cases you might realize that some particular code constructions / markup should be rewritten to satisfy compiler.

If issue is in web project or in some other project type, which simply could not be compiled as single DLL, you can try to "turn off" whole parts of project - like, exclude several folders with web pages and check results.

Hope this helps, but for sure all these steps require significant efforts to just isolate the issue; but in most cases having the particular issue isolated solves a significant part of a problem.