Visual Studio Website build fails intermittently on CI server

Question

We have a fairly complex Visual Studio solution (57 projects, 19 of which are website which fails to build almost every time when triggered by pushing code, but then we manually trigger the build and on the retry it builds just fine.

The solution contains 57 projects, 19 of which are website projects. (Not web application projects, there is no .csproj file.) The rest are class libraries and background jobs. The 19 website projects are structured in IIS virtual directories into one big multifunctional content management system.

The build server is Hudson v1.395. The command used to build is:

"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\devenv.com" "SolutionName.sln" /rebuild Debug

When the build fails, it always does so on the exact same website project, with the exact same message:

------ Rebuild All started: Project: C:\...\WebsiteName\, Configuration: Debug Any CPU ------
Validating Web Site
: Build (web): The application domain in which the thread was running has been unloaded.

Validation Complete

A Google Search for this message is currently less than helpful. This link gets closest to the actual issue but has no resolution. Obviously we are not changing any solution files during the build because it is occurring on the build server.

When it fails, we trigger the build manually, and we get as we expect (sorry, redacted):

------ Rebuild All started: Project: C:\...\News2\, Configuration: Debug Any CPU ------
Validating Web Site
Building directory '/WebsiteName/Dir1/Dir2/'.
Building directory '/WebsiteName/'.
Building directory '/WebsiteName/Dir3/'.
// 22 more but you get the point

// A few warnings caused by our own use of the ObsoleteAttribute, nothing to be concerned about
Validation Complete

What could cause this application domain unloaded message?

Some other notes:

• We thought it could be Hudson running out of memory, because we do observe java leaking it quite a bit. So, we added a task to restart the Hudson service every morning at 6am. Even with this and a sane amount of available memory ready to go during the build, it still failed.
• A push to the same repository also causes the build of a much simpler (only 22 projects, no website projects) solution at the same time. That one always succeeds. Also, manually triggering them both to run at the same time succeeds.
• I know we should upgrade Hudson but that's always one of those backburner projects we never seem to have time for. In any case, I feel fairly strongly that this is a Visual Studio / MSBuild problem, not a Hudson problem.

Edit 1: MSBuild

The problem with MSBuild is that there are so many little quirks that differ from a build in Visual Studio. It's extremely frustrating for a solution to compile in Visual Studio on a developer machine and then fail on the build server. Even the output from msbuild is drastically different (much more verbose for one thing) from what our developers see in their build output window. Are there additional command line flags that bring the MSBuild output more in line with what you get in the Visual Studio build window?

There are other things that are awkward too. If you happen to have a solution folder named the same as a project, MSBuild throws an error but Visual Studio deals with it just fine. Those are the quirks that really make you pull your hair out.

Answer 1

I had a problem with C++ builds on Hudson/Jenkins which is likely related, if you have two builds going at once then bad things can happen.

This is because Hudson/Jenkins will run a process tree killer to clean up processes at the end of a build, and MsBuild/VisualStudio will share some common processes between builds.

The actual problem I had with the C++ builds manifested itself as another error:

fatal error C1090: PDB API call failed, 
error code '23' : '( 

The issue was raised here:

https://issues.jenkins-ci.org/browse/JENKINS-9104

Turning off the process tree killer may resolve your issue.