MSBuild & TeamBuild - BuildInParallel failing because of MSB3021 file permission violation

Question

I maintain the build of a fairly large piece of software, consisting of roughly 350 csharp projects. Our build time for a debug built clocks in at about 17 minutes.

I have been looking at ways to improve build time, and the BuildInParallel property did look intriguing. Especially since we have a quad-core server doing our builds, it should really be able to take advantage of the computing power.

But alas... After setting the property, modifying the config file for the build agent and restarting it, the first run really did look promising, way faster than normal, right up until the point where it failed.

After looking at the build logs, it looks like the build fails when it attempts to copy references marked as CopyLocal=true to the ouput dir. If C# project A and C# project B are built in parallel, and the both reference the same third-party dll, and tries to copy it at the same time, the second process to attempt to copy the file will get a file access violation - the file is being used by another process.

Anyone experienced this, and been able to get multi-proc builds working on Team Build?

---

Here's one of the failures, kind of hard to figure out which other project was being built at the same time.

I have removed all the non-relevant stuff:

54>Target "_CopyFilesMarkedCopyLocal" in file "c:\WINDOWS\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets" from project "d:\temp\PCM\1.3-Maint_CI\Sources\Modules\Core\Test\UnitTest\TestDIPS.Core.Data.Server.NUnit\TestDIPS.Core.Data.Server.NUnit.csproj": 54>Task "Copy" Copying file from "..........\Bin\3rdParty\Oracle\Oracle.DataAccess.dll" to "d:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll". Command: copy /y "..........\Bin\3rdParty\Oracle\Oracle.DataAccess.dll" "d:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll" 54>c:\WINDOWS\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets(2703,9): error MSB3021: Unable to copy file "..........\Bin\3rdParty\Oracle\Oracle.DataAccess.dll" to "d:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll". The process cannot access the file 'd:\temp\PCM\1.3-Maint_CI\Binaries\Debug\Oracle.DataAccess.dll' because it is being used by another process. Done executing task "Copy" -- FAILED. 54>Done building target "_CopyFilesMarkedCopyLocal" in project "TestDIPS.Core.Data.Server.NUnit.csproj" -- FAILED.

Answer 1

The default targets that ship with MSBuild are designed for CopyLocal behavior - which is what VS relies on. CopyLocal is problematic when you're outputting to a single output directory.

To be able to truly build in parallel, you'll need to disable several CopyLocal specific behaviors in the Microsoft.*.Common.*.targets files. I've spoken with some folks in the MSBuild team in the past, and this is a particularly hairy thing to do. Even if you do blank-out some of the CopyLocal behavior, the VS test accessors don't behave well with build in parallel.

Some of the things you can start with:

1. Disable CopyLocal behavior for references by setting private to true.
2. Disable CopyLocal for CopyToOutputPath files when building dependent Projects.