MSBuild C++ link-time dependencies

Question

I'd like to increase parallelism of build using MSBuild in the following scenario:

There are three C++ projects (targets). The first is a static library called A, the second is a dynamic library called B which depends on A, and the third is an executable C that depends on B. All of the projects contain simple source code and no header generation or other shenanigans.

There is nothing in this configuration that should preclude all of the translation units from being compiled in parallel as the only real dependencies are at link time. However, the standard way of setting up references and dependencies in MSBuild forces all of A to build before it will even start building anything in B, and so on. This unnecessarily serializes some of the build process.

The projects and the general solution need to remain approachable to the "average" developer meaning that maintaining the projects and build setup should be achievable entirely in the Visual Studio UI (new libraries and executables may depend on preexisting property sheets but it should not be required to edit new projects' .vcxproj files by hand, nor to manually write MSBuild XML files).

My current thought is to move all the code from B and C into separate static libraries, called B' and C'. Then B would depend on both A and B', allowing them to compile in parallel. Likewise, as C ultimately depends on A, B', and C' those can all compile in parallel. The only remaining serial processes would be the linker steps, as expected. That is a little more cumbersome and non-idiomatic than I'd like though it wouldn't be the end of the world.

I'd really like to just be able to make dependencies for a project's link step that don't also act as dependencies for the project's compile step. In most other build systems I'm familiar with this is trivial if not implicit. Is it possible in MSBuild for C++ projects given the Visual Studio ease-of-use requirement; if so, how?

Answer 1

I found a not-ideal polling workaround, but it works and is easy. Within just a few minutes after implementing was able to drop a full rebuild from ~17s to ~14s, and expect to get it down further. Will update once I use this more.

First remove dependencies among projects so that they all build at the same time. This will cause build race conditions which we will manually fix. At bottom of vcxproj for project B (and likewise for C to B) put:

<Target Name="WaitForA" BeforeTargets="Link">
  <Exec Command="$(SolutionDir)Build\A.bat $(OutputPath)A.lib"/>
</Target>

Targets only run once which's good, but make sure the name is unique. WaitFor.bat is a bat that just runs powershell:

param([string] $file)

$sleeps = 0
while($true)
{
    try
    {
        $lock = [System.IO.File]::Open($file,'Open','ReadWrite','None');
        $lock.dispose();
        break;
    }
    catch
    {
        $sleeps = $sleeps + 1
        Start-Sleep -m 100
    }
}

Write-Host ("Slept {0} times waiting for '{1}'" -f $sleeps,$file)

Answer 2

They easiest way to do this is to put your three projects into a visual studio solution file (i.e. *.sln). I know it's ugly, and that a solution file is NOT a true msbuild file. In order for this to work, you also have to specify the project dependencies using the visual studio tools. (i.e. Right click on the solution, select 'Project Dependencies...' )

Once you do that, inside of visual studio, it should perform the compilation step in parallel for all the projects, but then link them in the correct order.

On the command line, to achieve the same effect, you have to pass in -m to msbuild.exe and also the name of the solution file.

This also assumes you have specified -MP in all your project files too.