MsBuild Compile and build dependencies in parallel

Question

I am working on a large C++ solution with lots of projects.

Some of those are build bottlenecks where dlls depend on a another that takes forever to build.

I have plenty of CPUs to build, but I can't get MSBuild to compile (not link) everything in parallel and only use dependencies at link time.

I essentially want to have on every project:

# build objects
msbuild /t:BuildCompile project.vcxproj

# only now build/wait for dependencies
msbuild /t:ResolveReferences;BuildLink project.vcxproj

I want the above to work as part of a single build (cascade to dependent projects).

I have been trying to mess around with MSBuild target build orders:

<PropertyGroup>
  <BuildSteps>
    SetBuildDefaultEnvironmentVariables;
    SetUserMacroEnvironmentVariables;
    PrepareForBuild;
    InitializeBuildStatus;
    BuildGenerateSources;
    BuildCompile;

    ResolveReferences;

    BuildLink;
  </BuildSteps>
</PropertyGroup>

does not work, Resolve Dependencies in this setup does not build dependent projects.

Any ideas? Only the linker actually depends on referenced projects, the objs do not.

Answer 1

Here's a possible solution: first gets a list of all projects by 'parsing' them from the solution file. Not needed if you have that list already. Then invokes msbuild for all projects twice, once with the BuildCompile target, then with the Build target. I specifically chose the Build target (which will skip compilation since that is done already) as I am not sure your proposed solution of calling just ResolveReferences and Link targets will make a successful build in all cases, for example it might skip resource compilation, skip custom build steps etc.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Build">
  <ItemGroup>
    <AllTargets Include="BuildCompile;Build" />
  </ItemGroup>
  <Target Name="Build">
    <ReadLinesFromFile File="mysolution.sln">
      <Output TaskParameter="Lines" ItemName="Solution" />
    </ReadLinesFromFile>

    <ItemGroup>
     <AllProjects Include="$([System.Text.RegularExpressions.Regex]::Match('%(Solution.Identity)', ', &quot;(.*\.vcxproj)&quot;').Groups[ 1 ].Value)"/>
    </ItemGroup>

    <MSBuild BuildInParallel="true" Projects="@(AllProjects)"
             Properties="Configuration=$(Configuration);Platform=$(Platform)"
             Targets="%(AllTargets.Identity)"/>
  </Target>
</Project>

Invoke like

msbuild mybuild.proj /p:Configuration=Debug;Platform=Win32

I'm curious to see if this will improve your build times.

edit since you are seeing what look likes full rebuilds, maybe the BuildCompile targets only works properly when the other targets from BuildSteps ran as well. You could try splitting the build explicitely:

<MSBuild BuildInParallel="true" Projects="@(AllProjects)"
         Properties="Configuration=$(Configuration);Platform=$(Platform)"
         Targets="SetBuildDefaultEnvironmentVariables;
                  SetUserMacroEnvironmentVariables;
                  PrepareForBuild;
                  InitializeBuildStatus;
                  BuildGenerateSources;
                  BuildCompile;"/>

<MSBuild BuildInParallel="true" Projects="@(AllProjects)"
         Properties="Configuration=$(Configuration);Platform=$(Platform)"
         Targets="Build"/>