Parallel compilation of delphi projects through MSBuild

Question

I have a script that compile all the projects (around 50) of my solution like following

msbuild "myProjName.dproj" /t:build /p:config="Release" /fileLogger /flp:ErrorsOnly /nologo

This works just fine but takes forever to compile. In order to make it faster to build i've been trying to leverage all the potential of our modern multi-core machines using the '/maxcpucount' switch explained here : http://msdn.microsoft.com/library/bb651793.aspx

I get about the same compilation time on my 4-core CPU dev machine. No perf gains.

Apparently this can only work when projects need dependencies to be built. The others "workers" would then build these dependencies projects in parallel as the main proj. So i tried to build a project group in delphi and adding all my projects to it and than run the msbuild command on this .groupproj but it is still as slow as it has always been.

Did any of you achieved to build multiple projets at the same time with msbuild? If yes can you provide me an explanation?

Thanks!

Answer 1

The following applies to RAD Studio XE4, but it may also apply to earlier or later versions. Also, the dependencies defined in the .groupproj will not be honored with this method. The .groupproj I was trying to parallelize had no inter-project dependencies, so I didn't figure out how to handle this.

When you build a .groupproj file with MSBuild using the Build, Clean or Make target, the build doesn't run in parallel because these targets use the CallTarget task to execute other targets, but CallTarget doesn't execute its targets in parallel.

In order to build separate projects in parallel, the MSBuild project must use a single MSBuild task to build multiple projects at once. The targets must be defined like this:

  <Target Name="Build">
    <MSBuild Projects="@(Projects)" BuildInParallel="true"/>
  </Target>
  <Target Name="Clean">
    <MSBuild Targets="Clean" Projects="@(Projects)" BuildInParallel="true"/>
  </Target>
  <Target Name="Make">
    <MSBuild Targets="Make" Projects="@(Projects)" BuildInParallel="true"/>
  </Target>

Add these to the .groupproj, then remove the other <Target> directives as well as the <Import> directive. (CodeGear.Group.Targets defines some targets to build the projects in the proper order and to build dependencies when you ask to build only a subset of the projects, but it overrides the Build, Clean and Make targets defined in the .groupproj.) Note that this only allows you to build all projects, not just a subset.

BuildInParallel was added in MSBuild 3.5. However, since .groupproj files don't specify the ToolsVersion attribute, MSBuild will use the MSBuild task as defined in version 2.0, which didn't support BuildInParallel. There are two options to fix this:

1. Add ToolsVersion="3.5" (or a later version) to the root <Project> element of your .groupproj file.
2. Run MSBuild with the /toolsversion:3.5 (or /tv:3.5 for short) command-line parameter (/toolsversion overrides the ToolsVersion specified in all project files.)

After doing this, run MSBuild with the /maxcpucount (or /m) argument and your projects should build in parallel. However, RAD Studio doesn't handle this transformed project group correctly, so you may want to give the file a different extension to make it clear that it's not a standard RAD Studio project group (any extension that ends in proj will do).

The following XSLT stylesheet performs the transformation described above:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
                exclude-result-prefixes="msbuild"
                xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
                xmlns:msbuild="http://schemas.microsoft.com/developer/msbuild/2003"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" indent="yes"/>

  <xsl:template match="//msbuild:Project">
    <xsl:copy>
      <xsl:attribute name="ToolsVersion">3.5</xsl:attribute>
      <xsl:apply-templates select="@* | node()"/>
      <Target Name="Build">
        <MSBuild Projects="@(Projects)" BuildInParallel="true"/>
      </Target>
      <Target Name="Clean">
        <MSBuild Targets="Clean" Projects="@(Projects)" BuildInParallel="true"/>
      </Target>
      <Target Name="Make">
        <MSBuild Targets="Make" Projects="@(Projects)" BuildInParallel="true"/>
      </Target>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="//msbuild:Target">
    <!-- Do not copy -->
  </xsl:template>

  <xsl:template match="//msbuild:Import">
    <!-- Do not copy -->
  </xsl:template>

  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@* | node()"/>
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

You can apply this stylesheet with MSBuild (4.0 or later: XslTransformation was added in MSBuild 4.0) using this project file (where groupproj2parallel.xslt is the XSLT file above):

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Build" Inputs="$(InputPaths)" Outputs="$(OutputPaths)">
    <XslTransformation
      XmlInputPaths="$(InputPaths)"
      XslInputPath="groupproj2parallel.xslt"
      OutputPaths="$(OutputPaths)" />
  </Target>
</Project>

You need to specify InputPaths and OutputPaths explicitly on the command line with /p:InputPaths="..." /p:OutputPaths="...", or by specifying them on the Properties parameter of an MSBuild task. (Alternatively, you can just hardcode the file names in the project file.)

---

The target definitions provided with MSBuild for C# and Visual Basic projects handle dependencies by using the <ProjectReference> items defined in project files, instead of defining dependencies in the solution file. Delphi .dproj files and C++ Builder .cbproj files don't support this, as the underlying CodeGear.Common.Targets doesn't reuse the machinery defined in Microsoft.Common.Targets for <ProjectReference>.

Answer 2

There are two ways to build a Delphi projects: MSBuild or DCC32.exe. MSBuild is recommended as the project files (dproj and groupproj) encapsulate all configuration settings.

However, there are extra over head using MSBuild compare to plain old DCC32.exe. Furthermore, using MSBuild to build Delphi Project Group (.groupproj) doesn't bring any benefifs for multi-core CPUs. The build performance is same as single core CPU.

Here are my statistics to build a 290 dproj files in one single groupproj:

MSBuild a `groupproj` contains 290 `dproj` on 2C/4T CPU: ~100s
MSBuild a `groupproj` contains 290 `dproj` on 4C/8T CPU: ~100s

MSBuild 290 `dproj` run in multi-threads on 2C/4T CPU: ~121s
MSBuild 290 `dproj` run in multi-threads on 4C/8T CPU: ~50s

DCC 290 `dproj` run in multi-threads on 2C/4T CPU: ~37s
DCC 290 `dproj` run in multi-threads on 4C/8T CPU: ~24s

From the reading, we can conclude that MSBuild introduce extra overhead compare to DCC32. To fully utilize CPU cores and threads available, DCC32 is the way to go by sacrifice the convenient of project configuration encapsulation design for .DPROJ.

A msbuild script to build Delphi groupproj in parallel is available at https://github.com/ccy/msbuild.delphi.parallel

Answer 3

Little bit off-topic: you could try the fastdcc part of the IDE fix pack to get faster builds: http://andy.jgknet.de/blog/ide-tools/ide-fix-pack/ For example, I got a build time of 1 minute going down to 22s!