Processing batch items in parallel

Question

I have an ItemGroup, and want to process all its items in parallel (using a custom task or an .exe).

• I could write my task/exe to accept the entire ItemGroup and process its items in parallel internally. However, I want this parallelism to work in conjunction with MSBuild's /maxCpuCount param, since otherwise I might end up over-parallelizing.
• This thread says there's no way.
• My testing shows that MSBuild's /maxCpuCount only works for building different projects, not items (see code below)

How can I process items from an ItemGroup in parallel?
Is there a way to author a custom task to work in parallel in conjunction with MSBuild's Parallel support?

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Build" >
    <!-- Runs only once - I guess MSBuild detects it's the same project -->
    <!--<MSBuild Projects="$(MSBuildProjectFullPath);$(MSBuildProjectFullPath)" Targets="Wait3000" BuildInParallel="true" />-->

    <!-- Runs in parallel!. Note that b.targets is a copy of the original a.targets -->
    <MSBuild Projects="$(MSBuildProjectFullPath);b.targets" Targets="Wait3000" BuildInParallel="true" />

    <!-- Runs sequentially -->
    <ItemGroup>
      <Waits Include="3000;2000"/>
    </ItemGroup>
    <Wait DurationMs="%(Waits.Identity)" />
  </Target>

  <Target Name="Wait3000">
    <Wait DurationMs="3000" />
  </Target>

  <UsingTask TaskName="Wait" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll" >
    <ParameterGroup>
      <DurationMs ParameterType="System.Int32" Required="true" />
    </ParameterGroup>
    <Task>
      <Code Type="Fragment" Language="cs">
        Log.LogMessage(string.Format("{0:HH\\:mm\\:ss\\:fff}  Start  DurationMs={1}", DateTime.Now, DurationMs), MessageImportance.High);
        System.Threading.Thread.Sleep(DurationMs);
        Log.LogMessage(string.Format("{0:HH\\:mm\\:ss\\:fff}  End    DurationMs={1}", DateTime.Now, DurationMs), MessageImportance.High);
      </Code>
    </Task>
  </UsingTask>
</Project>   

Answer 1

I know this is old, but if you get a few minutes, revisit your attempt to use the MSBuild task. Using the Properties and/or AdditionalProperties reserved item metadata elements* will resolve the issue you described in your code sample ("Runs only once - I guess MSBuild detects it's the same project").

The MSBuild file below processes items from an ItemGroup in parallel via MSBuild's parallel support (including /maxCpuCount). It does not use BuildTargetsInParallel from the MSBuild Extension Pack, nor any other custom or inline task.

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="Build" >
    <ItemGroup>
      <Waits Include="3000;2000"/>
    </ItemGroup>

    <ItemGroup>
      <ProjectItems Include="$(MSBuildProjectFullPath)">
        <Properties>
          WaitMs=%(Waits.Identity)
        </Properties>
      </ProjectItems>
    </ItemGroup>
    <MSBuild Projects="@(ProjectItems)" Targets="WaitSpecifiedMs" BuildInParallel="true" />
  </Target>

  <Target Name="WaitSpecifiedMs">
    <Wait DurationMs="$(WaitMs)" />
  </Target>

</Project>

* Well-hidden under "Properties Metadata" on the MSBuild Task reference page.