MSVC10 /MP builds not multicore across folders in a project

Question

I'm hoping someone points out something wrong or a workaround for what we are experiencing.

When compiling a project with /MP it appears that only files in the same folder are compiled concurrently. I used process explorer to swipe the command line and confirm the behavior.

Project filters seem to have all no impact on the what gets compiled concurrently.

Project structure on disk:

Folder\
  project.vcxproj  
  source\  
    foo.cpp  
    foo1.cpp  
  other_folder\  
    bar.cpp
    bar1.cpp
    bar3.cpp

Initial process tree:

MSBuild.exe
  cl.exe  ( passed: source\foo.cpp source\foo1.cpp )
    cl.exe  ( passed: source\foo.cpp )
    cl.exe  ( passed: source\foo1.cpp )

After the 2 child instances of cl.exe complete the parent closes and the following process tree appears:

MSBuild.exe
  cl.exe  ( passed: other_folder\bar.cpp other_folder\bar1.cpp other_folder\bar2.cpp )
    cl.exe  ( passed: other_folder\bar.cpp )
    cl.exe  ( passed: other_folder\bar1.cpp )
    cl.exe  ( passed: other_folder\bar2.cpp )

Our source is nicely organized in many levels of nested folders that match the layout of the headers on disk - I would hate to have to give that up to take advantage of /MP.

Answer 1

The use of %(RelativeDir) in "Object File Name" ( in the vcxproj XML, /Fo on the CL.exe command line ) project causes msbuild to batch the cpp files to cl.exe on a per directory basis. This can have a significant impact on the benefits gained from using /MP.

Note that if your project uses %(RelativeDir) for object files its likely that the configuration is trying to avoid colliding .obj files that from cpp files with the same name in different folders.

The /Fo command line parameter is typically a folder that the compiler dumps the obj files into - there is only ONE passed, so all of the cpp files for a given directory can only be passed to CL.exe at a time.

That was a pain - but I'm glad there is a reason and a solution. Hope it helps.

---

Update

A team mate found that anytime an MSBuild parameter is sent to CL.exe it seems to break or severely limit /MP. This is most likely because for /MP to work well the top level CL.exe needs to have a bundle of cpp files.

Our solution was to not use any msbuild params ( I think its the %params% ) for 'Object File Name'. This required that we rename some cpp files so they did not collide.

Hope this has changed in VS2012 or VS2013.

Answer 2

According it MSDN, the files should be compiled when ever there is a thread to handle them, it at the same time makes no guarantee's about the order in which files are to be compiled:

The source files might not be compiled in the same order in which they appear on the command line. Although the compiler creates a set of processes that contain copies of the compiler, the operating system schedules when each process executes. Consequently, you cannot guarantee that the source files will be compiled in a particular order.

A source file is compiled when a process is available to compile it. If there are more files than processes, the first set of files is compiled by the available processes. The remaining files are processed when a process finishes handling a previous file and is available to work on one of the remaining files.

It also states that the number of created process will be bound by the number of threads and files in the command line:

That value is the lesser of the number of source files that you specify on the command line

Combining these to things, we can see that the compiler handles the compilation incrementally(file wise) so that it can dispatch work to children correctly, it dose this folder by folder.

You might be able to get around this if you generated a custom make file, where you should be able to get more than one folder being processed at the same time (or try using the MSBUILD.exe tool).