g++ much slower on multiple files vs. monolithic single file using Google mock

Question

I'm having a problem that appears to be g++ related. Basically, g++ takes substantially more time to compile a program when it is split into multiple files versus a single monolithic file. In fact, if you cat the individual files together and compile that, it runs much faster than if you list the individual files on the g++ command line. For example, with nine files, it takes 1 minute, 39 seconds to compile; when I cat them together it only takes 13 seconds to compile. I've tried using strace but it just gets stuck in cc1plus; when I use the -f option I still can't sort out what's causing the problem.

I've isolated the problem. Here is how to reproduce it. I wrote a very simple program, like so:

void func_01(int i) 
{
  int j;
  volatile int *jp;

  jp = &j;

   for (; i; i--) ++*jp;
}

void call_01(void)
{
  func_01(10000);
}

int main(int argc, char *argv[])
{
  call_01();
}

Then I replicated it, removing main and substituting increasing numbers, 999 times. Then I built:

% time g++ -c test*.cpp

real    0m18.919s
user    0m10.208s
sys     0m5.595s
% cat test*.cpp > mon.cpp
% time g++ -c mon.cpp  

real    0m0.824s
user    0m0.776s
sys     0m0.040s

Because I intend to scale to hundreds of files much more complex than this, it's important to get the build time down. Can anyone help explain why this is happening, or offer a less gross workaround? I think it has in part to do with the preprocessor and the savings caused by include guards, because if I include even one file, the time difference is dramatically increased (a factor of five in one case) but it still remains, without includes, a factor of twenty faster to go with the monolithic file.

The version of g++ is 4.4.2, but I checked the latest version, 8.2.0, and it exists there as well.

Answer 1

There are two different effects:

1. Compiler invocation overhead: Compilers are complicated executables and sometimes they are even split into a frontend and a backend executable and the frontend spawns the backend for each individual source file, even when all source files are passed to the same compiler invocation of the frontend. Gcc and llvm do that for example.

   • Specify g++ -v to see these redundant compiler invokations. This answers your main question I think of why this happens even without header files.

2. Overhead due to parsing and compiling the same header files over and over again from scratch. In real-world examples this header overhead will be much more significant than the compiler invocation itself.

because if I include even one file, the time difference is dramatically increased (a factor of five in one case)

Yes! And this could also be well 1000 times slower instead of 5 times. With template intensive code the compiler needs to do a lot, at compile time.

The slowdown when splitting across many source files is hitting you especially for C++ code because C++ is header intensive. All your source *.cpp are compiled separately and all the headers they include are included redundantly for each individual source file.

Now if you cat all the source files together all the headers are, as you said, parsed only once because of the include guards. Since the compiler spends a great proportion of its time parsing and compiling the headers this is very significant, especially with template heavy code (e.g. using STL is enough).

The number of source files for hand-written C++ source code and also for generated C++ source code is a tradeoff between:

1. My full rebuild time is fast, but my incremental build time is slow.

   • This is the case when you have only one source file (meaning *.cpp files) or very little source files.

2. My full build time is slow, but my incremental build time is fast.

   • This is the case when you have lots of small source files (meaning *.cpp files).

(In any case, the number of header files does not matter much (except when you always pull in too much redundant stuff. This is about the number of compiler invocations which is the number of *.cpp or *.o files.)

For 1. the full compilation time from scratch is short since the compiler sees all the headers only once, which is significant in C++ and especially with template based header-only (or header intensive) libraries like STL or boost.

For 2. the individual compilation time is fast since only very few code in the *.cpp file is to be compiled when just a single of hundreds of files changed.

It strongly depends on your use case.

In case you generate the C++ code you should add an option to your generator to allow the user to choose which way to go with this tradeoff.