Improving CI build time (.NET)

Question

We are developing an application framework + "plugins" using TeamCity as a CI server.

Project Details

1. 4 Visual Studio solutions
2. ~70 projects (and increasing)
3. Currently running 2 builds using TeamCity: CI and FULL build.

CI - triggered on every commit.

FULL - runs nightly.

I would like to improve the performance of both builds (especially the CI build, as it needs to give its output as quickly as possible).

Are there any guidelines in general on what can be effectively and easily improved?

The build process simply builds a .sln file and runs some unit tests.

Directions Considered:

• MSBuild parallelization
• Overriding CopyFilesToLocal

Not sure these are applicable/will result in a performance gain.

I am looking for more ways to improve the build time (which takes around 3-4 minutes).

Answer 1

Minimise the work your ci builds do.

• set up the workspace with any unneeded folders cloaked, to minimise the number of files you need to get from source control. If necessary reorganise your source structure so that it's easy to knock whole folders of data out of the build with cloaking.

• make sure the ci build uses incremental get and incremental build.

• only build solutions/projects that you need to. If your libraries only get changed infrequently, can you precompile them and check the binaries into source control? If a project is not currently being actively developed, don't bother building it in ci builds.

• do you need to run unit tests for every check in? We run a simple ci build of code only, with a separate test build running as ci, but no more frequently than once per hour. This slashes our ci build time but still lets us know within one hour if we break any unit tests.

• Similarly, don't build documentation, obfuscate, build installers, sign assemblies with certificates, etc, and disable any build processes that copy the outputs to the drop folder. CI builds are there to tell you if you've broken the build asap, you don't care about generating useful binary outputs.

• optimise the build in general - merge projects together, use multi-threaded builds, use several build agents so ci builds don't have to wait for other build types to complete. Only do full builds overnight so your build server is dedicated to ci while you are working. Keep source files tidy (delete unused code rather than just commenting it out, remove unused usings/includes etc.)

• invest in better build server hardware. If you don't have a top spec machine, drop more RAM and an SSD into it for a cheap speed boost. Make sure your build server is dedicated to ci builds, and isn't being used for anything else that might slow it down. Make sure the network between the build server and tfs sever is gigabit. Ensure you don't have any anti-virus software running on the server, or at least that its scheduled-scans are run overnight and your build folders are in the real-time-scan exclusion lists.

• use tfs check in policies to stop devs checking in if the ci build has failed, so that you stop and fix breakages immediately.

Answer 2

I'm working on 500+ projects C# application. Projects are compiled in parallel and copylocal set to false. Compile time is about 37min without unit tests and code coverage. 13min for incremental build without any change in the code. If I turn off parallel compilation and set copylocal to true, compile time is > 1h40min. I have different configuration for local build, gated check-in build and server builds with deploy phase (night builds).

Here are my experiences:

1. Copying output files to one directory is not good idea if you want to build your projects in parallel without CopyLocal set to false. My assemblies were sometimes locked when multiple projects referenced the same assembly and MSBuild tried to copy this reference to the output folder at the same time. This solution was very helpful for me. I set copylocal to false for all references and my build dir size was lowered 10x (10 times less I/O). I have different setup for local build and for server build. Different setup for gated check-in build and for full deploy build.
2. If I enable parallel build, builds are faster, much faster. If you have strong build server your /m:2 build should be 2x faster as /m:1 build. It has nothing to do with dependencies between projects (if copylocal is set to false).
3. You should reduce dependencies between the projects if you want to have fast incremental build. It has no impact on full build (copylocal false). Incremental compile time depends on the changed project location in the build tree.

Yes, MSBuild uses a timestamp of dependend projects to determine if a project needs a rebuild. It compares input files (code files, referenced assemblies, temporary files,..) timestamp with output assembly. If something is changed, your project is recompiled. Try to reduce number of depedencies between projects to minimize recompilation. If your change was only in the 'private' part of the project, your output assembly will be changed, assembly timestamp will be changed and all related projects will be rebuild also. You cannot do much with this.

Run your build 2 times with diagnostic verbosity without any change in your code and check for "Building target "CoreCompile" completely" like I described here. You can have something wrong in your project files and your projects are recompiled every time. If you don't change anything your build log should not contain "Building target "CoreCompile" completely" logs.

Our build server is virtual machine, not real piece of hardware. It is not good idea to use VM for build server, but it was not my decision.

If you have multi GB RAM try to use part of it as a in-memory hard drive. Your build should be much faster :)

SSD drives are sensitive to high I/O per day. It have an impact on warranty.

Hope it helps someone ... ;)