optimize compiletime in Continuous Integration

Question

Maybe that's just a mad man's dream, but..

In my company we have a big C# .NET project, with ~25 solutions (very old) and ~3.5 mio. loc. The problems I'm facing is: too slow build times, right now it takes 7 minutes with SSD (dev machines), 15 minutes+ in VM with normal harddrives (would be the TeamCity build system I'd like to get deployed). I know, build system should be fastest, but that's nothing I can change in the short term.

I want to shorten the commit-build-unittest feedback-loop for the devs (preferably on Teamcity machine right now) by just compiling the project(s) which were touched by the last commit, taking all other assemblies from e.g. a local nuget server (teamcity server itself with version 7.0).

Now that would immensely cut down the feedback loop (15 minutes to less then a minute, given real unit tests) for small commits.

I know the problem of such a partial compile is the possibility of skipping compile errors (mismatching interfaces could go unnoticed), but that would be mitigated by running a second (Teamcity?) build server instance which runs the whole enchilada, in parallel. But getting first feedback immediately is very important to me.

Now my question: is there any build system/continuous integration system which can handle this task? Or would I have to write my own commit-aware background service? Which would be a bit nasty, as we use FinalBuilder Scripts, and that Format doesn't seem to be readable by any API (but didn't dig deep enough into that).

P.S.: Also, I would like to only run the unit tests for the projects which were changed by the last commit, or at least prioritize them. But that's an afterthought.

Answer 1

Most of the available CI engines adopts a deployment pipeline process, which is specifically designed to reduce the feedback time in the development loop. It works like this, with a FAIL status immediately if any of the steps went wrong.

It is suggested (by this book) to have the first 4 steps to be less then 2-5 mins even for the most complicated projects, if it goes above that then there is a problem with your configuration and the way you use CI process.

Code commit
 triggers --->

 Step 1. Automatic checkout on CI side
 Step 2. Compile code, ideally 1-2 mins  
 Step 3. Save binaries to the artifact repository
 Step 4. Unit test, ideally 1-2 mins
 Step 5. Deploy to staging
 Step 6. Automated integration testing
 Step 7. Automated acceptance testing
 ------------------------------------
 Manual testing
 Manual deploy to production

Specifically to Step 2 you can:

a. Split large solution into separate tiers. Each tier would have it's own Visual Studio solution and only projects relevant to that tier, in a sense you perform decentralisation of the initial bulky solution. On step 5 you would know how to assemble tiers into usable application.

b. In TeamCity configuration you can specify whether to perform a clean checkout or use the source already available (can save time on step 1). Check that the target to MSBuild is set to Build which will pick up only the source files that have changed since the last build (save time on step 2).

From personal experience option a) is the best one, but if needed you can also use both a) and b), which however may bring some latent bugs with old files being kept for longer than needed. Teamcity also supports several agents (up to 3 in free edition) which will allow you to run tasks in parallel.