Menu
We have the "problem" of a large automated suite of integration tests. While our build times are reasonable (< 1 hour), the tests typically take > 6 hours to complete.
While it's great to have this large chunk of functionality tested in our build runs, it obviously is a barrier to implementing CI, which I've found to be a very helpful for keeping source trees in a "always buildable" state.
I've reviewed threads of discussion like this one, which elaborate on the distinctions.
This leads me to a few questions:
Does CI dictate or recommend Unit vs. Integration testing automation? I've heard Unit-only in the past, but am not finding any such statements (or rationale) for this in a quick search.
What is a good "best practice" for combined build + automated test times/ratios to have effective CI for a team? My gut tells me that this should be < 2 hours as a worst case, and probably < 1 hour to be really effective. In theory, we could break up the tests to run in parallel and probably get them running in under 2 hours, but this would still be a 3 hour run.
What's the best way forward from long-running Nightly Builds + Integration Tests to CI? I'm thinking of a CI build with a few skeletal Unit Tests only, in combination with nightly builds that continue with the integration tests.
Any tooling recommendations are also welcome (Windows-only C#/C++ codebase)
593
Automation Testing is at the core of any CI/CD pipeline. This is because the whole concept of CI/CD revolves around “build fast, test fast, fail fast.” Tests have to be run as fast as possible so that the feedback reaches the developer early. This enables the early detection of bugs.
A daily build or nightly build is the practice of completing a software build of the latest version of a program, on a daily basis. This is so it can first be compiled to ensure that all required dependencies are present, and possibly tested to show no bugs have been introduced.
Continuous integration refers to the build and unit testing stages of the software release process. Every revision that is committed triggers an automated build and test. With continuous delivery, code changes are automatically built, tested, and prepared for a release to production.
Test automation is designed to produce a set of pass/fail data points correlated to user stories or application requirements. Continuous testing, on the other hand, focuses on business risk and provides insight on whether the software can be released.
For most projects, however, the XP guideline of a ten minute build is perfectly within reason. Most of our modern projects achieve this. It's worth putting in concentrated effort to make it happen, because every minute you reduce off the build time is a minute saved for each developer every time they commit. Since CI demands frequent commits, this adds up to a lot of time.
Source: http://martinfowler.com/articles/continuousIntegration.html#KeepTheBuildFast
Why does it takes 6 hours? How many tests do you have? What are the ratio of the unit-test compared to integrated ones? You probrably have many more integrated tests or your unit-test are not really unit. Are your unit tests touching the DB? This may be the problem.
6 hours is a long long time. The article above has some tips.
148
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
