Grouping test results : Karma / Protractor integration with Jenkins

Question

I have successfully setup a Jenkins job which executes my Karma tests (with Angular-CLI) and also my Protractor tests for an Angular4 App (with Angular-CLI). Both of these tests will output an XML file, which will be then consumed by Jenkins to show the results and details.

My issue is very simple and visible with the following screenshot : I cannot differenciate whether my test suites are from my protractor tests or karma tests.

• Highlighted in red are my protractor test suites
• Not-Highlighted are my Karma test suites.

Therefore I was asking myself the following questions :

• Is there a good practice that is enforced for managing today unit tests and end-to-end tests with continuous integration ?
• Do I have to separate the tests in separate Jenkins jobs ? And does that mean I need to have two different JenkinsFiles later on with the Jenkins-pipeline workflow ?
• Or on the contrary I can simply set a prefix for every test suite so I can manage everything easily ?

Both of the last options seem impractical.

---

EDIT : Be thoughful about the question and read this completly before voting to close it, as IT IS NOT OPINION-BASED. I will make myself clear : I am facing impractical implementation of continuous integration, with a clear definition of the problem, and I expect a solution that would solve my current problem, and if possible, some good practices that are enforced about testing front-end application that could also clear this problem.

Answer 1

Although this is a common issue, I don't believe there is a best or good practise. It simply depends on how the developer wishes to display test results on the build server.

I have a solution for your particular use case which I hope solves your technical issue.

I use Teamcity myself, however I have adapted this for use with Jenkins also. I've set up a demo app using Angular CLI to demonstrate.

Build servers such as Jenkins and TeamCity simply read either from a results file (Jenkins, usually JUnit which I will demonstrate below) or service messages via stdout (TeamCity). Therefore we can manipulate the test result data prior to these records being written, which allows us to be able to distinguish between karma (unit) and Protractor (e2e) without needing to run the tests on seperate jobs or bloat the test suites with unnecessary prefixes.

You will require two additional packages, karma-junit-reporter and jasmine-reporters. We will use karma-junit-reporter for Karma and jasmine-reporters for Protractor.

These packages allow us to modify the suite name prior to the results being written to a JUnit file, which will help us distinguish between unit and e2e.

You can follow the installations instructions provided by each tool or follow my implementation below:

Install karma-junit-reporter:

npm install karma-junit-reporter --save-dev

Make the the following changes to your karma.conf.js file:

Add karma-junit-reporter to the plugins array:

require('karma-junit-reporter')

Add junit to the reporters array:

reporters: ['progress', 'kjhtml', 'junit']

Add a junitReporter object with the following :

junitReporter: {
        outputDir: 'testresults', 
        outputFile: 'karmatest.xml', 
        suite: 'unit',  // whichever prefix you wish to use
        useBrowserName: false, 
      }

Install jasmine-reporters:

npm install jasmine-reporters --save-dev

Please note, as of writing; the jasmine-reporters document for Protractor states you should run npm install --save-dev jasmine-reporters@^2.0.0 - however this will cause issues without exception handling. I’d recommend installing latest.

Add the following code to the onPrepare function of the protractor.conf.js file:

var jasmineReporters = require('jasmine-reporters');
return browser.getProcessedConfig().then(function (config) {
      var junitReporter = new jasmineReporters.JUnitXmlReporter({
        consolidateAll: true,
        savePath: 'testresults',
        filePrefix: 'protractor-test-results',
        modifySuiteName: function (generatedSuiteName, suite) {
          return 'e2e.' + generatedSuiteName; // whichever prefix you wish to use
    }
  });
  jasmine.getEnv().addReporter(junitReporter);
});

You should now configure your Jenkins job to run ng test, ng e2e and also add a post build action to publish JUnit test result report as follows:

testresults\*.xml

Your Jenkins test result dashboard will now look like this:

For the purposes of this demonstration, I've failed both a unit test and an e2e test, as you can see each test name has a prefix determining whether it is unit or e2e. You can also drill down into all e2e or all unit tests by clicking the link under packages.

If you are brave and have a lot of time you can write your own utility which does the same job as these packages for a more customized outcome.

I hope this helps you achieve the outcome you require.

Thanks

Answer 2

Your question is reasonable, however with the current state of things solution to this problem is closer to opinion based.

We have the same problem with Teamcity, there is no visual difference between NUnit, Karma or Protractor tests. It happens because for CI all tests are the same, it just parses stdout searching for specific patterns/format (test suite, test name, etc.), but there is no such thing as test type or test kind, so there is no grouping by type.

So the only way to distinguish tests is to use naming conventions or create special versions of describe functions:

export describeProtractor(name:string, ...) {
     describe('(Protractor)' + name, ...);
} 

Having multiple jobs for this purpose is really not convenient.

Another option is to use HTML Publisher Plugin to display test results generated by karma/protractor html reporters, I've never used so its up to you.