Automatically detect test coupling in Protractor (randomizing test execution order)

Question

The Problem:

We have a rather large test codebase. From time to time, instead of executing all the tests, we execute them individually or in packs. But, sometimes, we see the unexpected test failures because of the tests being interconnected, coupled. For example, one test assumes there is some data created by a previous test - running this kind of test individually will fail.

The Question:

Is it possible to automatically detect which Protractor tests are coupled in the project?

Our current idea is to somehow randomize the test execution order or randomly pick up a pack of tests from all the available tests and check if there are no failures. Hence, the other question: is it possible to change/randomize the Protractor test discovery and change the order of test execution?

---

Inspired by the Ned Batchelder's "Finding test coupling" blogpost and the Python nose test runner's nose-randomly plugin:

Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.

By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced - a technique used in many places, for example Google’s C++ test runner googletest.

Answer 1

You can run tests randomly (at the file level) by setting the random property in your config . You can also set your salt/seed so it's reproducibly random.

/**
 * If true, run specs in semi-random order
 */
random?: boolean,
/**
 * Set the randomization seed if randomization is turned on
 */
seed?: string,

You could also turn on shardTestFiles (parallel test runs), which should also be very telling in how coupled your tests are.

Answer 2

Did you try shuffling "it" blocks like below:

var shuffle = function (items) {
  var item, randomIndex;      
  for(var i = 0; i < items.length; i++){
    randomIndex= (Math.random() * items.length) | 0;
    item = items[i];
    items[i] = items[randomIndex];
    items[randomIndex] = item;
  }
}

describe('Suite', function() {

  it("should a", function () {
      console.log("execute a");
  });

  it("should b", function () {
      console.log("execute b");
  });

  it("should c", function () {
      console.log("execute c");
  });

  shuffle(this.children);    // shuffle the 'it' blocks

});

Source: Can protractor tests be run in a random order?