Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Protractor flakiness

I maintain a complex Angular (1.5.x) application that is being E2E tested using Protractor (2.5.x). I am experiencing a problem with this approach, which presents primarily in the way the tests seem flaky. Tests that worked perfectly well in one pull request fail in another. This concerns simple locators, such as by.linkTest(...). I debugged the failing tests and the app is on the correct page, the links are present and accessible.

Has anyone else experienced these consistency problems? Knows of a cause or workaround?

like image 652
Jürgen Simon Avatar asked Jul 26 '16 09:07

Jürgen Simon


People also ask

Is protractor going to end?

Protractor launched in 2013 before WebDriver APIs were standard and E2E tests were hard to write, and it tests Angular and AngularJS apps by running them in Google Chrome. It will continue to run until the end of 2022 when Angular 15 will be the last update.

Is protractor going to be deprecated?

According to the Protractor's announcement on Github the Angular team will stop the development of Protractor by the end of 2022 in conjunction with Angular v15. This means that users need to migrate from Protractor since there will not be any further updates to the framework.

How do you measure flakiness?

The pieces of aggregates are made to pass through the slot of specified thickness of gauge and then they are weighed. Then the flakiness index is calculated as the total weight of material passed through various thickness gauges, expressed as a percentage of total weight of the sample gauged.

What is test flakiness?

Flaky tests are defined as tests that return both passes and failures despite no changes to the code or the test itself.


3 Answers

Just Say No to More End-to-End Tests!

That said, here are the few things you can do to tackle our mutual merciless "flakiness" enemy:

  • update to the latest Protractor (currently 4.0.0) which also brings latest selenium and chromedriver with it
  • turn off Angular animations
  • use dragons browser.wait() with a set of built-in or custom Expected Conditions. This is probably by far the most reliable way to approach the problem. Unfortunately, this is use-case and problem specific, you would need to modify your actual tests in the problematic places. For example, if you need to click an element, wait for it to be clickable:

    var EC = protractor.ExpectedConditions;
    var elm = $("#myid");
    
    browser.wait(EC.elementToBeClickable(elm), 5000);
    elm.click();
  • maximize the browser window (to avoid random element not visible or not clickable errors). Put this to onPrepare():

    browser.driver.manage().window().maximize();
  • increase the Protractor and Jasmine timeouts
  • slow Protractor down by tweaking the Control Flow (not sure if it works for 4.0.0, please test)
  • manually call browser.waitForAngular(); in problematic places. I am not sure why this helps but I've seen reports where it definitely helped to fix a flaky test.
  • use the jasmine done() callback in your specs. This may help to, for example, not to start the it() block until done is called in beforeEach()
  • return a promise from the onPrepare() function. This usually helps to make sure things are prepared for the test run
  • use protractor-flake package that would automatically re-run failed tests. More like a quick workaround to the problem

There are also other problem-specific "tricks" like slow typing into the text box, clicking via JavaScript etc.

like image 155
alecxe Avatar answered Oct 19 '22 15:10

alecxe


Yes, I think all of us experienced such flakiness issue.

Actually, the flakiness is quite common issue with any browser automation tool. However, this is supposed to be less in case of Protractor as Protractor has built-in wait consideration which performs actions only after loading the dom properly. But, in few cases you might have to use some explicit waits if you see intermittent failures.

I prefer to use few intelligent wait methods like:

function waitForElementToClickable(locator) {
        var domElement = element(by.css(locator)),
            isClickable = protractor.ExpectedConditions.elementToBeClickable(domElement);

        return browser.wait(isClickable, 2000)
            .then(function () {
                return domElement;
            });
    }

Where 2000 ms is used as timeout, you can make it configurable using a variable.Sometimes I also go with browser.sleep() when none of my intelligent wait works.

like image 2
Priyanshu Shekhar Avatar answered Oct 19 '22 17:10

Priyanshu Shekhar


It's been my experience that some methods (eg. sendKeys()) do not always fire at the expected time, within the controlFlow() queue, and will cause tests to be flakey. I work around this by specifically adding them to the controlFlow(). Eg:

this.enterText = function(input, text) {
    return browser.controlFlow().execute(function() {
        input.sendKeys(text);
    });
};
like image 1
Brine Avatar answered Oct 19 '22 15:10

Brine