Tape "test exited without ending" error with asynchronous forEach loops

Question

What I'm doing

Edit: I created a repo with a simplified version of my problem reproducing the issue.

I'm trying to set up automated frontend testings with browserstack, selenium-webdriver and tape.

• More about tape

The idea is to define multiple browsers and devices which have to be tested one after another with X amount of given tests. In the example below I define only one test and two browsers on OSX.

In order to define the browsers only once and handle tests I created a repo test-runner which should be added as dev-dependency to the repos which need to be tested on the given devices and browsers. The test-runner gets all needed tests passed, starts the first browser, runs the tests on that browser and once all tests are done the browser is closed quit() and the next browser gets started and tests again.

test-runner

/index.js

const webdriver = require( 'selenium-webdriver' )

// ---
// default browser configs
// ---
const defaults = {
  "os" : "OS X",
  "os_version" : "Mojave",
  "resolution" : "1024x768",
  "browserstack.user" : "username",
  "browserstack.key" : "key",
  "browserstack.console": "errors",
  "browserstack.local" : "true",
  "project" : "element"
}

// ---
// browsers to test
// ---
const browsers = [
  {
    "browserName" : "Chrome",
    "browser_version" : "41.0"
  },
  {
    "browserName" : "Safari",
    "browser_version" : "10.0",
    "os_version" : "Sierra"
  }
]

module.exports = ( tests, url ) => {

  // ---
  // Asynchronous forEach loop
  // helper function
  // ---
  async function asyncForEach(array, callback) {
    for (let index = 0; index < array.length; index++) {
      await callback(array[index], index, array)
    }
  }

  // ---
  // runner
  // ---
  const run = async () => {

    // ---
    // Iterate through all browsers and run the tests on them
    // ---
    await asyncForEach( browsers, async ( b ) => {

      // ---
      // Merge default configs with current browser
      // ---
      const capabilities = Object.assign( {}, defaults, b )

      // ---
      // Start and connect to remote browser
      // ---
      console.info( '-- Starting remote browser hang on --', capabilities.browserName )
      const browser = await new webdriver.Builder().
        usingServer( 'http://hub-cloud.browserstack.com/wd/hub' ).
        withCapabilities( capabilities ).
        build()

      // ---
      // Navigate to page which needs to be checked (url)
      // ---
      console.log('-- Navigate to URL --')
      await browser.get( url )

      // ---
      // Run the tests asynchronously
      // ---
      console.log( '-- Run tests --- ' )
      await asyncForEach( tests, async ( test ) => {
        await test( browser, url, capabilities, webdriver )
      } )

      // ---
      // Quit the remote browser when all tests for this browser are done
      // and move on to next browser
      // Important: if the browser is quit before the tests are done
      // the test will throw an error beacause there is no connection
      //  anymore to the browser session
      // ---
      browser.quit()

    } )

  }

  // ---
  // Start the tests
  // ---
  run()

}

If you're wondering how this asyncForEach function works I got it from here.

my-repo

/test/front/index.js

const testRunner = require( 'test-runner' )
const url = ( process.env.NODE_ENV == 'development' ) ? 'http://localhost:8888/element/...' : 'https://staging-url/element/...'

// tests to run
const tests = [
  require('./test.js')
]

testRunner( tests, url )

/test/front/test.js

const tape = require( 'tape' )

module.exports = async ( browser, url, capabilities, driver ) => {

  return new Promise( resolve => {

    tape( `Frontend test ${capabilities.browserName} ${capabilities.browser_version}`, async ( t ) => {

      const myButton = await browser.wait( driver.until.elementLocated( driver.By.css( 'my-button:first-of-type' ) ) )

      myButton.click()

      const marked = await myButton.getAttribute( 'marked' )
      t.ok(marked == "true", 'Button marked')

      //---
      // Test should end now
      //---
      t.end()

      resolve()

    } )

  })

}

/package.json

{
  ...
  "scripts": {
    "test": "NODE_ENV=development node test/front/ | tap-spec",
    "travis": "NODE_ENV=travis node test/front/ | tap-spec"
  }
  ...
}

When I want to run the tests I execute npm run test in my-repo

Remember, that we have only one test (but could also be multiple tests) and two browsers defined so the behaviour should be:

1. Start browser 1 and navigate(Chrome)
2. One test on browser 1 (Chrome)
3. Close browser 1 (Chrome)
4. Start browser 2 and navigate (Safari)
5. One test on browser 2 (Safari)
6. Close browser 2 (Safari)
7. done

The Problem

The asynchronous stuff seems to be working just fine, the browsers are started one after another as intended. The problem is, that the first test does not finish even when i call t.end() and I don't get to the second test (fails right after 4.).

What I tried

I tried using t.pass() and also running the CLI with NODE_ENV=development tape test/front/ | tap-spec but it didn't help. I also noticed, that when I don't resolve() in test.js the test ends just fine but of course I don't get to the next test then.

I also tried to adapt my code like the solution from this issue but didn't manage to get it work.

Meanwhile I also opened an issue on tapes github page.

So I hope the question is not too much of a pain to read and any help would be greatly appreciated.

Answer 1

It seems like tape does not work so well with asynchronous code. See these discussions on their Github issues page:

https://github.com/substack/tape/issues/223
https://github.com/substack/tape/issues/160

The solutions seems to be to declare your tests with tape.add in the beginning, before any async code gets called.

I would also try to refactor some of that async code that might not be needed, if you're just opening browsers in a sequence.

Answer 2

So unfortunately I got no answer yet with the existing setup and managed to get the things work in a slightly different manner.

I figured out, that tape() processes can not .end() as long as any other proscess is running. In my case it was browser. So as long as the browser runs, I think tape can not end.

In my example repo there is no browser but something else must be still running in order to prevent tape to end.

So I had to define the tests in only one tape process. Since I managed to open the browsers in sequence and test it's totally fine for now.

If there are a lot of different things to test, i will just split these things in different files and import them into the main test file.

I also import the browser capabilities from a dependency in order to define them only once.

So here is the code:

dependency main file

{
  "browsers": [{
      "browserName": "Chrome",
      "browser_version": "41",
      "os": "Windows",
      "os_version": "10",
      "resolution": "1024x768",
      "browserstack.user": "username",
      "browserstack.key": "key"
    },
    }
      "browserName": "Safari",
      "browser_version": "10.0",
      "os": "OS X",
      "os_version": "Sierra",
      "resolution": "1024x768",
      "browserstack.user": "username",
      "browserstack.key": "key"
    }
  ]
}

test.js

const tape = require( "tape" )
const { Builder, By, until } = require( 'selenium-webdriver' );
const { browsers } = require( "dependency" )
const browserStack = 'http://hub-cloud.browserstack.com/wd/hub'

tape( "Browsers", async ( t ) => {

  await Promise.all( browsers.map( async ( capa ) => {

    const { browserName, browser_version, os } = capa

    const browser = new Builder().usingServer( browserStack ).withCapabilities( capa ).build();

    await browser.get( 'http://someurl.com' )

    const myButton = await browser.wait( until.elementLocated( By.css( 'my-button:first-of-type' ) ) )

    myButton.click()

    const marked = await myButton.getAttribute( 'marked' )

    t.ok(marked == "true", `${browserName} ${browser_version} ${os}`)

    await browser.quit()

  } ) )

  t.end()

} )