Trouble with Stryker and Jest

Question

I'm testing Stryker with Jest. Stryker seems to don't apply Jest test.

The two mutants don't pass test if I introduce them manually in code, but they pass when I use Stryker.

Test doesn't seem to run on mutants. How can I make it work?

Here is my configuration :

package.json :

{
  "name": "test-stryker",
  "version": "1.0.0",
  "description": "",
  "main": "src/index.js",
  "scripts": {
    "test": "jest",
    "test-mutation": "stryker run"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "npm": "^6.0.0",
    "@stryker-mutator/javascript-mutator": "^3.3.0",
    "@stryker-mutator/jest-runner": "^3.3.0",
    "jest": "^26.0.0",
    "jest-cli": "^26.0.0"
  }
}

config.jest.js :

'use strict'

module.exports = {
  clearMocks: true,
  collectCoverage: true,
  collectCoverageFrom: [
    '**/*.js',
    '!**/reports/**/*.js',
    '!**/test/**/*.js',
    '!/node_modules/',
    '!**/*.config.js',
    '!**/*.conf.js'
  ],
  coverageDirectory: 'reports/coverage',
  testEnvironment: 'node'
}

stryker.conf.js :

/**
 * @type {import('@stryker-mutator/api/core').StrykerOptions}
 */
module.exports = {
  mutator: "javascript",
  reporters: ["html", "clear-text", "progress"],
  testRunner: "jest",
  coverageAnalysis: "off",
  jest: {
    configFile: "jest.config.js",
  },
};

sum.js :

'use strict'

module.exports = (a, b) => {
    return a + b
}

sum.test.js:

'use strict'

const sum = require('../../src/sum')

describe('sum', () => {
  test('With two values should return two values added.', () => {
    expect(sum(5,2)).toBe(7)
  })
})

Here is the result :

16:23:30 (21936) INFO ConfigReader Using stryker.conf.js
16:23:31 (21936) INFO InputFileResolver Found 1 of 12 file(s) to be mutated.
16:23:31 (21936) INFO InitialTestExecutor Starting initial test run. This may take a while.
16:23:35 (21936) INFO InitialTestExecutor Initial test run succeeded. Ran 1 tests in 4 seconds (net 3 ms, overhead 2707 ms).
16:23:35 (21936) INFO MutatorFacade 2 Mutant(s) generated
16:23:35 (21936) INFO SandboxPool Creating 4 test runners (based on CPU count)
Mutation testing  [==================================================] 100% (elapsed: <1m, remaining: n/a) 2/2 tested (2 survived, 0 timed out)

0. [Survived] ArithmeticOperator
C:\webdev\experiment\node\test-stryker\src\sum.js:3:11
-       return a + b
+       return a - b

1. [Survived] BlockStatement
C:\webdev\experiment\node\test-stryker\src\sum.js:2:27
-   module.exports = (a, b) => {
-       return a + b
-   }
+   module.exports = (a, b) => {}

Ran 0.00 tests per mutant on average.
----------|---------|----------|-----------|------------|----------|---------|
File      | % score | # killed | # timeout | # survived | # no cov | # error |
----------|---------|----------|-----------|------------|----------|---------|
All files |    0.00 |        0 |         0 |          2 |        0 |       0 |
 sum.js   |    0.00 |        0 |         0 |          2 |        0 |       0 |
----------|---------|----------|-----------|------------|----------|---------|

Answer 1

I've had the same problem and found that it was apparently this issue: https://github.com/stryker-mutator/stryker/issues/1566

Namely, the "find related tests" feature of jest does not seem to work on Windows. Stryker works for me when I add this to stryker.config.json:

"jest": { "enableFindRelatedTests" : false }

But of course that is really bad for performance on larger projects. Maybe with some additionally research the feature can be made to work. I'm not sure if it's a general issue with jest or only happens when driven by stryker.

Answer 2

There are better workarounds available. A workaround without performance impact:

Renaming the .stryker-tmp directory to be a non-hidden directory.

// stryker.conf.[js/json]
"tempDirName": "stryker-tmp",

(source: https://github.com/stryker-mutator/stryker-js/issues/2122#issuecomment-605783668)