Is it possible to skip a scenario with Cucumber-JVM at run-time

Question

I want to add a tag @skiponchrome to a scenario, this should skip the scenario when running a Selenium test with the Chrome browser. The reason to-do this is because some scenario's work in some environments and not in others, this might not even be browser testing specific and could be applied in other situation for example OS platforms.

Example hook:

@Before("@skiponchrome") // this works
public void beforeScenario() {
  if(currentBrowser == 'chrome') { // this works
    // Skip scenario code here
  }
}

I know it is possible to define ~@skiponchrome in the cucumber tags to skip the tag, but I would like to skip a tag at run-time. This way I don't have to think about which steps to skip in advance when I starting a test run on a certain environment.

I would like to create a hook that catches the tag and skips the scenario without reporting a fail/error. Is this possible?

Answer 1

I realized that this is a late update to an already answered question, but I want to add one more option directly supported by cucumber-jvm:

@Before //(cucumber one)
public void setup(){
    Assume.assumeTrue(weAreInPreProductionEnvironment);
}

"and the scenario will be marked as ignored (but the test will pass) if weAreInPreProductionEnvironment is false."

You will need to add

import org.junit.Assume;

The major difference with the accepted answer is that JUnit assume failures behave just like pending

Important Because of a bug fix you will need cucumber-jvm release 1.2.5 which as of this writing is the latest. For example, the above will generate a failure instead of a pending in cucumber-java8-1.2.3.jar

Answer 2

I really prefer to be explicit about which tests are being run, by having separate run configurations defined for each environment. I also like to keep the number of tags I use to a minimum, to keep the number of configurations manageable.

I don't think it's possible to achieve what you want with tags alone. You would need to write a custom jUnit test runner to use in place of @RunWith(Cucumber.class). Take a look at the Cucumber implementation to see how things work. You would need to alter the RuntimeOptions created by the RuntimeOptionsFactory to include/exclude tags depending on the browser, or other runtime condition.

Alternatively, you could consider writing a small script which invokes your test suite, building up a list of tags to include/exclude dynamically, depending on the environment you're running in. I would consider this to be a more maintainable, cleaner solution.

Answer 3

It's actually really easy. If you dig though the Cucumber-JVM and JUnit 4 source code, you'll find that JUnit makes skipping during runtime very easy (just undocumented).

Take a look at the following source code for JUnit 4's ParentRunner, which Cucumber-JVM's FeatureRunner (which is used in Cucumber, the default Cucumber runner):

@Override
public void run(final RunNotifier notifier) {
    EachTestNotifier testNotifier = new EachTestNotifier(notifier,
            getDescription());
    try {
        Statement statement = classBlock(notifier);
        statement.evaluate();
    } catch (AssumptionViolatedException e) {
        testNotifier.fireTestIgnored();
    } catch (StoppedByUserException e) {
        throw e;
    } catch (Throwable e) {
        testNotifier.addFailure(e);
    }
}

This is how JUnit decides what result to show. If it's successful it will show a pass, but it's possible to @Ignore in JUnit, so what happens in that case? Well, an AssumptionViolatedException is thrown by the RunNotifier (or Cucumber FeatureRunner in this case).

So your example becomes:

@Before("@skiponchrome") // this works
public void beforeScenario() {
  if(currentBrowser == 'chrome') { // this works
    throw new AssumptionViolatedException("Not supported on Chrome")
  }
}

If you've used vanilla JUnit 4 before, you'd remember that @Ignore takes an optional message that is displayed when a test is ignored by the runner. AssumptionViolatedException carries the message, so you should see it in your test output after a test is skipped this way without having to write your own custom runner.