Rerunning failed cucumber tests using cucumber-jvm

Question

I have a Cucumber-JVM, JUnit, Selenium setup. I initiate the run by running RunSmokeTests.java using JUnit within Eclipse. I have also set up a maven profile to run the tests from command line, and possibly Jenkins in the future.

When the tests are run then some of them may fail sometimes, mainly due to the application taking longer than expected. I would then have to re-run these scenarios. At the moment I run them by manually attaching @rerun tag to the ones that failed and then running RunReruns.java, which is similar to RunSmokeTest.java but with @rerun tag.

With the increasing number of automated tests it is time consuming to tag the tests and start the run and clear the tags. Is there a automated way with Cucumber-JVM to re-run failed tests?

RunSmokeTests.java

package testGlueClasses;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@Cucumber.Options(features = "src/test/java", strict = true, format = {
        "html:target/CucumberReport", "json:target/JSON/Cucumber.json",
        "FrameworkCore.CustomTestReporter" }, tags = { "@SmokeTest" }, glue = {
        "FrameworkCore", "MyApp.Utils", "MyApp.StepDefinitions" })
public class RunSmokeTests {

} 

Maven snippet:

    <profile>
        <id>smoke</id>
        <properties>
            <include.tests>
                **/RunSmokeTests.java
            </include.tests>
        </properties>
    </profile>

Answer 1

I came up with another solution to rerun just failed test using maven & cucumber.

1) Record test failures using a RunNotifier

public class RerunningCucumber extends Cucumber {

    private final String className;

    @SuppressWarnings("rawtypes")
    public RerunningCucumber(Class clazz) throws InitializationError, IOException {
        super(clazz);
        className = clazz.getSimpleName();
    }


    @Override
    public void run(RunNotifier notifier) {
        notifier.addListener(new RunListener(){

            public void testFailure(Failure failure) throws Exception {

                Throwable error = failure.getException();
                if (error instanceof AssertionError){
                    //Nothing. This is a normal failure. Continue
                    return;
                }

                //No! A wild exception has appeared!
                //Let's run this test again.
                RerunningCucumber.addFile(className);
            }

        });
        super.run(notifier);
    }


    private static final String filename = "target/rerun.properties";
    private static final Set<String> addedClasses = new HashSet<String>();
    public static synchronized void addFile(String className) throws IOException{
        //First find the file

        if (addedClasses.contains(className)){
            return;
        }

        File file = new File(filename);
        if (!file.exists()){
            //Need to create the file
            PrintWriter writer = new PrintWriter(file, "UTF-8");
            writer.print("retryclasses=**/"+className+".class");
            writer.close();
        }
        else {
            PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(file, true)));
            out.print(",**/"+className+".class");
            out.close();
        }

        addedClasses.add(className);
    }
}

2) Use custom class as a runner for the cucumber tests.

This will run the tests, and whenever there is a failure, output the failed class to a file. Trick is to keep features short and create a lot of test classes to avoid repeating tests.

@RunWith(RerunningCucumber.class)
@CucumberOptions(features = {"classpath:features/testFeature.feature}, format = {
        "html:target/cucumber-html-report/testFeature.html",
        "json:target/cucumber-json-report/testFeature.json"},
        tags = {"@testFeature"})

public class RunTestFeature {
}

3) Add a Rerun profile to maven.

This does three things: 1) it loads the failed classes into memory, 2) cleans JUST the failed classes properties file, and 3) reruns ONLY the failed tests as loaded from the properties file:

    <profile>
        <id>retry</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>properties-maven-plugin</artifactId>
                    <version>1.0-alpha-2</version>
                    <executions>
                        <!-- Associate the read-project-properties goal with the initialize 
                            phase, to read the properties file. -->
                        <execution>
                            <phase>pre-clean</phase>
                            <goals>
                                <goal>read-project-properties</goal>
                            </goals>
                            <configuration>
                                <files>
                                    <file>target/rerun.properties</file>
                                </files>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-clean-plugin</artifactId>
                    <version>2.6.1</version>
                    <configuration>
                        <filesets>
                            <fileset>
                                <directory>target</directory>
                                <includes>
                                    <include>rerun.properties</include>
                                </includes>
                            </fileset>
                        </filesets>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-antrun-plugin</artifactId>
                    <version>1.6</version>
                    <executions>
                        <execution>
                            <phase>compile</phase>
                            <goals>
                                <goal>run</goal>
                            </goals>
                            <configuration>
                                <target>
                                    <echo>Retrying the following classes: "${retryclasses}"</echo>
                                </target>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.17</version>
                    <configuration>
                        <includes>
                            <include>${retryclasses}</include>
                        </includes>
                        <testFailureIgnore>true</testFailureIgnore>
                    </configuration>
                    <executions>
                        <execution>
                            <phase>test</phase>
                            <goals>
                                <goal>test</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>

4) Usage

First test run:

mvn clean test

Next test runs:

mvn clean test -Pretry
mvn clean test -Pretry
mvn clean test -Pretry
...

You can repeat as many times as you want until there are no errors.

Answer 2

I don't have an executable example at hand, but you can do this also on the jvm. There is a RerunFormatter that writes a text file listing the file and line numbers of failed scenarios:

@CucumberOptions(format = {"rerun:target/rerun.txt"})

You should be able to specify this file as input for another test class by prefixing it with @:

@CucumberOptions(features = {"@target/rerun.txt"})