How can I run a specific set of Android instrumentation tests with Gradle?

Question

I have multiple test packages:

com.mypackage.blackbox    - Robotium UI tests
com.mypackage.integration - REST integration tests
com.mypackage.unit        - low level unit tests

Our server team needs to be able to run just the integration tests on every push (they take a couple of minutes), but then run all tests every night (the black box UI tests take more than 10 minutes).

This great answer provides a slightly hacky (but effective) way to do it by overloading an existing JUnit annotation like @SmallTest or @LargeTest.

The Gradle documentation suggests that test filters are the way to do this, e.g.

./gradlew connectedAndroidTestDevDebug --tests com.mypackage.integration.*

However, that fails with an > Unknown command-line option '--tests'. error (presumably because the Android Gradle plugin doesn't support everything that vanilla Gradle does?).

The same documentation says in future they plan to support these alternatives:

• Filtering based on custom annotations (future)
• Filtering based on test hierarchy; executing all tests that extend ceratain base class (future)
• Filtering based on some custom runtime rule, e.g. particular value of a system property or some static state (future)

Does anybody know a clean way to get this to work right now? For now I'm planning to use the @MediumTest annotation on the base class that all my integration tests extend, but I'd love to be able to specify particular package(s) instead. Using @MediumTest or @LargeTest abuses those annotations, as both my integration and black box tests are large tests according to the guidelines.

Answer 1

This is now possible with the addition of Android's recent Testing Support Library, you can now use AndroidJUnitRunner and filter the tests you run by your own custom annotations.

Filter test run to tests with a custom annotation (com.myapp.MyAnnotation in this example):

adb shell am \
  instrument -w -e annotation com.myapp.MyAnnotation \
  com.myapp/android.support.test.runner.AndroidJUnitRunner

Complete AndroidJUnitRunner Documentation

You'll need to annotate your test cases with your custom annotation to get this to work. Example test case:

import android.support.test.runner.AndroidJUnit4;
import com.myapp.MyAnnotation;

@RunWith(AndroidJUnit4.class)
public class CalculatorTest {

    @MyAnnotation
    @Test
    public void testAddition() {
        //Do testing here
    }

}

Here is what your "MyAnnotation" would look like:

package com.myapp;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * My custom Annotation to specify a type of tests to run.
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface MyAnnotation {
}

Answer 2

To run specific tests using gradle you have to create custom instrumentation test runner class and run tests using that class. I.e. create class

package com.my.package;

public class MyInstrumentationTestRunner extends InstrumentationTestRunner {
    @Override
    public void onCreate(Bundle instrumentationArguments) {
        instrumentationArguments.putString("size", "medium"); // Run medium tests
        // Add more ...
        super.onCreate(instrumentationArguments);
    }
}

Then in your build.gradle set testInstrumentationRunner (it's under android -> defaultConfig, i.e.

// ...

android {
    // ...

    defaultConfig {
        // ...
        testInstrumentationRunner "com.my.package.MyInstrumentationTestRunner"
    }
}

// ...

Hope it helps!

Note. build.gradle is from :lib, the tests are located under src/androidTest/java where the MyInstrumentationTestRunner is created.

Answer 3

Sam's answer is the most versatile answer. However, the simplest solution is probably to use the -e package option on the InstrumentationTestRunner:

Running all tests in a java package: adb shell am instrument -w -e package com.android.foo.subpkg com.android.foo/android.test.InstrumentationTestRunner

You can combine this option with using Square's Spoon library, as it allows you to specify either individual classes, or use -e to pass options through to the test runner (e.g. the package option):

--class-name        Test class name to run (fully-qualified)

--method-name       Test method name to run (must also use --class-name)

--e                 Arguments to pass to the Instrumentation Runner. This can be used
                    multiple times for multiple entries. Usage: --e <NAME>=<VALUE>.
                    The supported arguments varies depending on which test runner 
                    you are using, e.g. see the API docs for AndroidJUnitRunner.

For the record, Shazam's Fork has a more powerful regex option:

android.test.classes=REGEX - comma separated regexes that specify a pattern for the classes/packages to run