Robolectric vs Android Test Framework

Question

Update Apr-2015: Gradle build tools and Android Studio now officially support unit testing and prevent android.jar from throwing stub (no real implementation) error. So, yes its possible to run tests on Java VM, when stubs are appropriately mocked. Its a start but is still not comparable with Robolectric's power. There is also a third alternative, Scroll to bottom of this answer.

Now, about Robolectric :

Pros: here are a few points about how it has proved to be useful in unit testing:

1. You need not run an emulator, So you can test some of the project's non UI parts without requiring an emulator or device. This also applies to running tests on continuous integration/ build servers, no emulator instances need be launched.

2. With Android Studio, you can quickly run one particular test class, when you are working on the implementation to satisfy the test cases. You can debug as you write code. This is a massive productivity gain.

3. Can fake almost every android related thing as shadow objects, even SQLite. Plus each shadow object exposes many useful functions which their normal android counterparts don't offer. With shadow counterparts of android Object, you can do internal checking or call special methods.

4. Really shines when testing multi threaded code like AsyncTasks, Loopers and Handlers etc. You can pause and fast-forward thread Loopers, even main thread. Excellent for Handler based callback testing.

5. JUnit 4 format supported. Android is still holding onto JUnit 3 last time I checked.

6. Can be combined with other test tools like Mockito, Espresso etc etc.

7. Supports mock Activity instance creationRobolectric.buildActivity() and its control via ActivityController. Fragment/View manipulation also works on such mock activity instances.

8. There are now provided add-on modules that cover multi-dex, v4-support, play services, maps and http client as well. So, its now easy to test code using these library functions as well.

Cons: Where I found it not so good:

1. Robolectric excels at aiding Unit testing, but does not cover all the functionality a real device or emulator can offer. For example sensors, gps, open-gl etc etc.

2. You'll need an emulator or real device when doing integration or UI testing, so that Activities and services can interact with full android environment (other apps, like using camera app to get a picture for your app), not a limited one. Here you'll need to use the default test framework as it has functions to test UI as well.

3. JNI loading seems not to be supported. So code with native dependency can't be tested.

4. As of now, Robolectric has a hard wired dependency on google maps jar to work. And will download another android.jar from maven. So, project setup may require a bit of a tinkering. Update: as of v3 it seems to pull all dependencies via Gradle without much fuss.

5. Newer Android tools support coverage and reports generation etc, but only when test are run on a device. So with Robolectric you'll have to create extra Gradle tasks (run Jacoco) to do it for you. Update: Gradle 2.9 + ships with jacoco plugin.

6. As both gradle and android build tools are shipping out newer build versions at a fast rate, stable Robolectric versions will sometimes start having problems with the changed build tooling. Most typical problems include: sdk version incompatible, manifest not found, build output paths mismatch, resources not loading, build config issues etc. Some issues are also related to bugs in android tools. At times you may even have to write your own custom test runner or apply workarounds till next version fixes those issues. Check out open issues and configure tests accordingly.

---

Another alternative is simply mock stuff on your own, no frameworks involved. Its the "hard way" but the most customizable way. Its plain JUnit with JMockit:

@RunWith(JMockit.class)
public class OtherTest {
    public void testHandlerCallback(@Mocked final FragmentTransaction transaction,
                                    @Mocked final FragmentManager manager,
                                    @Mocked final Activity activity,
                                    @Mocked final LayoutInflater inflater,
                                    @Mocked final ViewGroup parent) {

        final List<Fragment> fragments = new ArrayList<>();
        new Expectations() {{
            activity.getFragmentManager(); result = manager;
            manager.beginTransaction(); result = transaction;
            transaction.add(withCapture(fragments), anyString);
            transaction.commit(); result = new Delegate<Void>() {
                public int commit() {
                    View v = fragments.get(0).onCreateView(inflater,parent,null);
                    Deencapsulation.invoke(v,"onMeasure",0,0);
                  return  0;  
                }
            };
        }};
    }
}

Above is a crude and inline example. You can actually create proper re-usable classes (say FragmentTestHarness) that will take a component (say a Fragment) under test and wrap it in completely isolated environment, preparing it for tests.

To share how i do...

Robolectric For SQL, activities flow, for those objects that needed context.

JUnit4 for api's java module to make sure data are return correctly.

Espresso For checking ui display correctly.

When I modified api...I only run jUnit4.

When I modified the data binding between api and UI or Sqlite, then I will only run Robolectric.

When I modified UI I only run Espresso.

Sometimes I will run Robolectric and espresso together, but very rare.

But I will run all before publish to play store.

Because I think there is not real benefit for now. But see how u use it to speed up your product quality and development speed.

Correct me if I am wrong.

You would use Robolectric only in cases you need to mock or fake the Android framework, e.g. if you need a context. When using the Android Test Framework you would have to run Instrumented tests for that which is ultra slow.

If you write your tests for letting them run frequently, e.g. because you follow a tdd approach this is not an option. So in this case Robolectric comes in handy.

So the main benefit about Robolectric is that it is a lot faster than Espresso or Instrumented tests in general.

The downside is that it fakes an Android environment which you should be aware of. To validate real world problems, better use a classic Android Framework approach.

The best is still to write your code in a way that you can unit test it and that you don't need a context or any other Android framework dependencies.

Robolectric is getting integrated into the Android Testing Framework since I/O 2018 - check out more at the official Robolectric page and watch this video from I/O 2018

The major benefit of Robolectric is speed. Unit tests with Robolectric do not require a running emulator or device to run the tests and are thus much much faster.

You might still want to have a suite of integration tests that run against a real device, just a much smaller suite.