How to speed up execution of Android UI tests as part of a CI system

Question

I'd like all unit & instrumentation tests (Espresso) to be ran after each commit / merge to the main develop branch. Unit tests are fast enough to allow this, but UI tests are not - 150 fully mocked UI tests take ~1h to run on a single device. Shazam's FORK library does great job in sharding these 150 tests across all connected devices. The current solution is a local machine with Jenkins running on it. Connecting 4 devices to it reduces the time to run UI tests to ~15 mins. That's not ideal, but bearable.

Ideally I'd like to find a cloud based CI system that allows me to run UI tests using Fork, so that the local Jenkins can be ditched and not maintained internally.

I tried AWS Device Farm and Firebase Test Lab, but both are using their own systems to run the tests. It seems they don't give the option to shard a single test suite across multiple devices. They seem to be great tools to run the whole test suite on different devices simultaneously, but that's not what I want for a CI solution (to split the test suite to multiple devices simultaneously).

Tried BuddyBuild as well, but internally their using Firebase Test Lab, so that didn't work for my case as well.

I was mainly thinking for solutions along these directions:

• find a way to run Fork on a cloud based solution
• find another way to shard a single test suite across multiple devices

Any suggestions are welcome! How do you guys solve this problem?

Answer 1

Congrats! Too much automation is a great problem to have. It sounds like you have already made a lot of progress speeding up the execution time. I'd say 15 minutes is a pretty reasonable number, but here are some alternate approaches to get that number even lower:

1. Create a smaller and faster test suite of the highest priority test cases to run per commit while the longer running full test suite runs continuously against the latest merged commits. This is a pretty common pattern across the industry. You can use Android's built-in @Small, @Medium, and @Large annotations or package names to divide tests up.

2. Optimize your test cases so multiple features are tested at once. Essentially, instead of testing every combination like A1,A2,B1,B2, you would instead test only A1,B2. See the pairwise testing Wikipedia page for a more detailed explanation.

3. If it's possible for your app, try using emulators. Performance has improved greatly the last couple years. Using Intel Hardware Accelerated Execution Manager (HAXM) drivers I found tests ran faster and more reliably than on most physical devices. This could also make it easier for your goal to run in the cloud.

4. Check how long each individual test takes to run corresponding to its priority. A low priority long running test could be moved to a less frequently running test.

5. Watch the tests run to identify sleeps or other slow running areas of the test to improve.

6. Look for tests that have never failed. Those could also be another candidate for moving to a less frequently running test.

7. If you are looking for alternate ways to shard tests, you could roll your own by kicking off parallel scripts across devices that target a subset of your tests using adb shell am instrument options. Although, this would create many separate test reports and you would need to evenly divide the tests up yourself.

Answer 2

Might take a look at the Flank open source project which seemed to be built with pretty much the same goal in mind: https://medium.com/walmartlabs/flank-smart-test-runner-for-firebase-cf65e1b1eca7

Flank is a Firebase Test Lab tool for massively-scaling your automated Android tests. Run large test suites across many devices/versions/configurations at the same time, in parallel. Flank can easily be used in a CI environment where Gradle (or similar) first builds the APK:s and then Flank is used to execute the tests.