Android weird test sharding

Question

I'm experimenting with test sharding on Android and I'm getting pretty weird results:

+ adb -s emulator-5580 shell am instrument -e numShards 2 -e shardIndex 0 -e class com.package.etc.automation.Tests.SanityTest.SanityTest -w com.package.etc.test/android.support.test.runner.AndroidJUnitRunner

com.package.etc.automation.Tests.SanityTest.SanityTest:..........

Time: 306.578

OK (10 tests)


+ adb -s emulator-5582 shell am instrument -e numShards 2 -e shardIndex 1 -e class com.package.etc.automation.Tests.SanityTest.SanityTest -w com.package.etc.test/android.support.test.runner.AndroidJUnitRunner

com.package.etc.automation.Tests.SanityTest.SanityTest:......................

Time: 645.723

OK (22 tests)

As you can see, adb split the tests into two uneven groups. The second one has twice as many tests as the first one and executes twice as long. Not the best parallelism if you ask me.

Is there a possibility to control the distribution of tests, or at least force adb to split the tests evenly?

Answer 1

Let's trace it down.

When test suite is started, TestRequestBuilder is built upon JUnit Filters. ShardingFilter is one of them and is added. Adding it means that previously added Filter is "intersected" with new one - method public boolean shouldRun(Description description) is invoked. If you look at it, more likely at this fragment:

if (description.isTest()) {
    return (Math.abs(description.hashCode()) % mNumShards) == mShardIndex;
}

And substituting with your numbers (numShards=2), you'll notice, that this is just a parity test. Statistically it might happen, that generated HashCode parity distribution is not 50%. Moreover, when some of the tests on your test class are ignored, disabled and are interweaving with enabled tests, you can even more disturb particular method hashcode (Junit Description uniqueId is generated from method and class name).

That is just statistics matter. As you might see in this answer:

How the groups are divided is arbitrary