in-app A|B Testing for Mobile

Question

Is there a good solution for A|B Testing in mobile apps like online? I know with iOS it's against the TOS to have different user experiences with identical actions, but what about Android? And what about firms like Apsalar which claim to offer A|B Testing in their analytics for apps? How would one implement that?

Answer 1

Artisan mobile makes an A/B testing solution for iOS and Android.

The basic idea is that you drop the SDK in your app and then put it out in the app store. You can use the service to create A/B tests and optimize your application without having to touch the code or go back through the app store for each test.

Answer 2

For mobile apps, A/B testing basically works by replacing static, hard-coded objects with dynamic objects that can be controlled from a remote server.

This methodology raises a potential performance issue: What if the end user's device is not connected to pull configuration data for an object being tested? We've built Splitforce (http://splitforce.com) to seamlessly setup and manage A/B testing in mobile apps while controlling for performance risk.

Los details

Once the SDK and experiment has been integrated, non-technical product or marketing folks can setup new tests or tweak existing tests on-the-fly - without having to resubmit to the app stores or hassle engineers.

On first app launch, the mobile app requests configuration data from the server and then caches that data locally on the device. This is to both ensure a consistent user experience on subsequent app launches, and prevent corrupt test results by guaranteeing accurate attribution of conversion events to variations.

If the end user's connection fails or is timed-out on first app launch, the library displays a hard-coded 'default' variation. And to make sure that everything is looking good before you go live, we've built a 'shake to preview' functionality in debug mode that does just that :-)

Once the app is deployed with Splitforce event data are stored locally and sent back to the website to be displayed for each variation alongside measurements of observed improvement and statistical confidence.

Instructions on integration of the SDKs and new tests can be found at https://splitforce.com/documentation.

And how is it used?

We've seen Splitforce used to A/B test: UI elements + layouts (color, text, images, ad/menu placements) UX workflows Game dynamics + rules Prices + promotions

We've also seen the tool used to control mobile apps remotely, by essentially setting one variation of a test subject to 100%.