Are there any good strategies for A/B testing on mobile devices?

Question

Based on what I know, A/B testing frameworks are for working with websites. For example, you want to know which of blue/red color of "Buy" button will encourage user to click it.

Since website information of user interface comes from server, and browsers display it, A/B testing is possible. However, since the user interface of mobile applications are native and static, it's not possible to implement A/B testing in the same manner (if I'm not mistaken).

So, my question is how to implement an A/B testing framework from the database/services all the way through to a native mobile application.

Any suggestion would be appreciated. Thanks.

Answer 1

Unlike on the web, changing content on mobile apps is a more involved process. Specifically: all code updates to your app have to be approved by Apple/Google.

Enter services such as Leanplum. Leanplum allows its customers to test nearly every aspect of their app in three primary ways:

1. Visual Interface Editor: This requires no coding, and Leanplum will automatically detect the elements and allow you to change them. No engineers or app store resubmissions required.
2. Content Management System: Enables more granular change to the app.
3. Mobile Marketing Automation: A full-service solution where every message and notification sent to your users (whether in-app, push, email, or in the news feed) can be A/B tested and personalized.

For more information on best practices for mobile A/B testing, you can check out this Leanplum blog post.

To learn more, check out our site at leanplum.com. We offer free 30-day trials of our platform, so you can experiment with building better brand relationships with your customers, without any commitment.

(Disclaimer: I am an Engineer at Leanplum.)

Answer 2

A/B testing on mobile is different from website in two main points:

• A website is always connected to the internet. A native mobile app is like a Windows app in the 90's: it does not always have a reliable internet connection
• The web is much more mature than the mobile. Modern web A/B testing tools can completely control all the elements on a web page.

The basic thing to do is to create a client-server architecture:

Tag manually the elements to test in your app

You will tag manually the element that you would like to A/B test and your framework will ask to the server which element to show. (The variations are stored on the server side). For example, the color of the button can be retrieved from the server and set at runtime when the button is displayed. It is not completely true that native mobile interfaces are static. They can be modified at runtime programmatically.

Send events to your server

The second step is to send an event to your server when the variation has been viewed and when the conversion succeeds.

Potential issues

By using this trivial architecture you will have the following problems:

Always show something

When the button should be displayed, the mobile device might not have an internet connection at all. You still need to show something to your user event if the internet connection is not there. The solution is to display a default value.

Store events

When a goal is performed (for example a click on a button) and if the mobile device is offline, the event will be dropped and you will end with wrong statistics. The solution is to save your events locally and to send them to the server later (when the internet connection is back).

Never wait for a variation

If you ask a variation from the server only when you need it, you application will be really slow (which is not acceptable). You need to retrieve your variation data as soon as possible in your application.

You mobile framework should be built to never block the application. That's a golden rule on mobile.

Save variation attributions

The server should save which variation has been assigned to a device to get reliable results.

Technology stack

Operating system/programming language

You can choose any technology stack. Just make sure that your server side can handle the load. If your mobile apps have many users, you will get many requests on your server. You might need to have a load balancer between your web servers. If you choose non-free technologies, watch out for licensing cost as high load means high cost.

Database

You might also have millions to database records. Generating reports might not be just easy as doing a "SELECT COUNT(*)" from you database table for performance issues. Be careful if you choose non-free technologies.

Disclaimer: I am the CTO of Arise.io