Apple TestFlight: Is it allowed to upload 2 beta apps for an A/B test?

Question

I need to run a beta test for an iOS 8 app through Apple TestFlight. Is it allowed to upload 2 versions of a beta app for an A/B test? (I would like to switch the app icon and some other things to check what works better for the users.)

---

Notes on bounty:

Especially interested if there are any experiences with submitting two similar builds for external testing, as builds have to go through the (albeit lighter) review process.

I would presume it would be OK, as one can submit freeform notes for the reviewers and explain the situation—AND because, in any case, externally tested apps have to go through the normal App Store review process before going live; so acceptance in external testing wouldn’t be a free ticket to App Store.

But these are just my assumptions and hence the bounty. Has someone done this? With negative or positive results? Or both—review processes are notoriously independent and, at times, arbitrary.

I’m not interested in interpretations of review guidelines, but how they are executed in practice.

Or should this all be handled inside the app with some A/B testing framework? (Which is unfortunately more work than just creating two bundle IDs… And this wouldn’t help in testing the app icon)

Answer 1

In order to have 2 active builds in test you need 2 apps (different bundle ids), the only choice you have is either through the same account or different accounts. It's easier to handle in the same account, less context switching for management.

We did both successfully without any issue raised from Apple. We did not put any special mentions in the review notes either.

Eventually you end up with more dead projects in your account but its only an issue for you, Apple wont complain.

Why tough

A/B testing can be done in the same build except for some things like you mentioned (app icon). Usually this is handled through surveys tough.

The only advantage of having 2 apps is that all users can test both versions in your A/B test mitigated by their willigness to spend time testing both.

Disadvantages that jumps to mind:

• sooner or later you will kill one of those version, potentially loosing some of your beta testers that chose only that version. Mitigated by the fact that you may not need beta testers after you release your appstore version
• you wont be able to control who has which version to insure proper distribution and move users from one to the other unless you are ready for lots of hands on communicating with your beta testers.

Answer 2

I have not been able to have multiple versions in TestFlight. If I were to do this I would use a separate provisioning file and iTunes Connect account. There are a few steps to change the provisioning account and the app name so it is unique, but I think that is the only way to get around it. However to use external testers (since internal testers are limited to 25) you will have to go through the review process. Apple might object to having to review the same app twice if they caught it. I would advise rereading the Terms and Conditions, remembering you are submitting for Beta App Review.

Answer 3

As of April 11, 2017, Testflight allows you to you can distribute and test multiple builds at the same time.

https://developer.apple.com/news/?id=04112017a