How many Android devices do I need for thorough testing?

Question

I am working on an Android app that I want to run on all Android devices without any problems, but I'm having a hard time trying to determine how many device configurations and how many hardware devices I should have for testing.

There are several features that I am using which are not supported in the emulator, as mentioned at the bottom of Using the Android Emulator in the Dev Guide:

Emulator Limitations

In this release, the limitations of the emulator include:

• No support for placing or receiving actual phone calls. You can simulate phone calls (placed and received) through the emulator console, however.
• No support for USB connections No support for camera/video capture (input).
• No support for device-attached headphones No support for determining connected state
• No support for determining battery charge level and AC charging state
• No support for determining SD card insert/eject
• No support for Bluetooth

The Multiple APK Support page states:

...we encourage you to develop and publish a single APK that supports as many device configurations as possible...

Android Market allows you to publish multiple APKs under the same application listing. Android Market then supplies each APK to the appropriate devices based on configuration support you've declared in the manifest file of each APK.

I would assume that it's not necessary to cover every single permutation of all the available feature sets, but it's not clear to me how many device configurations would be considered sufficient to cover all the devices available.

If I wanted to confirm on hardware that my app is almost guaranteed to work on every device, how many phones and how many tablets would I need? Also, are there any special considerations I need to make for manufacturer- or carrier-specific modifications?

Answer 1

Answer Updated on 11/15/2011

1. Use android website stats to glean info on the most popular os version(s).

2. Use android website stats to get the most popular DPI/screensize combination(s).

3. Use a combination of 1&2 to determine the specs of devices you want to cover.

4. Prepare a rough listing of devices which would meet (3) above.

5. If you have A lot of money, -or- if you work for a big company then order the phones. If not, then as others have already mentioned choose the one device which can provide you with max coverage. Sadly, the difference between the emulator and devices can be ridiculous at times. The other option that i found out only recently is that you can outsource this process to companies which exclusively focus on this market need. Unfortunately, I don't have any specific company to recommend in this area. Good luck !

Updates:

• I came across a couple of companies that are offering Android Testing as a Service. These would be good options to test your APKs without having to purchase the devices.

• You may want to give APKUDO a try: http://www.apkudo.com/ " See your app run on every Android device. Free. Apkudo helps developers release with confidence by installing, executing, and exercising their apps on every Android™ device, with detailed test reports & streaming video. Currently in private beta. "

[Update - 3/25/2013]

• TestDroid also has a cloud testing service http://testdroid.com/product/testdroid-cloud " Test online on Real devices Testdroid Cloud is the simplest way to test your application against various real Android devices – from different manufacturers, with different HW platforms, OS versions and screen resolutions. You can also pick and choose on what devices to run your tests, just on the devices you are interested in "

• AppThwack is another solution

• Ranorex is yet another solution (http://www.ranorex.com/mobile-automation-testing/android-test-automation.html)

Answer 2

One of the big problems with android is that each manufacturer may change pretty much whatever he wants in the firmware. It may have many advantages, but it might be a big problem when trying to test compatibility.

You should have at least one device, but there is no limitation, or even a number that is "enough", the more devices you test on - the better.

Regarding carrier modifications, usually those are smaller and will change less than manufacturer modifications, but this might change as well.

Answer 3

I have encountered many issues with many different phones for Android. Each phone has its little quirks and so the more phones you test on the better. I had an app that had well over 200K downloads and was really surprised about some of the issues. Now there are things I do and stay away from due to these quirks I discovered. Also each version of Android has its issues to. So I guess my answer is, get as many phones from at least each of the versions of OS's from say 1.6 to 3.x.

ListViews with custom Adapters is one, I created custom adapter to add icons to each item in a ListView. I had repeated crashes from 2.2 phones like DroidX to name one but on others it worked fine. It ran fine on a G1 too. Also with ListViews and SimpleExpandableListAdapter are fraught with issues on other phones where the icons don't line up or on large lists it just crashes the app. I stay away from ListView now since I want to try and keep my apps deployable across many different OS's

Pertaining to the other answer here are the list of phones I encountered for mostly US deployment. Not sure why the Samsung Galaxy S is listed 3 times or Motorola Droid II is twice, may have to do with OS versions.

1   Motorola Droid X     9.3% 
2   HTC Evo 4G           6.2% 
3   Samsung Galaxy S     4.9% 
4   HTC Desire HD        4.4% 
5   Motorola Droid       4.4% 
6   HTC Droid Incredible 4.1% 
7   Motorola Droid II    3.4% 
8   Samsung Galaxy S     3.3% 
9   Samsung Galaxy S     2.7% 
10  Motorola Droid II    2.4% 

This is why when I program I keep in mind to be agnostic to the OS API and just try to make sure it works for the majority of the users.