iPhone OS Testing Best Practices

Question

Not so long ago iPhone development was quite simple, only a few OS versions and even less devices.

Now however, there are 2 major OS versions and 5 different devices to consider.

As a company about to release several applications testing has been become more and more of an issue.

What are the best ways to test all combinations, do I need to acquire every generation of iPhone and iPod Touch? Are there any gotchas with specific hardware/OS combinations I might need to code around?

I guess my question is, "What's the minimum amount of testing required to cover all the bases?"

Answer 1

In my experience, you won't have much compatibility trouble between iPod/iPhone. There are other gotchas to be aware of:

• The devices run at different speeds. iPhone v1 and iPhone 3G run at 412MHz; iPod Touch runs at 532MHz, and the new 3GS runs at 600MHz. This can have a big impact on performance and even functionality if you're getting fancy.
• There's a huge performance difference between EDGE/3G/WiFi networks; often the differences are counter-intuitive. EDGE can often have better latency (time to first byte) than 3G, while 3G has 10x better bandwidth. You'll want to test your app under all three conditions.
• Are you using Core Location? iPhone v1 and iPod Touch do not have GPS.
• Are you using the camera? The iPod Touch does not have a camera.
• Is your app compatible with jailbroken phones? A lot of people have done it, and if your app crashes on them, they will blame you, not the Dev Team hackers, and this will be reflected in your App Store ratings. Note especially that background apps can use up memory that you might have thought would be available exclusively for your app. Leave yourself some overhead.

So, what do you need to buy? If you're a serious developer, yes, you should have all 5 devices available. But do you need to test every build on all 5 devices? Does every one of your developers need 5 devices each? No.

One developer can probably test everything that matters with one iPhone 3G and an iPod Touch. Toss in a 3GS and your coverage is probably nearly perfect. (Note that development on 3GS is much nicer just because the CPU is faster, so your apps deploy more quickly.)

At work we have one device per developer, but they're a mix of 3G and iPod devices (and, today, one 3GS).

Answer 2

I don't have much experience with the iPhone itself, but in general this might be a good application of pairwise testing. In practice, you can get 90% coverage with a small fraction of the testing of an exhaustive test pass. Then later if you find certain configurations have gotchas you can add them to your set of configurations and still not need to do an exhaustive pass.