When testing an iPhone application, is it enough to just use the Leaks instrument?

Question

I am about to finish my first iPhone application, and I'm am wondering if there are a set of steps that are used to check the app for memory leaks, performance, etc.?

Is checking with the Leaks instrument enough?

Are there any series of tests that need to be run? Is there any tutorial/document that you guys can point me to?

Answer 1

Running with "Leaks" is important. I'm not aware of a tutorial/checklist for final testing, though something like that would be handy. A couple things I would add:

1) Be sure to test with actual hardware and not just the simulator to make sure that performance is reasonable on ALL hardware you intend to support. In my experience, the simulator does not give you an accurate feel for the performance of the device and there can be significant differences between older and newer hardware (the extreme example being iPhone 4 vs. Gen1 iPhone). For example, in one of my apps, I generate a 1-page PDF report. On iPhone 4 and even iPad, it takes about 1 second. On Gen1 iPhone, the same code takes close to 8 seconds. There wasn't much I could do to speed it up, but it was clear I needed to add a progress indicator to let the user know that the app was not frozen. This is something I would not have noticed running with just the simulator and/or the latest hardware.

2) You might want to spend a little time running with NSZombieEnabled. This can find memory issues that may be lurking behind the scenes even if there is currently no visible sign of trouble. More information:

http://www.cocoadev.com/index.pl?NSZombieEnabled

Answer 2

Testing with the Leaks instrument should be part of your strategy, but not all of it. You will want to test your application from multiple angles.

My strategy towards testing tends to focus first on functional testing, followed by performance tests, and then a last round of functional tests. There's very little point in tuning performance if you have a crashing bug somewhere in your code, unless that crash is due to resource exhaustion of some kind.

Hammer on the application to try and make it break by running through every option under every condition you can think of. If that passes, I usually employ my "crazy monkey on crack" test where I hammer random buttons and areas on the screen as fast as I can to see if I expose any further crashers.

Only then do I turn to Instruments. Run the application on the device (no performance tuning should ever be done in the Simulator) using the Time Profiler and Memory Monitor instruments. Look for both performance hotspots and memory spikes, as well as memory accumulations. Repeat the same sort of testing you used for functional issues earlier while doing this.

Once you deal with the hotspots and obvious buildups, you can step down to a finer-grained examination of memory. I actually prefer using the Object Allocations instrument with its new heapshot analysis capability to the Leaks instrument for finding subtle memory buildups and leaks. The Leaks instrument tends to be conservative, and can miss some buildups. Nathaniel points out Bill Bumgarner's excellent post on the subject.

The Object Allocation instrument and its heapshots are particularly powerful when combined with the UI Automation instrument, where you can do hundreds or thousands of cycles of testing within parts of your application to make even the tiniest memory accumulation stand out. I've started to do more of this kind of testing now.

I think it works best to see this in action rather than described in text, so I'd recommend watching the video for my "Testing" and "Performance tuning" classes as part of my advanced iOS course on iTunes U. I demonstrate each of these tools and how I use them in the testing of my own applications before App Store submission. My course notes (in VoodooPad format) also describe this in detail.