iOS app crash rate - background noise level?

Question

We've just released an app using the Crittercism framework. After some time, we've had about 125K app loads, and 95 crashes - a rate of less than 0.08%.

One crash happened 19 times, another 10, but the other 41 all occurred 3 or less. If there were any major problems with the app, I would expect to see significantly more failures in particular areas, so I'm happy with the level of figures I'm seeing.

A quick look shows many of them to be low level failures, not obviously caused but programmer error.

Examples

• The largest group are all to do with CFNetworking on a background thread while static HTML is being being rendered in a web view on the main thread.
• There are some KVO failures in free_list_checksum_botch

But my question is, in a sufficiently complex OS (iOS in this case), with a sufficiently complex app (which I think it is), should I, as a developer, expect to see this level of "background noise"?

Should I expect to see one app crash per 1-2000 loads, just because the OS isn't perfect? Has anyone else had a similar experience?

(I'm not looking for solutions to the errors themselves.. thanks!)

Answer 1

Crittercism conducted an analysis on app crashes. Their report was based on Android vs iOS crashes.

They concluded that the most popular apps on iOS crash 0.51% of app launches. So @Ashley Mills, if you're getting 0.08%... you're doing well. (i think I have my figures correct, anyway).

Not sure where the original report is, but I read it here:

Forbes app crash rates, conducted by Crittercism

Answer 2

Actually another shot in the dark non-technical answer could be. What added value will it bring you (the developer) to keep digging into that particular issue, when you could be spending more time and effort either developing more capability within your tool, or another tool completely.

If your app is simply for fun and learning, then I could see digging into this issue as a fun adventure. From a business perspective, what is your time worth and is figuring out this 0.08% problem going to sell enough (more) copies to make your efforts worth it.

An analogous would be, what requirements are essential, and what requirements are desirements? Just food for thought. I know many of the companies I have worked for don't see the value in stressing about that low of a yielding bug.

Answer 3

I am an iOS developer, employed professionally. I take it personally when my apps crash, because that is not the user experience I was aiming for. A crash is a bad user experience. One crash, per user, is too many. A crash is a bug.

That being said, I have definitely seen crash logs that appear to be unsolvable because they seem to be indicating a problem way down deep in the SDK. What I have learned, however, is that more than likely there is something about my own code that is ultimately the cause.

There are any number of bizarre crashes that can be caused by timing issues between threads or blocks or just because I did something wrong. Just recently I discovered I was doing something entirely wrong with respect to a complex table I was updating. The crash logs for this problem provided almost no clues except for the general area of code I might look at. As I dug into the code and started experimenting, I realized my mistake, which ultimately was a timing issue caused by what I thought was a clever separation of main-thread vs. non-main-thread activity. I was too clever for my own good, in this case. :-)

So, to sum up:

• One crash is too many crashes and ultimately a bad user experience.
• Often, bizarre low-level crashes are the result of your own code's complexity and possibly timing issues there in.

Finally, I offer this question to consider:

• Are you willing to piss off or dismiss some of your users simply because they fall into the 0.08% of users experiencing crashes?

Food for thought. :-)

Answer 4

I'm a professional iPhone developer, and what I've seen is the crash frequency isn't what upsets users, it's the funnel for how the crashes happen.

If they are intermittent you are usually okay, the problem starts to occur when one user experiences a specific crash multiple times. This is unacceptable.

Striving to remove every crash is always a good thing, but in many cases it's simply not realistic, you have to decide where your time is best spent. If you have the opportunity to rework a portion of the UX flow or fix an intermittent crash, you need to decide which one benefits more users.

The important thing to remember is, if you choose not to fix a crash that happens 0.08% of the time, you aren't writing off the users experiencing it unless they are experiencing it multiple times.