OCR for Equations and Formulae on the iOS Platform (Xcode)

Question

I'm currently developing an application which uses the iOS enabled device camera to recognise equations from the photo and then match these up to the correct equation in a library or database - basically an equation scanner. For example you could scan an Image of the Uncertainty Principle or Schrodinger Equation and the iOS device would be able to inform the user it's name and certain feedback.

I was wondering how to implement this using Xcode, I was thinking of using an open-source framework such as Tesseract OCR or OpenCV but I'm not sure how to apply these to equations.

Any help would be greatly appreciated. Thanks.

Answer 1

Here's the reason why this is super ambitious. What OCR is doing is basically taking a confined set of dots and trying to match it to one of a number of members of a very small set. What you are talking about doing is more at the idiom than the character level. For instance, if I do a representation of Bayes' Rule as an equation, I have something like:

P(A|B) = P(B|A)P(A)/P(B)

Even if it recognizes each of those characters successfully, you have to have it then patch up features in the equation to families of equations. Not to mention, this is only one representation of Bayes Rule. There are others that use Sigma Notation (LaPlace's variant), and some use logs so they don't have to special case 0s.

This, btw, could be done with Bayes. Here are a few thoughts on that:

1. First you would have to treat the equations as Classifications, and you would have to describe them in terms of a set of features, for instance, the presence of Sigma Notation, or the application of a log.
2. The System would then be trained by being shown all the equations you want it to recognize, presumably several variations of each (per above). Then these classifications would have feature distributions.
3. Finally, when shown a new equation, the system would have to find each of these features, and then loop through the classifications and compute the overall probability that the equation matches the given classification.

This is how 90% of spam engines are done, but there, they only have two classifications: spam and not spam, and the feature representations are ludicrously simple: merely ratios of word occurrences in different document types.

Interesting problem, surely no simple answer.