What is Oracle experiment?

Question

I have read a paper about machine learning and it contains an Oracle experiment to compare between his study and another study? But it does not seem to be so clear what is Oracle experiment?

Answer 1

An "oracle" is an imaginary entity that always gives the right answer. An oracle experiment is used to compare your actual system to how your system would behave if some component of it always did the right thing.

For example, in the NLP domain, let's assume you built a parser that takes part-of-speech (POS) tagged sentences as input. In the real world, you would have to run real sentences through an actual POS tagger. This tagger would probably produce results with accuracy above 90%, but less than 100%. Since the accuracy of your parser depends on the accuracy of the incoming tags, your parser's performance will be negatively affected by this loss.

In order to see how well your parser would perform if the POS tagger was perfect, you could run an experiment with an oracle tagger. In this experiment, you would replace the real POS tagger with a program that knows the actual POS tags for the sentences, thus always returning tag results with 100% accuracy.

So, if your parser gets 85% accuracy in an experiment with a real tagger, and 90% in an experiment with an oracle tagger, then you know that 5% of your performance loss is directly due to the mistakes of the tagger.