Python NLTK: How to retrieve percentage confidence in classifier prediction

Question

I am currently training an NLTK classifier to recognize motion commands. These commands can include "move left", "please move forward", "halt!", "move towards the right", etc.

I am currently using the classifier based on a few key features (such as the existence of "halt" and "left") to classify information, and it works fine.

However, let us presume the following text, "move to the left right", is given. In this case, both keywords are conflicting each other, and presumably, the classifier should have a low confidence level when offering its resultant prediction.

As such, is there any way to retrieve the confidence of its predicted "direction", after using <CLASSIFIER>.classify() ?

NOTE: I have tried to use nltk.classify.accuracy() but it is only for use on a test data-set, not a single query.

Answer 1

Some classifiers in NLTK have a prob_classify method, which returns a probability distribution over all possible outcomes. From this you can compute a confidence scores such as odds or log-odds. For a percentage score, you can just take the probability of the most likely outcome.