What are the common methods to determine intent

Question

Many NLP APIs offer intent extraction like API.ai and wit.ai. However I'm unclear about their details. Do they do dependency parsing then extract relations, or simply taking out keywords from a sentence? How to parse "check if tomorrow is going to rain"?

Answer 1

There are a handful of approaches that I know of. They can be used together as an ensemble that outputs a score.

(1) Map intent to string literals. Compare these string literals for an exact match, or cosine similarity.

(2) Narrow down scope of possible intents based on context.

(3) Regex matches: if a sentence contains a characteristic regex (like phone number), then it can at least "narrow the scope" of intents to search for.

(4) Word Movers Distance: It is like word embeddings (i.e. deep learning NLP), but the whole sentence is passed in, and the aggregate distance from another sentence is compared.

(5) Use bidirectional LSTM: See tutorial or tensorflow.

(6) Keep a list of "candidate intents" using Named Entity Recognition (NER). spaCy does this. Even better is to use it for subject-object extraction.

(7) Use "fallback intents" if one isn't found. This could refer to "hierarchical intents" where the bottom-level leaves represent intents as you are referring to. This could also refer to an intent along the lines of "the bot has no idea what to say".