wit.ai: how does it identify intent and classifies entities from user expressions

Question

I have been studying wit.ai for several days. I have found the key points of wit.ai bot engine:

• Story based - create story for greeting, order pizza, order laptop, ask forecast
• Role based entity - location:form, location:to. Here “from” and “to” are role of “location” entity
• Composite/nested entity - car(model, color, modelYear). Here model, color, modelYear can be nested under car entity
• Search strategies: trait, free-text, keywords
• Understanding bot by creating some stories
• Scoring on match termed as confidence
• User expression length 256 at max
• Search from predefined list of keywords, expressions for match
• Nested context
• User defined entities, predefined entities
• Actions based on entities: if has only, if always has conditions
• For a given user expression wit searches for match in keyword list, free-text
• For a given user expression wit searches keywords position in the listed expressions under an entity
• Branching for missing information in given user expression
• Pronoun support in wit? No Story #1 User: How much is Widget X 2000? Bot: It costs $30. Story #2 User: Where can I buy Widget X 2000? Bot: At your local Best Buy. Story #2 User: How much is Widget X 2000? Bot: It costs $30. User: Where can I buy it? --error-- Use context
• Conversation-aware entity extraction is yet to be implemented
• Is it possible to make a wit.ai bot remember/reuse a context across stories? Check if context.key exists or not
• Is it possible to set a default intent in Wit.ai? No, confidence value can be checked, set threshold for confidence, below the threshold specific response
• Can you rank entities by priority in Wit.ai? No

Now I want to know how does wit detect the intent of user expression and classifies the entities using the stories of the created bot in wit.ai.

If anyone have understood the underlying technique/ML algorithms used in wit.ai please share in this thread. I hope it will be helpful for people like me.

Thanks in advance.

Answer 1

The two main part of wit.ai are:

1. intent classification
2. entity extraction

For entity extraction it uses duckling library that they recently open sourced it, and you can find detailed description on the algorithm there.

For intent classification, I guess they use text classification methods that are base on bag of words or more advanced word embedding methods like word2vec.

You can also take a look at similar fully open source projects like Rasa, or spaCy for both above tasks.