Latent Dirichlet Allocation Solution Example

Question

I am trying to learn about Latent Dirichlet Allocation (LDA). I have basic knowledge of machine learning and probability theory and based on this blog post http://goo.gl/ccPvE I was able to develop the intuition behind LDA. However I still haven't got complete understanding of the various calculations that goes in it. I am wondering can someone show me the calculations using a very small corpus (let say of 3-5 sentences and 2-3 topics).

Answer 1

Edwin Chen (who works at Twitter btw) has an example in his blog. 5 sentences, 2 topics:

• I like to eat broccoli and bananas.
• I ate a banana and spinach smoothie for breakfast.
• Chinchillas and kittens are cute.
• My sister adopted a kitten yesterday.
• Look at this cute hamster munching on a piece of broccoli.

Then he does some "calculations"

• Sentences 1 and 2: 100% Topic A
• Sentences 3 and 4: 100% Topic B
• Sentence 5: 60% Topic A, 40% Topic B

And take guesses of the topics:

• Topic A: 30% broccoli, 15% bananas, 10% breakfast, 10% munching, …
  • at which point, you could interpret topic A to be about food
• Topic B: 20% chinchillas, 20% kittens, 20% cute, 15% hamster, …
  • at which point, you could interpret topic B to be about cute animals

---

Your question is how did he come up with those numbers? Which words in these sentences carry "information":

• broccoli, bananas, smoothie, breakfast, munching, eat
• chinchilla, kitten, cute, adopted, hampster

Now let's go sentence by sentence getting words from each topic:

• food 3, cute 0 --> food
• food 5, cute 0 --> food
• food 0, cute 3 --> cute
• food 0, cute 2 --> cute
• food 2, cute 2 --> 50% food + 50% cute

So my numbers, differ slightly from Chen's. Maybe he includes the word "piece" in "piece of broccoli" as counting towards food.

---

We made two calculations in our heads:

• to look at the sentences and come up with 2 topics in the first place. LDA does this by considering each sentence as a "mixture" of topics and guessing the parameters of each topic.
• to decide which words are important. LDA uses "term-frequency/inverse-document-frequency" to understand this.