Shannon's Entropy measure in Decision Trees

Question

Why is Shannon's Entropy measure used in Decision Tree branching?

Entropy(S) = - p(+)log( p(+) ) - p(-)log( p(-) )

I know it is a measure of the no. of bits needed to encode information; the more uniform the distribution, the more the entropy. But I don't see why it is so frequently applied in creating decision trees (choosing a branch point).

Answer 1

Because you want to ask the question that will give you the most information. The goal is to minimize the number of decisions/questions/branches in the tree, so you start with the question that will give you the most information and then use the following questions to fill in the details.