Logistic regression: one-vs-all method for multi classification

Question

I'm new to Machine learning. When I learn about the Logistic regression, using one-vs-all (one-vs-rest) method for multiclassification: In logistic regression, the hypothesis function try to estimate the probability of the positive class. Assume we have 3 classes, then each class, we should predict the hypothesis function h(x)

h1(x)=P(y=1|x)
h2(x)=P(y=2|x)
h3(x)=P(y=3|x)

However,the sum of the three probabilities doesn't equal to 1? I "feel" that it equal to 1, and I don't understand why it doesn't. Can someone explain why?

Answer 1

Your results are correct and the sum of h1(x), h2(x) and h3(x) shouldn't be equal to 1.

As you perform one-vs-all classification, then for each class (e.g., class 1) you have two probabilities p(y=1|x) and p(y!=1|x) which sum up to 1:

p(y=1|x) + p(y!=1|x) = 1.

But, as you one-vs-all classifications are independent, then

p(y!=1|x) != p(y=2|x) + p(y=3|x) (at least not necessarily).

Maybe, it is easier to understand with an example:

• the first classifier says that p(y=1|x) = 0.7 and p(y!=1|x) = 0.3;
• the second classifier says that p(y=2|x) = 0.7 and p(y!=2|x) = 0.3;
• the third classifier says that p(y=3|x) = 0.7 and p(y!=3|x) = 0.3.

All of them are valid classifiers, but

p(y=1|x) + p(y=2|x) + p(y=3|x) != 1.