Menu
Just like we use the Normal Equation to find out the optimum theta value in Linear Regression, can/can't we use a similar formula for Logistic Regression ? If not, why ? I'd be grateful if could someone could explain the reasoning behind it. Thank You.
549
Unfortunately no, only two methods in classification theory have closed form solutions - linear regression and linear discriminant analysis/fischer discriminant.
First, logistic regression does not require a linear relationship between the dependent and independent variables. Second, the error terms (residuals) do not need to be normally distributed. Third, homoscedasticity is not required.
Normal Equation is an analytical approach to Linear Regression with a Least Square Cost Function. We can directly find out the value of θ without using Gradient Descent. Following this approach is an effective and time-saving option when working with a dataset with small features.
For logistic regression, there is no longer a closed-form solution, due to the nonlinearity of the logistic sigmoid function.
Unfortunately no, only two methods in classification theory have closed form solutions - linear regression and linear discriminant analysis/fischer discriminant.
In general it is considered a miracle that it "works" even for linear regression. As far as I know it is nearly impossible to prove that "you cannot solve logistic reggresion in closed form", however general understanding is that it will not ever be the case. You can do it, if your features are binary only, and you have very few of them (as a solution is exponential in number of features), which has been shown few years ago, but in general case - it is believed to be impossible.
So why it worked so well for linear regression? Because once you compute your derivatives you will notice, that resulting problem is set of linear equations, m equations with m variables, which we know can be directly solved through matrix inversions (and other techniques). When you differentiate logistic regression cost, resulting problem is no longer linear... it is convex (thus global optimum), but not linear, and consequently - current mathematics does not provide us with tools strong enough to find the optimum in closed form solution.
That being said there exists (absolutely impractical computationally) closed form solution if all your input variables are categorical (they can only take finitely many values that you can enumerate): https://www.tandfonline.com/doi/abs/10.1080/02664763.2014.932760?journalCode=cjas20
67
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
