Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can someone explain to me the difference between a cost function and the gradient descent equation in logistic regression?

I'm going through the ML Class on Coursera on Logistic Regression and also the Manning Book Machine Learning in Action. I'm trying to learn by implementing everything in Python.

I'm not able to understand the difference between the cost function and the gradient. There are examples on the net where people compute the cost function and then there are places where they don't and just go with the gradient descent function w :=w - (alpha) * (delta)w * f(w).

What is the difference between the two if any?

like image 934
oktapodi Avatar asked Nov 29 '12 09:11

oktapodi


People also ask

What is gradient descent algorithm and how it helps in minimizing the cost function?

Learn about gradient descent, an optimization algorithm used to train machine learning models by minimizing errors between predicted and actual results.

What is cost function logistic regression?

Non-convex function. For logistic regression, the Cost function is defined as: −log(hθ(x)) if y = 1. −log(1−hθ(x)) if y = 0.

What is the difference between gradient descent and linear regression?

Simple linear regression (SLR) is a model with one single independent variable. Ordinary least squares (OLS) is a non-iterative method that fits a model such that the sum-of-squares of differences of observed and predicted values is minimized. Gradient descent finds the linear model parameters iteratively.


1 Answers

Whenever you train a model with your data, you are actually producing some new values (predicted) for a specific feature. However, that specific feature already has some values which are real values in the dataset. We know the closer the predicted values to their corresponding real values, the better the model.

Now, we are using cost function to measure how close the predicted values are to their corresponding real values.

We also should consider that the weights of the trained model are responsible for accurately predicting the new values. Imagine that our model is y = 0.9*X + 0.1, the predicted value is nothing but (0.9*X+0.1) for different Xs. [0.9 and 0.1 in the equation are just random values to understand.]

So, by considering Y as real value corresponding to this x, the cost formula is coming to measure how close (0.9*X+0.1) is to Y.

We are responsible for finding the better weight (0.9 and 0.1) for our model to come up with a lowest cost (or closer predicted values to real ones).

Gradient descent is an optimization algorithm (we have some other optimization algorithms) and its responsibility is to find the minimum cost value in the process of trying the model with different weights or indeed, updating the weights.

We first run our model with some initial weights and gradient descent updates our weights and find the cost of our model with those weights in thousands of iterations to find the minimum cost.

One point is that gradient descent is not minimizing the weights, it is just updating them. This algorithm is looking for minimum cost.

like image 100
Reihan_amn Avatar answered Sep 29 '22 16:09

Reihan_amn