python linear regression implementation

Question

I've been trying to do my own implementation of a simple linear regression algorithm, but I'm having some trouble with the gradient descent.

Here's how I coded it:

def gradientDescentVector(data, alpha, iterations):
    a = 0.0
    b = 0.0
    X = data[:,0]
    y = data[:,1]
    m = data.shape[0]
    it = np.ones(shape=(m,2))
    for i in range(iterations):
        predictions = X.dot(a).flatten() + b

        errors_b = (predictions - y)
        errors_a = (predictions - y) * X

        a = a - alpha * (1.0 / m) * errors_a.sum()
        b = b - alpha * (1.0 / m) * errors_b.sum()
    return a, b

Now, I know this won't scale well with more variables, but I was just trying with the simple version first, and follow up from there.

I was following the gradient descent algorithm from the machine learning course at coursera:

But I'm getting infinite values after ~90 iterations (on a specific dataset), and haven't been able to wrap my head around this so far.

I've tried iterating over each value before I learned about numpy's broadcasting and was getting the same results.

If anyone could shed some light on what could be the problem here, it would be great.

Answer 1

It is clear that the parameters are diverging from the optimum ones. One possible reason may be that you are using too large a value for the learning rate ("alpha"). Try decreasing the learning rate. Here is a rule of thumb. Start always from a small value like 0.001. Then try increasing the learning rate by taking a three time higher learning rate than previous. If it gives less MSE error (or whatever error function you are using), then its fine. If not try taking a value between 0.001 and 0.003. Next if the latter hold true, then try this recursively until you reach a satisfying MSE.