Linear Regression model On tensorflow can't learn bias

Question

I am trying to train a linear regression model in Tensorflow using some generated data. The model seems to learn the slope of the line, but is unable to learn the bias.

I have tried changing the no. of epochs, the weight(slope) and the biases, but every time , the learnt bias by the model comes out to be zero. I don't know where I am going wrong and some help would be appreciated.

Here is the code.

import numpy as np
import tensorflow as tf

# assume the linear model to be Y = W*X + b
X = tf.placeholder(tf.float32, [None, 1])
Y = tf.placeholder(tf.float32, [None,1])
#  the weight and biases
W = tf.Variable(tf.zeros([1,1]))
b = tf.Variable(tf.zeros([1]))
# the model
prediction = tf.matmul(X,W) + b
# the cost function
cost = tf.reduce_mean(tf.square(Y - prediction))
# Use gradient descent

learning_rate = 0.000001
train_step = 
tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
steps = 1000
epochs = 10
Verbose = False
# In the end, the model should learn these values
test_w = 3
bias = 10

for _ in xrange(epochs):  
    for i in xrange(steps):
    #     make fake data for the model
    #     feed one example at a time
#     stochastic gradient descent, because we only use one example at a time
        x_temp = np.array([[i]])
        y_temp = np.array([[test_w*i + bias]])
    #     train the model using the data
        feed_dict = {X: x_temp, Y:y_temp}
        sess.run(train_step,feed_dict=feed_dict)
        if Verbose and i%100 == 0:
            print("Iteration No: %d" %i)
            print("W = %f" % sess.run(W))
            print("b = %f" % sess.run(b))

print("Finally:")
print("W = %f" % sess.run(W))
print("b = %f" % sess.run(b))
# These values should be close to the values we used to generate data

https://github.com/HarshdeepGupta/tensorflow_notebooks/blob/master/Linear%20Regression.ipynb

Outputs are in the last line of code. The model needs to learn test_w and bias (in the notebook link, it is in the 3rd cell, after the first comment), which are set to 3 and 10 respectively.

The model correctly learns the weight(slope), but is unable to learn the bias. Where is the error?

Answer 1

The main problem is that you are feeding just one sample at a time to the model. This makes your optimizer very inestable, that's why you have to use such a small learning rate. I will suggest you to feed more samples in each step.

If you insist in feeding one sample at a time, maybe you should consider using an optimizer with momentum, like tf.train.AdamOptimizer(learning_rate). This way you can increase the learning rate and reach convergence.