How can I train a simple, non-linear regression model with tensor flow?

Question

I've seen this example for linear regression and I would like to train a model

where

What I've tried

#!/usr/bin/env python

"""Example for learning a regression."""


import tensorflow as tf
import numpy

# Parameters
learning_rate = 0.01
training_epochs = 1000
display_step = 50

# Generate training data
train_X = []
train_Y = []
f = lambda x: x**2
for x in range(-20, 20):
    train_X.append(float(x))
    train_Y.append(f(x))
train_X = numpy.asarray(train_X)
train_Y = numpy.asarray(train_Y)
n_samples = train_X.shape[0]

# Graph input
X = tf.placeholder("float")
Y = tf.placeholder("float")

# Create Model
W1 = tf.Variable(tf.truncated_normal([1, 10], stddev=0.1), name="weight")
b1 = tf.Variable(tf.constant(0.1, shape=[1, 10]), name="bias")
mul = X * W1
h1 = tf.nn.sigmoid(mul) + b1
W2 = tf.Variable(tf.truncated_normal([10, 1], stddev=0.1), name="weight")
b2 = tf.Variable(tf.constant(0.1, shape=[1]), name="bias")
activation = tf.nn.sigmoid(tf.matmul(h1, W2) + b2)

# Minimize the squared errors
l2_loss = tf.reduce_sum(tf.pow(activation-Y, 2))/(2*n_samples)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(l2_loss)

# Initializing the variables
init = tf.initialize_all_variables()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(optimizer, feed_dict={X: x, Y: y})

        # Display logs per epoch step
        if epoch % display_step == 0:
            cost = sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y})
            print("Epoch: {:04d}, cost={:.9f}".format((epoch+1), cost),
                  "W=", sess.run(W1))  # "b=", sess.run(b1)

    print("Optimization Finished!")
    print("cost=", sess.run(cost, feed_dict={X: train_X, Y: train_Y}),
          "W1=", sess.run(W1), )  # "b2=", sess.run(b2)

When I execute it, I get:

$ python nnetstest.py
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 2
I tensorflow/core/common_runtime/local_session.cc:45] Local session inter op parallelism threads: 2
W tensorflow/core/common_runtime/executor.cc:1027] 0x314df50 Compute status: Invalid argument: Incompatible shapes: [40] vs. [1,10]
     [[Node: mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_Placeholder_0, weight)]]
Traceback (most recent call last):
  File "nnetstest.py", line 56, in <module>
    cost = sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y})
  File "/home/moose/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 345, in run
    results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)
  File "/home/moose/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 419, in _do_run
    e.code)
tensorflow.python.framework.errors.InvalidArgumentError: Incompatible shapes: [40] vs. [1,10]
     [[Node: mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_Placeholder_0, weight)]]
Caused by op u'mul', defined at:
  File "nnetstest.py", line 32, in <module>
    mul = X * W1
  File "/home/moose/.local/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 403, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/moose/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 728, in mul
    return _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
  File "/home/moose/.local/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py", line 633, in apply_op
    op_def=op_def)
  File "/home/moose/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1710, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/moose/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 988, in __init__
    self._traceback = _extract_stack()

I've tried a couple of slight variations in the input data, but I can't get it to work.

How can I train such a simple, non-linear regression model with Google TensorFlow?

Answer 1

The InvalidArgumentError is due to the values that you are feeding (train_X and train_Y) not having the necessary shape to be multiplied by W1.

There are a few issues here:

1. The statement mul = X * W1 should be mul = tf.matmul(X, W1), since * computes an elementwise multiplication, which is not what your equation specifies.

2. The input data X should be a one-column matrix. To handle scalar and vector data - as you have in your feed calls, you could reshape it as follows:

   X = tf.placeholder(tf.float32)
   reshaped_X = tf.reshape(X, [-1, 1])
   # ...
   mul = reshaped_X * W1
   

3. When you fetch the final cost, the first argument to sess.run should be l2_loss (and not cost):

   print("cost=", sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y}),
         "W1=", sess.run(W1))