tensorflow neural net with continuous / floating point output?

Question

I'm trying to create a simple neural net in tensorflow that learns some simple relationship between inputs and outputs (for example, y=-x) where the inputs and outputs are floating point values (meaning, no softmax used on the output).

I feel like this should be pretty easy to do, but I must be messing up somewhere. Wondering if there are any tutorials or examples out there that do something similar. I looked through the existing tensorflow tutorials and didn't see anything like this and looked through several other sources of tensorflow examples I found by googling, but didn't see what I was looking for.

Here's a trimmed down version of what I've been trying. In this particular version, I've noticed that my weights and biases always seem to be stuck at zero. Perhaps this is due to my single input and single output?

I've had good luck altering the mist example for various nefarious purposes, but everything I've gotten to work successfully used softmax on the output for categorization. If I can figure out how to generate a raw floating point output from my neural net, there are several fun projects I'd like to do with it.

Anyone see what I'm missing? Thanks in advance! - J.

  # Trying to define the simplest possible neural net where the output layer of the neural net is a single
  # neuron with a "continuous" (a.k.a floating point) output.  I want the neural net to output a continuous
  # value based off one or more continuous inputs.  My real problem is more complex, but this is the simplest
  # representation of it for explaining my issue.  Even though I've oversimplified this to look like a simple
  # linear regression problem (y=m*x), I want to apply this to more complex neural nets.  But if I can't get
  # it working with this simple problem, then I won't get it working for anything more complex.

  import tensorflow as tf
  import random
  import numpy as np


  INPUT_DIMENSION  = 1
  OUTPUT_DIMENSION = 1
  TRAINING_RUNS    = 100
  BATCH_SIZE       = 10000
  VERF_SIZE        = 1


  # Generate two arrays, the first array being the inputs that need trained on, and the second array containing outputs.
  def generate_test_point():
    x = random.uniform(-8, 8)

    # To keep it simple, output is just -x.  
    out = -x

    return ( np.array([ x ]), np.array([ out ]) )

  # Generate a bunch of data points and then package them up in the array format needed by
  # tensorflow
  def generate_batch_data( num ):
     xs = []
     ys = []

     for i in range(num):
       x, y = generate_test_point()

       xs.append( x )
       ys.append( y )

     return (np.array(xs), np.array(ys) )

  # Define a single-layer neural net.  Originally based off the tensorflow mnist for beginners tutorial

  # Create a placeholder for our input variable
  x = tf.placeholder(tf.float32, [None, INPUT_DIMENSION])

  # Create variables for our neural net weights and bias
  W = tf.Variable(tf.zeros([INPUT_DIMENSION, OUTPUT_DIMENSION]))
  b = tf.Variable(tf.zeros([OUTPUT_DIMENSION]))

  # Define the neural net.  Note that since I'm not trying to classify digits as in the tensorflow mnist
  # tutorial, I have removed the softmax op.  My expectation is that 'net' will return a floating point
  # value.
  net = tf.matmul(x, W) + b

  # Create a placeholder for the expected result during training
  expected = tf.placeholder(tf.float32, [None, OUTPUT_DIMENSION])

  # Same training as used in mnist example
  cross_entropy = -tf.reduce_sum(expected*tf.log(tf.clip_by_value(net,1e-10,1.0)))
  train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

  sess = tf.Session()

  init = tf.initialize_all_variables()
  sess.run(init)

  # Perform our training runs

  for i in range( TRAINING_RUNS ):
    print "trainin run: ", i,

    batch_inputs, batch_outputs = generate_batch_data( BATCH_SIZE )

    # I've found that my weights and bias values are always zero after training, and I'm not sure why.
    sess.run( train_step, feed_dict={x: batch_inputs, expected: batch_outputs})

    # Test our accuracy as we train...  I am defining my accuracy as the error between what I 
    # expected and the actual output of the neural net.
    #accuracy = tf.reduce_mean(tf.sub( expected, net))  
    accuracy = tf.sub( expected, net) # using just subtract since I made my verification size 1 for debug

    # Uncomment this to debug
    #import pdb; pdb.set_trace()

    batch_inputs, batch_outputs = generate_batch_data( VERF_SIZE )
    result = sess.run(accuracy, feed_dict={x: batch_inputs, expected: batch_outputs})

    print "    progress: "
    print "      inputs: ", batch_inputs
    print "      outputs:", batch_outputs
    print "      actual: ", result

Answer 1

Your loss should be the squared difference of output and true value:

loss = tf.reduce_mean(tf.square(expected - net))

This way the network learns to optimize this loss and make the output closer to the real result. Cross entropy should only be used for output values between 0 and 1 i.e. for classification.

Answer 2

If anyone is interested, I got this example to work. Here's the code:

# Trying to define the simplest possible neural net where the output layer of the neural net is a single
# neuron with a "continuous" (a.k.a floating point) output.  I want the neural net to output a continuous
# value based off one or more continuous inputs.  My real problem is more complex, but this is the simplest
# representation of it for explaining my issue.  Even though I've oversimplified this to look like a simple
# linear regression problem (y=m*x), I want to apply this to more complex neural nets.  But if I can't get
# it working with this simple problem, then I won't get it working for anything more complex.

import tensorflow as tf
import random
import numpy as np

INPUT_DIMENSION = 1
OUTPUT_DIMENSION = 1
TRAINING_RUNS = 100
BATCH_SIZE = 10000
VERF_SIZE = 1


# Generate two arrays, the first array being the inputs that need trained on, and the second array containing outputs.
def generate_test_point():
    x = random.uniform(-8, 8)

    # To keep it simple, output is just -x.
    out = -x

    return (np.array([x]), np.array([out]))


# Generate a bunch of data points and then package them up in the array format needed by
# tensorflow
def generate_batch_data(num):
    xs = []
    ys = []

    for i in range(num):
        x, y = generate_test_point()

        xs.append(x)
        ys.append(y)

    return (np.array(xs), np.array(ys))


# Define a single-layer neural net.  Originally based off the tensorflow mnist for beginners tutorial

# Create a placeholder for our input variable
x = tf.placeholder(tf.float32, [None, INPUT_DIMENSION])

# Create variables for our neural net weights and bias
W = tf.Variable(tf.zeros([INPUT_DIMENSION, OUTPUT_DIMENSION]))
b = tf.Variable(tf.zeros([OUTPUT_DIMENSION]))

# Define the neural net.  Note that since I'm not trying to classify digits as in the tensorflow mnist
# tutorial, I have removed the softmax op.  My expectation is that 'net' will return a floating point
# value.
net = tf.matmul(x, W) + b

# Create a placeholder for the expected result during training
expected = tf.placeholder(tf.float32, [None, OUTPUT_DIMENSION])

# Same training as used in mnist example
loss = tf.reduce_mean(tf.square(expected - net))
# cross_entropy = -tf.reduce_sum(expected*tf.log(tf.clip_by_value(net,1e-10,1.0)))
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)

sess = tf.Session()

init = tf.initialize_all_variables()
sess.run(init)

# Perform our training runs

for i in range(TRAINING_RUNS):
    print("trainin run: ", i, )

    batch_inputs, batch_outputs = generate_batch_data(BATCH_SIZE)

    # I've found that my weights and bias values are always zero after training, and I'm not sure why.
    sess.run(train_step, feed_dict={x: batch_inputs, expected: batch_outputs})

    # Test our accuracy as we train...  I am defining my accuracy as the error between what I
    # expected and the actual output of the neural net.
    # accuracy = tf.reduce_mean(tf.sub( expected, net))
    accuracy = tf.subtract(expected, net)  # using just subtract since I made my verification size 1 for debug
    # tf.subtract()
    # Uncomment this to debug
    # import pdb; pdb.set_trace()
    print("W=%f, b=%f" % (sess.run(W), sess.run(b)))

    batch_inputs, batch_outputs = generate_batch_data(VERF_SIZE)
    result = sess.run(accuracy, feed_dict={x: batch_inputs, expected: batch_outputs})

    print("    progress: ")
    print("      inputs: ", batch_inputs)
    print("      outputs:", batch_outputs)
    print("      actual: ", result)