tensorflow loss minimization type error

Question

I have a loss function implemented in TensorFlow that computes mean squared error. All tensors being used to compute the objective are of type float64 and therefore the loss function itself is of dtype float64. In particular,

print cost
==> Tensor("add_5:0", shape=TensorShape([]), dtype=float64)

However, when I attempt to minimize I obtain a value error with respect to type of the tensor:

GradientDescentOptimizer(learning_rate=0.1).minimize(cost)
==> ValueError: Invalid type <dtype: 'float64'> for add_5:0, expected: [tf.float32].

I don't understand why the expected dtype of the tensor is a single precision float when all variables leading up to the computation are of type float64. I have confirmed that when I coerce all variables to be float32 the computation executes correctly.

Does anyone have any insight as to why this could be happening? My computer is a 64bit machine.

Here is an example that reproduces the behavior

import tensorflow as tf
import numpy as np

# Make 100 phony data points in NumPy.
x_data = np.random.rand(2, 100) # Random input
y_data = np.dot([0.100, 0.200], x_data) + 0.300

# Construct a linear model.
b = tf.Variable(tf.zeros([1], dtype=np.float64))
W = tf.Variable(tf.random_uniform([1, 2], minval=-1.0, maxval=1.0, dtype=np.float64))
y = tf.matmul(W, x_data) + b

# Minimize the squared errors.
loss = tf.reduce_mean(tf.square(y - y_data))
optimizer = tf.train.GradientDescentOptimizer(0.5)
train = optimizer.minimize(loss)

# For initializing the variables.
init = tf.initialize_all_variables()

# Launch the graph
sess = tf.Session()
sess.run(init)

# Fit the plane.
for step in xrange(0, 201):
    sess.run(train)
    if step % 20 == 0:
        print step, sess.run(W), sess.run(b)

Answer 1

Currently the tf.train.GradientDescentOptimizer class only supports training on 32-bit floating-point variables and loss values.

However, it looks like the kernel is implemented for double-precision values, so it should be possible to train in your scenario.

A quick workaround would be to define a subclass that supports tf.float64 values as well:

class DoubleGDOptimizer(tf.train.GradientDescentOptimizer):
  def _valid_dtypes(self):
    return set([tf.float32, tf.float64])

...and then use DoubleGDOptimizer in place of tf.train.GradientDescentOptimizer.

EDIT: You'll need to pass in the learning rate as tf.constant(learning_rate, tf.float64) to make this work.

(N.B. This isn't a supported interface and it may be subject to change in future, but the team is aware of the desire for optimizing double-precision floats, and intends to provide a built-in solution.)