Tensorflow, train_step feed incorrect

Question

I am switching from convnetjs to tensorflow and am tying to get the basics of reading images and training a cnn with tensorflow.

i have a bunch of images 160*120*1 in two folders: train/go and train/no so i use two classes.

somehow i can get my head around how the connection between a tf.train.slice_input_producer and the sess.run(train_step.

My code:

import tensorflow as tf

def read_my_list( minId, maxId ):
    """ create list with train/no and train/go from 1 to maxid
        max maxId = 50000
    """ 

    filenames = []
    labels = []
    for num in range( minId, maxId ):

        filenames.append( "/media/boss/tensor/train/go/" + str( num ) + ".jpg" )
        labels.append( int( 1 ) )

        filenames.append( "/media/boss/tensor/train/no/" + no_go_name( num ) + ".jpg" )
        labels.append( int( 0 ) )

        # return list with all filenames
    return filenames, labels

def no_go_name( id ):

    # create string where id = 5 becomes 00005

    ret = str( id )
    while ( len( ret ) < 5 ):
      ret = "0" + ret;

    return ret;


def read_images_from_disk(input_queue):
    """Consumes a single filename and label as a ' '-delimited string.
    Args:
      filename_and_label_tensor: A scalar string tensor.
    Returns:
      Two tensors: the decoded image, and the string label.
    """
    label = input_queue[1]
    print( "read file "  )
    file_contents = tf.read_file(input_queue[0])
    example = tf.image.decode_jpeg(file_contents, channels=1)

    # do i need to set shape??????????
    example.set_shape([160, 120, 1])
    print( "file read " )
    return  example, label



# some stuff to create a cnn etc
x = tf.placeholder(tf.float32, [None, 19200])

W = tf.Variable(tf.zeros([19200, 2]))
b = tf.Variable(tf.zeros([2]))

y = tf.nn.softmax(tf.matmul(x, W) + b)

y_ = tf.placeholder(tf.float32, [None, 2])

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

init = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init)

    # get filelist and labels
    image_list, label_list = read_my_list( 1, 10 )

    # conver to tensors for input_queue
    images = tf.convert_to_tensor(image_list, dtype=tf.string)
    labels = tf.convert_to_tensor(label_list, dtype=tf.int32)

    # Makes an input queue
    input_queue = tf.train.slice_input_producer([images, labels],
                                        num_epochs=10,
                                        shuffle=True)

    image, label = read_images_from_disk(input_queue)

    for i in range(100):
        print( i )

        image_batch, label_batch = tf.train.batch([image, label],
                                          batch_size=2)


        #gives error see below                                     
        sess.run(train_step, feed_dict={x: image_batch, y_: label_batch})


    # test accuracy, unsure if something is wrong here
    correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))

    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

    batch_xs, batch_ys = tf.train.batch([image, label],
                                      batch_size=10)

    print(sess.run(accuracy, feed_dict={x: batch_xs, y_: batch_ys}))

the following line gives an error:

sess.run(train_step, feed_dict={x: image_batch, y_: label_batch})

Here is the error:

Traceback (most recent call last):

File "detectGoNo.py", line 95, in <module>
sess.run(train_step, feed_dict={x: image_batch, y_: label_batch})

File "/home/boss/anaconda2/envs/tensor2/lib/python2.7/site-
packages/tensorflow/python/client/session.py", line 340, in run

run_metadata_ptr)

File "/home/boss/anaconda2/envs/tensor2/lib/python2.7/site-
packages/tensorflow/python/client/session.py", line 545, in _run

raise TypeError('The value of a feed cannot be a tf.Tensor object. '

TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable 
feed values include Python scalars, strings, lists, or numpy ndarrays.

UPDATE 02-06-2016

I got everything to work with the solution from nessuno, training and validation (Code below) Mrry indicated a pipeline is more typical unfortunately this one does not work ( code below ) no errors are given but calculated cost remains the same and validation shows me that the network is not improving.

My best guess is that something is wrong with the way i feed the labels to the trainer or the way i use the one_hot function.

The validation part seems to be working, when i feel images with always the label 0 accuracy becomes 100%, labels 1 accuracy 0% and 50/50 it is 50%. of course it could be the other way around but since the cost is not changing while training i think something goes wrong while training

i know, the model i use right now is to simple but for debugging it is good enough, the working verion manages to get to 80% accuracy within 1500 images.

label = tf.cast( label, tf.int64 )

label = tf.one_hot( label, 2, 0, 1 )
label = tf.cast( label, tf.float32 )

My code: ( working )

import tensorflow as tf
import numpy      as np
import math

IMAGE_WIDTH  = 160
IMAGE_HEIGHT = 120
IMAGE_DEPTH  = 1
IMAGE_PIXELS = IMAGE_WIDTH * IMAGE_HEIGHT
NUM_CLASSES  = 2

STEPS         = 50000
STEP_PRINT    = 100
STEP_VALIDATE = 100
LEARN_RATE    = 0.0014
DECAY_RATE    = 0.4
BATCH_SIZE    = 5

def read_my_list( minId, maxId, folder ):
    """ create list with train/no and train/go from 1 to maxid
        max maxId = 50000
    """ 

    filenames = []
    labels    = []
    #labels = np.zeros( ( ( maxId - minId ) * 2, 2 ) )
    for num in range( minId, maxId ):

        filenames.append( "/media/boss/2C260F93260F5CE8/tensor/" + folder + "/go/" + str( num ) + ".jpg" )
        #labels[ ( num - minId ) * 2 ][ 1 ] = 1
        labels.append( int( 1 ) )

        filenames.append( "/media/boss/2C260F93260F5CE8/tensor/" + folder + "/no/" + no_go_name( num ) + ".jpg" )
        #labels[ ( ( num - minId ) * 2 ) + 1 ][ 0 ] = 1
        labels.append( int( 0 ) )

        # return list with all filenames
    print( "label: " + str( len( labels ) ) )
    print( "image: " + str( len( filenames ) ) )
    return filenames, labels

def no_go_name( id ):

    # create string where id = 5 becomes 00005

    ret = str( id )
    while ( len( ret ) < 5 ):
      ret = "0" + ret;

    return ret;

# Create model
def conv_net(x):

    img_width  = IMAGE_WIDTH
    img_height = IMAGE_HEIGHT
    img_depth  = IMAGE_DEPTH

    weights    = tf.Variable( tf.random_normal( [ img_width * img_height * img_depth, NUM_CLASSES ] ) )
    biases     = tf.Variable( tf.random_normal( [ NUM_CLASSES ] ) )

    # softmax layer
    out        = tf.add( tf.matmul( x, weights ), biases )
    return out 

def read_images_from_disk(input_queue):
    """Consumes a single filename and label as a ' '-delimited string.
    Args:
      filename_and_label_tensor: A scalar string tensor.
    Returns:
      Two tensors: the decoded image, and the string label.
    """
    label = input_queue[1]
    print( "read file "  )
    file_contents = tf.read_file(input_queue[0])
    example = tf.image.decode_jpeg( file_contents, channels = 1 )

    example = tf.reshape( example, [ IMAGE_PIXELS ] )
    example.set_shape( [ IMAGE_PIXELS ] )

    example = tf.cast( example, tf.float32 )
    example = tf.cast( example, tf.float32 ) * ( 1. / 255 ) - 0.5

    label = tf.cast( label, tf.int64 )

    label = tf.one_hot( label, 2, 0, 1 )
    label = tf.cast( label, tf.float32 )

    print( "file read " )
    return  example, label

with tf.Session() as sess:

    ########################################
    # get filelist and labels for training
    image_list, label_list = read_my_list( 501, 50000, "train" )

    # create queue for training
    input_queue = tf.train.slice_input_producer( [ image_list, label_list ],
                                                num_epochs = 100,
                                                shuffle = True )

    # read files for training
    image, label = read_images_from_disk( input_queue )

    # `image_batch` and `label_batch` represent the "next" batch
    # read from the input queue.
    image_batch, label_batch = tf.train.batch( [ image, label ], batch_size = BATCH_SIZE )

    # input output placeholders

    x = tf.placeholder(tf.float32, [None, IMAGE_PIXELS])
    y_ = tf.placeholder(tf.float32, [None, NUM_CLASSES])

    # create the network
    y = conv_net( x )

    # loss
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits( y, y_) )

    learning_rate = tf.placeholder(tf.float32, shape=[])

    # train step
    train_step   = tf.train.AdamOptimizer( 1e-3 ).minimize( cost )


    ########################################
    # get filelist and labels for validation
    image_list_test, label_list_test = read_my_list( 1, 500, "validation" )

    # create queue for validation
    input_queue_test = tf.train.slice_input_producer( [ image_list_test, label_list_test ],
                                                shuffle=True )

    # read files for validation
    image_test, label_test = read_images_from_disk( input_queue_test )

    # `image_batch_test` and `label_batch_test` represent the "next" batch
    # read from the input queue test.
    image_batch_test, label_batch_test = tf.train.batch( [ image_test, label_test ], batch_size=200 )

    correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))

    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

    init = tf.initialize_all_variables()
    sess.run(init)

    # N.B. You must run this function before `sess.run(train_step)` to
    # start the input pipeline.
    #tf.train.start_queue_runners(sess)
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)

    for i in range(STEPS):
        # No need to feed, because `x` and `y_` are already bound to
        # the next input batch.   
        if i % STEP_PRINT == 0:
            LEARN_RATE = LEARN_RATE * DECAY_RATE
            print( str( i ) + " " + str( LEARN_RATE ) )

        if i % STEP_VALIDATE == 0:

            imgs, lbls = sess.run([image_batch_test, label_batch_test])

            print(sess.run(accuracy, feed_dict={
                    x: imgs,
                    y_: lbls}))

        imgs, lbls = sess.run([image_batch, label_batch])

        sess.run(train_step, feed_dict={
         x: imgs,
         y_: lbls}) 
#         ,learning_rate:LEARN_RATE})      

    imgs, lbls = sess.run([image_batch_test, label_batch_test])

    print(sess.run(accuracy, feed_dict={
         x: imgs,
         y_: lbls}))

    coord.request_stop()
    coord.join(threads)

My code: ( not working )

with tf.Session() as sess:

    ########################################
    # get filelist and labels for training
    image_list, label_list = read_my_list( 501, 50000, "train" )

    # create queue for training
    input_queue = tf.train.slice_input_producer( [ image_list, label_list ],
                                                num_epochs = 100,
                                                shuffle = True )

    # read files for training
    image, label = read_images_from_disk( input_queue )

    # `image_batch` and `label_batch` represent the "next" batch
    # read from the input queue.
    image_batch, label_batch = tf.train.batch( [ image, label ], batch_size = BATCH_SIZE )

    x = image_batch
    y_ = label_batch

    # create the network
    y = conv_net( x )

    # loss
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits( y, y_) )

    # train step
    train_step   = tf.train.AdamOptimizer( 1e-3 ).minimize( cost )


    ########################################
    # get filelist and labels for validation
    image_list_test, label_list_test = read_my_list( 1, 500, "validation" )

    # create queue for validation
    input_queue_test = tf.train.slice_input_producer( [ image_list_test, label_list_test ],
                                                shuffle=True )

    # read files for validation
    image_test, label_test = read_images_from_disk( input_queue_test )

    # `image_batch_test` and `label_batch_test` represent the "next" batch
    # read from the input queue test.
    image_batch_test, label_batch_test = tf.train.batch( [ image_test, label_test ], batch_size=200 )

    xval = image_batch_test
    yval_ = label_batch_test

    # network for validation
    yval = conv_net( xval )

    # validate network
    correct_prediction = tf.equal( tf.argmax( yval, 1 ), tf.argmax( yval_, 1 ) )

    # calculate accuracy
    accuracy = tf.reduce_mean( tf.cast( correct_prediction, tf.float32 ) )

    # init all variables
    init = tf.initialize_all_variables()
    sess.run( init )

    # N.B. You must run this function before `sess.run(train_step)` to
    # start the input pipeline.
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners( coord = coord )

    for i in range(STEPS):
        # No need to feed, because `x` and `y_` are already bound to
        # the next input batch.   
        if i % STEP_PRINT == 0:
            print( i )

        # validate accuracy
        if i % STEP_VALIDATE == 0:
            print( sess.run( accuracy ) )

        # train one step
        sess.run( train_step )  

    # validate accuracy
    print( sess.run( accuracy ) )

    coord.request_stop()
    coord.join( threads )

UPDATE 10-06-2016
It took me a while to realize that a training pipeline and a validating pipeline do not share the same weights and biases.
right now i train, save the model and load the model in a separate script, works like a charm.

Answer 1

image_batch and label_batch are tf.Tensor objects.

The idea behind the feed_dict is to pass values from outside the graph, into the graph. Tensor objects are values that are into the graph.

Therefore, you have to evaluate inside the graph the two tensor objects (to extract the content) and than feed the graph with the computed values (that now are python values).

So, eval the values with

imgs, lbls = image_batch.eval(), label_batch.eval()

or (better, because it uses a single call)

imgs, lbls = sess.run([image_batch, label_batch])

and than feed the graph, replacing the content of the placeholders with the returned values:

    sess.run(train_step, feed_dict={
         x: imgs,
         y_: lbls})

Answer 2

As nessuno points out, the results of tf.train.batch()—image_batch and label_batch—are tf.Tensor objects, so you can't use these as a value to feed into a subgraph.

The typical way to use tf.train.batch() is to use it to define the inputs to a pipeline (rather than using tf.placeholder() for x and y_), so that batching and prefetching will be handled inside the TensorFlow graph. Here's an approximate restructuring of the first part of your program that performs batching as desired:

with tf.Session() as sess:

    # get filelist and labels
    image_list, label_list = read_my_list( 1, 10 )

    input_queue = tf.train.slice_input_producer([image_list, label_list],
                                                num_epochs=10,
                                                shuffle=True)

    image, label = read_images_from_disk(input_queue)

    # `image_batch` and `label_batch` represent the "next" batch
    # read from the input queue.
    image_batch, label_batch = tf.train.batch([image, label], batch_size=2)

    x = image_batch
    y_ = label_batch

    # Define your model in terms of `x` and `y_` here....
    train_step = ...

    # N.B. You must run this function after creating your graph.
    init = tf.initialize_all_variables()
    sess.run(init)

    # N.B. You must run this function before `sess.run(train_step)` to
    # start the input pipeline.
    tf.train.start_queue_runners(sess)

    for i in range(100):
        # No need to feed, because `x` and `y_` are already bound to
        # the next input batch.                                     
        sess.run(train_step)