Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Train Tensorflow model with estimator (from_generator)

I am trying train an estimator with a generator, but I want to feed this estimator with a package of samples for each iteration. I show the code:

def _generator():
for i in range(100):
    feats  = np.random.rand(4,2)
    labels = np.random.rand(4,1)

    yield feats, labels


def input_func_gen():
    shapes = ((4,2),(4,1))
    dataset = tf.data.Dataset.from_generator(generator=_generator,
                                         output_types=(tf.float32, tf.float32),
                                         output_shapes=shapes)
dataset = dataset.batch(4)
# dataset = dataset.repeat(20)
iterator = dataset.make_one_shot_iterator()
features_tensors, labels = iterator.get_next()
features = {'x': features_tensors}
return features, labels


x_col = tf.feature_column.numeric_column(key='x', shape=(4,2))
es = tf.estimator.LinearRegressor(feature_columns=[x_col],model_dir=tf_data)
es = es.train(input_fn=input_func_gen,steps = None)

When I run this code, it raises this error:

    raise ValueError(err.message)
ValueError: Dimensions must be equal, but are 2 and 3 for 'linear/head/labels/assert_equal/Equal' (op: 'Equal') with input shapes: [2], [3].

How do I have to call to this structure??

thx!!!

like image 800
Julio CamPlaz Avatar asked Apr 05 '18 13:04

Julio CamPlaz


People also ask

What kind of estimator model does TensorFlow recommend using for classification?

It is recommended using pre-made Estimators when just getting started. To write a TensorFlow program based on pre-made Estimators, you must perform the following tasks: Create one or more input functions. Define the model's feature columns.

How do I iterate over a TensorFlow dataset?

To iterate over the dataset several times, use . repeat() . We can enumerate each batch by using either Python's enumerator or a build-in method.

What is TensorFlow prefetch?

Prefetching. Prefetching overlaps the preprocessing and model execution of a training step. While the model is executing training step s , the input pipeline is reading the data for step s+1 . Doing so reduces the step time to the maximum (as opposed to the sum) of the training and the time it takes to extract the data ...

What is estimator in TensorFlow?

estimator —a high-level TensorFlow API. Estimators encapsulate the following actions: Training. Evaluation. Prediction.


1 Answers

The batch size is automatically computed and added to the tensors shapes by Tensorflow, so it doesn't have to be done manually. Your generator should also be defined to output single samples.

Assuming the 4 in position 0 of your shapes are for the batch size, then:

import tensorflow as tf
import numpy

def _generator():
    for i in range(100):
        feats  = numpy.random.rand(2)
        labels = numpy.random.rand(1)

        yield feats, labels


def input_func_gen():
    shapes = ((2),(1))
    dataset = tf.data.Dataset.from_generator(generator=_generator,
                                         output_types=(tf.float32, tf.float32),
                                         output_shapes=shapes)
    dataset = dataset.batch(4)
    # dataset = dataset.repeat(20)
    iterator = dataset.make_one_shot_iterator()
    features_tensors, labels = iterator.get_next()
    features = {'x': features_tensors}
    return features, labels


x_col = tf.feature_column.numeric_column(key='x', shape=(2))
es = tf.estimator.LinearRegressor(feature_columns=[x_col])
es = es.train(input_fn=input_func_gen,steps = None)
like image 58
benjaminplanche Avatar answered Oct 23 '22 13:10

benjaminplanche