Menu
My Tensorflow model takes in a sequence of sequence data for each example, namely, sequences of character tokens in a sequence of words (e.g., [[3], [4,3],[6,1,20]]). I was able to do this before by padding a 3D numpy array [batch_size, max_words_len, max_chars_len] and feeding that into a placeholder.
in_question_chars = tf.placeholder(tf.int32,
[None, None, None],
name="in_question_chars")
# example of other data
in_question_words = tf.placeholder(tf.int32,
[None, None],
name="in_question_words")
But now I would like to use Google Cloud Machine Learning Engine for online prediction/deployment. Based on the example from Tensorflow Serving: https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/mnist_saved_model.py
I came up with something like this but don't really know what to use for the feature to parse the sequence of sequence char tokens:
serialized_tf_example = tf.placeholder(tf.string, name='tf_example')
feature_configs = {'in_question_chars':tf.FixedLenSequenceFeature(shape=[None],
allow_missing=True,
dtype=tf.int32,
default_value=0),
'in_question_words':tf.FixedLenSequenceFeature(shape=[],
allow_missing=True,
dtype=tf.int32,
default_value=0)
}
tf_example = tf.parse_example(serialized_tf_example, feature_configs)
in_question_chars = tf.identity(tf_example['in_question_chars'],
name='in_question_chars')
# example of other data
in_question_words = tf.identity(tf_example['in_question_words'],
name='in_question_words')
Should I use VarLenFeature, which turns it into a SparseTensor (eventhough it's not really sparse), and then use tf.sparse_tensor_to_dense to convert it back to dense?
For the next step, I get the embedding for each char token.
in_question_char_repres = tf.nn.embedding_lookup(char_embedding,
in_question_chars)
So another option is to keep it a SparseTensor and then use tf.nn.embedding_lookup_sparse
I wasn't able to find an example of how this should be done. Please let me know what is best practice. Thanks!
Edit 8/25/17
It doesn't seem to allow me to set None for the 2nd dimension.
Here's an abridged version of my code
def read_dataset(filename, mode=tf.contrib.learn.ModeKeys.TRAIN):
def _input_fn():
num_epochs = MAX_EPOCHS if mode == tf.contrib.learn.ModeKeys.TRAIN else 1
input_file_names = tf.train.match_filenames_once(str(filename))
filename_queue = tf.train.string_input_producer(
input_file_names, num_epochs=num_epochs, shuffle=True)
reader = tf.TFRecordReader()
_, serialized = reader.read_up_to(filename_queue, num_records=batch_size)
features_spec = {
CORRECT_CHILD_NODE_IDX: tf.FixedLenFeature(shape=[],
dtype=tf.int64,
default_value=0),
QUESTION_LENGTHS: tf.FixedLenFeature(shape=[], dtype=tf.int64),
IN_QUESTION_WORDS: tf.FixedLenSequenceFeature(shape=[],
allow_missing=True,
dtype=tf.int64
),
QUESTION_CHAR_LENGTHS: tf.FixedLenSequenceFeature(shape=[],
allow_missing=True,
dtype=tf.int64
),
IN_QUESTION_CHARS: tf.FixedLenSequenceFeature(shape=[None],
allow_missing=True,
dtype=tf.int64
)
}
examples = tf.parse_example(serialized, features=features_spec)
label = examples[CORRECT_CHILD_NODE_IDX]
return examples, label # dict of features, label
return _input_fn
When I have 'None' for the shape, it gives me this error:
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f57fc309c18>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
per_process_gpu_memory_fraction: 1.0
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'outputdir'}
WARNING:tensorflow:From /home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:269: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py in _call_cpp_shape_fn_impl(op, input_tensors_needed, input_tensors_as_shapes_needed, require_shape_fn)
653 graph_def_version, node_def_str, input_shapes, input_tensors,
--> 654 input_tensors_as_shapes, status)
655 except errors.InvalidArgumentError as err:
/home/jupyter-admin/anaconda3/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
88 try:
---> 89 next(self.gen)
90 except StopIteration:
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in raise_exception_on_not_ok_status()
465 compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466 pywrap_tensorflow.TF_GetCode(status))
467 finally:
InvalidArgumentError: dense_shapes[2] has unknown rank or unknown inner dimensions: [?,?] for 'ParseExample/ParseExample' (op: 'ParseExample') with input shapes: [?], [0], [], [], [], [], [], [], [], [], [], [0], [1], [], [], [0], [], [0], [0], [0].
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
<ipython-input-45-392858a0e7b4> in <module>()
48
49 shutil.rmtree('outputdir', ignore_errors=True) # start fresh each time
---> 50 learn_runner.run(experiment_fn, 'outputdir')
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py in run(experiment_fn, output_dir, schedule, run_config, hparams)
207 schedule = schedule or _get_default_schedule(run_config)
208
--> 209 return _execute_schedule(experiment, schedule)
210
211
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py in _execute_schedule(experiment, schedule)
44 logging.error('Allowed values for this experiment are: %s', valid_tasks)
45 raise TypeError('Schedule references non-callable member %s' % schedule)
---> 46 return task()
47
48
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/experiment.py in train_and_evaluate(self)
500 name=eval_dir_suffix, hooks=self._eval_hooks
501 )]
--> 502 self.train(delay_secs=0)
503
504 eval_result = self._call_evaluate(input_fn=self._eval_input_fn,
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/experiment.py in train(self, delay_secs)
278 return self._call_train(input_fn=self._train_input_fn,
279 max_steps=self._train_steps,
--> 280 hooks=self._train_monitors + extra_hooks)
281
282 def evaluate(self, delay_secs=None, name=None):
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/experiment.py in _call_train(self, _sentinel, input_fn, steps, hooks, max_steps)
675 steps=steps,
676 max_steps=max_steps,
--> 677 monitors=hooks)
678
679 def _call_evaluate(self, _sentinel=None, # pylint: disable=invalid-name,
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
294 'in a future version' if date is None else ('after %s' % date),
295 instructions)
--> 296 return func(*args, **kwargs)
297 return tf_decorator.make_decorator(func, new_func, 'deprecated',
298 _add_deprecated_arg_notice_to_docstring(
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in fit(self, x, y, input_fn, steps, batch_size, monitors, max_steps)
456 hooks.append(basic_session_run_hooks.StopAtStepHook(steps, max_steps))
457
--> 458 loss = self._train_model(input_fn=input_fn, hooks=hooks)
459 logging.info('Loss for final step: %s.', loss)
460 return self
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in _train_model(self, input_fn, hooks)
954 random_seed.set_random_seed(self._config.tf_random_seed)
955 global_step = contrib_framework.create_global_step(g)
--> 956 features, labels = input_fn()
957 self._check_inputs(features, labels)
958 model_fn_ops = self._get_train_ops(features, labels)
<ipython-input-44-fdb63ed72b90> in _input_fn()
35 )
36 }
---> 37 examples = tf.parse_example(serialized, features=features_spec)
38
39 label = examples[CORRECT_CHILD_NODE_IDX]
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/parsing_ops.py in parse_example(serialized, features, name, example_names)
573 outputs = _parse_example_raw(
574 serialized, example_names, sparse_keys, sparse_types, dense_keys,
--> 575 dense_types, dense_defaults, dense_shapes, name)
576 return _construct_sparse_tensors_for_sparse_features(features, outputs)
577
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/parsing_ops.py in _parse_example_raw(serialized, names, sparse_keys, sparse_types, dense_keys, dense_types, dense_defaults, dense_shapes, name)
698 dense_keys=dense_keys,
699 dense_shapes=dense_shapes,
--> 700 name=name)
701 # pylint: enable=protected-access
702
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_parsing_ops.py in _parse_example(serialized, names, sparse_keys, dense_keys, dense_defaults, sparse_types, dense_shapes, name)
174 dense_defaults=dense_defaults,
175 sparse_types=sparse_types,
--> 176 dense_shapes=dense_shapes, name=name)
177 return _ParseExampleOutput._make(result)
178
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
765 op = g.create_op(op_type_name, inputs, output_types, name=scope,
766 input_types=input_types, attrs=attr_protos,
--> 767 op_def=op_def)
768 if output_structure:
769 outputs = op.outputs
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)
2630 original_op=self._default_original_op, op_def=op_def)
2631 if compute_shapes:
-> 2632 set_shapes_for_outputs(ret)
2633 self._add_op(ret)
2634 self._record_op_seen_by_control_dependencies(ret)
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in set_shapes_for_outputs(op)
1909 shape_func = _call_cpp_shape_fn_and_require_op
1910
-> 1911 shapes = shape_func(op)
1912 if shapes is None:
1913 raise RuntimeError(
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in call_with_requiring(op)
1859
1860 def call_with_requiring(op):
-> 1861 return call_cpp_shape_fn(op, require_shape_fn=True)
1862
1863 _call_cpp_shape_fn_and_require_op = call_with_requiring
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py in call_cpp_shape_fn(op, require_shape_fn)
593 res = _call_cpp_shape_fn_impl(op, input_tensors_needed,
594 input_tensors_as_shapes_needed,
--> 595 require_shape_fn)
596 if not isinstance(res, dict):
597 # Handles the case where _call_cpp_shape_fn_impl calls unknown_shape(op).
/home/jupyter-admin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py in _call_cpp_shape_fn_impl(op, input_tensors_needed, input_tensors_as_shapes_needed, require_shape_fn)
657 missing_shape_fn = True
658 else:
--> 659 raise ValueError(err.message)
660
661 if missing_shape_fn:
ValueError: dense_shapes[2] has unknown rank or unknown inner dimensions: [?,?] for 'ParseExample/ParseExample' (op: 'ParseExample') with input shapes: [?], [0], [], [], [], [], [], [], [], [], [], [0], [1], [], [], [0], [], [0], [0], [0].
Currently, I'm getting around this by turning the 2D sequence of sequence into 1D sequence by setting the second dimension to a max_char_length and then concatenating it into a 1d array. So I keep only the first max_char_length char if it's longer than max_char_length or pad it with zeros if it's shorter. This seems to work but perhaps there's a way where it can accept variable length sequence for the second dimension and do padding in tf.parse_example or tf.train.batch.
945
asked Mar 03 '26 10:03
EDIT: fixed confusing/wrong answer =)
So what you want is a tf.SequenceExample which uses tf.parse_single_sequence_example rather than tf.parse_example. This allows you to have each feature in the feature_list within an example be part of a sequence, in this case each Feature can be a VarLenFeature representing the number of characters in the word. Unfortunately, this doesn't work as well when you want to pass multiple sentences. So we have to do some hacking around with higher order functions and tf.sparse_concat:
I produced a test program that does this here: https://gist.github.com/elibixby/1c7a2497f96a457130241c59c676ebd4
The input (before serialization to a batch of SequenceExamples) looks like:
[[[5, 10], [5, 10, 20]],
[[0, 1, 2], [2, 1, 0], [0, 1, 2, 3]]]
The resulting SparseTensor looks like:
SparseTensorValue(indices=array([[[0, 0, 0],
[0, 0, 1],
[0, 1, 0],
[0, 1, 1],
[0, 1, 2],
[1, 0, 0],
[1, 0, 1],
[1, 0, 2],
[1, 1, 0],
[1, 1, 1],
[1, 1, 2],
[1, 2, 0],
[1, 2, 1],
[1, 2, 2],
[1, 2, 3]]]), values=array([[ 5, 10, 5, 10, 20, 0, 1, 2, 2, 1, 0, 0, 1, 2, 3]]), dense_shape=array([[2, 3, 4]]))
Which appears to be a SparseTensor where index=[sentence, word, letter]
148
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
