Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to stack two tensorflow datasets?

I want to stack two datasets objects in Tensorflow (rbind function in R). I have created one dataset A from tfRecord files and one dataset B from numpy arrays. Both have same variables. Do you know if there is a way to stack these two datasets to create a bigger one ? Or to create an iterrator that will randomly read data from this two sources ?

Thanks

like image 365
Kent930 Avatar asked Feb 13 '18 16:02

Kent930


People also ask

What is a prefetch dataset?

Dataset. prefetch transformation. It can be used to decouple the time when data is produced from the time when data is consumed. In particular, the transformation uses a background thread and an internal buffer to prefetch elements from the input dataset ahead of the time they are requested.

How do I iterate over a dataset in TF?

To iterate over the dataset several times, use . repeat() . We can enumerate each batch by using either Python's enumerator or a build-in method.

What does from_tensor_slices do?

from_tensor_slices(tensor) creates a Dataset whose elements are slices of the given tensors.


1 Answers

The tf.data.Dataset.concatenate() method is the closest analog of tf.stack() when working with datasets. If you have two datasets with the same structure (i.e. same types for each component, but possibly different shapes):

dataset_1 = tf.data.Dataset.range(10, 20)
dataset_2 = tf.data.Dataset.range(60, 70)

then you can concatenate them as follows:

combined_dataset = dataset_1.concatenate(dataset_2)
like image 70
mrry Avatar answered Sep 28 '22 05:09

mrry