How to convert Tensorflow dataset to 2D numpy array

Question

I have a TensorFlow dataset which contains nearly 15000 multicolored images with 168*84 resolution and a label for each image. Its type and shape are like this:

< ConcatenateDataset shapes: ((168, 84, 3), ()), types: (tf.float32, tf.int32)>

I need to use it to train my network. That's why I need to pass it as a parameter to this function that I built my layers in:

def cnn_model_fn(features, labels, mode):

  input_layer = tf.reshape(features["x"], [-1, 168, 84, 3])
  # Convolutional Layer #1
  conv1 = tf.layers.conv2d(
     inputs=input_layer,
     filters=32,
     kernel_size=[5, 5],
     padding="same",
     activation=tf.nn.relu)
.
.
.

I tried to convert each tensor into np.array(which is the proper type for the function above, i guess) by using tf.eval() and np.ravel(). But I failed.

So, how can I convert this dataset into the proper type to pass it to the function?

Plus

I am new to python and tensorflow and I don't think I understand why there are datasets if we can not use them directly to build layers (I am following the tutorial in TensorFlow's website btw).

Thanks.

Answer 1

You could try eager execution, previously I gave an answer with session run (showed below).
During eager execution using .numpy() on a tensor will convert that tensor to numpy array.
Example code (from my use case):

    #enable eager execution
    from __future__ import absolute_import, division, print_function, unicode_literals
    import tensorflow as tf
    tf.enable_eager_execution()
    print('Is executing eagerly?',tf.executing_eagerly())      

    #load datasets
    import tensorflow_datasets as tfds
    dataset, metadata = tfds.load('cycle_gan/horse2zebra',
                                  with_info=True, as_supervised=True)
    train_horses, train_zebras = dataset['trainA'], dataset['trainB']

    #load dataset in to numpy array 
    train_A=train_horses.batch(1000).make_one_shot_iterator().get_next()[0].numpy()
    print(train_A.shape)

    #preview one of the images
    import matplotlib.pyplot as plt
    %matplotlib inline
    import numpy as np
    print(train_A.shape)
    plt.imshow(train_A[1])
    plt.show()

Old, session run, answer:

I recently had this problem, and I did it like this:

    #load datasets
    import tf
    import tensorflow_datasets as tfds
    dataset, metadata = tfds.load('cycle_gan/horse2zebra',
                                  with_info=True, as_supervised=True)
    train_horses, train_zebras = dataset['trainA'], dataset['trainB']

    #load dataset in to numpy array
    sess = tf.compat.v1.Session()
    tra=train_horses.batch(1000).make_one_shot_iterator().get_next()
    train_A=np.array(sess.run(tra)[0])
    print(train_A.shape)
    sess.close()

    #preview one of the images
    import matplotlib.pyplot as plt
    %matplotlib inline
    import numpy as np
    print(train_A.shape)
    plt.imshow(train_A[1])
    plt.show()

Answer 2

It doesn't sound like you set up things using the Tensorflow Dataset pipeline, here is the guide for doing so:

https://www.tensorflow.org/programmers_guide/datasets

You can either follow that (it's the right approach, but there's a small learning curve to get used to it), or you can just pass in the numpy array to sess.run as part of the feed_dict parameter. If you go this way then you should just create a tf.placeholder which will be populated by the value in feed_dict. Many of the basic tutorial examples here follow this approach:

https://github.com/aymericdamien/TensorFlow-Examples