Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to read data into Tensorflow?

I'm trying to read data from CSV files to tensorflow,

https://www.tensorflow.org/versions/r0.7/how_tos/reading_data/index.html#filenames-shuffling-and-epoch-limits

The sample code in official document is like this:

col1, col2, col3, col4, col5 = tf.decode_csv(value, record_defaults=record_defaults)

To read the file, I need to know how many columns and lines in the file beforehand, and if there are 1000 columns, I need to define 1000 variables like col1, col2, col3, col4, col5,..., col1000 , this doesn't look like an efficient way to read data.

My questions

  1. What is the best way to read CSV files into Tensorflow ?

  2. Is there any way to read Database (such as mongoDB) in Tensorflow ?

like image 593
V Y Avatar asked Feb 26 '16 05:02

V Y


People also ask

How do I create a dataset in TensorFlow?

There are two distinct ways to create a dataset: A data source constructs a Dataset from data stored in memory or in one or more files. A data transformation constructs a dataset from one or more tf.data.Dataset objects. import tensorflow as tf

What is the best way to pass data to TensorFlow?

Updated to TensorFlow 1.8. As you should know, feed-dict is the slowest possible way to pass information to TensorFlow and it must be avoided. The correct way to feed data into your models is to use an input pipeline to ensure that the GPU has never to wait for new stuff to come in.

What file formats does TensorFlow support?

The tf.data API supports a variety of file formats so that you can process large datasets that do not fit in memory. For example, the TFRecord file format is a simple record-oriented binary format that many TensorFlow applications use for training data.

Should I use TensorFlow or Python when parsing data?

For performance reasons, use TensorFlow operations for preprocessing your data whenever possible. However, it is sometimes useful to call external Python libraries when parsing your input data. You can use the tf.py_function () operation in a Dataset.map () transformation.


1 Answers

def func()
    return 1,2,3,4

b = func() 

print b #(1, 2, 3, 4)

print [num for num in b] # [1, 2, 3, 4]

Hi its nothing to do with tensorflow its simple python need not define 1000 variable. tf.decode_csv returns a tuple.

No idea on database handling, I think u can use python and just input the data in the form of array to the tensorflow.

Hope this is helpful

like image 80
Aravind Pilla Avatar answered Oct 15 '22 14:10

Aravind Pilla