I'm trying to read data from CSV files to tensorflow,
https://www.tensorflow.org/versions/r0.7/how_tos/reading_data/index.html#filenames-shuffling-and-epoch-limits
The sample code in official document is like this:
col1, col2, col3, col4, col5 = tf.decode_csv(value, record_defaults=record_defaults)
To read the file, I need to know how many columns and lines in the file beforehand, and if there are 1000 columns, I need to define 1000 variables like col1, col2, col3, col4, col5,..., col1000 ,
this doesn't look like an efficient way to read data.
My questions
What is the best way to read CSV files into Tensorflow ?
Is there any way to read Database (such as mongoDB) in Tensorflow ?
There are two distinct ways to create a dataset: A data source constructs a Dataset from data stored in memory or in one or more files. A data transformation constructs a dataset from one or more tf.data.Dataset objects. import tensorflow as tf
Updated to TensorFlow 1.8. As you should know, feed-dict is the slowest possible way to pass information to TensorFlow and it must be avoided. The correct way to feed data into your models is to use an input pipeline to ensure that the GPU has never to wait for new stuff to come in.
The tf.data API supports a variety of file formats so that you can process large datasets that do not fit in memory. For example, the TFRecord file format is a simple record-oriented binary format that many TensorFlow applications use for training data.
For performance reasons, use TensorFlow operations for preprocessing your data whenever possible. However, it is sometimes useful to call external Python libraries when parsing your input data. You can use the tf.py_function () operation in a Dataset.map () transformation.
def func()
return 1,2,3,4
b = func()
print b #(1, 2, 3, 4)
print [num for num in b] # [1, 2, 3, 4]
Hi its nothing to do with tensorflow its simple python need not define 1000 variable. tf.decode_csv returns a tuple.
No idea on database handling, I think u can use python and just input the data in the form of array to the tensorflow.
Hope this is helpful
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With