Read Mysql Database in Tensorflow

Question

I can't find the way that using MySQL database in TensorFlow.

I made the table and sensor data for reference.

Here is the question.

1. What should I use to read MySQL database in TensorFlow?

2. I find the way that making CSV file in MySQL, and read it in TensorFlow. But it is not real-time data. I want to use the data in real-time.

Please help me. Thank you.

Answer 1

Late to the party. But i did it like that:

1. Read data from mysql in chunks (limit+offset) with pandas df.read_sql() in a separate thread.
2. yield the data with a generator
3. create a tf.Dataset with .from_tensor_slice() from the pandas dataframe.
4. Use model.fit() on the dataset.
5. Get the next chunk of data from the generator and train on that.

So, while the model is training on the data chunk, the next chunk is being loaded in the background. Ideally, the GPU will never sit idle. This can be adjusted by choosing a good chunk size and defining how many epochs the model should be trained on each chunk.

Relevant links: Make python generator run in background https://stackoverflow.com/a/29528804/5292996 https://www.tensorflow.org/tutorials/load_data/pandas_dataframe