Realtime data stream to Python from CSV file

Question

I have a CSV file that has data from a random sensor recorded over a few minutes time. Now I want to stream that data from a CSV file to my python code as if it were receiving data from the sensor itself directly. (The code is for taking readings from two different sensors/csv files and averaging them.) Someone suggested to use Apache Spark to stream data, but I feel that's a bit too complex for me. Might there be a simpler solution?

Answer 1

You could also use pandas read_csv() function to read the big csv file in small chunks, the basic code is written below:

import pandas as pd
chunksize = 100
for chunk in pd.read_csv('myfile.csv', chunksize=chunksize):
    print(chunk)

This link explains how this works: http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking