Python Flask date update real-time

Question

I am building a web app with Python Flask with JavaScript. I am a beginner of Javascript.

The process I do now:

In Flask Python code,
1. I get data by scrapping the web (numeric data that updates every minute). 2. use the data and calculate something and get the final numbers. 3. make a list that contains the final numbers 4. feed the list to a page by adding the list to the page's definition of Flask 5. Now in HTML get the list by capturing it with {{ data|safe }} tag

6. use it with Javascript to make a chart.

The problem is: In the step 1, the data I get is being updated every minute. For example, on that web page now there are 15 data points. I parse the last 10 data points from that web page and then I put them in a list in my Python and then do the following steps and make a chart on my webpage. One minute later, in the data source web page, there will be 16 data point available, and I need to get the last 10 data points. In that case, I need to run the python code again to get the latest 10 data points to use them to make a chart on my web page.

So, I need to always run the whole python code which is the whole Flask app init.py file and re-render my webpage to see the updated chart. If I do not re-run the init.py file in my server, then even after 10 minutes or 2 hours, I will only see the data that I parsed for the first time forever.

How should I run the Flask and always get the updated data without always re-run the flask init.py every time.

I thought about using time.sleep(60) so that the flask app python file is run every 1 minutes. But this is really taking alot of time when my code gets much more thinks to calculate. And do not really work.

How should I solve this problem??

Should I use time.sleep ? or is threre better way?

Answer 1

This is a classic "push" update problem. Your web server is updated with information that you would like to frequently update (or "push") to your web client front-end.

As PJ Santoro suggests, you can repeatedly poll your server (every 10 seconds say) along the lines of "Is there new data? No. Okay. Is there new data? No. Okay. Is there new data? Yes, here it is! Great! Is there new data? ...." It is rather inefficient, but for low data volumes you might never see that problem.

A more efficient update system would have the server send an update signal only when new data is ready. There are two major ways to do that:

1. Long polling, also known as reverse AJAX and Comet. Your web page opens an AJAX request on a server URL such as /update_data with a really long timeout (say 30 minutes or an hour). When the server has data, it sends it through the waiting AJAX connection. There are complications with this, including managing concurrency on the server side (not historically Python's strong suit) and the fact that even the maximum timeout (~1 hour last I used this technique) might expire without data, and some error handling is needed to manage that. If you wan to try long polling / reverse AJAX, here is an example using Flask.

2. WebSockets. WebSockets are a much more recent approach to interactive and "push" updates between servers and clients. They're very well-used; projects like the Jupyter Notebook depend on them extensively. They're extremely efficient and well-fit for this kind of incremental update of client state, and the code is often much less Byzantine and confusing than retrofits like reverse AJAX. But...there are complexities with WebSockets too. Managing concurrency in the Flask server side is still a significant issue, for example. If you'd like to consider WebSockets as your update mechanism, here is an example of how to use them with Flask.

Either approach you use, if your data communicated over time grows, you will also need to structure the data transmitted on each update so that it's an incremental update. No mater how good the plumbing, if you transmit thousands of data points every update, it's not going to be fast. But, long polling and WebSockets plumbing should at least get you a great distance toward a legitimate real-time update capability.

Answer 2

Well... in an ideal world you would separate this into 2 applications:

• Scrape & Parse Data From the Web (doesn't need to be exposed to the web)

• Deliver data to a user via a web app

You could use some sort of CI tool (e.g. Jenkins) to monitor and add the external data into a database and then your Flask app to serve this pre-processed data to your users.

If Steps 1-6 are relatively quick what you could do is setup an XHR endpoint within Flask and then use a setInterval() javascript function to call it on a interval to tell your application to update the data. E.g.:

setInterval(function() {
    var req = new XMLHttpRequest();
    req.open('GET', '/refresh_data?time=234', true);
    req.onreadystatechange = function(e) {
        if(req.readyState !== 4) {
            return;
        }
        if ([200, 304].indexOf(req.status) === -1) {
            console.warn('Error! XHR failed.');
        }
        else {
            data = JSON.parse(e.target.responseText);
            processData();
            updateChart();
        }
    };
    req.send();
}, 10000);  // time in milliseconds (e.g. every 10 seconds)

And have a flask endpoint like:

@app.route('/refresh_data')
def refresh_data():
    time = requests.args.get('time', type=int)
    if not time:
        return jsonify({status: 'error: incorrect parameters'})
    data = requests.get('http://api.getmoredata.com?time=%s' % time)
    # process the results...
    return jsonify({status: 'success', data: data})

Ideally you'd have some sort of a chart that has a refresh() method where you can pass it new data and just keep adding to it... (e.g. I'm a fan of D3 for stuff like this).

I'd probably remove this from you're init.py code, but it might be acceptable for the "first load". a flask app (e.g. any web app) responds to HTTP requests and nothing persists on the server between requests so a time.sleep() would not do much for you... if you want to run persistent code on your server you'd need to look into something like celery to manage background tasks.