How to automate API get data request? when using web sockets

Question

As far as I know Web Sockets allows bi-directional communication. and web sockets (for example: Socket.io) connections are always open. so, whenever new data has arrived data should be automatically pushed to the view via socket.

but in below code I am using set_interval to make a http.get call. and set_interval is called once every 1 second.

now, doing these does not give a real-time feel that is, the new data is pulled once every 1 second. which is statically defined.

in-short, I want to automate what set_interval does in below code. I don't want a static fetch interval value. This is because at-times stock price could change within 100ms and at times it would change once in few seconds.

Now, if I set interval to 1 sec, that is make a call every 1 second. the real feel of high fluctuation in market move would not be seen.

I am not sure how usually developers fetch data in IOT applications. for example car is monitored in real-time and let's say speed of the car is fetched in real time and graphed on a web or mobile application.

How do I achieve something similar like that in Stock Ticker? I want to simply plugin the application to an API and when new data arrives instantly push it to all the viewers (subscribers) in real-time.

Code below

////
// CONFIGURATION SETTINGS
////
var FETCH_INTERVAL = 1000;
var PRETTY_PRINT_JSON = true;

////
// START
////
var express = require('express');
var http = require('http');
var https = require('https');
var io = require('socket.io');
var cors = require('cors');

function getQuote(socket, ticker) {
    https.get({
        port: 443,
        method: 'GET',
        hostname: 'www.google.com',
        path: '/finance/info?client=ig&q=' + ticker,
        timeout: 1000
    }, function(response) {
        response.setEncoding('utf8');
        var data = '';

        response.on('data', function(chunk) {
            data += chunk;
        });

        response.on('end', function() {
            if(data.length > 0) {
                var dataObj;

                try {
                    dataObj = JSON.parse(data.substring(3));
                } catch(e) {
                    return false;
                }

                socket.emit(ticker, dataObj[0].l_cur);
            }
        });
    });
}

I am making a call to method getQuote depending on FETCH_INTERVAL set above

function trackTicker(socket, ticker) {
    // run the first time immediately
    getQuote(socket, ticker);

    // every N seconds
    var timer = setInterval(function() {
        getQuote(socket, ticker);
    }, FETCH_INTERVAL);

    socket.on('disconnect', function () {
        clearInterval(timer);
    });
}

var app = express();
app.use(cors());
var server = http.createServer(app);

var io = io.listen(server);
io.set('origins', '*:*');

app.get('/', function(req, res) {
    res.sendfile(__dirname + '/index.html');
});

io.sockets.on('connection', function(socket) {
    socket.on('ticker', function(ticker) {
        trackTicker(socket, ticker);
    });
});

server.listen(process.env.PORT || 4000);

Edits - Update

Okay, so I would need real-time feed. (this bit is sorted)

As far as I know, Real-time feeds are quite expensive and buying 10,000+ end points for each online client is quite expensive.

1) How do I make use of real-time feed to serve 1000s of end users? Can I use web sockets, Redis, publish/subscribe, broadcasting or some technology that copies real-time feed to tonnes of users? I want a efficient solution because I want to keep the expense of real-time data feed as low as possible.

How do I tackle that issue?

2) Yes, I understand polling needs to be done on server side and not on a client-side (to avoid doing polling for each client). but then what tech do I need to use? websockets, redis, pub/sub etc..

I have API URL and a token to access the API.

3) I am not just in need to fetch the data and push it to end users. But I would need to do some computation on the fetched data, will need to pull data from Redis or database as well and do calculations on it then push it to the view.

for example:

1) data I get in real-time market feed {"a":10, "b":20}
2) get data from DB or Redis  {"x":2, "y":4} 
3) do computation : z = a * x + b * y
4) finally push value of z in the view. 

How do I do all these in real-time at the same-time push it to multiple clients? Can you share a roadmap with me? I got the first piece of the puzzle getting real-time datafeed.

Answer 1

1) How do I make use of real-time feed to serve 1000s of end users? Can I use web sockets, Redis, publish/subscribe, broadcasting or some technology that copies real-time feed to tonnes of users? I want a efficient solution because I want to keep the expense of real-time data feed as low as possible.

How do I tackle that issue?

To "push" data to browser clients, you would want to use a webSocket or socket.io (built on top of webSockets). Then, anytime your server knows there's an update, it can immediately send that update to any currently connected client that is interested in that info. The basic idea is that the client connects to your server as soon as the web page is loaded and keeps that connection open for as long as the web page(s) are open.

2) Yes, I understand polling needs to be done on server side and not on a client-side (to avoid doing polling for each client). but then what tech do I need to use? websockets, redis, pub/sub etc..

It isn't clear to me what exactly you're asking about here. You will get updated prices using whatever the most efficient technology is that is offered by your provider. If all they provide is http calls, then you have to poll regularly using http requests. If they provide a webSocket interface to get updates, then that would be preferable.

There are lots of choices for how to keep track of which clients are interested in which pieces of information and how to distribute the updates. For a single server, you could easily build your own with just a Map of stock prices where the stock symbol is the key and an array of client identifiers is the value in the Map. Then, any time you get an update for a given stock, you just fetch the list of client IDs that are interested in that stock and send the update to them (over their webSocket/socket.io connection).

This is also a natural pub/sub type of application so anyone of the backends that support pub/sub would work just fine too. You could even use an EventEmitter where you .emit(stock, price) and each separate connection adds a listener for the stock symbols they are interested in.

For multiple servers at scale, you'd probably want to use some external process that manages the pub/sub process. Redis is a candidate for that.

3) I am not just in need to fetch the data and push it to end users. But I would need to do some computation on the fetched data, will need to pull data from Redis or database as well and do calculations on it then push it to the view.

I don't really see what question there is here. Pick your favorite database to store the info you need to fetch so you can get it upon demand.

How do I do all these in real-time at the same-time push it to multiple clients? Can you share a roadmap with me? I got the first piece of the puzzle getting real-time datafeed.

1. Real-time data feed.
2. Database to store your meta data used for calculations.
3. Some pub/sub system, either home built or from a pre-built package.

Then, follow this sequence of events.

1. Client signs in, connects a webSocket or socket.io connection.
2. Server accepts client connection and assigns a clientID and keeps track of the connection in some sort of Map between clientID and webSocket/socket.io connection. FYI, socket.io does this automatically for you.
3. Client tells server which items it wants to monitor (probably message sent over webSocket/socket.io connection.
4. Server registers that interest in pub/sub system (essentially subscribing the client to each item it wants to monitor.
5. Other clients do the same thing.
6. Each time client requests data on a specific item, the server makes sure that it is getting updates for that item (however the server gets its updates).
7. Server gets new info for some item that one or more clients is interested in.
8. New data is sent to pub/sub system and pub/sub system broadcasts that information to those clients that were interested in info on that particular item. The details of how that works depend upon what pub/sub system you choose and how it notifies subscribers of a change, but eventually a message is sent over webSocket/socket.io for the item that has changed.
9. When a client disconnects, their pub/sub subscriptions are "unsubscribed".