How insure indexing every object with Firebase FlashLight in a ElasticSearch Bonsai Free Instance

Question

Thanks to the tutorial of the FlashLight https://github.com/firebase/flashlight, this is somehow easy to do fulltextsearch with Firebase.

However, if you keep the free ES instance, it is limited in term of concurrency access, and when you will start your node app, you see the following message in the log :

failed to index firebase/xxx/-KHLhdwGplb3lHWjm8RS: Error: Concurrent request limit exceeded. Please consider batching your requests, or contact [email protected] for help.

How to solve this ?

Answer 1

If you have a bunch of data to index, the Flashlight app will ask ES to index every object on the fly, without any resource access constraint. You have to control/limit the access this share resource with a Semaphore.

Install the Semaphore lib

npm i --save semaphore

Edit the PathMonitor.js file, and limit access to ES resource to 1

PathMonitor.prototype = {
    _init: function () {
        this.sem = require('semaphore')(1);
        this.addMonitor = this.ref.on('child_added', this._process.bind(this, this._childAdded));
        this.changeMonitor = this.ref.on('child_changed', this._process.bind(this, this._childChanged));
        this.removeMonitor = this.ref.on('child_removed', this._process.bind(this, this._childRemoved));
    },
    ...
    _index: function (key, data, callback) {
        var that = this;
        that.sem.take(function () {
            that.esc.index({
                index: that.index,
                type : that.type,
                id   : key,
                body : data
            }, function (error, response) {
                that.sem.leave();
                if (callback) {
                    callback(error, response);
                }
            }.bind(that));
        });
    },
    ...
}

This may not be required in case of paid plan.