A solution for streaming JSON using oboe.js in AngularJS?

Question

I'm pretty new to Angular so maybe I'm asking the impossible but anyway, here is my challenge.

As our server cannot paginate JSON data I would like to stream the JSON and add it page by page to the controller's model. The user doesn't have to wait for the entire stream to load so I refresh the view fo every X (pagesize) records.

I found oboe.js for parsing the JSON stream and added it using bower to my project. (bower install oboe --save).

I want to update the controllers model during the streaming. I did not use the $q implementation of pomises, because there is only one .resolve(...) possible and I want multiple pages of data loaded via the stream so the $digest needs to be called with every page. The restful service that is called is /service/tasks/search

I created a factory with a search function which I call from within the controller:

'use strict';

angular.module('myStreamingApp')
    .factory('Stream', function() {
        return {
            search: function(schema, scope) {
                var loaded = 0;
                var pagesize = 100;
                // JSON streaming parser oboe.js
                oboe({
                    url: '/service/' + schema + '/search'
                })
                        // process every node which has a schema
                        .node('{schema}', function(rec) {
                            // push the record to the model data
                            scope.data.push(rec);
                            loaded++;
                            // if there is another page received then refresh the view
                            if (loaded % pagesize === 0) {
                                scope.$digest();
                            }
                        })
                        .fail(function(err) {
                            console.log('streaming error' + err.thrown ? (err.thrown.message):'');
                        })
                        .done(function() {
                            scope.$digest();
                        });
            }
        };
    });

My controller:

'use strict';
angular.module('myStreamingApp')
    .controller('MyCtrl', function($scope, Stream) {
         $scope.data = [];
         Stream.search('tasks', $scope);
     });

It all seams to work. After a while however the system gets slow and the http call doesn't terminate after refreshing the browser. Also the browser (chrome) crashes when there are too many records loaded. Maybe I'm on the wrong track because passing the scope to the factory search function doesn't "feel" right and I suspect that calling the $digest on that scope is giving me trouble. Any ideas on this subject are welcome. Especially if you have an idea on implementing it where the factory (or service) could return a promise and I could use

$scope.data = Stream.search('tasks');

in the controller.

Answer 1

I digged in a little further and came up with the following solution. It might help someone:

The factory (named Stream) has a search function which is passed parameters for the Ajax request and a callback function. The callback is being called for every page of data loaded by the stream. The callback function is called via a deferred.promise so the scope can be update automatically with every page. To access the search function I use a service (named Search) which initially returns an empty aray of data. As the stream progresses the factory calls the callback function passed by the service and the page is added to the data.

I now can call the Search service form within a controller and assign the return value to the scopes data array.

The service and the factory:

'use strict';
angular.module('myStreamingApp')
        .service('Search', function(Stream) {
            return function(params) {
                // initialize the data
                var data = [];
                // add the data page by page using a stream
                Stream.search(params, function(page) {
                    // a page of records is received.
                    // add each record to the data
                    _.each(page, function(record) {
                        data.push(record);
                    });
                });
                return data;
            };
        })
        .factory('Stream', function($q) {
            return {
                // the search function calls the oboe module to get the JSON data in a stream
                search: function(params, callback) {
                    // the defer will be resolved immediately
                    var defer = $q.defer();
                    var promise = defer.promise;
                    // counter for the received records
                    var counter = 0;
                    // I use an arbitrary page size.
                    var pagesize = 100;
                    // initialize the page of records
                    var page = [];
                    // call the oboe unction to start the stream
                    oboe({
                        url: '/api/' + params.schema + '/search',
                        method: 'GET'
                    })
                            // once the stream starts we can resolve the defer.
                            .start(function() {
                                defer.resolve();
                            })
                            // for every node containing an _id
                            .node('{_id}', function(node) {
                                //  we push the node to the page
                                page.push(node);
                                counter++;
                                // if the pagesize is reached return the page using the promise
                                if (counter % pagesize === 0) {
                                    promise.then(callback(page));
                                    // initialize the page
                                    page = [];
                                }
                            })
                            .done(function() {
                                // when the stream is done make surethe last page of nodes is returned
                                promise.then(callback(page));
                            });
                    return promise;
                }
            };
        });

Now I can call the service from within a controller and assign the response of the service to the scope:

$scope.mydata = Search({schema: 'tasks'});

Update august 30, 2014

I have created an angular-oboe module with the above solution a little bit more structured. https://github.com/RonB/angular-oboe