How to merge REST call results in Angular app more efficiently

Question

I have an Angular SPA running on a SharePoint 2013 page. In the code, I'm using $q to pull data from 10 different SharePoint lists using REST and then merging them into one JSON object for use in a grid. The code runs and outputs the intended merged data but it's leaky and crashes the browser after a while.

Here's the code in the service:

factory.getGridInfo = function() { 
    var deferred = $q.defer();

    var list_1a = CRUDFactory.getListItems("ListA", "column1,column2,column3");
    var list_1b = CRUDFactory.getListItems("ListB", "column1,column2,column3");
    var list_2a = CRUDFactory.getListItems("ListC", "column4");
    var list_2b = CRUDFactory.getListItems("ListD", "column4");
    var list_3a = CRUDFactory.getListItems("ListE", "column5");
    var list_3b = CRUDFactory.getListItems("ListF", "column5");
    var list_4a = CRUDFactory.getListItems("ListG", "column6");
    var list_4b = CRUDFactory.getListItems("ListH", "column6");
    var list_5a = CRUDFactory.getListItems("ListI", "column7");
    var list_5b = CRUDFactory.getListItems("ListJ", "column7");

    $q.all([list_1a, list_1b, list_2a, list_2b, list_3a, list_3b, list_4a, list_4b, list_5a, list_5b]) 
    .then(function(results){
        var results_1a = results[0].data.d.results;
        var results_1b = results[1].data.d.results;
        var results_2a = results[2].data.d.results;
        var results_2b = results[3].data.d.results;
        var results_3a = results[4].data.d.results;
        var results_3b = results[5].data.d.results;
        var results_4a = results[6].data.d.results;
        var results_4b = results[7].data.d.results;
        var results_5a = results[8].data.d.results;
        var results_5b = results[9].data.d.results;

        var combined_1 = results_1a.concat(results_1b);
        var combined_2 = results_2a.concat(results_2b);
        var combined_3 = results_3a.concat(results_3b);
        var combined_4 = results_4a.concat(results_4b);
        var combined_5 = results_5a.concat(results_5b);

        for(var i = 0; i < combined_1.length; i++){ 
            var currObj = combined_1[i];
            currObj["column4"] = combined_2[i].column4;
            currObj["column5"] = combined_3[i].column5;
            currObj["column6"] = combined_4[i].column6;
            currObj["column7"] = combined_5[i].column7;

            factory.newObjectArray[i] = currObj;

        }
        deferred.resolve(factory.newObjectArray);
    },
    function (error) {
        deferred.reject(error);
    });         
    return deferred.promise;
};  

Here's the REST call in CRUDFactory:

factory.getListItems = function (listName, columns){
    var webUrl = _spPageContextInfo.webAbsoluteUrl + "/_api/web/lists/getByTitle('"+listName+"')/items?$select="+columns+"&$top=5000";
    var options = {
        headers: { "Accept": "application/json; odata=verbose" }, 
        method: 'GET', 
        url: webUrl                 
    };
    return $http(options);
};

And then here's the controller bit:

$scope.refreshGridData = function(){ 
    $scope.hideLoadingGif = false;
    $scope.GridData = "";
    GlobalFactory.getGridInfo()
    .then(function(){
        $scope.GridData = GlobalFactory.newObjectArray;
        $scope.hideLoadingGif = true;
    });
};  

UPDATE 1: Per request, Here's the HTML (just a simple div that we're using angular-ui-grid on)

<div ui-grid="GridOptions" class="grid" ui-grid-selection ui-grid-exporter ui-grid-save-state></div>

This code starts by declaring some get calls and then uses $q.all to iterate over the calls and get the data. It then stores the results and merges them down to 5 total arrays. Then, because my list structure is proper and static, I'm able to iterate over one of the merged arrays and pull data from the other arrays into one master array that I'm assigning to factory.newObjectArray, which I'm declaring as a global in my service and using as my grid data source.

The code runs and doesn't kick any errors up but the issue is with (I believe) the "getGridInfo" function. If I don't comment out any of the REST calls, the browser uses 45 MB of data that doesn't get picked up by GC which is then compounded for each click until the session is ended or crashes. If I comment out all the calls but one, my page only uses 18.4 MB of memory, which is high but I can live with it.

So what's the deal? Do I need to destroy something somewhere? If so, what and how? Or does this relate back to the REST function I'm using?

UPDATE 2: The return result that the grid is using (the factory.newObjectArray) contains a total of 5,450 items and each item has about 80 properties after the merge. The code above is simplified and shows the pulling of a couple columns per list, but in actuality, I'm pulling 5-10 columns per list.

Answer 1

At the end of the day you are dealing with a lot of data, so memory problems are potentially always going to be an issue and you should probably consider whether you need to have all the data in memory.

The main goal you should probably be trying to achieve is limiting duplication of arrays, and trying to keep the memory footprint as low as possible, and freeing memory as fast as possible when you're done processing.

Please consider the following. You mention the actual number of columns being returned are more than your example so I have taken that into account.

factory.getGridInfo = function () {

  var deferred = $q.defer(),

    // list definitions
    lists = [
      { name: 'ListA', columns: ['column1', 'column2', 'column3'] },
      { name: 'ListB', columns: ['column1', 'column2', 'column3'], combineWith: 'ListA' },
      { name: 'ListC', columns: ['column4'] },
      { name: 'ListD', columns: ['column4'], combineWith: 'ListC' },
      { name: 'ListE', columns: ['column5'] },
      { name: 'ListF', columns: ['column5'], combineWith: 'ListE' },
      { name: 'ListG', columns: ['column6'] },
      { name: 'ListH', columns: ['column6'], combineWith: 'ListG' },
      { name: 'ListI', columns: ['column7'] },
      { name: 'ListJ', columns: ['column7'], combineWith: 'ListI' },
    ],

    // Combines two arrays without creating a new array, mindful of lenth limitations
    combineArrays = function (a, b) {
      var len = b.length;
      for (var i = 0; i < len; i = i + 5000) {
        a.unshift.apply(a, b.slice(i, i + 5000));
      }
    };

  $q.all(lists.map(function (list) { return CRUDFactory.getListItems(list.name, list.columns.join()); }))
  .then(function (results) {

    var listResultMap = {}, var baseList = 'ListA';

    // map our results to our list names
    for(var i = 0; i < results.length; i++) {
      listResultMap[lists[i].name] = results[i].data.d.results;
    }

    // loop around our lists
    for(var i = 0; i < lists.length; i++) {
      var listName = lists[i].name, combineWith = lists[i].combineWith;
      if(combineWith) {
        combineArrays(listResultMap[combineWith], listResultMap[listName]);
        delete listResultMap[listName];
      }
    }

    // build result
    factory.newObjectArray = listResultMap[baseList].map(function(item) {
      for(var i = 0; i < lists.length; i++) {
        if(list.name !== baseList) {
          for(var c = 0; c < lists[i].columns.length; c++) {
            var columnName = lists[i].columns[c];
            item[columnName] = listResultMap[list.name][columnName];
          }
        }
      }
      return item;
    });

    // clean up our remaining results
    for (var i = 0; i < results.length; i++) {
      delete results[i].data.d.results;
      delete results[i];
    }

    deferred.resolve(factory.newObjectArray);

  },
  function (error) {
    deferred.reject(error);
  });
  return deferred.promise;
};