AngularJS and big JSON data

Question

For a custom AngularJS application that uses data from an API I've been creating I've come across the use of Angular oboe. Oboe is a bower package that helps streaming large JSON files to the view. So after some trial and error, I've managed to build a decent oboe GET method that obtains around 4000 JSON items in around 2 seconds. But my problem starts when adding more GET methods to the same view.

At first there weren't any problems but eventually, the loading time keeps getting bigger and bigger. So I've tried using the Oboe Cached: true configuration. Sadly it doesn't work at all. Every time I load the page all the data is loaded again instead of obtained from the browser Cache

In my example below, you're able to see the structure of one of my oboe function's that I've been trying to cache. A JSfiddle link is added down below as well.

The Function and the view

 function createProduct(id, name) {
        this.id = id;
        this.name = name;
    }

    $scope.products = [];

    oboe({
        url: 'config/get/getProducts.php',
        method: 'GET',
        cached: true
    }).path('products.product.*', function () {
        // we don't have the person's details yet but we know we
        // found someone in the json stream. We can eagerly put
        // their div to the page and then fill it with whatever
        // other data we find:
    }).start(function () {
        console.log("start");
    }).node('products.product.*', function (products) {
        // we just found out their name, lets add it
            // to their div:
            $scope.products.push({
                id: products.id,
                name: products.name.language
            });
            $scope.totalItems = $scope.products.length;
            return new createProduct(products.id, products.name);
    }).done(function () {
        console.log( $scope.products );
    });

    // Refresh data
    $scope.refreshData = function() {
        cartService.refreshData()
            .then(function(response) {
                $scope.cartItems = response.cartItems;
                $scope.totalCartItems = response;
                $scope.selectedCustomer = response;
            })
    };

<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.2.23/angular.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<div class="productimg col-lg-4 col-md-4" ng-repeat="product in products | limitTo : limit : (currentPage - 1) * limit track by product.id"
             ng-class="{lastItem: $last}" scroll-bottom="event">
            <div class="row">
                <div class="col-md-12" ng-bind="product.id"></div>
                <div class="col-md-12">
                    <a ng-bind="product.name" href="{{product.id}}.nl"></a>
                </div>
            </div>
        </div>
        <div class="col-lg-12 text-center margin-t-30">
            <ul uib-pagination
                total-items="totalItems"
                ng-model="currentPage"
                items-per-page="limit">
            </ul>
        </div>

In the JSfiddle You're able to see the code. I couldn't get the JSON to work on on JSfiddle but see it as the following line but then around 10000 "product" rows.

{"products":{"product":[{"id":"1240","id_manufacturer":"0","id_supplier":"0","id_category_default":"2","id_tax_rules_group":"8","quantity":"0","id_shop_default":"1","reference":{},"ean13":"0","price":"0.000000","active":"0","date_add":"2014-07-15 12:06:34","date_upd":"2018-04-21 12:22:37","name":{"language":"zie voorschot factuur 03"}},{"id":"1241","id_manufacturer":"0","id_supplier":"0","id_category_default":"2","id_tax_rules_group":"8","quantity":"0","id_shop_default":"1","reference":{},"ean13":"0","price":"0.000000","active":"0","date_add":"2014-07-15 12:06:41","date_upd":"2018-04-21 12:22:37","name":{"language":"zie voorschot factuur 04"}},{"id":"8908","id_manufacturer":"0","id_supplier":"15","id_category_default":"2","id_tax_rules_group":"8","quantity":"0","id_shop_default":"1","reference":"041002","ean13":"5712084210057","price":"45.454545","active":"1","date_add":"2015-11-12 18:03:47","date_upd":"2017-11-18 09:57:27","name":{"language":"Vaavud Sleipnir smartphone wind meter"}}}}

So the real struggle I'm facing is that getting the data from the network tab takes around ten seconds. (there is an API request at "getProducts.php"). Then parsing that to the view costs around 30 seconds. (way too long). Secondly, I would like to cache the getProducts request so that the products are directly obtained the next time the view is loaded. With a normal $http.get() and cache: true. It worked but then I'm still facing the slow binding, even with Oboe.

If there is any more information needed please let me know in the comments below.

As always, thanks in advance!

Answer 1

We had a project like this which had too much API with heavy object data. Some of tips we used are these:

1-Arrays instead of Objects

With a bit searching and reading about these data structures, you'll find that the memory consumed by objects is more than the memory used by Arrays. So convert your Array of objects to array of arrays in server side.

2-Use one more level cache

Store data in a server side cache, something like MongoDb and Redis. Handle requests with these steps:

1. Check the browser cache. If data is available and up-to-date, use this data.

2. If data is not available in browser, send request to server. Check the server side cache. If data is available and up-to-date, Send this data to client and store it in cache and also use it.

3. If data is not available in server cache, Send request to API and get data, Store it in server cache, and then sent it back to client and store it in cache and also use it.

3-Send minimum data required by client each time

A client may not stay in a page and see all data shown. Prepare minimum data you need for each page and if client request, bring the others.

For your case, instead of 10,000 product, Just prepare 1000 of them for example and if client needs more, prepare more data, append to your client cache and show it.

If all 10,000 products are required by client, prepare them in several requests.

Read more here

4-Cache data in browser with good module and in correct way for reading data again

There are some modules prepared for AngularJs. Read about them, their standards and best practices. This is so important to have a good client cache.

ngStorage

Hope these tips help you.