Query from minimongo of large number of records stucks and hangs browser

Question

I am building a page for admin in angular-meteor.

I have published all the records from a collection: "posts" and have taken the subscription of all the records on front end.

$meteor.subscribe('posts');

In the controller, if I select the cursors of all records from minimongo it works fine like:

$scope.posts = $meteor.collection(Posts);

But I want to display pagination, so for that I want limited records at a time like:

$scope.posts = $meteor.collection(function(){

        return Posts.find(

            {}, 
            {
                sort:   {'cDate.timestamp': -1},
                limit:  10
            }
        );
    });

It stucks with the query in minimongo. And the browser hangs.

"posts" collection contains only 500 records. It was working fine when I had 200 records.

Can anyone give me an idea whats wrong with my code and concepts?

EDIT:

Okay! It worked fine when I commented the $sort line from query like this:

$scope.posts = $meteor.collection(function(){

            return Posts.find(

                {}, 
                {
                    //sort:   {'cDate.timestamp': -1},
                    limit:  10
                }
            );
        });

But I need to sort the records. So what should I do now?

EDIT:

Also tried adding index to the sort attribute like this:

db.Posts.ensureIndex({"cDate.timestamp": 1})

Still same issue.

Answer 1

Change your publication to accept a parameter called pageNumber like this

Meteor.publish('posts', function (pageNumber) {
   var numberOfRecordsPerPage = 10;
   var skipRecords = numberOfRecordsPerPage * (pageNumber - 1);
   return Post.find({
        "user_id": user_id
   }, {
        sort: { 'cDate.timestamp': -1 }
        skip: skipRecords, 
        limit: numberOfRecordsPerPage
   });
});

On client side, I didn't work with angular-meteor much. You can create a pageNumber property under your current scope using this.pageNumber or $scope.pageNumber. Update this pageNumber variable whenever your pagination page is clicked. Whenever this variable is changed, subscribe using the current page number.

If it is using standard blaze template, I would do it using a reactive var or session var in an autorun like this. In template html:

<template name="postsTemplate">
     <ul>
         <!-- you would want to do this list based on total number of records -->
         <li class="pagination" data-value="1">1</li>
         <li class="pagination" data-value="2">2</li>
         <li class="pagination" data-value="3">3</li>
     </ul>
</template>

In template js:

Template.postsTemplate.created = function () {
    var template = this;
    Session.setDefault('paginationPage', 1);
    template.autorun(function () {
       var pageNumber = Session.get('paginationPage');
       Meteor.subscribe('posts', pageNumber);
    });
}

Template.postsTemplate.events(function () {
    'click .pagination': function (ev, template) {
         var target = $(ev.target);
         var pageNumber = target.attr('data-value');
         Session.set('paginationPage', pageNumber);
     }
});

This way, you will have a maximum of 10 records at any point in time on the client, so it will not crash the browser. You might also want to limit the fields that you send to client using something like this

Meteor.publish('posts', function (pageNumber) {
   var numberOfRecordsPerPage = 10;
   var skipRecords = numberOfRecordsPerPage * (pageNumber - 1);
   return Post.find({
        "user_id": user_id
   }, {
        sort: { 'cDate.timestamp': -1 }
        skip: skipRecords, 
        limit: numberOfRecordsPerPage,
        fields: {'message': 1, 'createdBy': 1, 'createdDate': 1 } //The properties inside each document of the posts collection.
   });
});

And finally you will need the total number of records in posts collection on client side, to show the pagination links. You can do it using a different publication and using the observeChanges concept as mentioned in the official documentation here

// server: publish the current size of a collection
Meteor.publish("posts-count", function () {
  var self = this;
  var count = 0;
  var initializing = true;

  // observeChanges only returns after the initial `added` callbacks
  // have run. Until then, we don't want to send a lot of
  // `self.changed()` messages - hence tracking the
  // `initializing` state.
  var handle = Posts.find({}).observeChanges({
    added: function (id) {
      count++;
      if (!initializing)
        self.changed("postsCount", 1, {count: count});
    },
    removed: function (id) {
      count--;
      self.changed("postsCount", 1, {count: count});
    }
    // don't care about changed
  });

  // Instead, we'll send one `self.added()` message right after
  // observeChanges has returned, and mark the subscription as
  // ready.
  initializing = false;
  self.added("postsCount", 1, {count: count});
  self.ready();

  // Stop observing the cursor when client unsubs.
  // Stopping a subscription automatically takes
  // care of sending the client any removed messages.
  self.onStop(function () {
    handle.stop();
  });
});

// client: declare collection to hold count object
PostsCount = new Mongo.Collection("postsCount");

// to get the total number of records and total number of pages
var doc = PostsCount.findOne(); //since we only publish one record with "d == 1", we don't need use query selectors
var count = 0, totalPages = 0;

if (doc) {
    count = doc.count;
    totalPages = Math.ceil(count / 10); //since page number cannot be floating point numbers..
}

Hope this helps.

Answer 2

Browser crashing because there is only so much data that it can load in it's cache before itself cashing out. To your question about what happens when you need to demand a large number of documents, take that process away from the client and do as much on the server through optimized publish and subscribe methods / calls. No real reason to load up a ton of documents in the browser cache, as minimongo's convenience is for small data sets and things that don't need to immediately sync (or ever sync) with the server.