Lazy Loading/More Data Scroll in Mongoose/Nodejs

Question

I was wondering how to implement lazy loading/more data on scroll with mongoose. I want to load 10 posts at a time, but I'm not sure how to best load the next 10 elements in a query.

I currently have:

var q = Post.find().sort("rating").limit(10);

To load the 10 posts with the highest "rating". How do i go about doing this for the next 10 posts?

Answer 1

The general concept of "paging" is to use .skip() which essentially "skips" over the results that have already been retrieved, so you can essentially do this:

var q = Post.find().sort( "rating" ).skip(10).limit(10);

But really, as you can imagine this is going to slow down considerably when you get a few "pages" in. So you really want something smarter. Essentially this is a "range query" where you want to grab higher (or lower if descending ) results than the last set of results retrieved. So given the last value of 5 then for greater than you would do:

var q = Post.find({ "rating": { "$gt": 5 } }).sort( "rating" ).limit(10);

Looks Okay, but really there is still a problem. What if the next "page" also contained results with a rating of 5? This query would skip over those and never display them.

The smart thing to do is "keep" all of the _id values from the document since they are unique keys. Basically apply the same sort of thing, except this time you make sure you are not including the results from the previous page in your new one. The $nin operator helps here:

var q = Post.find({ "rating": { "$gte": 5 }, "_id": { "$nin": seenIds } })
    .sort( "rating" ).limit(10);

Whether the seenIds is just the last page of results or some more depends on the "density" of the value you are sorting on, and of course you need to "keep" these in a session variable or something.

But try to adapt this, as range queries are usually your best performance result.