Slow pagination over tons of records in mongodb

Question

I have over 300k records in one collection in Mongo.

When I run this very simple query:

db.myCollection.find().limit(5); 

It takes only few miliseconds.

But when I use skip in the query:

db.myCollection.find().skip(200000).limit(5) 

It won't return anything... it runs for minutes and returns nothing.

How to make it better?

Answer 1

One approach to this problem, if you have large quantities of documents and you are displaying them in sorted order (I'm not sure how useful skip is if you're not) would be to use the key you're sorting on to select the next page of results.

So if you start with

db.myCollection.find().limit(100).sort({created_date:true}); 

and then extract the created date of the last document returned by the cursor into a variable max_created_date_from_last_result, you can get the next page with the far more efficient (presuming you have an index on created_date) query

db.myCollection.find({created_date : { $gt : max_created_date_from_last_result } }).limit(100).sort({created_date:true});  

Answer 2

From MongoDB documentation:

Paging Costs

Unfortunately skip can be (very) costly and requires the server to walk from the beginning of the collection, or index, to get to the offset/skip position before it can start returning the page of data (limit). As the page number increases skip will become slower and more cpu intensive, and possibly IO bound, with larger collections.

Range based paging provides better use of indexes but does not allow you to easily jump to a specific page.

You have to ask yourself a question: how often do you need 40000th page? Also see this article;