Retrieving entire data collection from a RavenDB

Question

I have a requirement where I need to fetch the entire data collection Users from RavenDB and compare the retrieved result set with another set of data. There are close to 4000 records in this particular collection.

Because Raven is Safe By Default, I keep getting an exception for either Number of requests per session exceeded or it returns the maximum 128 records.

I don't want to set the property Session.Advanced.MaxNumberOfRequestsPerSession to a higher value.

What query should I use to get the count of all the records? What is the ideal approach to handle this situation?

Answer 1

You use paging, and read this 1024 items at a time.

int start = 0;
while(true)
{
   var current = session.Query<User>().Take(1024).Skip(start).ToList();
   if(current.Count == 0)
          break;

   start+= current.Count;
   allUsers.AddRange(current);

}

Answer 2

This question was posted before this feature was available in RavenDB, but in case anyone else stumbles upon this now...

The encouraged way to do this is via the Streaming API. The RavenDB client batches the stream so it can automatically 'page' the requests/responses to/from the server. If you opt in to using the Streaming API, the client assumes you "know what you're doing" and does not check the 128/1024/30 limits that are used for regular queries.

var query = session.Query<User>();
 
using (var enumerator = session.Advanced.Stream(query)) {
    while (enumerator.MoveNext()) {
        allUsers.Add(enumerator.Current.Document);
    }
}

var count = allUsers.Count;

Tip: Though this is the encouraged way to solve the problem... As a general rule it is best to avoid the situation to start with. What if there are a million records? That allUsers list is going to get huge. Maybe an index or transform could be done first to filter out what data you actually need to display to the user/process? Is this for reporting purposes? Maybe RavenDB should be automatically exporting to a SQL server with reporting services on it? Etc...