Iterate over large collection in MongoDB via spring-data

Question

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 1

Late response, but maybe will help someone in the future. Spring data doesn't provide any API to wrap Mongo DB Cursor capabilities. It uses it within find methods, but always returns completed list of objects. Options are to use Mongo API directly or to use Spring Data Paging API, something like that:

        final int pageLimit = 300;         int pageNumber = 0;         Page<T> page = repository.findAll(new PageRequest(pageNumber, pageLimit));         while (page.hasNextPage()) {             processPageContent(page.getContent());             page = repository.findAll(new PageRequest(++pageNumber, pageLimit));         }         // process last page         processPageContent(page.getContent()); 

UPD (!) This method is not sufficient for large sets of data (see @Shawn Bush comments) Please use Mongo API directly for such cases.