Scala Slick Lazy Fetch

Question

I want to be able to fetch all records from a very big table using Slick. If I try to do this through foreach, for or list fetching; I get an Out Of Memory Exception.

Is there any way to use "cursors" with Slick or lazy loading that only fetch the object when needed reducing the amount of memory used?

Answer 1

Not sure what do you mean by cursors, but you can fetch partial data using pagination:

query.drop(0).take(1000) will take the first 1000 records

query.drop(1000).take(1000) will take from 1001 to 2000 lines of the table.

But this query efficiency will depend on your database, if it will support it, if the table is right indexed.

Answer 2

you could use the combination of iterator which returns an iterator:

 val object = Objects.where(...).map(w => w).iterator()

and a groupby:

val chunkSize = 1000
val groupedObjects = objects.grouped(chunkSize)
groupedObjects.foreach {objects => objects.par.map(h => doJob(h))}

as suggest in this answer