What to do when hitting the queue size of slick?

Question

I am using Slick 3.0 with HikariCP 2.3.8 (also play 2.4)

I do a lot of database IO and constantly hit the queue limit. Is there a way to get the current queue size and how can it be increased? Or is it even advised to do that or would a wrapper around the database with its own queue be the better option?

The exception i am referring to:

java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2@39496345 rejected from java.util.concurrent.ThreadPoolExecutor@21ee20b4[Running, pool size = 20, active threads = 20, queued tasks = 1000, completed tasks = 7021]

Answer 1

1000 queued tasks seems like a lot to me. Clearly, slick is using an executor with a fixed size queue (1000 elements), and you're running into that limit because tasks are not being retired quickly enough.

The most obvious cause is SQL execution times. If you can get your SQL execution times down, you'll buy yourself a lot of headroom in the queue.

Typically, this would be done database-side by checking query execution plans, and depending on the database, asking the database what the long running queries are.

On the HikariCP side, you might want to enable DropWizard metrics (not sure how to do that through slick, tho) and enable a DropWizard log reporter to log pool statistics every 10 seconds or so.

Probably the most interesting metric there would be usage, as that will show you how long connections are out of the pool between getConnection() and close(). As you tune your database and/or queries, you want to see that number start dropping.

One crucial point is, if the database cannot keep up with your application load, increasing the slick queue from 1000 to 5000 (or even 10000) will buy you nothing except a small amount of time before it reaches that limit. You have to find the source of the performance bottleneck and eliminate it, such that queues are retired faster than your application generates them (excepting temporal spikes of course).

Answer 2

When using Database.forConfig a different value for the queue size can be provided.

slick documentation