I have a query that is basically a select *
. In development this table is only 30000 rows, but in production it will much bigger. So I want to consume this query lazily. Why is the query below not lazy?
I am using Postgres 9.5.4.1.
(do
(def pg-uri {:connection-uri "jdbc:postgresql://localhost/..."})
(def row (atom 0))
(take 10 (clojure.java.jdbc/query
pg-uri
["select * from mytable"]
{:fetch-size 10
:auto-commit false
:row-fn (fn [r] (swap! row inc))}))
@row) ;;=> 300000
First, see https://jdbc.postgresql.org/documentation/83/query.html#query-with-cursor.
Solved it like this.
(jdbc/with-db-transaction [tx connection]
(jdbc/query tx
[(jdbc/prepare-statement (:connection tx)
"select * from mytable"
{:fetch-size 10})]
{:result-set-fn (fn [result-set] ...)}))
where :result-set-fn
is a function that consumes the lazy result set.
with-db-transaction
takes care of autoCommit
set to false
.
:fetch-size
is not passed from query
so you have to make a prepare-statement
yourself.
clojure.java.jdbc
supports lazy processing of large result sets natively these days (the other answers here predate that native support). See the community documentation about it here:
http://clojure-doc.org/articles/ecosystem/java_jdbc/using_sql#processing-a-result-set-lazily
In particular, see the Additional Options? section for database-specific tweaks you might need. You can specify :auto-commit? false
on any function that would open a new connection, and you can specify :fetch-size
and the various cursor controls on any query-related function. See this StackOverflow question & answer for details of what PostgreSQL might need:
Java JDBC ignores setFetchSize?
Currently, you'll have to dig in the clojure.java.jdbc
source or the prepare-statement
reference documentation for more of those options. I'm continuing to work on the community documentation to surface all of that information.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With