Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Out of memory when doing a big query?

I received this error when trying to do big query.

java.lang.OutOfMemoryError: Java heap space

I've searched and found that applying setAutoCommit(false) and setFetchSize methods to my prepared statement might help handling big query. However, when I used it, I received this error.

java.sql.SQLException: Illegal value for setFetchDirection().

What is the proper and easy way to handle large query?

What is the proper way to use setFetchSize?

like image 404
Atthapon Junpun-eak Avatar asked Apr 26 '13 11:04

Atthapon Junpun-eak


People also ask

Is BigQuery in memory database?

BigQuery is a fast petabyte-scale analytics database. To achieve that level of performance, BigQuery executes queries completely in memory. Most databases and data processing systems achieve scalability using hybrid executors that operate on both disk and memory.

How can I improve my BigQuery performance?

To further improve query performance, consider the benefits of purchasing more reserved slots, in addition to optimizing your data model and queries. BigQuery offers two pricing models for queries: on-demand pricing and flat-rate pricing. On-demand pricing is based on the amount of data processed by each query you run.

Can SQL run out of memory?

In-Memory OLTP uses more memory and in different ways than SQL Server does. It is possible that the amount of memory you installed and allocated for In-Memory OLTP becomes inadequate for your growing needs. If so, you could run out of memory.


1 Answers

Assuming you are using the MySQL Connector/J driver provided by MySQL, I believe the solution is found in this manual page (notice parameter 1 of Connection::createStatement()):

If you are working with ResultSets that have a large number of rows or large values, and cannot allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.

To enable this functionality, create a Statement instance in the following manner:

stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
              java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
like image 76
RandomSeed Avatar answered Oct 28 '22 04:10

RandomSeed