Java: How to efficiently read from database?

Question

Im trying to read out a column from an sql database in java, I want the result to be returned in an array. Here is the function:

public static double[] open(Connection conn,String symbol,int startdate,int enddate)                 throws SQLException {
    int id = database.get_stockid(conn, symbol);
    Statement stat = conn.createStatement();
    ResultSet rs = stat.executeQuery("select price_open from stock_data where stock_id="+id+" and date>="+startdate+" and date<="+enddate+";");
    ArrayList<Double> data = new ArrayList<Double>();
    while(rs.next()) {
        data.add(rs.getDouble("open"));
    }
    double[] data1 = new double[data.size()];
    for(int a = 0; a < data1.length; ++a) {
        data1[a]=data.get(a);
    }
    return data1;
}

This is quite slow. It Takes 1.5 secs with my sqlite database. Is this the standard way to read out a column or am I doing something wrong? This is the bottleneck in my application, so I need it to be as fast as possible.

---

edited: thanks. I just found out that the ArrayList is not causing the problem. The bottleneck must lie in the sql part: If I load data for just 10 days it takes as long as loading data for 10 years. SO I have to improve on my sql but how??

Here the improved code:

public static double[] open(Connection conn,String symbol,int startdate,int enddate) throws SQLException {
    int id = database.get_stockid(conn, symbol);

    PreparedStatement stat = conn.prepareStatement("select price_open from stock_data where (stock_id="+id +") and (date between "+startdate+" and "+enddate+");");
    ResultSet rs = stat.executeQuery();
    ArrayList<Double> data = new ArrayList<Double>();
    while(rs.next()) {
        data.add(rs.getDouble(1));
    }
    double[] data1 = new double[data.size()];
    for(int a = 0; a < data1.length; ++a) {
        data1[a]=data.get(a);
    }
    return data1;
}

Answer 1

1. Replace

   double[] data1 = new double[data.size()];
   for(int a = 0; a < data1.length; ++a) {
       data1[a]=data.get(a);
   }
   

   with

   double[] data1 = data.toArray(new double[data.size()]);
   

2. Check what is the query running time (by profiling of this application or investing the logs on the database side), check if it can be reduced by e.g. introducing indexes on the columns used in the where clause i.d. stock_id and date.

3. If you are able to estimate the amount of records your query will return or you know that it will be at least N records then instead of:

   ArrayList<Double> data = new ArrayList<Double>();
   

   invoke:

   ArrayList<Double> data = new ArrayList<Double>(AMOUNT_OF_RECORDS);
   

   this will allow to prevent expanding of the ArrayList (creating a new array of greater size and copying of the elements from the smaller array to the new bigger array).

   BTW. For the ArrayList class the default initial capacity is 10.

4. Are the results returned from your query unique? Maybe most of the values returned from the query are duplicated? If yes, then append the DISTINCT keyword to your query:

   select distinct price_open from stock_data ...
   

   this will allow to save the time on the communication with the database and also less results returned, less results have to be processed.

5. Use PreparedStatement instead of Statement to:

   • protect from the SQL injection
   • but also because of performance, as using PreparedStatement allows the database to reuse the already parsed query

---

Update #1

1. Remember to always free all the resources, i.d. ResultSet and PreparedStatement.
   • In Java 1.7+, you can use the new try-with-resources statement (The Java® Language Specification)
   • In older Java versions, you have to put calls to the close methods in the finally block and have separate exception handling for each call to prevent a scenario when an exception thrown in the first close prevents the second close from being called.