SQLite/C# Connection Pooling and Prepared Statement Confusion

Question

I have been spending some time reading different best practices for databases and for SQLite specifically. While reading I found I was doing many things I shouldn't be doing and when attempting to fix these issues I became confused when thinking about some of the finer details of using SQLite with it's ADO implementation.

My confusion stems specifically from prepared statements and connection pooling.

While reading http://msdn.microsoft.com/en-us/library/ms971481.aspx I found that connections should only be opened for a transaction. Once the transaction is complete then the connection should be closed. I do not have a firm grasp as to why this is the case, but I have been working off the assumption that the author(s) know better then I. I understand that when a connection is closed it doesn't mean it has actually been closed. It simply means that it has been put back into the pool.

Now to improve my queries and inserts I read about using prepared statements. In SQLite, do prepared statements really improve performance? and http://petesbloggerama.blogspot.com/2007/02/sqlite-adonet-prepared-statements.html both seemed to indicate that when executing a query that will be done multiple times prepared statements are the way to go. I have also read that a prepared statement is specific to a connection and that once the connection is closed the prepared statement is lost.

My confusion is this. If I am opening and closing my connection (which may or may not mean the connection is being closed due to the thread pool) then how much use am I really getting from a prepared statement? I can understand that if I have 1000 objects I need to save in a single transaction that the prepared statement can help a lot. However I don't believe I would see a benefit from saving a single object in a transaction because once I close the connection the prepared statement that was generated from the first object is now lost. Is this a true statement?

My confusion is furthered by the fact that I believe a prepared statement is linked to the scope of my SQLiteCommand object.

If I create a SQLiteCommand that represents a query that I will be executing often do I need to keep that SQLiteCommand in memory for the prepared statement to stay active?

If I create a new SQLiteCommand with the same SQLite statement is it recognized that the new SQLiteCommand is the same as the previous and thus has a prepared statement that can be used?

If I keep a SQLiteCommand in memory and change it's parameters and connection as I open and close the connection for different transactions am I essentially keeping a prepared statement alive between different connections?

I am most likely over thinking things at this point but I hope you can help me better understand how these things interact so I can get the most benefit out of them.

Answer 1

It helps to remember that both connection pooling and prepared (compiled) statements are just tools that have their limits and no approach can be equally suitable to all possible situations. With this in mind, let's remember when one might want to use connection pooling and prepared statements.

Possible Reasons to Use Connection Pooling

Connection pooling is useful when connections are expensive, for example:

• It takes significant time to establish a connection (network connections to a SQL Server or Oracle DB) and it is beneficial to "cache" open connections in an attempt to improve system performance.
• Connections are limited and shared within an application (connections from a web application serving multiple concurrent requests) or between applications so they have to be released as soon as possible to let the other clients continue.

Possible Reasons to Use Prepared Statements

Prepared statements are simply meant to improve performance of re-usable queries by cutting down the parsing time.

SQLite: What's the Best Choice?

The answer depends on your application requirements. Personally, I'm not sure if SQLite connection pooling is necessarily a good choice. If your application is single-threaded, it might be best to use a single permanent connection to the SQLite DB, which could be much faster than pooling and would allow you to use prepared statements too. This is different from SQL Server where connection pooling is a very reasonable default.

If performance matters, you should definitely profile the application to see if the SQLite connection pooling is beneficial for your scenario.

Specific Questions

Most of the answers are related to the current System.Data.SQLite provider source.

If I am opening and closing my connection (which may or may not mean the connection is being closed due to the thread pool) then how much use am I really getting from a prepared statement?

Generally, you should treat a connection coming out of the pool as new, i.e. you should not expect to get any benefit from statements prepared previously. The statement will be "re-prepared" unless you keep both the command and connection.

However I don't believe I would see a benefit from saving a single object in a transaction because once I close the connection the prepared statement that was generated from the first object is now lost. Is this a true statement?

This is a true statement.

If I create a SQLiteCommand that represents a query that I will be executing often do I need to keep that SQLiteCommand in memory for the prepared statement to stay active?

Yes, you need to keep it. SQLiteCommand holds a reference to the prepared statement.

If I create a new SQLiteCommand with the same SQLite statement is it recognized that the new SQLiteCommand is the same as the previous and thus has a prepared statement that can be used?

I don't think it's supported.

If I keep a SQLiteCommand in memory and change it's parameters and connection as I open and close the connection for different transactions am I essentially keeping a prepared statement alive between different connections?

If you change the SQLiteCommand's connection, the statement will be "re-prepared".

Answer 2

I didn't catch exactly what is the core problem, but if the problem is how to insert bulk insert statements in one transaction in so little time.

Here is a helper class I found earlier that could help you:

SQLiteBulkInsertHelper.cs

You can use it like this:

SQLiteBulkInsertHelper ContactBlk = new SQLiteBulkInsertHelper("<SQLiteConnection>","<Table Name>"); ContactBlk.AllowBulkInsert = true; ContactBlk.AddParameter("<Column Name>", /*Column Data Type*/System.Data.DbType.Int64); ContactBlk.AddParameter("<Column Name>", /*Column Data Type*/System.Data.DbType.String); ContactBlk.Insert(new object[] {<First Column Value>,<Second Column Value>}); ContactBlk.Flush(); 

Give it a try if you see it as a solution to your problem.