Why does SQLite give a "database is locked" for a second query in a transaction when using Perl's DBD::SQLite?

Question

Is there a known problem with SQLite giving a "database is locked" error for a second query in a single transaction when using Perl DBD::SQLite? Scenario: Linux, Perl DBI, AutoCommit => 0, a subroutine with two code blocks (using the blocks to localize variable names). In the first code block a query handle is created by prepare() on a select statement, it is executed() and the block closed. The second code block another query handle is created by prepare for an update statement, and frequently (30% of the time) SQLite/DBI gives a database locked error at this stage. I think the error happens during prepare() and not during the execute().

My work around is to commit after the first query. ( Calling finish on the first query did not help). I prefer not to commit for several reasons relating to elegance and performance. The original code has worked fine for many years with Postgres as the database. I tried sqlite_use_immediate_transaction with no effect.

In all other situations, I've found SQLite to perform very well, so I suspect this is an oversight in the DBD driver, rather than an issue with SQLite. Sadly, my current code is a big pile of scripts and modules, so I don't have a short, single file test case.

Answer 1

Not related to this in anyway is it: Transaction and Database Locking from the DBD::SQLite perldoc?

Transaction by AutoCommit or begin_work is nice and handy, but sometimes you may get an annoying "database is locked" error. This typically happens when someone begins a transaction, and tries to write to a database while other person is reading from the database (in another transaction). You might be surprised but SQLite doesn't lock a database when you just begin a normal (deferred) transaction to maximize concurrency. It reserves a lock when you issue a statement to write, but until you actually try to write with a commit statement, it allows other people to read from the database. However, reading from the database also requires shared lock, and that prevents to give you the exclusive lock you reserved, thus you get the "database is locked" error, and other people will get the same error if they try to write afterwards, as you still have a pending lock. busy_timeout doesn't help in this case.

To avoid this, set a transaction type explicitly. You can issue a begin immediate transaction (or begin exclusive transaction) for each transaction, or set sqlite_use_immediate_transaction database handle attribute to true (since 1.30_02) to always use an immediate transaction (even when you simply use begin_work or turn off the AutoCommit.).

my $dbh = DBI->connect("dbi:SQLite::memory:", "", "", {
  sqlite_use_immediate_transaction => 1,
});

Note that this works only when all of the connections use the same (non-deferred) transaction. See http://sqlite.org/lockingv3.html for locking details.