how to debug locked sqlite3 database

Question

I am writing an application on iOS that uses sqlite3 wrapped by fmdatabase. I am running into the problem that at some point, my program is stuck in a loop inside of FMDatabase libraries, particularly, a function that calls sqlite3_step and finds out that the database is busy, then retries over and over again.

I am looking for general debugging tools and tips, as it will be too much to give my whole setup here. Some things that might be of consequence, I opened a database handle to a database that already has a handle in another thread. sqlite3_threadsafe() returns 2, so I know its enabled. I have also tested this new connection by making a very simple select and update statement. When I let my program run, and when it tries to run an update on the database, i get stuck.

The update statement that my program makes itself is not at fault, because this query runs fine when I do NOT open up two connections. Yet, I fail to see where I could be going wrong...

Any help or tips on where I might be wrong would be greatly appreciated.

Answer 1

SQLite locks the entire database during a write operation (i.e. when a write is happening on any table, no other write, to any table anywhere can happen at the same time). Some databases provide concurrent writes via table-level locks, or sometimes row-level locks. To contrast this to SQLite's implementation, a table-level lock basically means that when you're writing data to a given table, no other thread can write to any record in that table, at the same time (however, writes to other tables can occur simultaneously, in some circumstances). Similarly, row-level locks take it even further, and allow only the necessary rows involved to be locked, allowing concurrent writes to the same table to happen from multiple threads. The idea here is to minimize the amount of data you need to lock for a write operation, which effectively increases the amount of concurrent writes possible across the database, and depending on your implementation/how you use your database, this can significantly increase throughput.

Now, back to your question...

The fact that SQLite is threadsafe doesn't mean that multiple threads can concurrently write to it - it means that it has a way of handling access from multiple threads - which is to (a) allow timeouts/retries, and (b) to return a useful error (SQLITE:Busy) when a lock is currently held on the database. That is, threadsafe means nothing more than, "Multiple threads can access this data in a way that won't result in data corruption due to simultaneous access."

Basically, somewhere in the code, one thread is trying to do its update before another thread has released its lock on the database. This is a common hurdle with SQLite, because the authors/documentation will tell you that SQLite can handle concurrency like a champ. The reality is that what SQLite considers "concurrency support" amounts to trying to be very fast so that locks on the database are only held for a very short time, and therefore locks on the database are released before timeouts are hit. In a lot of cases, this works just fine and never gets in your way. However, having very short-lived locks is not the same as actually allowing concurrent writes from multiple threads.

Think of it like the way that iOS does multitasking (at least as of iOS 5, when I'm writing this) - really what it's doing is putting other apps on pause, and coming back to them. This has the effect that (a) battery life is much better due to lower CPU utilization, and (b) you don't have to start an app from scratch every time you launch it. This is great, but the actual word "multitasking" as used in iOS doesn't technically mean the same thing as "multitasking" in other environments (even Mac OS X).

SQLite is the same way. Do they have "concurrency" support? Well sort of, but the way they define the word "concurrency" isn't the way the rest of the DB world defines "concurrency".

No one is really wrong, but in cases like these, it adds to implementation confusion.