Python sqlite3 and concurrency

Question

I have a Python program that uses the "threading" module. Once every second, my program starts a new thread that fetches some data from the web, and stores this data to my hard drive. I would like to use sqlite3 to store these results, but I can't get it to work. The issue seems to be about the following line:

conn = sqlite3.connect("mydatabase.db") 

• If I put this line of code inside each thread, I get an OperationalError telling me that the database file is locked. I guess this means that another thread has mydatabase.db open through a sqlite3 connection and has locked it.
• If I put this line of code in the main program and pass the connection object (conn) to each thread, I get a ProgrammingError, saying that SQLite objects created in a thread can only be used in that same thread.

Previously I was storing all my results in CSV files, and did not have any of these file-locking issues. Hopefully this will be possible with sqlite. Any ideas?

Answer 1

Contrary to popular belief, newer versions of sqlite3 do support access from multiple threads.

This can be enabled via optional keyword argument check_same_thread:

sqlite.connect(":memory:", check_same_thread=False)