Can I achieve scalable multi-threaded access to an in-memory SQLite database

Question

I have a multi-threaded Linux C++ application that needs a high performance reference data lookup facility. I have been looking at using an in-memory SQLite database for this but can't see a way to get this to scale in my multi-threaded environment.

The default threading mode (serialized) seems to suffer from a single coarse grained lock even when all transactions are read only. Moreover, I don't believe I can use multi-thread mode because I can't create multiple connections to a single in-memory database (because every call to sqlite3_open(":memory:", &db) creates a separate in-memory database).

So what I want to know is: is there something I've missed in the documentation and it is possible to have multiple threads share access to the same in-memory database from my C++ application.

Alternatively, is there some alternative to SQLite that I could be considering ?

Answer 1

Yes! see the following extracted from the documentation at: http://www.sqlite.org/inmemorydb.html

But its not a direct connection to DB memory, instead to the shared cache.Its a workaround. see the picture.

In-memory Databases And Shared Cache

In-memory databases are allowed to use shared cache if they are opened using a URI filename. If the unadorned ":memory:" name is used to specify the in-memory database, then that database always has a private cache and is this only visible to the database connection that originally opened it. However, the same in-memory database can be opened by two or more database connections as follows:

rc = sqlite3_open("file::memory:?cache=shared", &db);

Or,

ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;

This allows separate database connections to share the same in-memory database. Of course, all database connections sharing the in-memory database need to be in the same process. The database is automatically deleted and memory is reclaimed when the last connection to the database closes.

If two or more distinct but shareable in-memory databases are needed in a single process, then the mode=memory query parameter can be used with a URI filename to create a named in-memory database:

rc = sqlite3_open("file:memdb1?mode=memory&cache=shared", &db);

Or,

ATTACH DATABASE 'file:memdb1?mode=memory&cache=shared' AS aux1;

When an in-memory database is named in this way, it will only share its cache with another connection that uses exactly the same name.