How can I attach an in-memory SQLite database in Python?

Question

I'd like to merge SQLite databases, and some may be in memory. I create the in-memory databases by specifying the database path as :memory:. Following this post, using the attach feature of SQLite seems like both a simple and efficient approach. But how can I specify my in-memory database as the source to attach?

For example, I'd want to do something like:

c1 = sqlite3.connect(":memory:")
c1.execute(...create table, insert a bunch, commit...)

c2 = sqlite3.connect(":memory:")
c2.execute("""
  ATTACH ? AS ToMerge;
  BEGIN; 
    INSERT INTO Records SELECT * FROM ToMerge.Records; 
  COMMIT;
""", (c1.get_attach_id(), ))

but, of course, c1.get_attach_id() is a method I made up for demonstration purposes, since using the string :memory: would be ambiguous. How can I specify the existing c1 database?

Answer 1

A plain :memory: string connecting to an in-memory database cannot be shared or attached to from other connections.

You need to use a file: URI filename connection string with a ?cache=shared parameter to be able to share the in-memory database between connections; then you can also attach to it:

# first connection
c1 = sqlite3.connect("file::memory:?cache=shared", uri=True)

# second connection, to the *same database*
c2 = sqlite3.connect("file::memory:?cache=shared", uri=True)

# third connection, to a different database altogether
c3 = sqlite3.connect('/tmp/sqlite3.db', uri=True)
# can attach to the shared in-memory database, but only if you used
# uri=True on the original connection
c3.execute("ATTACH DATABASE 'file::memory:?cache=shared' AS inmem")

See the In-Memory Databases documentation.

Note that there can only be one such shared in-memory database; all other in-memory databases must remain private to their connection. Use databases with an actual filesystem storage if you need more complex setups; these are easy enough to clean up afterwards anyway if you create these in a tempfile.mkdtemp() temporary directory each.

Demo:

>>> import sqlite3
>>> c1 = sqlite3.connect("file::memory:?cache=shared", uri=True)
>>> c1.execute('CREATE TABLE foo (bar, baz)')
<sqlite3.Cursor object at 0x106839490>
>>> c1.execute("INSERT INTO foo VALUES ('spam', 'ham')")
<sqlite3.Cursor object at 0x106839500>
>>> c1.commit()
>>> c2 = sqlite3.connect("file::memory:?cache=shared", uri=True)
>>> list(c2.execute('SELECT * FROM foo'))
[(u'spam', u'ham')]
>>> c3 = sqlite3.connect('/tmp/sqlite3.db', uri=True)
>>> c3.execute("ATTACH DATABASE 'file::memory:?cache=shared' AS inmem")
<sqlite3.Cursor object at 0x1068395e0>
>>> list(c3.execute('SELECT * FROM inmem.foo'))
[(u'spam', u'ham')]

Support for in-memory shared-cache connections was added to SQLite version 3.7.13; for Python you can check the version of the underlying library with sqlite3.sqlite_version (string) or sqlite3.sqlite_version_info (tuple with integers):

>>> sqlite3.sqlite_version_info
(3, 8, 10, 2)