ImportError: No module named 'pysqlite2' when running tests in Python 3 Ubuntu

Question

Some background: we have a codebase written in Python 3 that uses Pyramid and the SqlAlchemy ORM to persist to a mysql database. To write tests for our classes using the ORM we are using Sqlite. All of this works fine together... locally.

Setting up our Jenkins (Ubuntu) server to run the test suite, inside a virtualenv, we run into a problem. The tests are executed like so:

coverage run --source src/ --omit=src/tests/ -m py.test

Tests not involving the ORM are fine. Those with the ORM throw this error:

____________________________________________________________ TestSGenre.test_get_all_success _____________________________________________________________

self = <tests.common.orm.models.test_s_genre.TestSGenre testMethod=test_get_all_by_discipline_success>

    def setUp(self):
        DBSession.remove()
>       self.engine = setup()

source/src/tests/common/orm/models/test_s_genre.py:13: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
source/src/tests/common/orm/orm_setup.py:7: in setup
    engine = create_engine('sqlite://', connect_args={'check_same_thread': False}, poolclass=StaticPool)
/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/engine/__init__.py:386: in create_engine
    return strategy.create(*args, **kwargs)
/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py:74: in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py:339: in dbapi
    raise e
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'sqlalchemy.dialects.sqlite.pysqlite.SQLiteDialect_pysqlite'>

    @classmethod
    def dbapi(cls):
        try:
>           from pysqlite2 import dbapi2 as sqlite
E           ImportError: No module named 'pysqlite2'

/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py:334: ImportError

Checking for sqlite3 (from inside the virtualenv) is successful:

(swapenv)user@jenkins:/var/lib/jenkins/workspace/SWAP_Unit_Test$ which sqlite3
/usr/bin/sqlite3

It is also successful outside the virtualenv. We've tried installing and reinstalling all number of sqlite packages, sqlite-dev, etc. Supposedly the Sqlite library is part of Python 3, but then why can't it be found when the tests are being run?

Answer 1

You are missing the sqlite3 Python module, which you can verify with:

bin/python -c 'import sqlite3'

The which sqlite3 command only shows you have the sqlite3 command line tool installed; this is not what Python uses. It uses the libsqlite3 shared library (which the command line tool also uses). If missing, it means Python was not able to find the SQLite development headers when you built Python.

On Ubuntu, you need to install libsqlite3-dev to get those headers.

You may be missing other dependencies; on Ubuntu I'd install:

libreadline6-dev
libbz2-dev
libssl-dev
libsqlite3-dev
libncursesw5-dev
libffi-dev
libdb-dev
libexpat1-dev
zlib1g-dev
liblzma-dev
libgdbm-dev
libmpdec-dev

Some of these are accelerator packages; Python will work without them but some modules will be slower (e.g. decimal without the mpdecimal library).

You may want to verify the Ubuntu Python 3.4 source package dependencies for your Ubuntu version.

Answer 2

Running Debian Buster, I discovered the same problem with python3.6, even though, python3.5 successfully imported sqlite3. And even though the sqlite3 module was installed and should have been available to python3.6. My solution was to run

export PYTHONPATH=$PYTHONPATH:/usr/lib/python3.6/lib-dynload

For some reason I have not yet determined, the module directory for python3.6 does not get loaded correctly for sqlite3. This solution worked both inside and outside of virtual environments.