sqlite3 error on AWS lambda with Python 3

Question

I am building a python 3.6 AWS Lambda deploy package and was facing an issue with SQLite.

In my code I am using nltk which has a import sqlite3 in one of the files.

Steps taken till now:

1. Deployment package has only python modules that I am using in the root. I get the error: Unable to import module 'my_program': No module named '_sqlite3'

2. Added the _sqlite3.so from /home/my_username/anaconda2/envs/py3k/lib/python3.6/lib-dynload/_sqlite3.so into package root. Then my error changed to:

   Unable to import module 'my_program': dynamic module does not define module export function (PyInit__sqlite3)

3. Added the SQLite precompiled binaries from sqlite.org to the root of my package but I still get the error as point #2.

My setup: Ubuntu 16.04, python3 virtual env

AWS lambda env: python3

How can I fix this problem?

Answer 1

This is a bit of a hack, but I've gotten this working by dropping the _sqlite3.so file from Python 3.6 on CentOS 7 directly into the root of the project being deployed with Zappa to AWS. This should mean that if you can include _sqlite3.so directly into the root of your ZIP, it should work, so it can be imported by this line in cpython:

https://github.com/python/cpython/blob/3.6/Lib/sqlite3/dbapi2.py#L27

Not pretty, but it works. You can find a copy of _sqlite.so here:

https://github.com/Miserlou/lambda-packages/files/1425358/_sqlite3.so.zip

Good luck!

Answer 2

Depending on what you're doing with NLTK, I may have found a solution.

The base nltk module imports a lot of dependencies, many of which are not used by substantial portions of its feature set. In my use case, I'm only using the nltk.sent_tokenize, which carries no functional dependency on sqlite3 even though sqlite3 gets imported as a dependency.

I was able to get my code working on AWS Lambda by changing

import nltk 

to

import imp import sys sys.modules["sqlite"] = imp.new_module("sqlite") sys.modules["sqlite3.dbapi2"] = imp.new_module("sqlite.dbapi2") import nltk 

This dynamically creates empty modules for sqlite and sqlite.dbapi2. When nltk.corpus.reader.panlex_lite tries to import sqlite, it will get our empty module instead of the standard library version. That means the import will succeed, but it also means that when nltk tries to use the sqlite module it will fail.

If you're using any functionality that actually depends on sqlite, I'm afraid I can't help. But if you're trying to use other nltk functionality and just need to get around the lack of sqlite, this technique might work.