python build from source: cannot build optional module sqlite3

Question

I am building python (version 3.5) from source in order to get the latest version. I can make python and install it, but several "optional" modules including _sqlite3 are not installing:

$>./configure --prefix=/my/prefix && make 

Yields the following:

...

Python build finished successfully!

The necessary bits to build these optional modules were not found:

_bz2 _lzma _sqlite3

_ssl _tkinter readline

To find the necessary bits, look in setup.py in detect_modules() for the module's name.

I (frustratedly) installed sqlite3 from source also to ensure sqlite3 development files exist somewhere, and I believe I have set the necessary lib and include paths as per the related lzma module install tutorial:

$> find / -name libsqlite3.so
/home/username/myproject/lib/libsqlite3.so

$> find / -name sqlite3.h
/home/username/myproject/include/sqlite3.h

$> echo $LD_LIBRARY_PATH
/home/username/myproject/lib

$> echo $LDFLAGS
-L/home/username/myproject/lib

$> echo $CFLAGS
-I/home/username/myproject/include

And yet, when I run ./configure --prefix=/my/prefix --enable-loadable-sqlite-extensions && make, I am given the above error that _sqlite3 (amongst others) was not installed because the necessary bits were not found.

Based on this related answer, it seems I may need to change setup.py? Is that correct?

Is there really no better way than hacking up the python setup.py script?

By the way, I realize that installing sqlite-dev with yum may fix this issue and put the relevant sqlite3 files somewhere obvious to the python installation, but I am not positive that I will be able to do that due that due to limited repository access.

Answer 1

This link provided the solution for me building Python 3.5. Specifically for Ubuntu but helped figure it out for CentOS6 as well.

Install missing packages before compiling Python3

More specifically for Ubuntu server 16.04:

for pkg in build-essential zlib1g-dev libbz2-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev libgdbm-dev liblzma-dev tk8.5-dev lzma lzma-dev libgdbm-dev
do
    apt-get -y install $pkg
done

Answer 2

The setup.py script does not check any environment variables for the location of the sqlite3.h file or any other related files, and therefore changing environment variables is insufficient to allow python to find the files, unless sqlite3-dev packages are installed into the "standard" directories.

The following snippet of possible include directories for sqlite3 is taken from setup.py (for Python-3.5.0):

sqlite_inc_paths = ['/usr/include',
                    '/usr/include/sqlite',
                    '/usr/include/sqlite3',
                    '/usr/local/include',
                    '/usr/local/include/sqlite',
                    '/usr/local/include/sqlite3',
                     ]

From that, its clear that if sqlite3 is not installed in a "standard" system location such as /usr or /usr/local, then the header files will not be found.

To fix the issue, add in /path/to/my/personal/sqlite/include into the above sqlite_inc_paths array:

sqlite_inc_paths = ['/path/to/my/personal/sqlite/include',
                   ...]                          

And sqlite module will be found.

Automated install. To automate the above change, a perl one liner can be used to make the above change:

$> perl -pi.orig -e "s|(?<=sqlite_inc_paths = )\[|['/path/to/my/personal/sqlite/include',\n|" setup.py

sed can also be used, but the -i in-place flag doesn't work on all systems.