WARNING: IPython History requires SQLite, your history will not be saved

Question

Hi I'm using Ubuntu release 12.10 (quantal) 32-bit with Linux Kernel 3.5.0-21-generic. I'm trying to get IPython's History to work. I've set it up using pythonbrew and a virtual environment. In there I use pip to install IPython. Currently, when I start up IPython in a terminal I get:

WARNING: IPython History requires SQLite, your history will not be saved
Python 2.7.3 (default, Nov  8 2012, 18:25:10) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Searching on the warning in the first line, I found this issue report, so I went back and installed the following:

sudo apt-get install libsqlite0 libsqlite0-dev libsqlite3-0 libsqlite3-dev

and then removed and reinstalled pysqlite using pip

pip uninstall pysqlite
pip install pysqlite

After that I thought I would check the installation by importing the module:

Python 2.7.3 (default, Nov  8 2012, 18:25:10) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

So now it seems the file _sqlite3.so can't be found. That's when I found this SO question. Either it doesn't exist or it's not in my PYTHONPATH environment variable. Searching for the file, I get:

$ locate _sqlite3.so
/home/me/Desktop/.dropbox-dist/_sqlite3.so
/home/me/epd/lib/python2.7/lib-dynload/_sqlite3.so
/usr/lib/python2.7/lib-dynload/_sqlite3.so

So the file is there, but when I looked in my python path:

import sys
for p in sys.path:
    print p

none of the above paths that contain _sqlite3.so were contained in my PYTHONPATH. For giggles, I added the path /usr/lib/python2.7/lib-dynload to my PYTHONPATH in a terminal and then tried to import sqlite3 again:

Python 2.7.3 (default, Nov  8 2012, 18:25:10) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/usr/lib/python2.7/lib-dynload")
>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: /usr/lib/python2.7/lib-dynload/_sqlite3.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

Uh oh. Now I'm completely stuck. Can anyone help me out? I've also read in a few places that I may have to rebuild Python. I have no idea how to do this in pythonbrew. Can anyone point me in the right direction?

Answer 1

I've also read in a few places that I may have to rebuild Python.

This is correct. SQLite is part of the standard library, and is built when you compile Python. There are a few 'optional' parts of the standard library, which Python will simply skip (with minimal warning, unfortunately) if the dependencies are missing at build time, and sqlite is one of these. You should be able to just install libsqlite3-dev, then rebuild Python and you should be set. Keep an eye on the build messages, as they do report which modules they are skipping due to missing dependencies.

Answer 2

Thanks to minrk for pointing me in the right direction. All I had to do was rebuild python. I've outlined the steps below for those that are using pythonbrew. Notice that I already installed the libsqlite3-dev package up in the question section.

First, with the proper version of python and virtual environment loaded up run the command:

$ pip freeze -l > requirements.txt

This gives us a text file list of all of the pip packages that have been installed in the virtual environment for this particular python release in pythonbrew. Then we remove the version of python from pythonbrew and reinstall it (this is the "rebuild python" step):

$ pythonbrew uninstall 2.7.3
$ pythonbrew install 2.7.3

After that, we switch over to the newly installed python version 2.7.3 and create a new virtual environment (which I've called "sci"):

$ pythonbrew switch 2.7.3
$ pythonbrew venv create sci
$ pythonbrew venv use sci

Ideally you should be able to run the command:

$ pip install -r requirements.txt

and according to this pip should reinstall all the modules that you had in the virtual environment before we clobbered that version of python (2.7.3). It didn't work for me for whatever reason so I manually installed all of the modules using pip individuality.

$ ipython --pylab

Python 2.7.3 (default, Jan  5 2013, 18:48:27) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

and IPython history works!

Answer 3

What worked for me (using osx + homebrew + brewed python):

# Reinstall Python 2.7 with sqlite
brew remove python
brew install readline sqlite gdbm --universal
brew install python --universal --framework

# Reinstall iPython with correct bindings
pip uninstall ipython    
pip install ipython

And you should be good to go.