ubuntu 11.04 lxml import etree problem for custom python

Question

ubuntu 11.04 has native python2.7 i build python2.5 from source to /usr/local/python2.5/bin, and try to install lxml for my custom python2.5 install. Also i use virtualenv. I switch to my env with python2.5. On import lxml i got an error.

from lxml import etree
ImportError: /home/se7en/.virtualenvs/e-py25/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so: undefined symbol: PyUnicodeUCS2_DecodeLatin1

With python2.7 env, all is ok but on python2.5 import fails. Please help to fix for python2.5 ?

ldd /home/se7en/.virtualenvs/e-py25/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so

results:

(e-py25)se7en@se7en-R510-P510:~/downloads/lxml-2.2.4$ ldd /home/se7en/.virtualenvs/e-py25/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so
    linux-gate.so.1 =>  (0x00968000)
    libxslt.so.1 => /usr/lib/libxslt.so.1 (0x005aa000)
    libexslt.so.0 => /usr/lib/libexslt.so.0 (0x00110000)
    libxml2.so.2 => /usr/lib/libxml2.so.2 (0x00db3000)
    libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0x00a22000)
    libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0x00564000)
    libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0x00123000)
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0x0013c000)
    libgcrypt.so.11 => /lib/i386-linux-gnu/libgcrypt.so.11 (0x0029d000)
    libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0x00d6e000)
    /lib/ld-linux.so.2 (0x004fc000)
    libgpg-error.so.0 => /lib/i386-linux-gnu/libgpg-error.so.0 (0x00879000)
(e-py25)se7en@se7en-R510-P510:~/downloads/lxml-2.2.4$ 

Answer 1

You cannot directly symlink different Python versions or native libraries, as Python DLL format changes across major Python versions.

Based on this:

"from lxml import etree" raise "ImportError: /home/se7en/.virtualenvs/e-py25/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so: undefined symbol: PyUnicodeUCS2_DecodeLatin1

It clearly states that lxml is somehow getting compiled against wrong version of Python. Usually this error stems from the problem that you have been mixing manually compiled Python interpreter with Ubuntu's default one, as Python interpreter can be compiled with different unicode flags and Ubuntu uses non-default flags (if I recall correctly).

Usually I solve this problem by

• Creating a fresh virtualenv

• Reinstalling lxml under this virtualenv using easy_install

• Running Python using -v switch and Python will print everything it tries to import

• If it is still importing stuff from wrong location either virtualenv or your native library setup has been corrupted

• Native library setup can be overridden with manual lib builds and LD_LIBRARY_PATH environment variable

• If virtualenv does not build lxml against your correct Python version it is virtualenv bug (as long as you can show how to reproduce this in a repeatable manner). However, we have been succesfully using lxml with Ubuntu, virtualenv and various Python versions so I doubt there is a bug.

There is also a way for static lxml installations using a tool called buildout (a little bit like virtualenv, but much much more complex):

http://groups.google.com/group/gomobile-dev/browse_thread/thread/7f5e34e991cfdaa9/c65b70e7a9422ebf?#c65b70e7a9422ebf