What am I doing wrong when installing lxml on Mac OS X 10.8.1?

Question

I'm building lxml on Mac OS X 10.8.1 and Python 2.7.2 and having problems. (I've cloned the Github repository and followed the instructions for Mac OS X here: http://lxml.de/build.html)

It appears something hangs in building libxml2; the following is the tail end of the terminal output:

configure: creating ./config.status
config.status: creating libxml2.spec
config.status: creating Makefile
config.status: creating include/Makefile
config.status: creating include/libxml/Makefile
config.status: creating doc/Makefile
config.status: creating doc/examples/Makefile
config.status: creating doc/devhelp/Makefile
config.status: creating example/Makefile
config.status: creating python/Makefile
config.status: creating python/tests/Makefile
config.status: creating xstc/Makefile
config.status: creating include/libxml/xmlversion.h
config.status: creating xml2-config
config.status: creating libxml-2.0.pc
config.status: creating libxml-2.0-uninstalled.pc
config.status: creating python/setup.py
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
rm: libtoolT: No such file or directory
Done configuring
make  all-recursive
Making all in include
Making all in libxml
make[3]: Nothing to be done for `all'.
make[3]: Nothing to be done for `all-am'.
Making all in .
  CC     error.lo

[snipped]

  CC     hash.lo
parser.c: In function 'xmlParseInNodeContext':
parser.c:13492: warning: pointer targets in passing argument 1 of 'xmlFindCharEncodingHandler' differ in signedness
  CC     list.lo

[snipped]

  CC     xpointer.lo
xpath.c: In function 'xmlXPathRegisterFuncNS':
xpath.c:4870: warning: ISO C forbids passing argument 4 of 'xmlHashAddEntry2' between function pointer and 'void *'
xpath.c: In function 'xmlXPathFunctionLookupNS':
xpath.c:4951: warning: ISO C forbids assignment between function pointer and 'void *'
xpath.c: In function 'xmlXPathCompOpEval':
xpath.c:13535: warning: ISO C forbids assignment between function pointer and 'void *'
xpath.c:13562: warning: ISO C forbids assignment between function pointer and 'void *'
xpath.c: At top level:
trionan.c:221: warning: 'trio_is_negative' defined but not used
  CC     xinclude.lo

[snipped]

  CC     xmlstring.lo
threads.c: In function 'xmlCleanupThreads':
threads.c:918: error: expected expression before '{' token
make[2]: *** [threads.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 225, in <module>
    **setup_extra_options()
  File "setup.py", line 139, in setup_extra_options
    STATIC_CFLAGS, STATIC_BINARIES)
  File "/Users/jedc/Downloads/lxml/setupinfo.py", line 57, in ext_modules
    multicore=OPTION_MULTICORE)
  File "/Users/jedc/Downloads/lxml/buildlibxml.py", line 338, in build_libxml2xslt
    cmmi(libxml2_configure_cmd, libxml2_dir, multicore, **call_setup)
  File "/Users/jedc/Downloads/lxml/buildlibxml.py", line 266, in cmmi
    cwd=build_dir, **call_setup)
  File "/Users/jedc/Downloads/lxml/buildlibxml.py", line 249, in call_subprocess
    raise Exception('Command "%s" returned code %s' % (cmd_desc, returncode))
Exception: Command "make -j5" returned code 512

I'm WAY outside my depth in trying to figure out what's wrong here. Can anyone point me in the direction of what I need to do to fix this?

Answer 1

This is a bug in libxml2 2.9.

The error you're getting is on this line:

once_control = PTHREAD_ONCE_INIT;

That's illegal code, that just happens to work on linux. PTHREAD_ONCE_INIT can only be used for initialization, not assignment, and there's a specific reason for that: so platforms can define PTHREAD_ONCE_INIT as an aggregate initializer. Which OS X does. From 10.8.1's /usr/include/pthread.h:

#define PTHREAD_ONCE_INIT {_PTHREAD_ONCE_SIG_init, {0}}

It looks like this has been reported to the libxml2 mailing list a few days ago (https://mail.gnome.org/archives/xml/2012-September/msg00036.html), filed in bugzilla two days later (https://bugzilla.gnome.org/show_bug.cgi?id=684024), and fixed the next day (http://git.gnome.org/browse/libxml2/commit/?id=3f6cfbd1d38d0634a2ddcb9a0a13e1b5a2195a5e), so presumably 2.9.1 will not have this problem.

So, here are your options:

Of course you could wait for 2.9.1 to come out.

Alternative, despite what the build page says, the version of libxml2 that comes with Mountain Lion, 2.7.8, is not "horribly outdated"—it's newer than the 2.7.3 version he suggests using. So, you can just skip the --static-deps and use the built-in 2.7.8.

If you do want something newer than 2.7.8, but not as new as the (non-working) 2.9.0, the build page shows how to specify it explicitly, and http://www.xmlsoft.org should have the version history somewhere so you can pick the version you want. For example:

python setup.py build --static-deps \
   --libxml2-version=2.8.0

Or you could manually apply the patch from http://git.gnome.org/browse/libxml2/commit/?id=3f6cfbd1d38d0634a2ddcb9a0a13e1b5a2195a5e and build that manually, or pull the top of tree instead of version 2.9.0, build that, and then tell lxml to use that build.

In the future, if you don't want to have to deal with debugging builds, I'd suggest just using pip to install Python packages, and Homebrew to install any missing dependencies. You may not always get the ideal build of the very latest version of everything, but it will be a lot easier, and usually good enough.