How to install python subversion bindings needed by hg convert on Mac OS X 10.8?

Question

I am looking for a solution, preferably clean and simple, to enable hg convert to work on OS X 10.8 with SVN repositories.

Currently, if you try to convert a SVN repository you will get a could not load Subversion python bindings error.

Note: Alex Martelli recommended in another answer to install CollabNet subversion distribution for OS X, still it seems that the latest CollabNet version cannot be installed on OS X 10.8 (the installer locks).

Answer 1

Using homebrew

The easiest way to get this working is to install subversion from source using homebrew and the --with-python flag.

First, make sure you have your command-line tools installed. With Mavericks and Xcode 5, most commands like cc just work even though the command-line tools aren’t fully installed. If you don’t have a /usr/include directory, then first you need to run

$ xcode-select --install

Once your command-line tools are installed, run:

$ brew install subversion --with-python
$ mkdir -p ~/Library/Python/2.7/lib/python/site-packages
$ echo $(brew --cellar)/subversion/1.8.5/lib/svn-python \
    > ~/Library/Python/2.7/lib/python/site-packages/svn.pth

You can test the bindings by running the unit tests:

$ svn co http://svn.apache.org/repos/asf/subversion/tags/1.8.5/subversion/bindings/swig/python/tests
$ cd tests && python run_all.py

Using Apple’s source code

This works for Mountain Lion, but needs tweaks for Mavericks and results in failing unit tests; see Simon Wright’s answer to this question.

It is possible to build the subversion bindings for Python using Apple’s version of the subversion source code. The resulting module will be exactly compatible and link against all the system libraries. And then hg convert will just work.

Here’s how to do it:

1. Download the subversion tarball from opensource.apple.com

2. Unpack it and configure it:

   cd subversion-52/subversion && ./configure
   

3. In subversion/bindings/swig/python, add this Makefile, being sure to change leading whitespace to tabs:

   SHELL = /bin/bash -eu
   
   CC = gcc -g -O2
   CFLAGS = -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK \
       -I ../proxy \
       -I ../../.. \
       -I ../../../include \
       -I /usr/include/apr-1 \
       -I libsvn_swig_py \
       -I /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 \
   
   LDFLAGS = \
       /System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib \
       /usr/lib/libsvn_*-1.0.dylib \
       /usr/lib/libapr-1.dylib \
   
   .PHONY: build egg test clean install
   
   test: egg
       mkdir -p tmp && cd tmp \
         && PYTHONPATH=../egg python -S -m svn.tests.run_all
   
   DESTDIR=$(HOME)/Library/Python/2.7/lib/python/site-packages
   install: egg
           mkdir -p "${DESTDIR}"
       rm -rf "$(DESTDIR)/svn.egg"
       cp -R egg "$(DESTDIR)/svn.egg"
       echo './svn.egg' > "$(DESTDIR)/svn.pth"
   
   egg: build
       rm -rf egg
       mkdir egg
       cp -R svn egg
       cp -R tests egg/svn
       touch egg/svn/tests/__init__.py
       mkdir egg/libsvn
       cp *.py egg/libsvn
       cp *.so *.dylib egg/libsvn
       # https://blogs.oracle.com/dipol/entry/dynamic_libraries_rpath_and_mac
       for F in egg/libsvn/*.so; do \
         install_name_tool -change libsvn_swig_py.dylib \
           '@loader_path/libsvn_swig_py.dylib' "$$F"; \
       done
       strip -x egg/libsvn/*.so
       touch egg/libsvn/__init__.py
       mkdir -p egg/EGG-INFO
       echo 'Version: 1.6.8' > egg/EGG-INFO/PKG-INFO
   
   build: libsvn_swig_py.dylib _client.so _core.so _delta.so _diff.so _fs.so _ra.so _repos.so _wc.so
   
   libsvn_swig_py.dylib:
       $(CC) $(CFLAGS) -shared -dynamic libsvn_swig_py/swigutil_py.c -o $@ $(LDFLAGS)
   
   _client.so: svn_client.c
   _core.so: core.c
   _delta.so: svn_delta.c
   _diff.so: svn_diff.c
   _fs.so: svn_fs.c
   _repos.so: svn_repos.c
   _wc.so: svn_wc.c
   
   _%.so: svn_%.c libsvn_swig_py.dylib
       $(CC) $(CFLAGS) -bundle $^ -o $@ $(LDFLAGS)
   _%.so: %.c libsvn_swig_py.dylib
       $(CC) $(CFLAGS) -bundle $^ -o $@ $(LDFLAGS)
   
   clean:
       rm -rf *.o *.so *.dylib *.dSYM
   

   Then run make in that directory.

4. If the unit tests passed in the last step, you now have fully-functional Python bindings for subversion! Run make install if you’re happy, and hg convert will start working.

---

As is so often the case, it was easier to rewrite the build system from scratch than to figure out the existing one.

All the scripts from Apple and Subversion really complicate it, but all you really need to do is copy the .py files, build a shared library with the common code, and then build each C-language Python module. Figuring out which include files and libraries to use is straightforward: try to build it starting without any includes or libraries, and when the build fails, add a reference to the missing include path or library that is causing the build failure. The tricky part is telling the C-language modules like _core.so where to find the dynamic library of common code.

Answer 2

Using the hg from MacPorts, I needed sudo port install subversion-python27bindings.

Answer 3

This is an update for Mavericks to andrewdotn's answer above.

Under Mavericks, the only SVN dynamic libraries in /usr/lib (the standard place) are, for example, /usr/lib/libsvn_client-1.0.0.0.dylib. This library’s internal names (find using otool -L) tell the dynamic loader to look for libsvn_client-1.0.dylib. Normally this’d also be in /usr/lib as a symlink to the real library.

Try this:

1. In andrewdotn's Makefile, edit the LDFLAGS (second line) to say

   /Library/Developer/CommandLineTools/usr/lib/libsvn_*-1.0.dylib \

2. Set DYLD_LIBRARY_PATH:

   export DYLD_LIBRARY_PATH=/Library/Developer/CommandLineTools/usr/lib

then make. I got 2 errors:

======================================================================
ERROR: test_get_pristine_copy_path (svn.tests.wc.SubversionWorkingCopyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../egg/svn/tests/wc.py", line 192, in test_get_pristine_copy_path
    self.assertEqual(path_to_text_base, wc.get_pristine_copy_path(path_to_file))
  File "../egg/libsvn/wc.py", line 2447, in svn_wc_get_pristine_copy_path
    return apply(_wc.svn_wc_get_pristine_copy_path, args)
SubversionException: ("The node '/var/folders/_q/fvnxz46903z9hjh38fz0lyhm0000gs/T/tmp7vMRZu/foo' was not found.", 155010)

======================================================================
ERROR: test_lock (svn.tests.wc.SubversionWorkingCopyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../egg/svn/tests/wc.py", line 48, in test_lock
    lock = wc.add_lock(self.path, core.svn_lock_create(core.Pool()), self.wc)
  File "../egg/libsvn/wc.py", line 2601, in svn_wc_add_lock
    return apply(_wc.svn_wc_add_lock, args)
SubversionException: ('sqlite: LOCK.lock_token may not be NULL', 200035)

----------------------------------------------------------------------

but I installed anyway and hg convert worked fine.

I suspect you'll need to set DYLD_LIBRARY_PATH as above every time you need to run hg convert; hopefully not that often!