Django A/B Split Testing Packages (None I've found are well-documented and up-to-date.)

Question

There are two main schools of thought for doing A/B (Split) Testing:

• Javascript-based solutions such as Optimizely, Google Analytics Content Experiments.
• Server-side solutions such as Django-AB, Splango, and django-lean. (Also, writing your own.)

My understanding is that Javascript-based solutions are spectacular for "which color button converts better," but not so great for switching out entire page layouts, and completely unworkable for trying out large functional changes such as the sequence of pages in a funnel.

That leads me towards a server-side solution. I'm not crazy about coding my own, and will do so only if there is no other option. I'm trying to add value by improving the core functionality of my site, not by creating a better split-testing framework.

The Django apps I've found for split testing are various mixtures of unmaintained, undocumented, documented incorrectly, and incompatible with Django 1.5. This surprises me, because the Django and Python communities seem to have a strong focus on good documentation. I'm also very surprised that none of the testing frameworks I've tried has been compatible with Django 1.5 -- is testing not as core a part of the philosophy in the Django/Python world as it is in Rails?

Here's what I've found:

• Splango https://github.com/shimon/Splango -- Not compatible with Django 1.5 (although most compatibility bugs I found were trivial to fix). Largely un-touched since October 2010, except for a fix August 2012 which claims to make sure templates get included in the install. Since templates don't get included in the install when Splango is installed via PyPI, either the fix didn't work or didn't get submitted to PyPI. Documentation is largely accurate, but doesn't completely cover how to set up tests and get reports. It tells you how to configure the template to gather the data, but there appears to be additional steps required in the admin interface which are completely undocumented, and I'm not sure I've done them properly.

• Django-lean. Original at https://bitbucket.org/akoha/django-lean has not been updated since July 2010. There is an apparently "blessed" fork at https://github.com/anandhenry2002/django-lean which has not been changed since May 2012, when it was copied over from the original. The original's documentation is incorrect in ways that make following the examples impossible. (Though you can probably muddle your way through, as I did.) The new version's documentation has formatting problems that make it difficult to read on github. (This appears to be because it's the unchanged documentation from the old project, and BitBucket syntax doesn't work on Github.) The django-lean Google Group has not had a message since July 2012.

• django-mini-lean https://github.com/DanAncona/django-mini-lean -- Updated as recently as February 2013, but undocumented.

• Leaner - https://bitbucket.org/brianjinwright/leaner -- Last updated July 2012, and no docs.

• Django-AB -- Last updated May 2009. Is not a package, and can't be installed via PIP or PyPI. After placing the checkout in my django app folder (and renaming the folder to ab) and following the installation instructions, I get an error loading the template loader that I have not tracked down further.

So far Splango appears to be the winner, as I've actually been able to get it more-or-less working (by manually installing the templates, and then editing them to fix Django 1.5 incompatibilities).

Can anyone point me to anything I've missed?

Answer 1

You have missed this app : https://github.com/mixcloud/django-experiments + https://github.com/disqus/gargoyle/