Cleanly Mocking Remote Servers and APIs for Django Unittests

Question

I have a thorny problem that I can't seem to get to grips with. I am currently writing unit tests for a django custom auth-backend. On our system we actually have two backends: one the built-in django backend and the custom backend that sends out requests to a Java based API that returns user info in the form of XML. Now, I am writing unit tests so I don't want to be sending requests outside the system like that, I'm not trying to test the Java API, so my question is how can I get around this and mock the side-effects in the most robust way.

The function I am testing is something like this, where the url settings value is just the base url for the Java server that authenticates the username and password data and returns the xml, and the service value is just some magic for building the url query, its unimportant for us:

@staticmethod
def get_info_from_api_with_un_pw(username, password, service=12345):
    url = settings.AUTHENTICATE_URL_VIA_PASSWORD
    if AUTH_FIELD == "username":
        params = {"nick": username, "password": password}
    elif AUTH_FIELD == "email":
        params = {"email": username, "password": password}
    params["service"] = service
    encoded_params = urlencode([(k, smart_str(v, "latin1")) for k, v in params.items()])
    try:
        # get the user's data from the api
        xml = urlopen(url + encoded_params).read()
        userinfo = dict((e.tag, smart_unicode(e.text, strings_only=True))
                        for e in ET.fromstring(xml).getchildren())
        if "nil" in userinfo:
            return userinfo
        else:
            return None

So, we get the xml, parse it into a dict and if the key nil is present then we can return the dict and carry on happy and authenticated. Clearly, one solution is just to find a way to somehow override or monkeypatch the logic in the xml variable, I found this answer:

How can one mock/stub python module like urllib

I tried to implement something like that, but the details there are very sketchy and I couldn't seem to get that working.

I also captured the xml response and put it in a local file in the test folder with the intention of finding a way to use that as a mock response that is passed into the url parameter of the test function, something like this will override the url:

@override_settings(AUTHENTICATE_URL_VIA_PASSWORD=(os.path.join(os.path.dirname(__file__), "{0}".format("response.xml"))))
def test_get_user_info_username(self):
    self.backend = RemoteAuthBackend()
    self.backend.get_info_from_api_with_un_pw("user", "pass")

But that also needs to take account of the url building logic that the function defines, (i.e. "url + encoded_params"). Again, I could rename the response file to be the same as the concatenated url but this is becoming less like a good unit-test for the function and more of a "cheat", the whole thing is just getting more and more brittle all the time with these solutions, and its really just a fixture anyway, which is also something I want to avoid if at all possible.

I also wondered if there might be a way to serve the xml on the django development server and then point the function at that? It seems like a saner solution, but much googling gave me no clues if such a thing would be possible or advisable and even then I don't think that would be a test to run outside of the development environment.

So, ideally, I need to be able to somehow mock a "server" to take the place of the Java API in the function call, or somehow serve up some xml payload that the function can open as its url, or monkeypatch the function from the test itself, or...

Does the mock library have the appropriate tools to do such things?

http://www.voidspace.org.uk/python/mock

So, there are two points to this question 1) I would like to solve my particular problem in a clean way, and more importantly 2) what are the best practices for cleanly writing Django unit-tests when you are dependent on data, cookies, etc. for user authentication from a remote API that is outside of your domain?

Answer 1

The mock library should work if used properly. I prefer the minimock library and I wrote a small base unit testcase (minimocktest) that helps with this.

If you want to integrate this testcase with Django to test urllib you can do it as follows:

from minimocktest import MockTestCase
from django.test import TestCase
from django.test.client import Client

class DjangoTestCase(TestCase, MockTestCase):
    '''
    A TestCase class that combines minimocktest and django.test.TestCase
    '''

    def _pre_setup(self):
        MockTestCase.setUp(self)
        TestCase._pre_setup(self)
        # optional: shortcut client handle for quick testing
        self.client = Client()

    def _post_teardown(self):
        TestCase._post_teardown(self)
        MockTestCase.tearDown(self)

Now you can use this testcase instead of using the Django test case directly:

class MySimpleTestCase(DjangoTestCase):
    def setUp(self):
        self.file = StringIO.StringIO('MiniMockTest')
        self.file.close = self.Mock('file_close_function')
    def test_urldump_dumpsContentProperly(self):
        self.mock('urllib2.urlopen', returns=self.file)
        self.assertEquals(urldump('http://pykler.github.com'), 'MiniMockTest')
        self.assertSameTrace('\n'.join([
            "Called urllib2.urlopen('http://pykler.github.com')",
            "Called file_close_function()",
        ]))
        urllib2.urlopen('anything')
        self.mock('urllib2.urlopen', returns=self.file, tracker=None)
        urllib2.urlopen('this is not tracked')
        self.assertTrace("Called urllib2.urlopen('anything')")
        self.assertTrace("Called urllib2.urlopen('this is mocked but not tracked')", includes=False)
        self.assertSameTrace('\n'.join([
            "Called urllib2.urlopen('http://pykler.github.com')",
            "Called file_close_function()",
            "Called urllib2.urlopen('anything')",
        ]))