How do I unit test a module that relies on urllib2?

Question

I've got a piece of code that I can't figure out how to unit test! The module pulls content from external XML feeds (twitter, flickr, youtube, etc.) with urllib2. Here's some pseudo-code for it:

params = (url, urlencode(data),) if data else (url,) req = Request(*params) response = urlopen(req) #check headers, content-length, etc... #parse the response XML with lxml... 

My first thought was to pickle the response and load it for testing, but apparently urllib's response object is unserializable (it raises an exception).

Just saving the XML from the response body isn't ideal, because my code uses the header information too. It's designed to act on a response object.

And of course, relying on an external source for data in a unit test is a horrible idea.

So how do I write a unit test for this?

Answer 1

urllib2 has a functions called build_opener() and install_opener() which you should use to mock the behaviour of urlopen()

import urllib2 from StringIO import StringIO  def mock_response(req):     if req.get_full_url() == "http://example.com":         resp = urllib2.addinfourl(StringIO("mock file"), "mock message", req.get_full_url())         resp.code = 200         resp.msg = "OK"         return resp  class MyHTTPHandler(urllib2.HTTPHandler):     def http_open(self, req):         print "mock opener"         return mock_response(req)  my_opener = urllib2.build_opener(MyHTTPHandler) urllib2.install_opener(my_opener)  response=urllib2.urlopen("http://example.com") print response.read() print response.code print response.msg 

Answer 2

It would be best if you could write a mock urlopen (and possibly Request) which provides the minimum required interface to behave like urllib2's version. You'd then need to have your function/method which uses it able to accept this mock urlopen somehow, and use urllib2.urlopen otherwise.

This is a fair amount of work, but worthwhile. Remember that python is very friendly to ducktyping, so you just need to provide some semblance of the response object's properties to mock it.

For example:

class MockResponse(object):     def __init__(self, resp_data, code=200, msg='OK'):         self.resp_data = resp_data         self.code = code         self.msg = msg         self.headers = {'content-type': 'text/xml; charset=utf-8'}      def read(self):         return self.resp_data      def getcode(self):         return self.code      # Define other members and properties you want  def mock_urlopen(request):     return MockResponse(r'<xml document>') 

Granted, some of these are difficult to mock, because for example I believe the normal "headers" is an HTTPMessage which implements fun stuff like case-insensitive header names. But, you might be able to simply construct an HTTPMessage with your response data.