Caching in urllib2?

Question

Is there an easy way to cache things when using urllib2 that I am over-looking, or do I have to roll my own?

Answer 1

If you don't mind working at a slightly lower level, httplib2 (https://github.com/httplib2/httplib2) is an excellent HTTP library that includes caching functionality.

Answer 2

You could use a decorator function such as:

class cache(object):
    def __init__(self, fun):
        self.fun = fun
        self.cache = {}

    def __call__(self, *args, **kwargs):
        key  = str(args) + str(kwargs)
        try:
            return self.cache[key]
        except KeyError:
            self.cache[key] = rval = self.fun(*args, **kwargs)
            return rval
        except TypeError: # incase key isn't a valid key - don't cache
            return self.fun(*args, **kwargs)

and define a function along the lines of:

@cache
def get_url_src(url):
    return urllib.urlopen(url).read()

This is assuming you're not paying attention to HTTP Cache Controls, but just want to cache the page for the duration of the application