How to make this Twisted Python Proxy faster?

Question

The code below is an HTTP proxy for content filtering. It uses GET to send the URL of the current site to the server, where it processes it and responds. It runs VERY, VERY, VERY slow. Any ideas on how to make it faster?

Here is the code:

from twisted.internet import reactor
from twisted.web import http
from twisted.web.proxy import Proxy, ProxyRequest
from Tkinter import *
#import win32api
import urllib2
import urllib
import os
import webbrowser

cwd = os.path.abspath(sys.argv[0])[0]
proxies = {}
user = "zachb"
class BlockingProxyRequest(ProxyRequest):
    def process(self):
        params = {}
        params['Location']= self.uri
        params['User'] = user
        params = urllib.urlencode(params)
        req = urllib.urlopen("http://weblock.zbrowntechnology.info/ProgFiles/stats.php?%s" % params, proxies=proxies)
        resp = req.read()
        req.close()
        if resp == "allow":
            pass
        else:
            self.transport.write('''BLOCKED BY ADMIN!''')
            self.transport.loseConnection()

        ProxyRequest.process(self)

class BlockingProxy(Proxy):
    requestFactory = BlockingProxyRequest
factory = http.HTTPFactory()
factory.protocol = BlockingProxy

reactor.listenTCP(8000, factory)
reactor.run()

Anyone have any ideas on how to make this run faster? Or even a better way to write it?

Answer 1

The main cause of slowness in this proxy is probably these three lines:

    req = urllib.urlopen("http://weblock.zbrowntechnology.info/ProgFiles/stats.php?%s" % params, proxies=proxies)
    resp = req.read()
    req.close()

A normal Twisted-based application is single threaded. You have to go out of your way to get threads involved. That means that whenever a request comes in, you are blocking the one and only processing thread on this HTTP request. No further requests are processed until this HTTP request completes.

Try using one of the APIs in twisted.web.client, (eg Agent or getPage). These APIs don't block, so your server will handle concurrent requests concurrently. This should translate into much smaller response times.