how to use two level proxy setting in Python?

Question

I am working on web-crawler [using python].

Situation is, for example, I am behind server-1 and I use proxy setting to connect to the Outside world. So in Python, using proxy-handler I can fetch the urls. Now thing is, I am building a crawler so I cannot use only one IP [otherwise I will be blocked]. To solve this, I have bunch of Proxies, I want to shuffle through.

My question is: This is two level proxy, one to connect to main server-1, I use proxy and then after to shuffle through proxies, I want to use proxy. How can I achieve this?

Answer 1

Update Sounds like you're looking to connect to proxy A and from there initiate HTTP connections via proxies B, C, D which are outside of A. You might look into the proxychains project which says it can "tunnel any protocol via a user-defined chain of TOR, SOCKS 4/5, and HTTP proxies".

Version 3.1 is available as a package in Ubuntu Lucid. If it doesn't work directly for you, the proxychains source code may provide some insight into how this capability could be implemented for your app.

Orig answer: Check out the urllib2.ProxyHandler. Here is an example of how you can use several different proxies to open urls:

import random
import urllib2

# put the urls for all of your proxies in a list
proxies = ['http://localhost:8080/']

# construct your list of url openers which each use a different proxy
openers = []
for proxy in proxies:
    opener = urllib2.build_opener(urllib2.ProxyHandler({'http': proxy}))
    openers.append(opener)

# select a url opener randomly, round-robin, or with some other scheme
opener = random.choice(openers)
req = urllib2.Request(url)
res = opener.open(req)

Answer 2

I recommend you take a look at CherryProxy. It lets you send a proxy request to an intermediate server (where CherryProxy is running) and then forward your HTTP request to a proxy on a second level machine (e.g. squid proxy on another server) for processing. Viola! A two-level proxy chain.

http://www.decalage.info/python/cherryproxy