Change IP address dynamically?

Question

Consider the case, I want to crawl websites frequently, but my IP address got blocked after some day/limit.

So, how can change my IP address dynamically or any other ideas?

Answer 1

An approach using Scrapy will make use of two components, RandomProxy and RotateUserAgentMiddleware.

Modify DOWNLOADER_MIDDLEWARES as follows. You will have to insert the new components in the settings.py:

DOWNLOADER_MIDDLEWARES = {     'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 90,     'tutorial.randomproxy.RandomProxy': 100,     'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,     'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware' : None,     'tutorial.spiders.rotate_useragent.RotateUserAgentMiddleware' :400,     } 

Random Proxy

You can use scrapy-proxies. This component will process Scrapy requests using a random proxy from a list to avoid IP ban and improve crawling speed.

You can build up your proxy list from a quick internet search. Copy links in the list.txt file according to requested url format.

Rotation of user agent

For each scrapy request a random user agent will be used from a list you define in advance:

class RotateUserAgentMiddleware(UserAgentMiddleware):     def __init__(self, user_agent=''):         self.user_agent = user_agent      def process_request(self, request, spider):         ua = random.choice(self.user_agent_list)         if ua:             request.headers.setdefault('User-Agent', ua)              # Add desired logging message here.             spider.log(                 u'User-Agent: {} {}'.format(request.headers.get('User-Agent'), request),                 level=log.DEBUG             )      # the default user_agent_list composes chrome,I E,firefox,Mozilla,opera,netscape     # for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php     user_agent_list = [         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",         "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",         "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",         "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",         "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",         "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",         "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",         "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",         "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",         "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",         "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",         "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",         "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",         "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",         "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"     ] 

More details here.