how to get old url when there is redirection in scrapy?

Question

scrapy version : 0.20

Problem:

start_urls=[URL1,URL2,URL3]

def parse(self,response):
    //suppose URL2 is redirected to other URL
    //I need to get current start URL(before redirection) 

I have tried with response.request.url but it is same as response.url

please help me out

Answer 1

If you've got RedirectMiddleware enabled (it should be enabled by default), you can try:

original_url = response.meta.get('redirect_urls', [response.url])[0]

See https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/redirect.py#L35 for implementation details