How can use scrapy shell with url and basic auth credentials?

Question

I want to use scrapy shell and test response data for url which requires basic auth credentials. I tried to check scrapy shell documentation but I couldn't find it there.

I tried with scrapy shell 'http://user:[email protected]' but it didn't work. Does anybody know how I can achieve it?

Answer 1

if you want to use only the shell, you could do something like this:

$ scrapy shell

and inside the shell:

>> from w3lib.http import basic_auth_header
>> from scrapy import Request
>> auth = basic_auth_header(your_user, your_password)
>> req = Request(url="http://example.com", headers={'Authorization': auth})
>> fetch(req)

as fetch uses the current request to update the shell session.

Answer 2

Yes with httpauth middleware.

Make sure HTTPAuthMiddleware is enabled in the settings then just define:

class MySpider(CrawSpider):
    http_user = 'username'
    http_pass = 'password'
    ...

as class variables in your spider.

Also, you don't need to specify the login credentials in the url if the middleware has been enabled in the settings.