Using Selenium after authenticated login session with Scrapy

Question

after looking around, it seems that if you login to a website through Scrapy, the authenticated login session doesn't transfer over if you try to use Selenium within the spider. Is there a way to transfer that session over to Selenium? Or would I have to login to the website all over again with Selenium?

Thanks!

Answer 1

The session is most likely just your cookie. So to convert to carry your session over to Selenium webdriver you need to set cookies of scrapy requests to selenium.

Scrapy is smart enough to keep track of cookies by itself you can find the cookies of the current request in response.headers.
Then you can set those cookies for your webdriver:

driver.add_cookie({'name': 'foo', 'domain': 'bar'})

You can convert response.headers['Set-Cookie'] to a dictionary using dict comprehension like:

import re
foo = response.headers['Set-Cookie']
values = {k.strip():v for k,v in re.findall(r'(.*?)=(.*?);', foo)}
driver.add_cookie(values)

Note: some websites can use more complicated sessions that also require other headers to match, but you can also replicated that by copying your scrapy response headers to your selenium webdriver.