Scrapy xpath error: 'Selector' object has no attribute '_default_type'

Question

I'm using scrapy shell to run some sample test

after I enter 'response.xpath('//h1/text()').get()'

and it gives me AttributeError: 'Selector' object has no attribute '_default_type'

Can anyone help with this?

Thank you very much

below are the things that I entered in the terminal

scrapy shell

fetch("https://www.worldometers.info/world-population/population-by-country/")

r = scrapy.Request(url="https://www.worldometers.info/world-population/population-by-country/")

fetch(r)

response.xpath('//h1/text()').get()

Everything went good before the last line. After I entered 'response.xpath('//h1/text()').get()' it throw an error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-7a322073163e> in <module>
----> 1 response.xpath('//h1/text()').get()

~/anaconda3/envs/virtual_work/lib/python3.7/site-packages/scrapy/http/response/text.py in xpath(self, query, **kwargs)
    117 
    118     def xpath(self, query, **kwargs):
--> 119         return self.selector.xpath(query, **kwargs)
    120 
    121     def css(self, query):

~/anaconda3/envs/virtual_work/lib/python3.7/site-packages/scrapy/http/response/text.py in selector(self)
    113         from scrapy.selector import Selector
    114         if self._cached_selector is None:
--> 115             self._cached_selector = Selector(self)
    116         return self._cached_selector
    117 

~/anaconda3/envs/virtual_work/lib/python3.7/site-packages/scrapy/selector/unified.py in __init__(self, response, text, type, root, _root, **kwargs)
     84                             % self.__class__.__name__)
     85 
---> 86         st = _st(response, type or self._default_type)
     87 
     88         if _root is not None:
AttributeError: 'Selector' object has no attribute '_default_type'

Answer 1

I met the same error. And the cause seems to be the incompatibility between scrapy and parsel

My requirements.txt was:

Scrapy==2.4.1
scrapy-splash==0.7.2

After adding this line, the exception is gone:

parsel==1.7.0

I suppose in the recently released parsel==1.8.0, parsel#181 removed _default_type, which scrapy only remove the usage of after scarpy#5006, hence the exception. I guess updating Scrapy and scrapy-splash is another way to fix it, like what Mayuresh K said.

Hope this helps.