Scrapy code throws TypeError: 'NoneType' object is not iterable

Question

Im trying to make a request for a parameter site_search_url, but I get the following error when I run: ``:

start_requests = iter(self.spider.start_requests())
TypeError: 'NoneType' object is not iterable

code:

class BrickSetSpider(scrapy.Spider):
    def __init__(self, site_search_url):
        self.site_search_url = site_search_url

    def start_requests(self):
        se_base = 'http://www.se.com/search?q=site:'
        start_urls = [ se_base + self.site_search_url, ]

    def parse(self, response):
        yield scrapy.Request(
            response.urljoin(next_page),
            callback=self.parse
        )

What am I doing wrong here?

Thank you

Answer 1

Your start_requests returns nothing, which mean returns None in Python, while it should return an iterable of Request objects. In your case easiest is to populate start_urls in __init__ and don't override start_requests:

class BrickSetSpider(scrapy.Spider):
    se_base = 'http://www.se.com/search?q=site:'
    def __init__(self, site_search_url):
        self.start_urls = [self.se_base + site_search_url]

    def parse(self, response):
        yield scrapy.Request(
            response.urljoin(next_page),
            callback=self.parse
        )