How can I organize my spiders into nested directories in Scrapy?

Question

I have the following directory structure:

my_project/
  __init__.py
  spiders/
    __init__.py
    my_spider.py
    other_spider.py
  pipeines.py
  # other files

Right now I can be in the my_project directory and start my crawl using scrapy crawl my_spider.

What I'd like to achieve is to be able to run scrapy crawl my_spider with this updated structure:

my_project/
  __init__.py
  spiders/
    __init__.py
    subtopic1/
      __init__.py # <-- I get the same error whether this is present or not
      my_spider.py
    subtopicx/
      other_spider.py
  pipeines.py
  # other files

But right now I get this error:

KeyError: 'Spider not found: my_spider'

What is the appropriate way to organize Scrapy spiders into directories?

Answer 1

I know this is long over due but this is the right way to organize your spiders in nested directories. You set the modules location in the settings defined here.

Example:

SPIDER_MODULES = ['my_project.spiders', 'my_project.spiders.subtopic1', 'my_project.spiders.subtopicx']