How to call Scrapy Spider through a Django App

Question

I'm having trouble calling a scrapy spider in my django view. How can I do this? I tried to follow this tutorial http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/ but did not work in the import settings.

Answer 1

If the error is coming from

from scrapy.conf import settings

it's likely because scrapy simply can't find the settings file it's expecting. Since it's customary in Django to use settings for django's own configuration, it would be clearest if you don't use that term.

You can specify your scrapy settings within your Django settings:

"""settings.py"""
# stuff

SCRAPY_SETTINGS = {
    ... # put your usual scrapy keys and values here
}

# more stuff

Then, instead of importing scrapy.conf.settings, you can instead use:

from django.conf import settings

and where you reference scrapy settings in your script, you should change the argument to CrawlerProcess to settings.SCRAPY_SETTINGS

self.crawler = CrawlerProcess(settings.SCRAPY_SETTINGS)

If you have any further trouble, please post the full error you're getting and the code for your view.