The user could input keyword in a form and submit it, so I can get the keyword in the views. Then I can make the start_url with the keyword. How can I pass the start_url to a scrapy spider and start it?
This is my view method.
def results(request):
"""Return the search results"""
key= request.GET['keyword'].strip()
books = Book.objects.filter(title__contains=key)
if books is None:
# I want to call the scrapy spider here.
pass
books = Book.objects.filter(title__contains=key)
context = {'books': books, 'key': title}
return render(request, 'search/results.html', context)
This is the init() method of my spider class.
def __init__(self, key):
self.key = key
url = "http://search.example.com/?key=" + key
self.start_urls = [url]
this worked for me :
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
if books is None:
# I want to call the scrapy spider here.
os.environ.setdefault("SCRAPY_SETTINGS_MODULE","whereyourscrapysettingsare")
crawler_settings = get_project_settings()
crawler = CrawlerRunner(crawler_settings)
crawler.crawl(yourspider, key=key)
from http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With