Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to run scrapy spider with arguments inside django view

The user could input keyword in a form and submit it, so I can get the keyword in the views. Then I can make the start_url with the keyword. How can I pass the start_url to a scrapy spider and start it?

This is my view method.

def results(request):
    """Return the search results"""
    key= request.GET['keyword'].strip()
    books = Book.objects.filter(title__contains=key)
    if books is None:
        # I want to call the scrapy spider here.
        pass
        books = Book.objects.filter(title__contains=key)
    context = {'books': books, 'key': title}
    return render(request, 'search/results.html', context)

This is the init() method of my spider class.

def __init__(self, key):
    self.key = key
    url = "http://search.example.com/?key=" + key
    self.start_urls = [url]
like image 689
PXY Avatar asked Apr 27 '16 16:04

PXY


1 Answers

this worked for me :

from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
if books is None:
    # I want to call the scrapy spider here.
    os.environ.setdefault("SCRAPY_SETTINGS_MODULE","whereyourscrapysettingsare")
crawler_settings = get_project_settings()
crawler = CrawlerRunner(crawler_settings)
crawler.crawl(yourspider, key=key)

from http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

like image 102
Tigrou Avatar answered Nov 01 '22 12:11

Tigrou