How to run Scrapy project in Jupyter?

Question

On a Mac, I have Jupyter installed and when I type jupyter notebook from the root folder of my Scrapy project, it opens the notebook. I can browse all of the project files at this point.

How do I execute the project from the notebook?

If I click the Running tab, under Terminals, I see:

There are no terminals running.

Answer 1

There are two main ways to achieve that:

1. Under the Files tab open a new terminal: New > Terminal
Then simply run you spider: scrapy crawl [options] <spider>

2. Create a new notebook and use CrawlerProcess or CrawlerRunner classes to run in a cell:

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

process.crawl('your-spider')
process.start() # the script will block here until the crawling is finished

Scrapy docs - Run Scrapy from a script

Answer 2

No Need of Terminal to run Spyder Class. Just add the following code in your jupyter-notebook cell:

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    # Your spider definition
    ...

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

For more information see here