In scrapy for example if i had two URL's that contains different HTML. Now i want to write two individual spiders each for one and want to run both the spiders at once. In scrapy is it possible to run multiple spiders at once.
In scrapy after writing multiple spiders, how can we schedule them to run for every 6 hours(May be like cron jobs)
I had no idea of above , can u suggest me how to perform the above things with an example.
Thanks in advance.
It would probably be easiest to just run two scrapy scripts at once from the OS level. They should both be able to save to the same database. Create a shell script to call both scrapy scripts to do them at the same time:
scrapy runspider foo &
scrapy runspider bar
Be sure to make this script executable with chmod +x script_name
To schedule a cronjob every 6 hours, type crontab -e
into your terminal, and edit the file as follows:
* */6 * * * path/to/shell/script_name >> path/to/file.log
The first * is minutes, then hours, etc., and an asterik is a wildcard. So this says run the script at any time where the hours is divisible by 6, or every six hours.
You can try using CrawlerProcess
from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess
from myproject.spiders import spider1, spider2
1Spider = spider1.1Spider()
2Spider = spider2.2Spider()
process = CrawlerProcess(get_project_settings())
process.crawl(1Spider)
process.crawl(2Spider)
process.start()
If you want to see the full log of the crawl, set LOG_FILE
in your settings.py
.
LOG_FILE = "logs/mylog.log"
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With