I had multiple spiders in my project folder and want to run all the spiders at once, so i decided to run them using scrapyd service. I have started doing this by seeing here
First of all i am in current project folder
I had opened the scrapy.cfg
file and uncommented the url line after
[deploy]
I had run scrapy server
command, that works fine and scrapyd server runs
I tried this command scrapy deploy -l
Result : default http://localhost:6800/
scrapy deploy -L scrapyd
i got following outputResult:
Usage
=====
scrapy deploy [options] [ [target] | -l | -L <target> ]
deploy: error: Unknown target: scrapyd
when i tried to deploy the project with this command scrapy deploy scrapyd -p default
got following error
Usage
=====
scrapy deploy [options] [ [target] | -l | -L <target> ]
deploy: error: Unknown target: scrapyd
I am really unable to identify whey scrapyd is showing the above errors, can lead me in to a correct way of how to deploy a project in to scrapyd
Thanks in advance..........
Edited Code:
After seeing the answer of Peter Kirby,i named target in scrapy.cfg and tried the following command in my project folder,
command:
scrapy deploy ebsite -p ebsite
then i got the below error
Building egg of ebsite-1341808241
'build/lib' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-2.7' does not exist -- can't clean it
zip_safe flag not set; analyzing archive contents...
Deploying ebsite-1341808241 to http://localhost:6800/addversion.json
Deploy failed: <urlopen error [Errno 111] Connection refused>
How to solve this.....
Then run the following command in your Scrapy projects root directory: This will then eggify your Scrapy project and deploy it to your locally running Scrapyd server. You should get a result like this in your terminal if it was successful:
Scrapyd is application that allows us to deploy Scrapy spiders on a server and run them remotely using a JSON API. Scrapyd allows you to: Run Scrapy jobs. Pause & Cancel Scrapy jobs. Manage Scrapy project/spider versions. Access Scrapy logs remotely.
To scrap the data from web pages, first you need to create the Scrapy project where you will be storing the code. To create a new directory, run the following command −. scrapy startproject first_scrapy. The above code will create a directory with name first_scrapy and it will contain the following structure −.
Scrapyd comes with a minimal web interface which can be accessed at http://localhost:6800/, however, this interface is just a rudimentary overview of what is running on a Scrapyd server and doesn't allow you to control the spiders deployed to the Scrapyd server.
From scrapyd service documentation: (http://scrapy.readthedocs.org/en/latest/topics/scrapyd.html?highlight=scrapyd)
You can define targets by adding them to your project’s scrapy.cfg file... Here’s an example of defining a new target scrapyd2 with restricted access through HTTP basic authentication:
[deploy:scrapyd2]
url = http://scrapyd.mydomain.com/api/scrapyd/
username = john
password = secret
Essentially what your error means is that your "target" name is not correct. If I remember correctly, the scrapy.cfg file sets the initial target name as "default". What you should be typing is something like:
scrapy deploy default -p project_name
Just type scrapy deploy
if you have no named targets and left settings at default!
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With