I have created a couple of web spiders that I intend to run simultaneously with scrapyd. I first successfully installed scrapyd in Ubuntu 14.04 using the command: pip install scrapyd, and when I run the command: scrapyd, I get the following output in the terminal:
2015-07-14 01:22:02-0400 [-] Log opened.
2015-07-14 01:22:02-0400 [-] twistd 13.2.0 (/usr/bin/python 2.7.6) starting up.
2015-07-14 01:22:02-0400 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2015-07-14 01:22:02-0400 [-] Site starting on 6800
2015-07-14 01:22:02-0400 [-] Starting factory <twisted.web.server.Site instance at 0x7f762f4391b8>
2015-07-14 01:22:02-0400 [Launcher] Scrapyd 1.1.0 started: max_proc=8, runner='scrapyd.runner'
I believe that the fact that I got this output suggests that scrapy is working; however, when I run the command: scrapyd-deploy as in the docs, I get the error: scrapyd-deploy: command not found. How can this be possible if the installation was successful? I included the following target in the config file:
[deploy:scrapyd2]
url = http://scrapyd.mydomain.com/api/scrapyd/
username = name
password = secret
I'm not exactly sure how the target works, but I basically copied it from the docs so I would think that it would work. Is there something that I am supposed to import or configure that I haven't? Thanks.
scrapyd-deploy
is a part of scrapyd-client.You can install it from PyPi. Try:
$ sudo pip install scrapyd-client
Can be installed from git directly:
pip install git+https://github.com/scrapy/scrapyd-client
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With