Best practice to web hosting a website with Scrapy Spiders running in backend

Question

Maybe I am missing something about Scrapy, but here is what I am about to do: I have created a website based on the information I am crawling from Internet using Scrapy Crawl Spiders. However, I am stuck in how to get my website going live. I am considerring web hosting but most of the service providers do not allow install those scripts on their server. Of couse I can rent a server but it is too expensive for me at the moment. Could anyone please shed some light on this if you have similar experience. The website is based on ASP.NET so will need the webhosting supports MS SQL, ASP.NET as well as Scrapy. Is there something in the scrapy so I can get the spiders running without installing? Much appreciated.

Cheers, Ray

Answer 1

You would need a hosting service where you could install the scrapyd service so that you can automate your screen scraping. I've never done it as I am just getting started playing around with Scrapy, but here is the information on scrapyd: http://readthedocs.org/docs/scrapy/en/latest/topics/scrapyd.html

You might want to look at Virtual Dedicated Servers for hosting as they are cheaper than co-located or dedicated servers but give you more control than shared hosting.