What are the best ways to run a Ruby script as a subprocess/another thread from Sinatra?

Question

I have a Sinatra app I plan on hosting on Heroku.

This application, in part, scrapes a lot of information from other pages around the net and stores the information to a database. These scrapping operations are a slow process, so I need them to run in another thread/process separate from my Sinatra app.

My plan is just to have a button for each process that I can click and the scrapping will take place in the background.

I'm unsure what's the best way to do this, complicated by what Heroku will allow.

Answer 1

There's a gem called hirefire specifically for that:

HireFire automatically "hires" and "fires" (aka "scales") Delayed Job and Resque workers on Heroku. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases.