django-rq and rqscheduler deployment on AWS Elastic Beanstalk

Question

I have a Django app that runs on the AWS EB environment. With recent updates, I had to integrate django-rq and rqscheduler for some queue-based background tasks. This all works fine on localhost with commands rqworker and rqscheduler. But I am having real trouble finding a way to make it run on the AWS EB environment. My analysis says the only way to go is to use ElastiCache. Can anyone guide me in the right direction or any blog posts that could help me with this?

Answer 1

Yeah! So you're probably going to want to separate your persistent store (Redis) from your workers. This is really well abstracted in Heroku (not saying you should necessarily use them, but their UI reflects reality very well) with Resources (not restarted between deploys) and Dynos (restarted between deploys).

You should likely have an ElastiCache (or self-hosted Redis) instance for each of your deployed environments (production, staging, etc.) with any URLs/credentials via YAML. That way, you won't lose jobs when your service is rebooted (because Redis will still be alive) but you can deploy new code whenever you want!