Implementing "idle" and "normal" priority queues for long-running tasks in Celery

Question

I'm attempting to implement a following scenario with Celery: two queues of (same) long-running tasks, one for "normal" and the other for "idle" priority.

I'd make the workers monitor both queues and take tasks from the "normal" priority queue first and if it's empty, it'd take tasks from the "idle" priority.

My question is: Is it possible to guarantee the order in which the workers will check their tasks queues? Also, is this the right approach for implementing priorities?

Background: The tasks are running ffmpeg transcoding jobs. The "normal" priority ones would be new videos coming in (which have to be transcoded ASAP) and the "idle" priority would be the tasks to re-transcode old (40.000+ video) archive to updated format settings. I do not have several servers available to do multi-server task dispatching.

Answer 1

Under the theory that idle workers are not consuming much system resources, I have implemented two priorities with a second exchange, a second set of queues, and a second set of workers. I don't have such intensive work processes so I don't make my second set of workers throttle themselves when the first set is active, but possible the UNIX renice command would work for this.

You could probably also do something with a control queue so that it is not sufficient for workers to receive a work request, they also need to get a token from a control queue. Then you spit out a small number of tokens to the idle control queue but a larger number of tokens to the normal control queue.