How I can specify SQS queue name in celery

Question

I need to replace my redis broker with SQS broker, while googleing it I came across many pages which tell how to use SQS with celery. As per my understanding, it creates own SQS queue, I have only one task and want to use already created SQS queue.

Answer 1

By default celery will create a new queue for you using the a Queue Prefix settings if defined.

However, if you want to use an existing queue, you can provide the name with the task-default-queue settings. Do make sure you don't define the queue-prefix mentioned above in this case.

Answer 2

You could set the queue name via broker_transport_options (in celery 4.0) like:

broker_transport_options = {"queue_name_prefix": "my-queue-"}

Documentation is here

Answer 3

The commit on 26th Feb, 2020 adds the ability to use predefined queues.

You should be able to use predefined Queues by adding predefined queues options to CELERY_BROKER_TRANSPORT_OPTIONS

CELERY_BROKER_TRANSPORT_OPTIONS={
    'predefined_queues':{
            'HIGH_PRIORITY': {
              'url': 'https://sqs.ap-south-1.amazonaws.com/030221/HGH_PRIORITY',
              'access_key_id': config('AWS_ACCESS_KEY'),
              'secret_access_key': config('AWS_SECRET_KEY'),
            },
        }
}

Following is the documentation update from the commit -

Other Features supported by this transport:
  Predefined Queues:
    The default behavior of this transport is to use a single AWS credential
    pair in order to manage all SQS queues (e.g. listing queues, creating
    queues, polling queues, deleting messages).
    If it is preferable for your environment to use a single AWS credential, you
    can use the 'predefined_queues' setting inside the  'transport_options' map.
    This setting allows you to specify the SQS queue URL and AWS credentials for
    each of your queues. For example, if you have two queues which both already
    exist in AWS) you can tell this transport about them as follows:
    transport_options = {
      'predefined_queues': {
        'queue-1': {
          'url': 'https://sqs.us-east-1.amazonaws.com/xxx/aaa',
          'access_key_id': 'a',
          'secret_access_key': 'b',
        },
        'queue-2': {
          'url': 'https://sqs.us-east-1.amazonaws.com/xxx/bbb',
          'access_key_id': 'c',
          'secret_access_key': 'd',
        },
      }
    }