What is the equivalent of AWS Dead Letter Queue in Google Cloud Platform?

Question

What is the equivalent of AWS Dead Letter Queue in Google Cloud Platform? How is the failed records managed in Google Cloud Platform?

Answer 1

As of beginning 2020, Google Pub/Sub now supports configuring dead letter topic when creating a subscription (just like other major queueing systems):

$ gcloud pubsub subscriptions create SUBSCRIPTION \
  --topic=TOPIC \
  --topic_project=TOPIC_PROJECT \
  --max-delivery-attempts=NUMBER_OF_RETRIES \
  --dead-letter-topic=DEAD_LETTER_TOPIC \
  --dead-letter-topic-project=DEAD_LETTER_TOPIC_PROJECT

After reaching the NUMBER_OF_RETRIES the message that failed to be delivered to TOPIC will be published to DEAD_LETTER_TOPIC instead (which can be useful for further analysis, triggering an alert or other automated action, etc.)

As stated in the docs, the Cloud Pub/Sub service account associated with the enclosing subscription's parent project (i.e., service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com) must have permission to Publish() to this topic and Acknowledge() messages on this subscription.

Docs: https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/create#--max-delivery-attempts

Answer 2

Short answer: there is none. Google PubSub is missing this core feature of every other queuing system.

Longer answer: You can try to implement a DLQ yourself, but there are several features missing from Google Pub/Sub that make it difficult to implement correctly.

1. PubSub does not track how many times a message has been delivered so that you can have it sent to a DLQ after X failed attempts. So you have to create your own datastore in something like Redis to track delivery count by message id.
2. PubSub does not support exponential back off for redelivery. If you want a message redelivered, you don't ack or nack (which just causes immediate redelivery), you instead do nothing and let the message timeout.
3. There is no way to configure a message to go to a DLQ. So you have to create a second PubSub topic and then use client side logic to determine if/when the message should be sent to the DLQ topic. If you need to reprocess the messages in the DLQ, then you have to pull them from the DLQ topic and push them back into the main topic. Note that this means it will get redelivered to every subscriber! There is no way to redeliver a message to a specific subscription. So every subscriber has to be idempotent to avoid double processing a message.