Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

GCloud Pub/Sub Push Subscription: Limit max outstanding messages

Is there a way in a push subscription configuration to limit the maximum number of outstanding messages. In the high level subscriber docs (https://cloud.google.com/pubsub/docs/push) it says "With slow-start, Google Cloud Pub/Sub starts by sending a single message at a time, and doubles up with each successful delivery, until it reaches the maximum number of concurrent messages outstanding." I want to be able to limit the maximum number of messages being processed, can this be done through the pub/sub config?

I've also thought of a number of other ways to effectively achieve this, but none seem great:

  • Have some semaphore type system implemented in my push endpoint that returns a 429 once my max concurrency level is hit?

  • Similar, but have it deregister the push endpoint (turning it into a pull subscription) until the current messages have been processed

My push endpoints are all on gae, so there could also be something in the gae configs to limit the simultaneous push subscription requests?

like image 580
Kevin Abbott Avatar asked Apr 03 '18 16:04

Kevin Abbott


People also ask

What happens if two subscribers use the same subscription to pull messages from a topic in Google pub sub?

In the process of redelivering the outstanding message, Pub/Sub holds back and tries not to deliver the outstanding message to any other subscriber on the same subscription. Once the outstanding message is acknowledged it can be delivered to other subscribers.

How long do messages stay in Pubsub?

The default message retention duration is 7 days and the default expiration period is 31 days. To create a subscription with retention of acked messages enabled, follow these steps: In the Google Cloud console, go to the Pub/Sub subscriptions page. Click Create subscription.

Can Pubsub hold millions of messages?

Resource limitsThere is no limit on the number of retained messages.

What is Unacked message count?

Unacked message count and oldest message age refer to the number and age of undelivered messages in your subscription.


Video Answer


1 Answers

Push subscriptions do not offer any way to limit the number of outstanding messages. If one wants that level of control, the it is necessary to use pull subscriptions and flow control.

Returning 429 errors as a means to limit outstanding messages may have undesirable side effects. On errors, Cloud Pub/Sub will reduce the rate of sending messages to a push subscriber. If a sufficient number of 429 errors are returned, it is entirely possible that the subscriber will receive a smaller number of messages than it can handle for a time while Cloud Pub/Sub ramps the delivery rate back up.

Switching from push to pull is a possibility, though still may not be a good solution. It would really depend on the frequency with which the push subscriber exceeds the desired number of outstanding messages. The change between push and pull and back may not take place instantaneously, meaning the subscriber could still exceed the desired limit for some period of time and may also experience a delay in receiving new messages when switching back to a push subscriber.

like image 130
Kamal Aboul-Hosn Avatar answered Jan 04 '23 06:01

Kamal Aboul-Hosn