IssueTracker

Cloud Pub/Sub's subscriber flow control is specified on a per-client basis. If the client has multiple concurrent streams, the flow control limits are enforced on each individual stream on the server and across all streams in the client. As a result, the client could receive and buffer more messages than the flow control limits. For example, if one specifies a max of 10 outstanding messages and opens 10 streams, then the client may receive 100 messages. Additionally, if a stream closes, the client reopens it and at this time, the client may receive additional messages up to the flow control limits specified.

In these scenarios, the client may hold on to messages that are not sent to the callback specified by the user for processing messages, delaying the delivery of the messages. The Pub/Sub team is aware of this issue and is working on improvements.

There are steps one can take in their subscribers to minimize the impact of this issue:

1. Set the number of concurrent streams to 1. Instead, horizontally scale to more subscriber instances. These need not be in different job instances; one job can just create multiple instances of the subscriber client.
2. Ensure that subscribers being flow controlled is a temporary, transient occurrence. This may require horizontally scaling to spread load out over more subscriber clients or raising the flow control limits.
3. Process and acknowledge messages quickly. Processing more quickly reduces the amount of time other messages may spend buffered in the client waiting to be able to be processed.