PubSub Emulator retry and dead letter policy broken when enableMessageOrdering is on [399692454]

Assigned

Bug

Status Update

No update yet.

Description

sh...@herobullion.com

created issue #1

Feb 28, 2025 03:34AM

Problem you have encountered:

The PubSub Emulator's dead letter and retry policies don't work for subscriptions that have enableMessageOrdering is true. The dead letter and retry policies do work properly when enableMessageOrdering is not specified.

This is a problem for me because all of my subscriptions need message ordering, so I am unable to test this functionality locally or in ci.

What you expected to happen:

When the enabledMessageOrdering is enabled, the retry policy should work, and the dead letter policy should work.

Steps to reproduce:

Run the emulator locally. I have it running in a docker container, I'm using your standard image: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators. Here is what I get when I run gcloud version:

docker run --rm gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators gcloud version
Google Cloud SDK 512.0.0
beta 2025.02.21
bigtable
bq 2.1.13
cloud-datastore-emulator 2.3.1
cloud-firestore-emulator 1.19.9
core 2025.02.21
gcloud-crc32c 1.0.0
gsutil 5.33
pubsub-emulator 0.8.19

Create a couple topics+subscriptions to test with. Note I am setting "enableMessageOrdering": true on the main topic. Also note I am using push subscriptions, and in order to test this I am configuring the main topic's push URL to be one that returns an error, whereas the dead letter topic's URL responds with a 200.

# Create deadletter topic
curl 'http://localhost:8085/v1/projects/proj1/topics/deadletter1' -X 'PUT' -H 'content-type: application/json' --data-raw '{}'

# Create deadletter subscription
curl 'http://localhost:8085/v1/projects/proj1/subscriptions/deadletter1-sub' -X 'PUT' -H 'content-type: application/json' --data-raw '{"topic":"projects/proj1/topics/deadletter1","pushConfig":{"pushEndpoint":"http://host.docker.internal:3000/order-management/async/deadletter1?deadletter=true"}}'

# Create main topic
curl 'http://localhost:8085/v1/projects/proj1/topics/topic1' -X 'PUT' -H 'content-type: application/json' --data-raw '{}'

# Create main subscription
curl 'http://localhost:8085/v1/projects/proj1/subscriptions/topic1-sub' -X 'PUT' -H 'content-type: application/json' --data-raw '{"topic":"projects/proj1/topics/topic1","enableMessageOrdering": true, "pushConfig":{"pushEndpoint":"http://host.docker.internal:3000/order-management/async/topic1"},"deadLetterPolicy":{"deadLetterTopic":"projects/proj1/topics/deadletter1","maxDeliveryAttempts":5},"retryPolicy":{"maximumBackoff":"30s","minimumBackoff":"10s"}}'

Since I am using push subscriptions, I created a simple app using bun (nodejs alternative). Here is my code. It is very simple, it just responds with a 401 status code to the topic1-sub subscription push URL, and a 200 status code to the deadletter1-sub subscription push URL.

import { Hono } from 'hono';
import { HTTPException } from 'hono/http-exception';

const app = new Hono();

app.all('/order-management/*', async (c) => {
  console.log('Request received at', new Date(), 'url:', c.req.url);
  console.log('Payload is:', await c.req.text());
  if (c.req.query('deadletter') === 'true') {
    console.log('Deadletter request received!');
    console.log('👍');
    return c.json({ message: 'Deadletter request received!' });
  }
  console.log('👎 ❌');
  throw new HTTPException(401, { message: 'Intentional error' });
});

const port = parseInt(process.env.PORT!) || 3000;
console.log(`Running at http://localhost:${port}`);

export default {
  port,
  fetch: app.fetch,
};

Publish a message to the main topic. Here is how I'm doing that.

curl 'http://localhost:8085/v1/projects/proj1/topics/topic1:publish' -H 'content-type: application/json' --data-raw '{"messages":[{"data":"aGVsbG8gd29ybGQK"}]}'

Observe the logs of the sample web app. Note that the retries all happened immediately, one after another, every second, and they don't wait and obey retry policy min/max backoff values. Also observe that when the maxDeliveryAttempts is reached the message stops retrying (which is good!), but the message doesn't get moved to the dead letter topic (which is bad!). Here are my app logs, take note of the timestamps and there are only 5 entries:

Request received at 2025-02-28T03:14:03.364Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:14:03.289Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:14:03.289Z","messageId":"23","attributes":{},"message_id":"23"}}
👎 ❌
Request received at 2025-02-28T03:14:04.405Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:14:03.289Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:14:03.289Z","messageId":"23","attributes":{},"message_id":"23"}}
👎 ❌
Request received at 2025-02-28T03:14:05.457Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:14:03.289Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:14:03.289Z","messageId":"23","attributes":{},"message_id":"23"}}
👎 ❌
Request received at 2025-02-28T03:14:06.486Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:14:03.289Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:14:03.289Z","messageId":"23","attributes":{},"message_id":"23"}}
👎 ❌
Request received at 2025-02-28T03:14:07.539Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:14:03.289Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:14:03.289Z","messageId":"23","attributes":{},"message_id":"23"}}
👎 ❌

Now lets "fix" this by taking off enableMessageOrdering of that one subscription. (Of course I don't want to have to do this because my app relies on ordered messaging!)

# Delete the subscription
curl 'http://localhost:8085/v1/projects/proj1/subscriptions/topic1-sub' -X 'DELETE' -H 'content-type: application/json' --data-raw '{}'

# Create the subscription again, this time without enableMessageOrdering
curl 'http://localhost:8085/v1/projects/proj1/subscriptions/topic1-sub' -X 'PUT' -H 'content-type: application/json' --data-raw '{"topic":"projects/proj1/topics/topic1","pushConfig":{"pushEndpoint":"http://host.docker.internal:3000/order-management/async/topic1"},"deadLetterPolicy":{"deadLetterTopic":"projects/proj1/topics/deadletter1","maxDeliveryAttempts":5},"retryPolicy":{"maximumBackoff":"30s","minimumBackoff":"10s"}}'

Now publish another message. Just use the publish command from above.
Observe the logs of the sample web app. Note that the retryPolicy values were obeyed - there were waits between retries. Also note that there is one last entry in the log, compared to last time, which is from when the dead letter subscription fired.

Request received at 2025-02-28T03:23:05.384Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:23:05.373Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:23:05.373Z","messageId":"24","attributes":{},"message_id":"24"}}
👎 ❌
Request received at 2025-02-28T03:23:06.455Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:23:05.373Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:23:05.373Z","messageId":"24","attributes":{},"message_id":"24"}}
👎 ❌
Request received at 2025-02-28T03:23:22.967Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:23:05.373Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:23:05.373Z","messageId":"24","attributes":{},"message_id":"24"}}
👎 ❌
Request received at 2025-02-28T03:23:44.983Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:23:05.373Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:23:05.373Z","messageId":"24","attributes":{},"message_id":"24"}}
👎 ❌
Request received at 2025-02-28T03:24:08.966Z url: http://host.docker.internal:3000/order-management/async/topic1
Payload is: {"subscription":"projects\/proj1\/subscriptions\/topic1-sub","message":{"publishTime":"2025-02-28T03:23:05.373Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:23:05.373Z","messageId":"24","attributes":{},"message_id":"24"}}
👎 ❌
Request received at 2025-02-28T03:24:33.994Z url: http://host.docker.internal:3000/order-management/async/deadletter1?deadletter=true
Payload is: {"subscription":"projects\/proj1\/subscriptions\/deadletter1-sub","message":{"publishTime":"2025-02-28T03:24:33.957Z","data":"aGVsbG8gd29ybGQK","publish_time":"2025-02-28T03:24:33.957Z","messageId":"25","attributes":{"CloudPubSubDeadLetterSourceSubscription":"topic1-sub","CloudPubSubDeadLetterSourceSubscriptionProject":"proj1","CloudPubSubDeadLetterSourceDeliveryCount":"5","CloudPubSubDeadLetterSourceTopicPublishTime":"2025-02-28T03:23:05.373Z"},"message_id":"25"}}
Deadletter request received!
👍

Other information (workarounds you have tried, documentation consulted, etc):

I didn't show it in my steps above but please note that when the enableMessageOrdering is set on the subscription, I did make sure to add an orderingKey to the published messages, but that made no difference.

I would really appreciate it if you could address this issue! 🙏

Comments

sh...@herobullion.com <sh...@herobullion.com> #2Feb 28, 2025 03:40AM

Also, just another note I forgot to mention in my post, I am 99.999% certain I followed these rough steps using the Google Cloud Console (I didn't use curl commands, for example, I just used the Console's UI) and it seems that enabledMessageOrdering works along with the retryPolicy and deadLetterPolicy in real environments.

va...@google.com <va...@google.com> Feb 28, 2025 06:30AM

Assigned to pu...@google.com.

pu...@google.com <pu...@google.com> #3Mar 3, 2025 01:05PM

Reassigned to gc...@google.com.

Hello,

Thank you for reaching out to us with your request.

We have duly noted your feedback and will thoroughly validate it. While we cannot provide an estimated time of implementation or guarantee the fulfillment of the issue, please be assured that your input is highly valued. Your feedback enables us to enhance our products and services.

We appreciate your continued trust and support in improving our Google Cloud Platform products. In case you want to report a new issue, Please do not hesitate to create a new issue on the Issue Tracker providing a detailed description of your issue.

Once again, we sincerely appreciate your valuable feedback; Thank you for your understanding and collaboration.

Issue 399692454

Description

Issue summary

Comments

sh...@herobullion.com <sh...@herobullion.com> #2Feb 28, 2025 03:40AM

va...@google.com <va...@google.com> Feb 28, 2025 06:30AM

pu...@google.com <pu...@google.com> #3Mar 3, 2025 01:05PM

Add comment

Issue metadata