Cloud Task Creation 502's in us-east1 [344943219]

Bug

Status Update

No update yet.

Description

an...@brighthire.ai

created issue #1

Jun 5, 2024 01:27AM

Starting on 5/29, we started to see an increased rate of 502's returned on cloud task creation that is still ongoing. Our application is designed to retry so when it does, we then end up getting a 409 Entity Already Exists.

I expect task creation to succeed on first attempt, but if it does return a 502, I would expect a retry to succeed.

Are there any GCP outages that would be responsible for this behavior?

Comments

je...@metalgear.xyz <je...@metalgear.xyz> #2Jun 6, 2024 07:15AM

I also experience a high number of 502 in europe-west1

an...@brighthire.ai <an...@brighthire.ai> #3Jun 6, 2024 01:53PM

We suspect that we may be running into some sort of rate limit. The Cloud Tasks API Quota for maximum requests per minute is rather high (6 million/min) and we're not even close to that, but we believe that when we have bursts of task creation that the API then throws the 502.

an...@gmail.com <an...@gmail.com> #4Dec 16, 2024 05:37PM

Any update on this? It seems to be happening again. We are experiencing both high latency and 502, the requests are seconds apart.

Issue 344943219

Description

Issue summary

Comments

je...@metalgear.xyz <je...@metalgear.xyz> #2Jun 6, 2024 07:15AM

an...@brighthire.ai <an...@brighthire.ai> #3Jun 6, 2024 01:53PM

an...@gmail.com <an...@gmail.com> #4Dec 16, 2024 05:37PM

Add comment

Issue metadata