GCP error reporting via webhooks is also rate limited [273978798]

Assigned

Feature Request

Status Update

No update yet.

Description

ka...@revolut.com

created issue #1

Mar 17, 2023 10:21AM

We are using GCP errors to catch any issues with our services, we are also using the webhook notification channel for automation of the process. Once we receive a webhook request from GCP, we then send messages via slack to service owners and create JIRA tickets.

The GCP error reporting feature has a rate limiter (

https://cloud.google.com/error-reporting/docs/notifications#limit) which does make sense for users of the gcp app & it's notifications, but that limit should not be present on service to service automations like webhooks as it doesn't allow us to catch all the errors that are happening which basically makes the integration useless for bigger projects that have more than 5 errors/hour

Comments

ma...@google.com <ma...@google.com> Mar 20, 2023 04:55AM

Assigned to ma...@google.com.

ma...@google.com <ma...@google.com> #2Mar 20, 2023 09:45AM

Hello,

Thank you for drawing our attention to this.

Could you please provide your issue in the following format :

Problem you have encountered :
What you expected to happen :
Steps to reproduce :
Other information (workarounds you have tried, documentation consulted, etc) :

Any additional information you provide will help us better understand and resolve the problem. Thank you for your trust and continued support to improve Google Cloud Platform products. In case you want to report a new issue, please do not hesitate to create a new Issue Tracker thread describing your issue.

Regards,

Mahaboob Subhani

ka...@revolut.com <ka...@revolut.com> #3Mar 20, 2023 09:53AM

Problem you have encountered : Notifications about new errors in "Error reporting" tab are behind rate limiter for all types of notification channels, including webhooks
What you expected to happen : Notifications that use a webhook as a channel should not be behind ratelimiting (currently 5 invocations per hour) as it doesn't allow us to build automations around it because we are not called about every new error. Rate limiting only makes sense for user focused channels like app/email/etc.
Steps to reproduce : On a project with > 5 new errors every hour, create a notification channel with webhook
Other information (workarounds you have tried, documentation consulted, etc) :

https://cloud.google.com/error-reporting/docs/notifications#limit The behaviour is described in your documentation, but I don't think that it makes sense for webhooks hence I'm raising this issue

ma...@google.com <ma...@google.com> #4Mar 20, 2023 01:12PM

Hello,

Thanks for reaching out to us!

I understand that you would like to integrate Stackdriver with a JIRA project and GCP error reporting via webhooks.

I can confirm that currently there isn’t a built-in integration for Stackdriver and JIRA, Please see the documents [1][2] for your reference. However, as per the document[3] it is possible to use webhooks, which I understand is an option for you.

unfortunately, it seems that currently there is not integration with JIRA.

[1]https://cloud.google.com/stackdriver/integrations [2]https://marketplace.atlassian.com/apps/1214005/google-stackdriver-monitoring?hosting=cloud&tab=overview [3]https://cloud.google.com/monitoring/support/notification-options#webhooks

I hope this is helpful to you.

Thanks and regards,

Mahaboob Subhnai Shaik,

ka...@revolut.com <ka...@revolut.com> #5Mar 20, 2023 01:14PM

Yes, I explained that I'm using webhooks already in this ticket twice, but webhooks are rate limited for unknown reason which makes them unusable for any service to service communication. Please escalate this issue to the team responsible for it.

ma...@google.com <ma...@google.com> #6Mar 21, 2023 06:26AM

Reassigned to gc...@google.com.

Hello,

Thanks for reaching out to us!

The issue request has been created and the Product Engineering Team is working on this request. Right now, there is no ETA to this request. Thank you for your trust and continued support to improve Google Cloud Platform products.

In case you want to report a new issue, please do not hesitate to create a new Issue Tracker thread describing your issue.

Regards, Mahaboob Subhani

je...@layern.so <je...@layern.so> #7Aug 22, 2023 05:02AM

I would like to echo the need for this request. It is very possible for a scenario where notifications are thrown for a new service added within the GCP Project, consuming the quota of 5 per hour, resulting in notifications for a critical service being missed.

Can we at least have the option of raising the limit at account level temporarily?

sa...@revolut.com <sa...@revolut.com> #8Sep 29, 2023 02:26PM

One again I would like to bump this thread.

Rate limiting of 5/hour (and silently ignoring any others) makes this feature not suitable for any real-world production system.

This ticket has nothing to do with Jira - it is solely about the fact that webhooks for GCP errors are rate limited, which makes them not useful for any serious project which relies on error notifications (via webhook).

Is there a way to increase the limit?

Message last modified on Sep 29, 2023 02:33PM

co...@gmail.com <co...@gmail.com> #9Oct 8, 2023 04:41PM

Comment has been deleted.

Message last modified on Oct 8, 2023 04:41PM

je...@layern.so <je...@layern.so> #10Oct 8, 2023 04:42PM

Seconding the comment above, as we're running into the same problem. Rate limit of 5/hour across all services in the same project makes it unsuitable for any large scale production system.

Please provide an update on the progress of this feature given its importance to the usability of the entire product in a production environment.

ba...@google.com <ba...@google.com> Oct 10, 2023 03:48AM

Reassigned to ba...@google.com.

ba...@google.com <ba...@google.com> #11Oct 10, 2023 01:56PM

Reassigned to gc...@google.com.

Hello,

According to the Public Issue Tracker Process, we wait for a feature request to gain a handful of stars, and hopefully, comments from several users before the engineering team works on the issue. Though we can't provide an ETA on this issue nor guarantee their implementation, rest assured that your feedback is always taken very seriously.

To ensure a faster resolution and dedicated support for your issue, I kindly request you to file a support ticket by clicking here. Our support team will prioritize your request and provide you with the assistance you need.

Please note that the Issue Tracker is primarily meant for reporting bugs and requesting new features. For individual support issues, it is best to utilize the support ticketing system. If you have any additional issues or concerns, please don’t hesitate to create a new thread on the Issue Tracker. We appreciate your cooperation!

Thanks & Regards,
Manish Bavireddy.
Google Cloud Support

me...@mikewooldridge.co.uk <me...@mikewooldridge.co.uk> #12Jul 10, 2024 12:49PM

Upvoting and bumping this from my side too - there aren't the same issues around MTA acceptance rates with non-email notification modes, would be good to have un-ratelimited Error Reporting for webhooks too.

Even better would be to have the limit per-service, not account-wide, but appreciate this is a bigger project.

Issue 273978798

Description

Issue summary

Comments

ma...@google.com <ma...@google.com> Mar 20, 2023 04:55AM

ma...@google.com <ma...@google.com> #2Mar 20, 2023 09:45AM

ka...@revolut.com <ka...@revolut.com> #3Mar 20, 2023 09:53AM

ma...@google.com <ma...@google.com> #4Mar 20, 2023 01:12PM

ka...@revolut.com <ka...@revolut.com> #5Mar 20, 2023 01:14PM

ma...@google.com <ma...@google.com> #6Mar 21, 2023 06:26AM

je...@layern.so <je...@layern.so> #7Aug 22, 2023 05:02AM

sa...@revolut.com <sa...@revolut.com> #8Sep 29, 2023 02:26PM

co...@gmail.com <co...@gmail.com> #9Oct 8, 2023 04:41PM

je...@layern.so <je...@layern.so> #10Oct 8, 2023 04:42PM

ba...@google.com <ba...@google.com> Oct 10, 2023 03:48AM

ba...@google.com <ba...@google.com> #11Oct 10, 2023 01:56PM

me...@mikewooldridge.co.uk <me...@mikewooldridge.co.uk> #12Jul 10, 2024 12:49PM

Add comment

Issue metadata