Please enable a way to create an alert policy for log-based custom metrics through the API [143436657]

Assigned

Feature Request

Status Update

No update yet.

Description

wi...@wietsemuizelaar.nl

created issue #1

Oct 28, 2019 02:58PM

Please describe your requested enhancement. Good feature requests will solve common problems or enable new use cases.

What you would like to accomplish:

See

https://issuetracker.google.com/issues/143232362 for reference. Currently, there is no way to create an alert-policy for log-based metrics that is API-based.

How this might work:

I see 3 ways of implementing this:
- allow the GUI way of working to be valid in the API as well (without a resource-type filter, and that the API server doesn't fail)
- create a resource-type for log-based metrics that can be used (resource.type="logging" or something like that)
- make resource.type="metric" valid for log-based metric.

If applicable, reasons why alternative solutions are not sufficient:
There are lots of recommendations from the Security Health Analytics component that include creating an alert for custom log metric events, which I really would like to automate. Currently, this is not possible, since the only way to achieve this is actually using the Stackdriver GUI.

Other information (workarounds you have tried, documentation consulted, etc):

The documentation

https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies#Condition now currently states for the API that a resource-type is option, but it actually isn't since when you try to create an alert-policy without a resource type filter, the API will fail:

ERROR: (gcloud.alpha.monitoring.policies.create) INVALID_ARGUMENT: Field alert_policy.conditions[0].condition_threshold.filter had an invalid value of "metric.type="

logging.googleapis.com/user/audit/create_service_account"": must specify a restriction on "resource.type" in the filter; see "

https://cloud.google.com/monitoring/api/resources" for a list of available resource types.

Comments

di...@google.com <di...@google.com> #2Oct 28, 2019 10:44PM

I have forwarded this request to the engineering team. We will update this issue with any progress updates and a resolution.

[Deleted User] <[Deleted User]> #3Oct 29, 2019 01:06AM

Hello! Sorry to bring up this issue after almost a year but I wanted to add that we have chosen metric identifier as

agent.googleapis.com/memory/percent_used but autoscaling didnt work out for us either. It would be appraciated if you can guide us.

[Deleted User] <[Deleted User]> #4Jan 8, 2020 09:40AM

Hi, at the moment we are using the cpu_utilization/target_utilization attribute (in app.yaml) for autoscaling in the app engine flexible environment, however it would be great if we can have the way to mention the memory_utilization metrics as well to decide on the auto scaling. It will give us more control of the auto scaling the instances than now.

[Deleted User] <[Deleted User]> #5Jan 9, 2020 08:15AM

Hi, I do not see any memory metrics in neither console nor stackdriver. Is this connected to this issue?

[Deleted User] <[Deleted User]> #6Feb 11, 2020 03:51AM

Hello Google team, I was directed to this issue by the support team when we raised concerns around the non availability of memory metrics for Auto Scaling. Is this feature 'released' or in roadmap or not considered ? Please provide some details around this

jo...@temp.yupiq.com <jo...@temp.yupiq.com> #7Apr 10, 2020 02:31AM

Hi ,

Can somebody tell the exact metric for memory based autoscaling

[Deleted User] <[Deleted User]> #8Jun 24, 2020 09:14AM

Is there an update on this issue? This is still present.

Message last modified on Jun 24, 2020 09:15AM

ad...@google.com <ad...@google.com> Jun 25, 2020 10:30PM

Assigned to gc...@google.com.

ew...@squareup.com <ew...@squareup.com> #9Jul 8, 2020 09:07PM

Ugh. I just spent three hours trying to debug this before finding this issue. +1!

mw...@gmail.com <mw...@gmail.com> #10Aug 19, 2020 10:45PM

kindly fix this issue

be...@gtempaccount.com <be...@gtempaccount.com> #11Aug 25, 2020 08:52AM

I too am running into this issue.

[Deleted User] <[Deleted User]> #12Aug 27, 2020 07:21AM

i'm also running into this!

br...@shopgiddy.com <br...@shopgiddy.com> #13Sep 3, 2020 08:42PM

For anyone still running into this issue, it looks like setting the type as `global` will allow you to use custom metrics

[Deleted User] <[Deleted User]> #14Sep 21, 2020 11:45AM

Same, here It would be very nice if I do not have to "click-ops" the gui. Regarding the global type, should we set it like this? Terraform example:

AND resource.type=\"global\"

[Deleted User] <[Deleted User]> #15Sep 21, 2020 12:01PM

https://issuetracker.google.com/issues/143436657#comment13 AND resource.type=\"global\" works, thank you very much!

ky...@intuit.com <ky...@intuit.com> #16Jul 13, 2021 12:51AM

Adding AND resource.type=\"global\" is not working for me.

When navigating to the metric editor in the GUI I get an error: "Chart definition invalid. The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series."

Interestingly when I remove "Resource type: Global" from the "find resource type and metric" selection the error goes away and it appears to be working. However Terraform will not let me create the alert without a resource type.

sy...@brightedge.com <sy...@brightedge.com> #17Aug 7, 2021 06:32PM

I'm having the same problem adding resource.type="global" just allows the API call to work, but the alert will never fire since the values are not checked. Please fix

te...@gmail.com <te...@gmail.com> #18Aug 19, 2021 12:10AM

Worked with my GCP CE, figured out it should be resource.type="api".

[Deleted User] <[Deleted User]> #19Sep 30, 2021 01:23AM

le dot

co...@thomsonreuters.com <co...@thomsonreuters.com> #20Nov 15, 2021 08:30PM

Adding resource.type=\"global\" to the filter worked for me

Still wasted a lot of time, and I was using a different iac provider not Terraform (Pulumi)

Message last modified on Nov 15, 2021 08:56PM

rc...@google.com <rc...@google.com> #21Nov 17, 2021 02:39PM

Hello is there any update on this? I have a similar case

Adding resource.type=\"global\" to the filter did not work.
Cx beleives the problem comes from the severity keyword that's not recognized:

Field alert_policy.conditions[0].condition_threshold.filter had an invalid value
Maybe it's the terraform documentation that's not clear about how to write filters, but the strange thing is that this filter works when I create my alert by hand in the gcp console.

Message last modified on Dec 2, 2021 09:53AM

rc...@google.com <rc...@google.com> #22Nov 29, 2021 03:14PM

Hello Are there any updates on this?

[Deleted User] <[Deleted User]> #23Dec 2, 2021 12:40PM

Adding AND resource.type=\"global\" is work for me in terraform

ns...@google.com <ns...@google.com> #24Mar 29, 2022 08:00PM

Running into the same issue. Workarounds like choosing resource types as global or api not working. Can you please provide fix or help identifying a workaround here.

to...@wikimedia.de <to...@wikimedia.de> #25Mar 31, 2022 03:06PM

Yeah whats the deal with letting this sit for 2 years without any resolve or hints. This problem still exists.

It's not so much a feature request but rather a bug report.

Message last modified on Mar 31, 2022 03:07PM

to...@wikimedia.de <to...@wikimedia.de> #26Mar 31, 2022 04:09PM

Ok after some debugging it was indeed just a matter of finding the correct resource.type.

The first problem is the dropdown list is very long. The difference between using the UI and the API (terraform) that errors when not providing a resource.type makes you think something like global or api would resolve it. In our case using k8s_container resolved the problem. For any future readers of this thread identifying where the original google_logging_metric is filtering on will help you determine the resource.type where the actual metrics ends up.

mi...@gree.net <mi...@gree.net> #27Apr 10, 2022 11:20PM

I am also encountering this problem.

Adding AND resource.type=\"global\"
This doesn't cause an error, but
Since we can't actually detect the metrics, we can't
I'm having trouble.

Message last modified on Apr 10, 2022 11:25PM

ar...@hedera.com <ar...@hedera.com> #28Apr 12, 2022 07:54PM

I can confirm that 'finding the correct resource.type' is a successful resolution. What is the correct resource type? "the monitored resource for the logs-based metrics should be the same as that of the corresponding logs (e.g., if the log is written against gce_instance, the metrics will be against gce_instance as well)." [1]

So basically whatever the resouce.type is for your logs based metric needs to be defined in the alert as well. In my case it was gcs_bucket

[1]

https://groups.google.com/g/google-stackdriver-discussion/c/TQksCmJP_CE?pli=1

ve...@huntington.com <ve...@huntington.com> #29Aug 11, 2022 03:50PM

Did one any one resolved this problem? please help me to resolve same
Adding AND resource.type=\"global\" is not working for me.

ve...@huntington.com <ve...@huntington.com> #30Aug 12, 2022 01:34AM

i think i have figured out the solution..we need to add resource.type="cloud_run_revision" for cloudrun logs..it works for me

br...@buddiez.app <br...@buddiez.app> #31Oct 21, 2022 11:44AM

Guys, for every future reader: the `resource.type` must be THE SAME that you put on the metric. For example, I wanted to capture a metric from my cloud run instance. So, the `resource.type` should be `cloud_run_revision`. Yeah, I know it is tricky.

qb...@cfgo.com <qb...@cfgo.com> #32Dec 15, 2022 02:57PM

This is not true for every use case. We are stuck with resource.type="gce_route".

ab...@worldline.com <ab...@worldline.com> #33Jan 17, 2023 09:22AM

Are there any plans to enable alert creation via gcloud/terraform where we do not have to specify resource.type?

[Deleted User] <[Deleted User]> #34Apr 26, 2023 05:48PM

I was able to get this working using:

AND resource.type=\"logging_bucket\"

[Deleted User] <[Deleted User]> #35Jul 12, 2023 02:57PM

For future readers:

If you are trying to measure instance group size:

      metric.type = "compute.googleapis.com/instance_group/size"
      AND resource.type="instance_group"

As someone mentioned earlier, " the resource.type must be THE SAME that you put on the metric "

Message last modified on Jul 12, 2023 02:58PM

sh...@gmail.com <sh...@gmail.com> #36Aug 31, 2023 10:20AM

someone wrote that you need to specify the resource from where this log is created...
I specified
"metric.type=\"

logging.googleapis.com/user/my-log-based-metric\" AND resource.type=\"gce_instance\" " - it's works

ab...@worldline.com <ab...@worldline.com> #37Oct 2, 2023 01:25PM

We are still facing this issue. Trying to create a custom alert using resource.type=\"project\" but it keeps giving the error:
The filter contains unknown resource type: project

We have been facing this issue since more than a year now, are there any plans to bring any improvement ? resource.type=project seems a very basic thing which should be present like "global".

dh...@google.com <dh...@google.com> Oct 4, 2023 06:20AM

Reassigned to dh...@google.com.

dh...@google.com <dh...@google.com> Oct 10, 2023 06:38AM

Reassigned to gc...@google.com.

th...@ustudio.com <th...@ustudio.com> #38Jan 19, 2024 09:05PM

We also ran into this issue, and resolved it by, rather than using a log-based metric as the basis for a threshold-based alert policy, we use just an alert policy, with a conditionMatchedLog condition (in Terraform, this is condition_matched_log in the condition block).

This resolves the issue of having to guess which resource type the generated log-based metric is in, because there is no log-based metric at all.

It also is considerably simpler, as there is only one resource per filter, and the threshold timing doesn't need to be figured out.

ab...@worldline.com <ab...@worldline.com> #39Jun 4, 2024 09:20AM

The workaround to directly create an alert metric does indeed solve the issue but there are multiple SCC findings which indicate we "must" create a log based metric first and then use it to create an alert policy. Even though we already have an alert policy created directly for the same security finding without using the log metrics.

The creation of alerts from log metrics is not the smoothest way in many cases because the API channel doesnt support many of the resource.types like gce_firewall_rule, gce_route etc. Can you let us know when there will be a fix for this available ?