Deleting individual log entries [168298994]

Assigned

Customer Issue

Status Update

No update yet.

Description

pe...@google.com

created issue #1

Sep 11, 2020 02:31PM

This will create a feature request which anybody can view and comment on.

Please describe your requested enhancement. Good feature requests will solve common problems or enable new use cases.

-What you would like to accomplish:

The ability to delete single log entries or log entries matching a particular filter and time frame would allow for preservation of critical logs.

-How this might work:
Currently the only way to perform a deletion is via the gcloud command

gcloud logging logs delete logToBeDeleted

Reading a log entry has a concept of a filter, e.g.

https://cloud.google.com/logging/docs/reference/tools/gcloud-logging#reading_log_entries

Applying a similar filter to the deletion command can allow customers to preserve logs that are usable for other purposes.

-If applicable, reasons why alternative solutions are not sufficient:

Deletion of all log entries means that investigations historically are hindered and all of our history around WAF rule compatibility research was unfortunately lost due to this feature not being available.

-Other information (workarounds you have tried, documentation consulted, etc):

We identified an information leak from our customers using our services in a way that was not intended. In order to prevent this happening we updated our inclusion filter in our sink for logs to drop these log entries BUT the only way to achieve this historically was to delete everything.

Comments

ah...@google.com <ah...@google.com> #2Sep 11, 2020 05:31PM

Assigned to gc...@google.com.

I have forwarded this request to the engineering team. We will update this issue with any progress updates and a resolution.

jo...@google.com <jo...@google.com> #3Jan 26, 2021 08:17PM

Hello! Sorry to bring up this issue after almost a year but I wanted to add that we have chosen metric identifier as

agent.googleapis.com/memory/percent_used but autoscaling didnt work out for us either. It would be appraciated if you can guide us.

jo...@google.com <jo...@google.com> #4Feb 5, 2021 07:30PM

Hi, at the moment we are using the cpu_utilization/target_utilization attribute (in app.yaml) for autoscaling in the app engine flexible environment, however it would be great if we can have the way to mention the memory_utilization metrics as well to decide on the auto scaling. It will give us more control of the auto scaling the instances than now.

si...@worldnettps.com <si...@worldnettps.com> #5Mar 31, 2021 09:53AM

Hi, I do not see any memory metrics in neither console nor stackdriver. Is this connected to this issue?

[Deleted User] <[Deleted User]> #6Nov 29, 2021 09:02PM

Hello Google team, I was directed to this issue by the support team when we raised concerns around the non availability of memory metrics for Auto Scaling. Is this feature 'released' or in roadmap or not considered ? Please provide some details around this

mm...@google.com <mm...@google.com> #7Apr 7, 2022 06:19PM

Hi ,

Can somebody tell the exact metric for memory based autoscaling

[Deleted User] <[Deleted User]> #8Jun 22, 2022 11:10AM

Imagine a single one of these teams accidentally deploys a container that exposes sensitive customer data. Due to customer security/privacy principles and/or compliance reasons, an organization must take swift action to redact or delete the log entries. Without the feature described in comment#1, the customer will have to delete ALL k8s_container logs for all containers on every GKE cluster in the project.

This is exactly our situation, and it we can confirm that this is extremely painful. Are there any updates regarding when/if this may be available?

hi...@elmy.fr <hi...@elmy.fr> #9Aug 5, 2022 11:48AM

si...@worldnettps.com <si...@worldnettps.com> #10Sep 2, 2022 11:32AM

Is there any movement in getting this issue scheduled?

br...@zenbusiness.com <br...@zenbusiness.com> #11Sep 26, 2022 07:23PM

I'd like to add my comment on this as well. We have a current need to redact/delete/obfuscate a particular log entry or even finite sensitive data within a log entry for data privacy regulations and general security remediations as described in

comment #4

and elsewhere. Can this be prioritized? With all the new privacy and sensitive data regulations currently in place and more on the horizon, this functionality is critical.

be...@banked.com <be...@banked.com> #12Nov 23, 2022 03:40PM

We got burned by this today too.

It's a bit mad that this is literally impossible, it seems like a pretty basic feature to have for a logging platform.

ds...@digitalmint.io <ds...@digitalmint.io> #13Feb 3, 2023 02:41AM

I imagine this feature would be necessary for companies that need to delete retained data on GCP without destroying the whole bucket for audit purposes.

li...@leantaas.com <li...@leantaas.com> #14Feb 17, 2023 12:44AM

To echo others, this would be very useful in situations where you need to retain logs for compliance.

ar...@seclogic.io <ar...@seclogic.io> #15Mar 2, 2023 04:40AM

+1 Similar issue here

ke...@foldapp.com <ke...@foldapp.com> #16Mar 29, 2023 01:16PM

2.5 years later and this is still an issue...

ka...@google.com <ka...@google.com> Jun 6, 2023 04:46AM

Reassigned to ka...@google.com.

ka...@google.com <ka...@google.com> Jun 23, 2023 09:44AM

Reassigned to gc...@google.com.

ru...@anz.com <ru...@anz.com> #17Jun 28, 2023 10:46PM

We are experiencing same issue here ...
Is there any good workarounds so far?

um...@google.com <um...@google.com> #18Aug 8, 2023 12:07AM

Hi team,one of my customer is also having same issue.Customer wants to delete a specific log entry by using its lable.

wa...@mb.com.br <wa...@mb.com.br> #19Aug 8, 2023 06:25PM

Just adding one more issue, the only delete command there is, only deletes logs from the _Default log storage, if you create sinks to filter which logs goes to specific storages, allowing to keep different retention periods per log type, you won't be able to delete, even if you want to delete it all

Message last modified on Aug 8, 2023 06:26PM

da...@ottogroup.com <da...@ottogroup.com> #20Dec 15, 2023 03:49PM

We encountered an issue with a dataflow job and needed to delete logs from a specific job. However, it proved impossible to do so, and we were compelled to delete all entries from the log name. Please consider adding a filter to the 'gcloud logging logs delete' command to enable the deletion of specific log entries.

Thanks in advance.

[Deleted User] <[Deleted User]> #21Jan 15, 2024 12:53AM

Any update on this feature request? Currently I need to delete the entire log if I need to remove some sensitive data or some data that was incorrectly written to the log which is causing an incompatibility with a log sink to big query.

ja...@google.com <ja...@google.com> #22Feb 22, 2024 08:58PM

Hello, I have a customer that has the same request, he needs to delete specific logs, but he is unable to do it, due to this limitation of filtering not available. Is there an estimate time when this will be released ?

lb...@league.com <lb...@league.com> #23Jul 12, 2024 03:06PM

Just ran into this today ourselves. The View filters do not work as a comparable solution. We need better view filters to hide offending entries, or we need to be able to delete. Please!