IssueTracker

Summary of problem:

• container-watcher is a GCP managed daemonset, in kube-system namespace.
• On Sept 13th, 2024 an automatic update was pushed to multiple clusters, that updated a container-watcher daemonset to image gke.gcr.io/watcher-daemonset:ktd_release.watcher_20240908_RC00
• The newly released image has a memory leak, and older versions of the image did not have this memory leak. Attached "new image has memory leak.png", gives some visual context.

About container-watcher:
Most people probably aren't familiar with it, as it's part of GKE's Enterprise Offering's integration with GCP's Security Command Center, here are some relevant doc's links for reference.

• https://cloud.google.com/security-command-center/docs/concepts-container-threat-detection-overview
• https://cloud.google.com/security-command-center/docs/how-to-use-container-threat-detection#enabling
• https://cloud.google.com/security-command-center/docs/concepts-scc-errors

Problem you have encountered:

• The main problem is it's causing noise on alerts.
• The issue is limited, because while a memory leak was introduced, it has a memory limit of 400mb that will reboot the pod before it can cause too many problems.
• The used memory is over its request, which creates a negligible, yet non-zero chance of a noisy neighbor problem.

What you expected to happen:

• CICD QA to be mature enough to detect memory leaks, and not push a bugged platform update that introduces a memory leak bug.

Steps to reproduce:

• This is hard to reproduce in most sandbox environments as you need Enterprise Tier and Security Command Center, but if you install the daemonset, you can just wait and see the problem.

Other information:

• This is happening across multiple clusters running different versions of kubernetes 1.25 - 1.30.
• This isn't a bid deal, but the point of CI is that issues and bugs are cheaper to fix when they're caught early, and this one wasn't caught by CI. If it's not fixed soon, there's a risk of it never getting fixed, and I'm concerned it might go unnoticed.
• As you can see in the screen capture the old version of the image didn't have a memory leak, so it's important to report, so the small diff between those 2 images can be captured to discover the change that introduced the memory leak, and potentially update CI testing to check for memory leaks in the future.