Assigned

Bug

Status Update

No update yet.

Description

lu...@surferseo.com

created issue #1

Jun 19, 2024 08:01AM

Problem you have encountered:

Prometheus collector pods running in gke-gmp-system are crashing.

We do run latest GKE Autopilot cluster: 1.29.4-gke.1043002. The collector pods which are running on arm64 nodes are in crashing loop:

% Kubectl describe pod collector-xtmwj
Name:                 collector-xtmwj
Namespace:            gke-gmp-system
Priority:             1000000000
Priority Class Name:  gmp-critical
Service Account:      collector
Node:                 gk3-production-k8s-clust-nap-1vwzuiv2-68965d5b-h4ff/10.20.0.91
Start Time:           Wed, 19 Jun 2024 09:28:35 +0200
Labels:               app=managed-prometheus-collector
                      app.kubernetes.io/name=collector
                      app.kubernetes.io/version=0.10.0
                      controller-revision-hash=65c88bbb46
                      pod-template-generation=8
Annotations:          cluster-autoscaler.kubernetes.io/safe-to-evict: true
                      components.gke.io/component-name: managed-prometheus
                      components.gke.io/component-version: 0.12.0-gke.0
Status:               Running
SeccompProfile:       RuntimeDefault
IP:                   10.225.137.74
IPs:
  IP:           10.225.137.74
Controlled By:  DaemonSet/collector
Init Containers:
  config-init:
    Container ID:  containerd://ce142f501f74b920f6c9af1c9e8384b966e1e7325135c1cfbcc93ff4d468694b
    Image:         gke.gcr.io/gke-distroless/bash:gke_distroless_20240307.00_p0@sha256:45349ac3e884819cd7b4f8955d3a87d00b91ff890329e6fe0fb201d6f1a3c77f
    Image ID:      gke.gcr.io/gke-distroless/bash@sha256:45349ac3e884819cd7b4f8955d3a87d00b91ff890329e6fe0fb201d6f1a3c77f
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      touch /prometheus/config_out/config.yaml
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 19 Jun 2024 09:29:28 +0200
      Finished:     Wed, 19 Jun 2024 09:29:28 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /prometheus/config_out from config-out (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lzwfz (ro)
Containers:
  prometheus:
    Container ID:  containerd://da390ff9512a0e64b1707d373991b4c1807f8b197c8c34365f0b1847e0cd86fb
    Image:         gke.gcr.io/prometheus-engine/prometheus:v2.45.3-gmp.3-gke.0@sha256:5b5c1721ace0a0876c21431196aa3a63ec3393d829d8a8c9b75271d487ac5624
    Image ID:      gke.gcr.io/prometheus-engine/prometheus@sha256:5b5c1721ace0a0876c21431196aa3a63ec3393d829d8a8c9b75271d487ac5624
    Port:          19090/TCP
    Host Port:     0/TCP
    Args:
      --config.file=/prometheus/config_out/config.yaml
      --enable-feature=exemplar-storage
      --enable-feature=google-kubernetes-secret-provider
      --storage.tsdb.path=/prometheus/data
      --storage.tsdb.no-lockfile
      --storage.tsdb.retention.time=30m
      --storage.tsdb.wal-compression
      --storage.tsdb.min-block-duration=10m
      --storage.tsdb.max-block-duration=10m
      --web.listen-address=:19090
      --web.enable-lifecycle
      --web.route-prefix=/
      --export.user-agent-mode=gke
      --log.format=json
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 19 Jun 2024 09:45:58 +0200
      Finished:     Wed, 19 Jun 2024 09:45:58 +0200
    Ready:          False
    Restart Count:  8
    Limits:
      memory:  1G
    Requests:
      cpu:      4m
      memory:   32M
    Liveness:   http-get http://:19090/-/healthy delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:19090/-/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      GOGC:        25
      EXTRA_ARGS:  --export.label.project-id="production-379914" --export.label.location="europe-west4" --export.label.cluster="production-k8s-cluster-ce560b4"
    Mounts:
      /etc/secrets from collection-secret (ro)
      /prometheus/config_out from config-out (ro)
      /prometheus/data from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lzwfz (ro)
  config-reloader:
    Container ID:  containerd://9026bf47e2ab8d8f0862979660a19d07d7ec08eab616becbd33ea7911f44b337
    Image:         gke.gcr.io/prometheus-engine/config-reloader:v0.12.0-gke.0@sha256:6de4ee5377044e2acc4d77ed3c9534a5419ccbef703bf0835abd5bb8249e30cc
    Image ID:      gke.gcr.io/prometheus-engine/config-reloader@sha256:6de4ee5377044e2acc4d77ed3c9534a5419ccbef703bf0835abd5bb8249e30cc
    Port:          19091/TCP
    Host Port:     0/TCP
    Args:
      --config-file=/prometheus/config/config.yaml
      --config-file-output=/prometheus/config_out/config.yaml
      --reload-url=http://127.0.0.1:19090/-/reload
      --ready-url=http://127.0.0.1:19090/-/ready
      --listen-address=:19091
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 19 Jun 2024 09:45:58 +0200
      Finished:     Wed, 19 Jun 2024 09:45:58 +0200
    Ready:          False
    Restart Count:  8
    Limits:
      memory:  32M
    Requests:
      cpu:     1m
      memory:  4M
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /prometheus/config from config (ro)
      /prometheus/config_out from config-out (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lzwfz (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  storage:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      collector
    Optional:  false
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  collection-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  collection
    Optional:    false
  kube-api-access-lzwfz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 :NoExecute op=Exists
                             :NoSchedule op=Exists
                             components.gke.io/gke-managed-components op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason           Age                  From                                   Message
  ----     ------           ----                 ----                                   -------
  Normal   Scheduled        21m                  gke.io/optimize-utilization-scheduler  Successfully assigned gke-gmp-system/collector-xtmwj to gk3-production-k8s-clust-nap-1vwzuiv2-68965d5b-h4ff
  Warning  FailedMount      21m (x5 over 21m)    kubelet                                MountVolume.SetUp failed for volume "kube-api-access-lzwfz" : object "gke-gmp-system"/"kube-root-ca.crt" not registered
  Warning  NetworkNotReady  21m (x8 over 21m)    kubelet                                network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  Warning  FailedMount      21m (x6 over 21m)    kubelet                                MountVolume.SetUp failed for volume "collection-secret" : object "gke-gmp-system"/"collection" not registered
  Warning  FailedMount      21m (x6 over 21m)    kubelet                                MountVolume.SetUp failed for volume "config" : object "gke-gmp-system"/"collector" not registered
  Warning  BackOff          117s (x98 over 20m)  kubelet                                Back-off restarting failed container prometheus in pod collector-xtmwj_gke-gmp-system(efa7289f-9382-4f22-903c-78f976223b93)

The node the pod is trying to start is:

% Kubectl describe node gk3-production-k8s-clust-nap-1vwzuiv2-68965d5b-h4ff | grep Architecture -A3 -B7
System Info:
  Machine ID:                 489e4239be6112d93a8216ff7d935122
  System UUID:                489e4239-be61-12d9-3a82-16ff7d935122
  Boot ID:                    8a393a66-bb3a-46b7-bd91-66a265a57628
  Kernel Version:             6.1.75+
  OS Image:                   Container-Optimized OS from Google
  Operating System:           linux
  Architecture:               arm64
  Container Runtime Version:  containerd://1.7.13
  Kubelet Version:            v1.29.4-gke.1043002
  Kube-Proxy Version:         v1.29.4-gke.1043002

The logs from container:

√ kubernetes-production % Kubectl logs collector-xtmwj prometheus 
exec /bin/prometheus: exec format error
√ kubernetes-production % Kubectl logs collector-xtmwj config-reloader 
exec /bin/config-reloader: exec format error

Collector pods on amd64 nodes are running fine.

It looks like multi arch build of gke.gcr.io/prometheus-engine/prometheus:v2.45.3-gmp.3 may be broken.

Steps to reproduce:

Spin up latest GKE autopilot, run a pod with arm64 and check it collector pod in gke-gmp-system on this node is running fine.

Comments

ba...@google.com <ba...@google.com> Jun 19, 2024 08:08AM

Assigned to ma...@google.com.

ma...@google.com <ma...@google.com> #2Jun 19, 2024 01:15PM

Reassigned to gc...@google.com.

Hello,

Thanks for reaching out to us!

The Product Engineering Team has been made aware of your feature request, and will address it in due course. Though we can't provide an ETA on feature requests nor guarantee their implementation, rest assured that your feedback is always taken very seriously, as it allows us to improve our products. Thank you for your trust and continued support to improve Google Cloud Platform products.

In case you want to report a new issue, please do not hesitate to create a new Issue Tracker thread describing your issue. Once again, thank you for the suggestions.

Thanks & Regards,
Manish Bavireddy.
Google Cloud Support

IssueTracker

Prometheus "collector" pods running in "gke-gmp-system" are crashing on arm64 nodes

Status Update

Description

Problem you have encountered:

Steps to reproduce:

Comments

ba...@google.com <ba...@google.com> Jun 19, 2024 08:08AM

ma...@google.com <ma...@google.com> #2Jun 19, 2024 01:15PM

Issue 347990504

Description

Problem you have encountered:

Steps to reproduce:

Issue summary

Comments

ba...@google.com <ba...@google.com> Jun 19, 2024 08:08AM

ma...@google.com <ma...@google.com> #2Jun 19, 2024 01:15PM

Add comment

Issue metadata