IssueTracker

Product team is working on this issue, there is no ETA yet.

My client is reporting he is still experiencing this issue (GKE 1.18.17-gke.1901)

Update:

We've confirmed that the PR which fixed an issue in kubelet, didn't fix this particular NodeAffitnity issue in GKE. We were only able to reproduce the issue on preemptible VMs. We've also only had reports of this issue on preemptible VMs. After multiple reproductions we've identified the NodeAffinity issue is mainly due to a non graceful termination of pods when a preemption occurs. We've also tested that the kubelet graceful node termination feature fixes the issue as pods are now gracefully terminated.

Hence to fix the, the cluster's NODE POOLS must be on 1.20+ versions to have the feature enabled. Another workaround is to simply not use preemptible VMs for node pools on 1.19 and lower.

Note that this issue has little to no impact on workloads. As long as the pod is backed by controller (deployment/statefulset, etc), when a pod runs into the NodeAffinity issue a new pod is immediately created and rescheduled. However, it creates a lot false positives as the pod will stay in a failed state and will make its controller page in the Cloud Console UI seem to be in a failed state as well even if all desired replicas are healthy and running.

Graceful node termination feature is also very noisy in the same way as the Shutdown failed status of pods bubbles up to the controllers page in Cloud Console UI as well. Our UI team is working on enhancing the UX for this feature. In the meantime, you can clean up pods as mentioned in the doc or ignore the false positives and wait on garbage collection to clean up the pods.

I think graceful node termination feature is now broken I think. See the PR trying to fix the issue, but it got concerns. https://github.com/kubernetes/kubernetes/pull/104798#issuecomment-924066206

So I think NodeAffinity issue might still exist.

tfmenard@ do you have some details about why NodeAffinity issue due to a non graceful termination of pods when a preemption occurs? Thanks!

After investigation it was determined that comment#5 is still valid, graceful node shutdown fixes the issue as pods are gracefully terminated. If you are still running into this issue please verify that the feature is enabled on your cluster version, if it is and you are still seeing this issue please open a support case.

A node pool version is 1.22.8-gke.202 and seeing Pod errors: NodeAffinity still. ( Pod was terminated in response to imminent node shutdown. )

Can we confirm if the issue was happened again?

Nodepool version 1.23.5-gke.1503 is also facing the same issue. The customer sees a "Pod Predicate NodeAffinity failed" error in one of the pods. Cx workload is not affected as a new pod with the same image managed to get started on the same node. The problematic pod is backed-up by ReplicaSet.

Seems to suffer from the same on 1.22.12-gke.2300 (bothmaster & nodes).

Is this issue being actively worked on by any team? Looks like there is a timing issue (race), where node label is not yet updated when pods are being scheduled?

/cc @msau @liggitt

IIRC, this was reported in https://github.com/kubernetes/kubernetes/issues/92067 and resolved in https://github.com/kubernetes/kubernetes/pull/94087 in 1.21

That PR ensured nodes have a current view of their API labels before they take action on API sourced pods.

https://github.com/kubernetes/kubernetes/pull/99336 improved the latency of that fix in 1.22 but kept the same guard in place. Kubelets do not handle pods sourced from the API until they have an up-to-date view of their labels from the API.

It doesn't look like this issue is on any team's radar. Is there a reproducer?

/cc @acondor for scheduler visibility /cc @porterdavid for node visibility

Along with any reproducer, it would help to have a trace of the following:

1. Output from a watch of the Node object (to show any label changes), e.g. kubectl get node/$name -o json --output-watch-events > nodes.json
2. Output from a watch of the Pod objects (to show any pod creations / schedulings), e.g. kubectl get pods -o json --output-watch-events -n $pod-namespace > pods.json

Adding another repro sample, we consistently encountered this on 1.24.11-gke.1000. We can post the relevant outputs if there is an appropriate thread to share.

Providing a consistent repro would be very useful! Thank you!