Context

I have two simple PHP apps running in GKE as Kubernetes Deployments. They are APIs supporting a website, named as api-deployment and plugin-id-deployment. The api-deployment is associated with a service named api-service, which is used by the plugin-id application, as it makes calls to the api-deployment workload. The api-deployment has a HPA associated with it.

The Problem:

We noticed that, when traffic at the website starts dropping after peak lead and the HPA starts scaling down the api-deployment, a few requests from the plugin-id to the api-deployment app fail. By looking into logs we can see that GKE is sending a command to terminate some of the pods before it updates the api-service (via endpoint slices) to mark these pods as "terminating" and "not ready". As a consequence, we can see that the pods start announcing that a SIGTERM has been received while the Kubernetes service is still sending new traffic to these pods, which results in some request failures.

You can see logs attached showing the issue.

Note1: Ignore the fact that SIGWINCH is being reported instead of SIGTERM, that's because Apache2 misuses these signals.

Note2: The api-deployment has a graceful termination period of 30s, but its container terminates very quickly, it only needs a second to respond to pending requests and shutdown. Incoming requests start being rejected when the termination signal is received.

What you expected to happen:

I expected the service to mark some of the pods as "terminating" and "not ready" BEFORE they were deleted, to ensure availability. In other words, I expected the service endpoint and endpoint slices requests to be logged before the deletion of the pods and the termination signal logs.

Steps to reproduce:

You don't really need a HPA to reproduce, you can manually force a scale down which produces the same result.

1. Create 2 simple deployments and one service that's used by one to connect to the other. App A calls app B via this service. Also, create an ingress and a service for app A, so you can invoke its endpoint externally and make A call your downstream app B on each request. Make sure app B logs the errors from the calls to app A and doesn't implement any retry mechanism
2. Increase the number of replicas (e.g. to 15) on app B, wait for them to be running.
3. Start a load test command locally to send requests to app A simulating a high traffic situation. Example of command: wrk -t15 -c20 -d30s --latency --timeout 3 https://myapp.com/app-a. This command tests for 30 seconds
4. Quickly, as soon as you've started the command on the previous step, scale down your app A deployment from 15 to 3 instances.

You should see that wrk reports failing requests

Workaround attempted

I've tried adding apreStop hook an deployment B (api-deployment) to make it sleep for 5 seconds. It seems to reduce the number of errors reported/logged, but doesn't completely resolve the issue.

This page, which was used as a reference, details the termination process for pods, which aligns with what was described here.