I am trying to set up Workload Identity Federation from workloads running in kubernetes pods on AWS EKS to Google Cloud Service Accounts. Testing on EC2 everything worked first time, but moving to EKS has been troublesome.

Google Cloud SDK and CLI are ignoring the presence of the AWS_WEB_IDENTITY_TOKEN_FILE environment variable, and instead going to the AWS metadata service in order to obtain an AWS STS Token. This means the obtained AWS STS token is for the AWS IAM Role of the underlying EC2 instance, rather than that of the pod where the workload is running.

Steps to prove

The principal with roles/iam.workloadIdentityUser should normally be:

principalSet://iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/attribute.aws_role/arn:aws:sts::<aws account>:assumed-role/<pod role>

Trying to use this from a kubernetes pod gives the error:

ERROR: (gcloud.config.set) There was a problem refreshing your current auth tokens: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n "error": {\n "code": 403,\n "message": "The caller does not have permission",\n "status": "PERMISSION_DENIED"\n }\n}\n')`

If <pod role> is changed to be the underlying <ec2 role> of the EKS worker node then federation is successful and API calls are accepted. This is showing the CLI/SDK is using the incorrect AWS IAM Role when contacting Google for federation.

Workaround

Based on reading the documentation at https://google.aip.dev/auth/4117 there is no accommodation for using Web Identity Tokens in the auth flow. AWS sets the environment variables AWS_WEB_IDENTITY_TOKEN_FILE & AWS_ROLE_ARN on EKS pods which are configured with a kubernetes service account that reference an AWS IAM Role.

We can force the correct role to be used by triggering this condition (from the above documentation):

Check the environment variables ACCESS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and the optional AWS_SESSION_TOKEN for the AWS security credentials. If found, skip using the AWS metadata server to determine these values

With this code

eval $(aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name test --web-identity-token $(cat $AWS_WEB_IDENTITY_TOKEN_FILE) | jq -r '.Credentials | "export AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=\(.SessionToken)\n"')

Now the principalSet can be updated with the correct <pod role>, and everything works.

Summary

The gcloud SDK/CLI are not using the correct credentials on AWS when running in EKS pods using service accounts linked to IAM roles.

Two more related bits of background reading:

First, documentation from AWS on IAM Roles for Service Accounts (IRSA)

Second, the AWS Java SDK documentation on Class DefaultCredentialsProvider. This shows the order of credential evaluation performed by the Java SDK and is useful to see what is happening in this case:

The bug in this case is because the gcloud SDK/CLI is falling through to case 6 (instance profile from metadata service)
The SDK/CLI should be implementing case 3 and using Web Identity Tokens
The workaround triggers case 2

Issue 238911014

Links (10)

Issue metadata