Status Update
Comments
de...@daangn.com <de...@daangn.com> #2
You suggested that we make temporary security credentials with using aws cli(assume-role-with-web-identity) and then impersonate gcp service account from IRSA.
But, all credentials made by aws cli(assume-role-with-web-identity) have Session Duration(Expiration time).
So, after the expiration time, we can't use the credentials anymore.
How do you deal with the credentials' duration time when you try to continously use gcp api from pod with irsa?
Thank you.
ta...@ujet.cx <ta...@ujet.cx> #3
We would like to use WIF from EKS workloads but not really possible due to the above limitations.
For the AWS STS token expiration when using the above workaround, I think the easiest would be to asynchronously update the env variables in the background with a cron or based on the expiration field coming from the sts service.
as...@bt.com <as...@bt.com> #4
as...@bt.com <as...@bt.com> #5
were you able to refresh the tokens periodically ? Could you help me how you did it?
ma...@google.com <ma...@google.com>
ma...@google.com <ma...@google.com> #6
Hello,
Thanks for reaching out to us!
The Feature request has been created and the Product Engineering Team is working on this request. Right now, there is no ETA to this request. Thank you for your trust and continued support to improve Google Cloud Platform products.
In case you want to report a new issue, please do not hesitate to create a new
Thanks & Regards,
Mahaboob
[Deleted User] <[Deleted User]> #7
This workaround did not work for me because the command above does not expose the session expiry. Thus the application cannot use the session. I've exposed the session expiry from the token and can confirm it works.
eval $(aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name test --web-identity-token $(cat $AWS_WEB_IDENTITY_TOKEN_FILE) | jq -r '.Credentials | "export AWS_ACCESS_KEY_ID=(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=(.SessionToken)\nexport AWS_SESSION_EXPIRATION=(.Expiration)\n"')
as...@bt.com <as...@bt.com> #8
Can you share full example please? I tried with Cronjob and normal linux cron. Both attempt were unsucessful.
1. Cronjob runs in a new pod where I cannot access the AWS_WEB_IDENTITY_TOKEN_FILE which is present in main pod.
2. If I run a linux cron in the main pod, I still cannot access the env variables, as the cron runs in a new shell, where normal env variables are not accessible.
I could think of updating a volume inside a pod, instead of env variable. I think that would work. But could not get it to work.
as...@bt.com <as...@bt.com> #9
Any updates on the fix ?
am...@sentra.io <am...@sentra.io> #10
ma...@google.com <ma...@google.com> #11
Hello,
The Product Engineering team is aware and working on this request. Currently we cannot provide any ETA. Please follow this thread for updates.
Thanks and Regards,
Mahaboob Subhani
ma...@google.com <ma...@google.com>
ma...@google.com <ma...@google.com>
dg...@google.com <dg...@google.com> #12
th...@lookout.com <th...@lookout.com> #13
ja...@booking.com <ja...@booking.com> #14
pb...@paloaltonetworks.com <pb...@paloaltonetworks.com> #15
le...@google.com <le...@google.com> #16
An option is to use the OIDC token in AWS_WEB_IDENTITY_TOKEN_FILE when using EKS. You need to set up a OIDC provider in your pool and use file-sourced credentials.
We are currently working on an SDK feature to make providing AWS creds to the library easier.
pb...@paloaltonetworks.com <pb...@paloaltonetworks.com> #17
as...@bt.com <as...@bt.com> #18
@Google team,
You have pointed back to the initial workaround mentioned by OP. What progress Google is making on this?
The option you mention to use OIDC token in AWS_WEB_IDENTITY_TOKEN_FILE is the one described in the workaround in the OP's post. i.e. You run this in the pod
eval $(aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name test --web-identity-token $(cat $AWS_WEB_IDENTITY_TOKEN_FILE) | jq -r '.Credentials | "export AWS_ACCESS_KEY_ID=(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=(.SessionToken)\nexport AWS_SESSION_EXPIRATION=(.Expiration)\n"')
When you run this command, you will have temporary AWS_ACCESS_KEY_ID
& AWS_SECRET_ACCESS_KEY
exported in the pod and then the Google client sdk would recognize this , and things would work as they should using the pod role instead of worker node role.
The problem is these temporary sessions expire, so how do we rotate/refresh the session effectively.
Any idea when will be the SDK feature to make providing AWS creds to the library easier would be available, any timelines ?
le...@google.com <le...@google.com> #19
I am suggesting using the OIDC token directly, skipping the assume role call. To do that you would have to set up an OIDC provider in your pool and use file-sourced credential config that points to AWS_WEB_IDENTITY_TOKEN_FILE.
le...@google.com <le...@google.com> #20
Can you folks please let us know which languages you'd like supported first? Thanks in advance.
as...@bt.com <as...@bt.com> #21
So, you mean point the AWS_WEB_IDENTITY_TOKEN_FILE in credential config file. would that work?
for e.g.
....
"credential_source": {
"file": "/AWS_WEB_IDENTITY_TOKEN_FILE"
}
...
I guess at first Python and Java would be mostly used.
ed...@starlingbank.com <ed...@starlingbank.com> #22
ma...@starlingbank.com <ma...@starlingbank.com> #23
So, you mean point the AWS_WEB_IDENTITY_TOKEN_FILE in credential config file
You can read the AWS_WEB_IDENTITY_TOKEN_FILE directly, but it is rotated very regularly so your application would. need to take that into consideration. This is what the Google SDK should be doing.
Can you folks please let us know which languages you'd like supported first?
Java for us, please.
ja...@booking.com <ja...@booking.com> #24
dg...@google.com <dg...@google.com> #25
+1 for Java here - Customer still actively wanting this.
de...@daangn.com <de...@daangn.com> #26
jp...@google.com <jp...@google.com> #27
Unless you're dealing with a large number of EKS clusters, it might be an option to federate your EKS cluster(s) with workload identity federation directly. We recently published a page that describes how to do this, see
ma...@starlingbank.com <ma...@starlingbank.com> #28
Is this a new feature of workload identity federation? Just trying to determine if this is a cryptic way of saying the ticket is resolved. At first glance this would remove the need for the workaround we depend on currently.
as...@bt.com <as...@bt.com> #29
Thanks Google team, Essentially, we will establish trust with GCP that says: "trusts this openid connect (OIDC) token issued to a k8s service account by a given K8s API server and map it to this GCP principal:// or principalSet://.
While this would work for EKS , as the OIDC discovery endpoint is public (
But not for bare kubernetes on-prem which might be private by design. We will have make the OIDC metadata for your kubernetes cluster visible to GCP externally.
I say this, since the new post you have created titled "Kubernetes" in general. Either the title need to be EKS specific Or the caveat needs to be added for bare k8s scenario.
eu...@gmail.com <eu...@gmail.com> #30
When I do `Create a credential configuration file` and then try to use it in `gcloud auth login --cred-file ./credential-configuration.json` followed by `gcloud auth print-access-token` I get `'Error code invalid_request: Invalid subject_token_type for specified provider. Allowed values: urn:ietf:params:aws:token-type:aws4_request'`.
If I patch a working (working with `imdsv2` endpoint, but not working with `AWS_WEB_IDENTITY_TOKEN_FILE`) `credential-configuration.json` that cointains `urn:ietf:params:aws:token-type:aws4_request`, replacing it's `credential_source` with the `"file": "/var/run/service-account/token"`, I get `ERROR: (gcloud.auth.login) The provided external account credentials are invalid or unsupported`.
It seems that to this day the only option to authenticate from EKS to GCP is to use `imdsv2` endpoint, or to use a workaround of generating AWS credential env vars from the `AWS_WEB_IDENTITY_TOKEN_FILE` token. Am I missing anything?
th...@lookout.com <th...@lookout.com> #31
da...@goodrx.com <da...@goodrx.com> #32
ei...@google.com <ei...@google.com> #33
In addition to the approach Johannes mentioned, we have also added support recently for defining custom logic to supply the 3rd party token or AWS security credentials for workload identity federation. This feature can be used to implement logic that returns the AWS security credentials from environment variables or any other method to the google credential to exchange for a GCP access token.
- Java:
https://github.com/googleapis/google-auth-library-java?tab=readme-ov-file#using-a-custom-supplier-with-aws - Go:
https://pkg.go.dev/golang.org/x/oauth2@v0.20.0/google/externalaccount#AwsSecurityCredentialsSupplier - Python:
https://googleapis.dev/python/google-auth/latest/user-guide.html#accessing-resources-using-a-custom-supplier-with-aws - Nodejs:
https://github.com/googleapis/google-auth-library-nodejs?tab=readme-ov-file#accessing-resources-from-aws-using-a-custom-aws-security-credentials-supplier
ch...@foresite.com <ch...@foresite.com> #34
sh...@thomsonreuters.com <sh...@thomsonreuters.com> #35
an...@splunk.com <an...@splunk.com> #36
The current solution using a custom provider works well and is rather easy to implement by letting AWS SDK handle the web identity credentials retrieval.
However, I believe this case is common enough that this should be automatically handled by the Google OAuth library alone - it could check for AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE variables and if they are set, send the request to AWS to assume role with web identity. It would be also great if the library would handle caching those credentials.
Description
I am trying to set up Workload Identity Federation from workloads running in kubernetes pods on AWS EKS to Google Cloud Service Accounts. Testing on EC2 everything worked first time, but moving to EKS has been troublesome.
Google Cloud SDK and CLI are ignoring the presence of the
AWS_WEB_IDENTITY_TOKEN_FILE
environment variable, and instead going to the AWS metadata service in order to obtain an AWS STS Token. This means the obtained AWS STS token is for the AWS IAM Role of the underlying EC2 instance, rather than that of the pod where the workload is running.Steps to prove
The principal with
roles/iam.workloadIdentityUser
should normally be:principalSet://iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/attribute.aws_role/arn:aws:sts::<aws account>:assumed-role/<pod role>
Trying to use this from a kubernetes pod gives the error:
If
<pod role>
is changed to be the underlying<ec2 role>
of the EKS worker node then federation is successful and API calls are accepted. This is showing the CLI/SDK is using the incorrect AWS IAM Role when contacting Google for federation.Workaround
Based on reading the documentation athttps://google.aip.dev/auth/4117 there is no accommodation for using Web Identity Tokens in the auth flow. AWS sets the environment variables
AWS_WEB_IDENTITY_TOKEN_FILE
&AWS_ROLE_ARN
on EKS pods which are configured with a kubernetes service account that reference an AWS IAM Role.We can force the correct role to be used by triggering this condition (from the above documentation):
With this code
eval $(aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name test --web-identity-token $(cat $AWS_WEB_IDENTITY_TOKEN_FILE) | jq -r '.Credentials | "export AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=\(.SessionToken)\n"')
Now the
principalSet
can be updated with the correct<pod role>
, and everything works.Summary
The gcloud SDK/CLI are not using the correct credentials on AWS when running in EKS pods using service accounts linked to IAM roles.
Two more related bits of background reading:
First, documentation from AWS on IAM Roles for Service Accounts (IRSA)
Second, the AWS Java SDK documentation on Class DefaultCredentialsProvider . This shows the order of credential evaluation performed by the Java SDK and is useful to see what is happening in this case: