IssueTracker

This will create a public issue which anybody can view and comment on.

Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.

Problem you have encountered: Creating a TPU job through AI Platform using a custom Service Accounts is not allowed. It is needed to use a custom SA in order to give extra permissions (customer needs fine-grained permissions and cannot give role to TPU default Service Account) to the account to access (in this case) the Secret Manager but users are unable to do so as TPU jobs do not currently accept custom Service Accounts like GPU jobs do.

What you expected to happen: The TPU job should accept using a custom SA as it can be done when it comes to GPU jobs.

Steps to reproduce: Follow the steps here to train using TPU. Follow the steps to create a custom service account Give the roles:

• AI platform service agent
• Cloud TPU V2 API Service Agent
• Secret Manager Admin (or any other role needed for the complementary access)

Add the custom SA you would like to use like:

--service-account=my-service-account@$PROJECT_ID.iam.gserviceaccount.com \

Launch the job and it will fail with the error:

ERROR: (gcloud.ai-platform.jobs.submit.training) INVALID_ARGUMENT: Field: service_account Error: Service account is currently not supported for TPU training jobs.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: Service account is currently not supported for TPU training jobs.
    field: service_account

Other information (workarounds you have tried, documentation consulted, etc): Tried giving additional roles to the TPU SA created at the beginning of [1] but did not work. Perhaps this is because AI Platform creates an additional VM and the permissions are not fully passed to that VM and thus it can’t use the permissions.

[1] https://cloud.google.com/ai-platform/training/docs/using-tpus