The Dataproc API returns different formats for the imageUri field, depending on how the user supplied it. If the caller supplies a full resource name for the image, imageUri will contain the full resource name. But if it the caller supplies a relative resource name, imageUri returns the relative resource name.

Stock images always seem to return the full resource name. The API should always normalize imageUri so apps can depend on the format being a full resource name.

Steps to reproduce:

1) Create two clusters via the API, one with a full name, the other with a relative name for the custom image. (I'm using python with the discovery API here):

>>> from googleapiclienthelpers.discovery import build_subresource
>>> dataproc = build_subresource('dataproc.projects.regions.clusters', 'v1beta2')
>>> dataproc.create(projectId='cleardata-lab-krv', region='us-central1', body={'clusterName': 'api-relative-name', 'config': {'masterConfig': {'imageUri': 'projects/cleardata-images/global/images/cleardataproc-debian9-1-4-10-20190804'}}}).execute()
{'name': 'projects/cleardata-lab-krv/regions/us-central1/operations/31d2b055-c099-4c58-9126-93dd05562df0',
'metadata': {'@type': 'type.googleapis.com/google.cloud.dataproc.v1beta2.ClusterOperationMetadata',
'clusterName': 'api-relative-name',
'clusterUuid': 'b0e9ebf2-e385-41a3-acef-2b5b5ff16b12',
'status': {'state': 'PENDING',
'innerState': 'PENDING',
'stateStartTime': '2019-08-20T21:55:57.801Z'},
'operationType': 'CREATE',
'description': 'Create cluster with 2 workers',
'warnings': ['For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance.']}}
>>> dataproc.create(projectId='cleardata-lab-krv', region='us-central1', body={'clusterName': 'api-full-name', 'config': {'masterConfig': {'imageUri': 'https://www.googleapis.com/compute/v1/projects/cleardata-images/global/images/cleardataproc-debian9-1-4-10-20190804'}}}).execute()
{'name': 'projects/cleardata-lab-krv/regions/us-central1/operations/95890614-cf09-4d03-ab3c-55a6fea36a1c',
'metadata': {'@type': 'type.googleapis.com/google.cloud.dataproc.v1beta2.ClusterOperationMetadata',
'clusterName': 'api-full-name',
'clusterUuid': '7f0a6cb3-87fd-4a1c-a39b-b8af4b060dcf',
'status': {'state': 'PENDING',
'innerState': 'PENDING',
'stateStartTime': '2019-08-20T21:56:23.205Z'},
'operationType': 'CREATE',
'description': 'Create cluster with 2 workers',
'warnings': ['For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance.']}}

2) wait for all clusters to complete creating.

3) Describe them and note the different formats for imageUri:
$ gcloud dataproc clusters describe api-relative-name --region=us-central1 | grep imageUri
imageUri: projects/cleardata-images/global/images/cleardataproc-debian9-1-4-10-20190804
imageUri: projects/cleardata-images/global/images/cleardataproc-debian9-1-4-10-20190804
$ gcloud dataproc clusters describe api-full-name --region=us-central1 | grep imageUri
imageUri: https://www.googleapis.com/compute/v1/projects/cleardata-images/global/images/cleardataproc-debian9-1-4-10-20190804
imageUri: https://www.googleapis.com/compute/v1/projects/cleardata-images/global/images/cleardataproc-debian9-1-4-10-20190804

Thanks,
Ross

Issue 139762442

Links (7)

Issue metadata