Feature Request P2
Status Update
No update yet.
Comments
ab...@e.lloydsbanking.com
Apologies, I intended to create a feature request not a bug. I will create a feature request instead.
Description
This will create a feature request which anybody can view and comment on.
Please describe your requested enhancement. Good feature requests will solve common problems or enable new use cases.
What you would like to accomplish:
The customer is facing an unexpected issue with the Dataproc API when uplifting the Composer and Python version.
The code for cluster status check which is required before checking Dataproc cluster compliance, this is currently working as expected on composer-2.6.5-airflow-2.5.3 (python 3.8.12).
Once the composer env and python are uplifted to composer-2.9.2-airflow-2.9.1 (python 3.11.8) which uses google-cloud-dataproc package version 5.10.2, the class documentation for this version shows enum value of ‘RUNNING’, they observe that the state is no longer being returned as ‘State.RUNNING’ therefore the task is getting stuck in a while loop.
To capture any changes to cluster status class, the cluster status check was updated to and they could observe that ‘ACKNOWLEDGE 3’ is printed to DAG logs and the response.status.state is printed as ‘2’ therefore proving that the response for this API has changed and this is not reflected in class documentation.
How this might work:
Could you please update the documentation if the documentation is incorrect, or the response of '2' is a bug and need to be remediated