IssueTracker

What you would like to accomplish:

The Dataflow pipelines might be using deprecated Beam versions. An API method that can verify if a version is deprecated or not can be helpful in this case.

>How this might work:

The method can be included in the pipeline code, and if it is validated, the pipeline can move forward. Otherwise not.

>If applicable, reasons why alternative solutions are not sufficient:

Besides consulting the documentation, there are no alternative options. Consulting the documentation of course is a good approach, but calling the API can be a faster way to achieve this goal.

>Other information (workarounds you have tried, documentation consulted, etc):

We know the supported versions can be checked as follows:
- Supported Beam SDK versions: https://cloud.google.com/dataflow/docs/support/sdk-version-support-status
- Runtime support: https://cloud.google.com/dataflow/docs/support/beam-runtime-support
- Information about each release: https://github.com/apache/beam/releases

According to the resource [1], the Google provided templates are updated to use the latest available Beam version. However, this is not working using custom code or if some of the deployments will still require a version that is still supported, but not the latest version.

Internally, on the GCP side, I also found that any pipeline with Beam versions >=2.45.0 returns an alert regarding potential data loss using a specific version. I think a similar warning can be implemented for deprecated versions as well, but including a method in the code can be more helpful.

[1]: https://stackoverflow.com/questions/72114980/how-to-update-sdk-version-for-dataflow-job