IssueTracker

I am attempting to specify a TTL for a Dataproc serverless PySpark Batch created via Airlfow's DataprocCreateBatchOperator using the 2.1 runtime version.

The pyspark batch config looks as follows:

{
    "pyspark_batch": {
        "main_python_file_uri": ""
    },
    "runtime_config": {
        "version": "2.1",
        "container_image": "",
        "properties": {}
    },
    "environment_config": {
        "execution_config": {
            "service_account": "",
            "ttl": "8h"
        }
    }
}

When creating the batch, the operation fails with:

Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 2105, in execute
    self.operation = hook.create_batch(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/common/hooks/base_google.py", line 462, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/hooks/dataproc.py", line 852, in create_batch
    result = client.create_batch(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/google/cloud/dataproc_v1/services/batch_controller/client.py", line 534, in create_batch
    request = batches.CreateBatchRequest(request)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/proto/message.py", line 516, in __init__
    pb_value = marshal.to_proto(pb_type, value)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/proto/marshal/marshal.py", line 211, in to_proto
    pb_value = rule.to_proto(value)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
    return self._descriptor(**value)
ValueError: Protocol message ExecutionConfig has no "ttl" field.

It looks like the ttl field may not be fully exposed for configuration in Dataproc serverless configs.

Somewhat related issue: https://github.com/kubeflow/pipelines/issues/10190