This will create a public issue which anybody can view and comment on.

Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.

Problem you have encountered

If a job has an execution execution that has failed previously, updating its definition returns the error:

google.api_core.exceptions.InternalServerError: 500 Task <job_id>-<execution_id>-<task_id> failed with message: The container exited with an error. 13: Task job_id>-<execution_id>-<task_id> failed with message: The container exited with an error.

The job definition is successfully updated, but the call to JobsClient.update_job fails.

What you expected to happen

The job definition is successfully updated and no error occurs.

Steps to reproduce

Here's essentially the program we're running:

jobs_client = run_v2.JobsClient()

# Check if the job already exists
request = run_v2.GetJobRequest(
    name=GCP_JOB_IDENTIFIER,
)
jobs_client.get_job(request=request)

# If the job already exists, update it
job.name = GCP_JOB_IDENTIFIER
update_request = run_v2.UpdateJobRequest(
    job=job,
)
update_job_operation = jobs_client.update_job(request=update_request)
update_job_response: Job = update_job_operation.result(timeout=200) # << this is where the error occurs

Other information (workarounds you have tried, documentation consulted, etc)

Python version 3.9.15

GCloud SDK versions:

• google-cloud-appengine-logging==1.6.0
• google-cloud-audit-log==0.3.0
• google-cloud-core==2.1.0
• google-cloud-logging==3.11.4
• google-cloud-pubsub==2.13.11
• google-cloud-run==0.10.14
• google-cloud-secret-manager==2.10.0
• google-cloud-storage==1.42.3

Slightly fuller stack trace

... <our code> ...
File "/opt/conda/lib/python3.9/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.InternalServerError: 500 Task <job_id>-<execution_id>-<task_id> failed with message: The container exited with an error. 13: Task job_id>-<execution_id>-<task_id> failed with message: The container exited with an error.

Workaround

try:
    ... above ...
except InternalServerError as e:
    if "The container exited with an error" in e.message:
        logging.info(
            f"Ignoring this error since execution failures are irrelevant to Cloud Run Job updates: {e.message}"
        )
    else:
        raise