IssueTracker

## FEATURE REQUEST

**Business Impact:**
Cannot trace all the logs for the jobs in a specific Vertex pipeline.
 
**What you would like to accomplish:**
We would like to query all the logs for the jobs in a specific Vertex pipeline. Currently, this isn’t possible as the logs for each job are related through the job ID, which changes every time the job runs.

As seen in the structure of the logs for the jobs [1] there isn’t anything that could represent the pipeline where they belong.

[1]
```
{
  "textPayload": "Job is preparing.",
  "insertId": "XXXXXXXXXXX",
  "resource": {
    "type": "ml_job",
    "labels": {
      "job_id": "XXXXX",
      "project_id": "PROJECT_ID",
      "task_name": "service"
    }
  },
  "timestamp": "2023-03-01T09:00:14.653381445Z",
  "severity": "INFO",
  "labels": {
    "ml.googleapis.com/job_state": "PREPARING",
    "ml.googleapis.com/endpoint": ""
  },
  "logName": "projects/PROJECT_ID/logs/ml.googleapis.com%2FXXXXXX",
  "receiveTimestamp": "2023-03-01T09:00:15.558850085Z"
}

```

**How this might work:**
It would be good to have a field in those logs referring to the pipeline, or to have something like custom tags or labels in them, so they could be filtered based on it. The main idea would be to be able to relate the logs for the jobs with the pipeline where they belong.

For instance, we could add the following field in the log entry that was mentioned before:

```
“pipeline_name”: “name-test-123”
```

**Other information (workarounds you have tried, documentation consulted, etc):**
Currently we can only use this query [1-2] to see the logs for all the pipelines and their jobs. But at the end, the main objective is to look for one specific pipeline job, not all of them.

[1]
```
logName="projects/PROJECT_ID/logs/aiplatform.googleapis.com%2Fpipeline_job_events" OR
logName=~"^projects/PROJECT_ID/logs/ml.googleapis.com"
```

[2] https://cloud.google.com/vertex-ai/docs/pipelines/logging