IssueTracker

Problem you have encountered:

I had a python flex template pipeline that was failing to start the worker using my custom SDK image, and finding the error in the logs was made much harder by the fact that everything was logged at "default" log level, even log.fatal() calls in beam's boot.go.

What you expected to happen:

Fatal logs should be logged at error severity, etc.

Steps to reproduce:

Modify a worker SDK image's entry point to eat up the argv, e.g. by wrapping in a call to bash: ENTRYPOINT["/bin/bash", "-c", "/opt/apache/beam/boot"]

The worker will then not get the "id" argument passed and will log fatal on startup (https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L108)

Go to logs explorer and run query:

resource.type="dataflow_step"
log_name="projects/delfina-gaia-dev/logs/dataflow.googleapis.com%2Fworker-startup"

And observe lines like "2025/01/28 22:14:30 No id provided." at default severity.

JSON Payload omitting our project-specific info:

{
insertId: "7709925706309102903:406986:0:76"
jsonPayload: {
message: "2025/01/28 22:14:30 No id provided."
stream: "stderr"
}
labels: {
compute.googleapis.com/resource_id: "7709925706309102903"
compute.googleapis.com/resource_name: "beamapp-root-0128220107-6-01281401-8l7w-harness-vt4r"
compute.googleapis.com/resource_type: "instance"
dataflow.googleapis.com/job_id: "2025-01-28_14_01_09-13363111400804503033"
dataflow.googleapis.com/job_name: "beamapp-root-0128220107-632595-h8eybl06"
dataflow.googleapis.com/log_type: "system"
dataflow.googleapis.com/region: "us-central1"
}
logName: "projects/delfina-gaia-dev/logs/dataflow.googleapis.com%2Fworker-startup"
receiveTimestamp: "2025-01-28T22:15:05.771228174Z"
resource: {
labels: {5}
type: "dataflow_step"
}
timestamp: "2025-01-28T22:14:30.954498455Z"
}

I believe this is also the case for log.Warning's, e.g. the following lines in logs explorer at info severity:

{
insertId: "6649504384075416827:423582:0:11645"
jsonPayload: {
line: "exec.go:66"
message: "WARNING:root:This input type hint will be ignored and not used for type-checking purposes. Typically, input type hints for a PTransform are single (or nested) types wrapped by a PCollection, or PBegin. Got: Tuple[apache_beam.pvalue.PCollection[int], apache_beam.pvalue.PCollection[dict[str, typing.Any]]] instead."
}
labels: {6}
logName: "projects/delfina-gaia-dev/logs/dataflow.googleapis.com%2Flauncher"
receiveTimestamp: "2025-01-29T18:58:57.587560865Z"
resource: {2}
severity: "INFO"
timestamp: "2025-01-29T18:58:51.183763Z"
}

Which ultimately gets logged at https://github.com/apache/beam/blob/3d4615e2fe916d5cb92763736a97b1573d36f2ba/sdks/python/apache_beam/typehints/decorators.py#L375

(from https://github.com/apache/beam/blob/3d4615e2fe916d5cb92763736a97b1573d36f2ba/sdks/python/apache_beam/typehints/decorators.py#L329)