When tracing with OpenTelemetry sending an X-Cloud-Trace-Context header that ends in `;o=1` is supposed to force tracing to occur. (https://cloud.google.com/trace/docs/setup#force-trace)

However, when concurrent requests are made to a Cloud Run service using the same TraceID, Cloud Run does not honor this flag.

--

I have a Cloud Run API service. When making sequential requests to it with an x-cloud-trace-context header as follows, each request is correctly traced.

for I in {1..3}; do
curl -D - -o /dev/null -H"X-Cloud-Trace-Context: 00fb1bfed62a3789a04a24d3ab8ddd14/${I};o=1" -H"Accept: */*" https://pinian-core-s6qd3oo4jq-uc.a.run.app/v1/user?userId=ff4WFv2tGQ949AEAAQ
done

In each of the three responses, the response X-Cloud-Trace-Context header correctly has `;o=1` at the end, indicating that tracing has happened. And, as expected, each span is sent to Cloud Trace correctly, and Cloud Run passes the context to the serving application with `;o=1`.

However, if multiple requests are made concurrently in a short enough window, only one of the requests will have tracing enabled.

for I in {1..3}; do
curl -D - -o /dev/null -H"X-Cloud-Trace-Context: 00fb1bfed62a3789a04a24d3ab8ddd15/${I};o=1" -H"Accept: */*" https://pinian-core-s6qd3oo4jq-uc.a.run.app/v1/user?userId=ff4WFv2tGQ949AEAAQ &
done

When run in parallel like this, one of the responses will correctly have `;o=1`. However, two of them will not. For the two that were not, no span is sent to Cloud Trace, and the Cloud Trace context passed to the serving application also does not have `;o=1`.

This breaks parent-based sampling. And even if the serving application is set up to Always Sample as a workaround, its spans are incorrectly nested below the Root Span because of the missing parent span from Cloud Run.

This makes it incredibly hard to visualize what's happening in Cloud Trace.

This problem also occurs when the requests to the service are different, e.g. requesting /v1/user?userId=XXX, then /v1/questions, then /v1/topic?topicId=YYY. It happens regardless of whether the same SpanID is used for each request.

I have also tested running many more requests concurrently. As long as the TraceID is the same and the requests happen concurrently, only one in a given window of time will have tracing enabled.

My expectation is that Cloud Run respects the `;o=1` flag, sends spans to Cloud Trace for each request sent with `;o=1`, and forwards the context to the serving application with `;o=1` as well.

Issue 325741117

Links (3)

Issue metadata