Skip to content

[Bug] TracingInterceptor never sets a span status on a failed workflow, only an exception event #1772

Description

@mattcamp

What are you really trying to do?

Telling failed workflow runs from successful ones in an OTel backend, without
having to know Temporal's span naming.

We run a workflow-orchestration platform on Temporal and export spans to
ClickHouse. The obvious way to find failures is StatusCode = 'Error'. That
returns essentially nothing, because a failed workflow's spans are exported with
status Unset.

Across our dev environment over the 7 days to 18 Aug 2026: 43,797 spans with
Unset and exactly one with Error
— and the one was an activity, not a
workflow. Real failures over that period were in the thousands.

Describe the bug

TracingInterceptor records an exception event on the span that closes a
failed workflow, but never sets that span's status. The two span-creation paths
in temporalio/contrib/opentelemetry/_interceptor.py disagree:

  • _start_as_current_span (activity and client spans) sets
    Status(StatusCode.ERROR, ...) when the wrapped call raises, and correctly
    skips it for ApplicationErrorCategory.BENIGN (added in Don't set error status on otel spans for benign exceptions #1085).
  • _completed_workflow_span — which every workflow-side span goes through,
    since a long-running span is not replay-safe so the span is started and ended
    at the same instant — calls span.record_exception(params.exception) and
    nothing else.

So CompleteWorkflow:<Type> for a workflow that failed carries an exception
event and a status of Unset, which is indistinguishable from success to
anything that reads status: backend queries, alerting rules, and the error
highlighting in most trace UIs.

The practical effect is that every consumer has to re-derive "did this fail?"
from the name of the span carrying the exception event. We ended up matching
CompleteWorkflow:%OrchestrationWorkflow in SQL, which couples our queries to
Temporal's span naming and to which of our workflow types happen to be roots. A
status code is exactly the contract that should have made that unnecessary.

Minimal Reproduction

_completed_workflow_span is what _top_level_workflow_context calls in its
finally when a workflow raises a FailureError, so calling it directly shows
the behaviour without needing a server:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind
from temporalio.contrib.opentelemetry import TracingInterceptor
from temporalio.contrib.opentelemetry._interceptor import _CompletedWorkflowSpanParams
from temporalio.exceptions import ApplicationError

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))

interceptor = TracingInterceptor(always_create_workflow_spans=True)
interceptor.tracer = provider.get_tracer(__name__)

interceptor._completed_workflow_span(
    _CompletedWorkflowSpanParams(
        context={},
        name="CompleteWorkflow:MyWorkflow",
        attributes={},
        time_ns=1,
        link_context=None,
        exception=ApplicationError("deliberate failure"),
        kind=SpanKind.INTERNAL,
        parent_missing=False,
    )
)

span = exporter.get_finished_spans()[0]
print(f"status : {span.status.status_code}")
print(f"events : {[e.name for e in span.events]}")

Output on 1.31.0:

status : StatusCode.UNSET
events : ['exception']

Expected StatusCode.ERROR.

Environment/Versions

  • OS and processor: macOS 15 / arm64 (also reproduced in Linux containers on EKS)
  • SDK version: temporalio 1.31.0, opentelemetry-sdk 1.44.0
  • Are you using Docker or Kubernetes or building Temporal from source? Temporal
    Cloud, workers on EKS

Suggested fix

Set the status in _completed_workflow_span when params.exception is present,
before span.end(), mirroring the check _start_as_current_span already makes:

 if params.exception:
     span.record_exception(params.exception)
+    if (
+        not isinstance(params.exception, ApplicationError)
+        or params.exception.category != ApplicationErrorCategory.BENIGN
+    ):
+        span.set_status(
+            Status(
+                status_code=StatusCode.ERROR,
+                description=f"{type(params.exception).__name__}: {params.exception}",
+            )
+        )
 span.end(end_time=params.time_ns)

That would leave the two paths consistent, and would keep #1041 / #1085's benign
behaviour intact on both.

Additional context

The status cannot be set by a downstream consumer after the fact, because
_completed_workflow_span calls span.end() before returning and the spec says
calls on an ended span are ignored. The only workaround we found that does not
override a private method is to pass the interceptor a wrapping Tracer whose
spans set the status themselves when record_exception is called — a modified
Tracer was floated in #1047 for the activity case, and it works here too, but
it is a lot of delegation boilerplate for something the interceptor could state
directly.

Related but not the same: #1047 is about activity spans and heartbeat-timeout
cancellations; #1041 / #1085 is about not flagging benign errors on the path
that does set a status. This is the workflow-completion path, which never sets
one at all.

Happy to open a PR with the change above and a test alongside the existing ones
in tests/contrib/test_opentelemetry.py if that is welcome.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions