You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Telling failed workflow runs from successful ones in an OTel backend, without
having to know Temporal's span naming.
We run a workflow-orchestration platform on Temporal and export spans to
ClickHouse. The obvious way to find failures is StatusCode = 'Error'. That
returns essentially nothing, because a failed workflow's spans are exported with
status Unset.
Across our dev environment over the 7 days to 18 Aug 2026: 43,797 spans with Unset and exactly one with Error — and the one was an activity, not a
workflow. Real failures over that period were in the thousands.
Describe the bug
TracingInterceptor records an exception event on the span that closes a
failed workflow, but never sets that span's status. The two span-creation paths
in temporalio/contrib/opentelemetry/_interceptor.py disagree:
_completed_workflow_span — which every workflow-side span goes through,
since a long-running span is not replay-safe so the span is started and ended
at the same instant — calls span.record_exception(params.exception) and
nothing else.
So CompleteWorkflow:<Type> for a workflow that failed carries an exception
event and a status of Unset, which is indistinguishable from success to
anything that reads status: backend queries, alerting rules, and the error
highlighting in most trace UIs.
The practical effect is that every consumer has to re-derive "did this fail?"
from the name of the span carrying the exception event. We ended up matching CompleteWorkflow:%OrchestrationWorkflow in SQL, which couples our queries to
Temporal's span naming and to which of our workflow types happen to be roots. A
status code is exactly the contract that should have made that unnecessary.
Minimal Reproduction
_completed_workflow_span is what _top_level_workflow_context calls in its finally when a workflow raises a FailureError, so calling it directly shows
the behaviour without needing a server:
Are you using Docker or Kubernetes or building Temporal from source? Temporal
Cloud, workers on EKS
Suggested fix
Set the status in _completed_workflow_span when params.exception is present,
before span.end(), mirroring the check _start_as_current_span already makes:
That would leave the two paths consistent, and would keep #1041 / #1085's benign
behaviour intact on both.
Additional context
The status cannot be set by a downstream consumer after the fact, because _completed_workflow_span calls span.end() before returning and the spec says
calls on an ended span are ignored. The only workaround we found that does not
override a private method is to pass the interceptor a wrapping Tracer whose
spans set the status themselves when record_exception is called — a modified Tracer was floated in #1047 for the activity case, and it works here too, but
it is a lot of delegation boilerplate for something the interceptor could state
directly.
Related but not the same: #1047 is about activity spans and heartbeat-timeout
cancellations; #1041 / #1085 is about not flagging benign errors on the path
that does set a status. This is the workflow-completion path, which never sets
one at all.
Happy to open a PR with the change above and a test alongside the existing ones
in tests/contrib/test_opentelemetry.py if that is welcome.
What are you really trying to do?
Telling failed workflow runs from successful ones in an OTel backend, without
having to know Temporal's span naming.
We run a workflow-orchestration platform on Temporal and export spans to
ClickHouse. The obvious way to find failures is
StatusCode = 'Error'. Thatreturns essentially nothing, because a failed workflow's spans are exported with
status
Unset.Across our dev environment over the 7 days to 18 Aug 2026: 43,797 spans with
Unsetand exactly one withError— and the one was an activity, not aworkflow. Real failures over that period were in the thousands.
Describe the bug
TracingInterceptorrecords anexceptionevent on the span that closes afailed workflow, but never sets that span's status. The two span-creation paths
in
temporalio/contrib/opentelemetry/_interceptor.pydisagree:_start_as_current_span(activity and client spans) setsStatus(StatusCode.ERROR, ...)when the wrapped call raises, and correctlyskips it for
ApplicationErrorCategory.BENIGN(added in Don't set error status on otel spans for benign exceptions #1085)._completed_workflow_span— which every workflow-side span goes through,since a long-running span is not replay-safe so the span is started and ended
at the same instant — calls
span.record_exception(params.exception)andnothing else.
So
CompleteWorkflow:<Type>for a workflow that failed carries an exceptionevent and a status of
Unset, which is indistinguishable from success toanything that reads status: backend queries, alerting rules, and the error
highlighting in most trace UIs.
The practical effect is that every consumer has to re-derive "did this fail?"
from the name of the span carrying the exception event. We ended up matching
CompleteWorkflow:%OrchestrationWorkflowin SQL, which couples our queries toTemporal's span naming and to which of our workflow types happen to be roots. A
status code is exactly the contract that should have made that unnecessary.
Minimal Reproduction
_completed_workflow_spanis what_top_level_workflow_contextcalls in itsfinallywhen a workflow raises aFailureError, so calling it directly showsthe behaviour without needing a server:
Output on 1.31.0:
Expected
StatusCode.ERROR.Environment/Versions
temporalio1.31.0,opentelemetry-sdk1.44.0Cloud, workers on EKS
Suggested fix
Set the status in
_completed_workflow_spanwhenparams.exceptionis present,before
span.end(), mirroring the check_start_as_current_spanalready makes:That would leave the two paths consistent, and would keep #1041 / #1085's benign
behaviour intact on both.
Additional context
The status cannot be set by a downstream consumer after the fact, because
_completed_workflow_spancallsspan.end()before returning and the spec sayscalls on an ended span are ignored. The only workaround we found that does not
override a private method is to pass the interceptor a wrapping
Tracerwhosespans set the status themselves when
record_exceptionis called — a modifiedTracerwas floated in #1047 for the activity case, and it works here too, butit is a lot of delegation boilerplate for something the interceptor could state
directly.
Related but not the same: #1047 is about activity spans and heartbeat-timeout
cancellations; #1041 / #1085 is about not flagging benign errors on the path
that does set a status. This is the workflow-completion path, which never sets
one at all.
Happy to open a PR with the change above and a test alongside the existing ones
in
tests/contrib/test_opentelemetry.pyif that is welcome.