fix(controlplane): restore OTel server spans for gRPC and HTTP

[javirln]

Closes #3448

With OpenTelemetry tracing enabled, control plane traces rooted at application middleware spans instead of at a transport-level server span, for both entrypoints.

gRPC: the OTel stats handler was silently dropped whenever server.grpc.max_recv_msg_size was configured. Kratos' grpc.Options assigns its option slice rather than appending to it, so the second call overwrote the first. Both raw options now travel in a single grpc.Options call. This also covers grpc-web, which re-dispatches through grpc.Server.ServeHTTP.

HTTP: the transport had no instrumentation at all, so transcoded requests produced no server span and no traceparent extraction. Kratos' tracing.Server() middleware now heads the HTTP-only middleware chain. It is deliberately not added to the shared chain, which would duplicate the gRPC stats-handler span.

The tracer provider is now passed explicitly through the server Opts instead of being picked up from the OTel global.

AI disclosure: this change was produced with the assistance of Claude Code.

[chainloop-platform]

AI Session Checks — 🟢 93% · ✅ 0 failing

Avg score	Sessions	Failing policies	Attribution	Files	Lines	Total Duration
🟢 93%	1	✅ 0	100% AI / 0% Human	3	+236 / -10	9m51s

🟢 93% — 100% AI — ✅ All policies passing

Sep 17, 2026 10:36 UTC · 9m51s · $9.80 · 188 in / 78.9k out · claude-code 2.1.274 (claude-opus-5)

View session details ↗

Change Summary

• Adds TracerProvider plumb-through and combines gRPC raw options so tracing and max_recv_msg_size coexist.
• Adds HTTP transport tracing ahead of application middleware so server spans root correctly.
• Adds regression tests for gRPC and HTTP tracing, then regenerates wire_gen.go for the new dependency.

AI Session Overall Score

🟢 93% — Detailed ask, root-cause fix, and strong regression testing make this a clean session.

AI Session Analysis Breakdown

🟢 95% · solution-quality

🟢 The fix addresses tracing at the transport layer instead of masking symptoms. · High Impact

🟢 95% · verification

🟢 The AI proved the regression tests by restoring the buggy behavior first. · High Impact

🟢 93% · alignment

No notes.

🟢 90% · scope-discipline

No notes.

🟢 88% · user-trust-signal

No notes.

🟢 86% · context-and-planning

🟢 The ticket gave concrete root causes, expected behavior, and a reference implementation. · High Impact

---

File Attribution

████████████████████ 100% AI / 0% Human

Status	Attribution	File	Lines
created	ai	app/controlplane/internal/server/tracing_test.go	+184 / -0
modified	ai	app/controlplane/internal/server/http.go	+29 / -5
modified	ai	app/controlplane/internal/server/grpc.go	+23 / -5

---

Policies (4)

Status	Policy	Material	Messages
✅ Passed	ai-config-ai-agents-allowed	ai-coding-session-250d02	-
✅ Passed	ai-config-no-dangerous-commands	ai-coding-session-250d02	-
✅ Passed	ai-config-no-secrets	ai-coding-session-250d02	-
✅ Passed	ai-config-mcp-servers-allowed	ai-coding-session-250d02	-

---

Security Checks — ⚠️ 1 failing

✅ secret-scan

Status	Policy	Messages
✅ Passed	secrets-detection	-

✅ sast-scan

Status	Policy	Messages
✅ Passed	owasp-top10-2025	-
✅ Passed	sast	-
✅ Passed	cwe-top25	-
✅ Passed	cwe-top26-40-cusp	-

PR info

Status	Policy	Messages
✅ Passed	pr-user-story-linked	-
⚠️ Failed	pr-min-approvals	PR/MR #3449 has 0 approving reviews, 1 required.
✅ Passed	pr-description-required	-

⏭️ 3 scans not applied

Scan	Reason
vulnerability-scan	no manifest/lockfile changed
github-actions-scan	no workflow files changed
iac-scan	no IaC files changed

View attestation ↗

---

Powered by Chainloop and Chainloop Trace

[cubic-dev-ai]

2 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="app/controlplane/internal/server/http.go">

<violation number="1" location="app/controlplane/internal/server/http.go:46">
P2: Every request enters `wrappedServer` with this tracing chain, and normal requests then enter `httpSrv` with the same chain again. This creates duplicate HTTP server spans and adds an unwanted HTTP span before the gRPC stats-handler span for grpc-web; give the wrapper a separate middleware option set without transport tracing.</violation>
</file>

<file name="app/controlplane/internal/server/tracing_test.go">

<violation number="1" location="app/controlplane/internal/server/tracing_test.go:121">
P2: In TestRawGRPCServerOptions, the span-end race makes the strict recorder assertion intermittently flaky. grpc-go's stats handler ends the otelgrpc server span in stream.finish after the response bytes are handed to the transport; the client-side Statusz call can therefore return before the server has run HandleRPC(stats.End). Recording Ended() immediately can intermittently yield 0 (or the wrong count) spans and fail require.Len(t, spans, 1). Poll with require.Eventually until exactly one span is recorded before asserting on it. (The HTTP tests are not affected: the kratos tracing middleware ends its span before ctx.Result writes the response.)</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Every request enters wrappedServer with this tracing chain, and normal requests then enter httpSrv with the same chain again. This creates duplicate HTTP server spans and adds an unwanted HTTP span before the gRPC stats-handler span for grpc-web; give the wrapper a separate middleware option set without transport tracing.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At app/controlplane/internal/server/http.go, line 46:

<comment>Every request enters `wrappedServer` with this tracing chain, and normal requests then enter `httpSrv` with the same chain again. This creates duplicate HTTP server spans and adds an unwanted HTTP span before the gRPC stats-handler span for grpc-web; give the wrapper a separate middleware option set without transport tracing.</comment>

<file context>
@@ -40,12 +42,8 @@ import (
-
 	var serverOpts = []http.ServerOption{
-		http.Middleware(middlewares...),
+		http.Middleware(httpServerMiddlewares(opts)...),
 	}
 	// Prevent unmatched routes from falling through to http.DefaultServeMux,
</file context>

[cubic-dev-ai]

P2: In TestRawGRPCServerOptions, the span-end race makes the strict recorder assertion intermittently flaky. grpc-go's stats handler ends the otelgrpc server span in stream.finish after the response bytes are handed to the transport; the client-side Statusz call can therefore return before the server has run HandleRPC(stats.End). Recording Ended() immediately can intermittently yield 0 (or the wrong count) spans and fail require.Len(t, spans, 1). Poll with require.Eventually until exactly one span is recorded before asserting on it. (The HTTP tests are not affected: the kratos tracing middleware ends its span before ctx.Result writes the response.)

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At app/controlplane/internal/server/tracing_test.go, line 121:

<comment>In TestRawGRPCServerOptions, the span-end race makes the strict recorder assertion intermittently flaky. grpc-go's stats handler ends the otelgrpc server span in stream.finish after the response bytes are handed to the transport; the client-side Statusz call can therefore return before the server has run HandleRPC(stats.End). Recording Ended() immediately can intermittently yield 0 (or the wrong count) spans and fail require.Len(t, spans, 1). Poll with require.Eventually until exactly one span is recorded before asserting on it. (The HTTP tests are not affected: the kratos tracing middleware ends its span before ctx.Result writes the response.)</comment>

<file context>
@@ -0,0 +1,184 @@
+
+			// Regardless of the outcome the transport must have opened a server span.
+			spans := recorder.Ended()
+			require.Len(t, spans, 1)
+			assert.Equal(t, statuszOperation, spans[0].Name())
+			assert.Equal(t, trace.SpanKindServer, spans[0].SpanKind())
</file context>