fix(trace): restore gRPC server spans and add HTTP server spans

[migmartri]

Problem

With tracing enabled, traces rooted at middleware spans (WithCurrentMembershipsMiddleware, WithCurrentAPITokenAndOrgMiddleware, ...) instead of transport-level server spans — for both gRPC and HTTP entrypoints. See #3448.

Root causes

1. gRPC: kratos' grpc.Options assigns s.grpcOpts instead of appending (kratos/v2@v2.9.2/transport/grpc/server.go:124). NewGRPCServer passed grpc.Options(grpcLib.StatsHandler(otelgrpc.NewServerHandler())) and then, when max_recv_msg_size was configured (Helm controlplane.grpcMaxRecvMsgSize), a second grpc.Options(grpcLib.MaxRecvMsgSize(...)) — silently wiping the stats handler and with it all gRPC server spans.
2. HTTP: the Kratos HTTP server had no server-side span instrumentation at all (no tracing.Server() middleware, no otelhttp), so transcoded REST requests started their trace at the first middleware span, with no traceparent extraction.

Fix

• grpc.go: pass all raw grpc-go server options in a single grpc.Options(...) call, with a comment explaining the assignment semantics so it doesn't regress.
• http.go: prepend kratos tracing.Server() as the outermost HTTP middleware. It uses the global tracer provider and extracts W3C tracecontext by default. gRPC is untouched there (already covered by the stats handler, so no double spans).

Verification

New regression tests in app/controlplane/internal/server/otel_test.go spin up real Kratos servers with an in-memory exporter:

• TestGRPCServerSpanWithMaxRecvMsgSize — gRPC server span grpc.health.v1.Health/Check is produced with MaxRecvMsgSize set. During development the split-grpc.Options variant was confirmed to produce zero spans, reproducing the bug.
• TestHTTPServerSpanFromTracingMiddleware — HTTP request produces a server span named after the route (/ping).

go build ./app/controlplane/... ./app/artifact-cas/..., go vet, and go test ./app/controlplane/internal/server/ all pass. artifact-cas checked: only one grpc.Options call, unaffected.

Fixes #3448

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

[chainloop-platform]

Security Checks — ✅ 8 passing

✅ secret-scan

Status	Policy	Messages
✅ Passed	secrets-detection	-

✅ sast-scan

Status	Policy	Messages
✅ Passed	owasp-top10-2025	-
✅ Passed	sast	-
✅ Passed	cwe-top25	-
✅ Passed	cwe-top26-40-cusp	-

PR info

Status	Policy	Messages
✅ Passed	pr-min-approvals	-
✅ Passed	pr-description-required	-
✅ Passed	pr-user-story-linked	-

⏭️ 3 scans not applied

Scan	Reason
vulnerability-scan	ran, but its output is not attested yet
github-actions-scan	no workflow files changed
iac-scan	no IaC files changed

View attestation ↗

---

Powered by Chainloop and Chainloop Trace

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[migmartri]

Addressed the review feedback in d75c189:

P2 — gRPC test now exercises NewGRPCServer: TestGRPCServerSpanWithMaxRecvMsgSize builds the server through the real NewGRPCServer constructor (minimal Opts: server config with MaxRecvMsgSize set, validator, logger, stub AuthService for the status-service registration) and asserts the grpc.health.v1.Health/Check server span is exported. A regression that splits the raw options into a second grpc.Options call makes this test fail (verified during development: split calls → zero spans).

P2 — HTTP test now exercises the production middleware list: the chain construction was extracted into httpMiddlewares(opts), which NewHTTPServer now consumes; TestHTTPServerSpanFromTracingMiddleware builds a Kratos HTTP server with that exact chain and asserts the /ping server span. Building full NewHTTPServer would require a working AuthService (OIDC config, login/callback handlers) — the extracted function covers the changed wiring without that coupling.

P3 — comment accuracy: the comment on httpMiddlewares now states that handlers registered via httpSrv.Handle (auth login/callback, Prometheus metrics) and /openapi.yaml bypass the middleware chain and still produce no server span. Instrumenting those raw handlers would need otelhttp wrapping — left out of this PR's scope.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

[migmartri]

Assessment of the review round, plus a follow-up fix in cc25bad:

• P2 (gRPC test path) — fully valid. The original test hand-wrote the combined options and asserted they work; that tests kratos, not our wiring, and the real regression mode (a second grpc.Options call in NewGRPCServer) would have slipped through. Fixed by building through the real constructor — cheaper than expected since only AuthSvc.AuthURLs.Login is dereferenced at construction, and the stats handler records the span even though the auth middleware rejects the health check.
• P2 (HTTP test path) — directionally right, impractical as stated. Full NewHTTPServer construction needs a working AuthService (OIDC config, login/callback handlers invoked at construction). Compromise: extracted httpMiddlewares(opts) (now consumed by NewHTTPServer) and test through that chain. Covers dropping/reordering tracing.Server(); the remaining gap (rewiring NewHTTPServer away from httpMiddlewares) is accepted.
• P3 (comment accuracy) — correct, and now fully fixed rather than documented. The auth login/callback and Prometheus metrics handlers registered via httpSrv.Handle bypass the middleware chain; they are now wrapped with otelhttp.NewHandler individually (go.mod: otelhttp promoted from indirect to direct dependency). The remaining untraced endpoints are /openapi.yaml and /download/{digest} — kratos Context handlers that can't be wrapped without adapters, and low-value static/redirect endpoints anyway. The comment on httpMiddlewares states this precisely.

No false positives in this round; each finding pointed at a real weakness.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri