Serve gRPC and HTTP from one endpoint per service #3628

Today the control plane and the Artifact CAS each expose two endpoints: a gRPC server and an HTTP server, on separate ports (control plane 9000/8000, CAS 9001/8001). Every installation therefore needs two Services, two ingresses (or Gateway routes), two hostnames, and two sets of TLS settings per service. Clients and docs must also know which endpoint to use for what.

Goal: serve both protocols from one endpoint per service, so a deployment has one port and one hostname per service.

Current status and selected approach

The original spec was merged in #3629. Implementation PR #3647 is draft and now covers the control plane and its Helm ingress. CAS remains separate work under this task.

For the control plane, the HTTP server owns the shared port and handles HTTP/1.1, HTTP/2, h2c, and optional TLS. It dispatches each request, including each HTTP/2 stream:

• Native gRPC over HTTP/2 → the existing grpc-go server through ServeHTTP.
• gRPC-Web and its CORS preflight → the existing gRPC-Web wrapper.
• Ordinary HTTP → the existing HTTP routes.

This removes both the custom connection classifier and cmux from the current implementation. Existing RPC services and interceptors remain in use. It does not migrate to Connect or add the Connect protocol.

flowchart TD
    clients["HTTP, gRPC-Web, and native gRPC clients"]
    ingress["Optional shared ingress hostname<br/>One root path, HTTP/2 upstream"]
    shared["Shared HTTP endpoint<br/>Dispatch every request / HTTP/2 stream"]
    grpc["grpc-go ServeHTTP"]
    web["Existing gRPC-Web wrapper"]
    http["Existing HTTP routes"]
    rpc["Existing RPC services and interceptors"]
    legacy["Legacy native gRPC endpoint<br/>Enabled by default"]
    clients --> ingress --> shared
    clients -->|"Direct VM / Compose access"| shared
    shared --> grpc --> rpc
    shared --> web --> rpc
    shared --> http
    clients -->|"Existing endpoint"| legacy --> rpc

The existing HTTP ingress exposes all three request types through the same hostname. The chart configures HTTP/2 upstream using the supported NGINX and Traefik settings and documents controller prerequisites and TLS overrides. The legacy gRPC endpoint and ingress stay available; no client endpoints or defaults change. Clients can test the shared hostname independently.

Options evaluated and outcomes

The research and references are in the research comment. The current decision comment supersedes the earlier cmux recommendation.

What the experiments established

Connection reuse matters even with separate clients. Direct CLI and native gRPC clients normally use gRPC connections, but a proxy can pool requests from independent clients onto one upstream connection. We have not measured how often that happens in Chainloop deployments. The decision follows a reproduced failure, rather than an assumed frequency.

In the NGINX experiment with the earlier connection-classified server, native gRPC succeeded, but a subsequent gRPC-Web request could receive 415 application/grpc. A single HTTP/2 upstream path to the current per-request server passed the focused native gRPC, browser HTTP, gRPC-Web, and preflight checks. See the routing reproduction; its earlier cmux recommendation is historical.

Current tests also exercise all request types on one real HTTP/2 connection in both protocol orders, over h2c and TLS. Local NGINX Open Source integration passed with versions 1.28.0 and 1.23.3. These checks do not certify every deployed ingress controller or the full authentication and authorization matrix.

CAS requires separate evidence. A paused-reader streaming probe showed that grpc-go's HTTP adapter can read ahead beyond the application's receive pace, with substantially greater live heap than the native transport in that probe. This was a transport experiment, not a real CAS throughput benchmark or isolated server-memory measurement. It prevents a claim of bounded-memory ByteStream parity; it does not establish the eventual CAS design. CAS keeps its current native gRPC transport and endpoints.

Relevant project precedents include Argo CD's cmux implementation, etcd's shared TLS HTTP dispatch, and ZITADEL's HTTP dispatch. Argo CD's shown HTTP branch is HTTP/1.1, so it does not demonstrate arbitrary ordinary HTTP/2 and native gRPC streams on one connection. Connect uses HTTP handlers with HTTP/2 deployment requirements; its deployment model informed this choice without requiring a Connect migration.

Specs and remaining work

The implementation PR adds a replacement control-plane spec and links it from the original merged spec.

Replacement D-001 supersedes the control-plane portions of original D-014, D-015, and D-016. Replacement D-002 supersedes the control-plane ingress design in original D-004. Replacement D-003 defers CAS transport selection. The original CAS requirements remain open.

• Compare connection classification, per-request dispatch, Connect/bridges, and proxy alternatives.
• Implement control-plane per-request dispatch and shared Helm ingress in draft PR feat(controlplane): serve HTTP and gRPC through one ingress #3647.
• Add the replacement control-plane spec and mark the original control-plane design as superseded.
• Complete the full original R-002 authentication, authorization, and behavior alignment matrix. Focused tests are insufficient evidence of full alignment.
• Validate the chart with deployed controller configurations, including F5 NGINX/NGINX Plus and Traefik. Local NGINX tests are not controller certification.
• Review and merge the control-plane implementation; retain legacy endpoint compatibility.
• Select and implement the CAS shared-endpoint approach.
• Establish original R-003 CAS streaming and performance evidence, including the 1 GB workload, bounded memory, and the specified 10% performance limit.

Related: #2642 (single hostname at the chart level, closed as stale). The server design also supports direct VM and Compose deployments and does not depend on Gateway API.

Original task posted by Maximus bot (Claude Code) on behalf of @migmartri. Decision history updated with Codex assistance.