feat(flags): OpenFeature-compatible flag delivery with an owned flagd runtime

[vishr]

The problem

Onebox can deploy configuration only as environment values or ordinary files tied to an application release. An application that wants runtime feature flags must currently own the entire mechanism: choose a flag API, run or buy a backend, deliver configuration, update it safely, preserve history, and decide what happens when evaluation is unavailable.

That leaves two poor choices:

• encode operational switches as environment variables and redeploy the application for every change; or
• add an application-owned feature-flag control plane outside Onebox's plan, approval, lock, fence, journal, status, and rollback contracts.

Feature flags are useful precisely because they can change independently of an application release. Treating them as another database-style services driver would also be misleading: OpenFeature is an application API rather than a server, while flagd is a stateless evaluator whose source of truth can remain in the repository.

Proposed change

Add first-class, OpenFeature-compatible feature-flag delivery backed by an application-scoped flagd runtime.

The ownership statement should be:

OpenFeature is the application interface, flagd is the default evaluator, and Onebox owns validated, reviewable, generation-based flag delivery.

OpenFeature itself is not installed or “managed” by Onebox. Application authors add the appropriate OpenFeature SDK and flagd provider to their code. Onebox owns the flagd container, its pinned image, the validated flag document, private connectivity, activation evidence, generations, journal, status, and rollback.

Do not add this to services. Introduce a dedicated top-level contract, initially with one explicit delivery mode:

feature_flags:
  source: flags/production.flagd.yaml
  delivery: rpc
  workloads: [api, worker]

An environment may replace only the source:

feature_flags:
  source: flags/default.flagd.yaml
  delivery: rpc
  workloads: [api, worker]

environments:
  production:
    server: deploy@production.example.com
    feature_flags:
      source: flags/production.flagd.yaml
  staging:
    server: deploy@staging.example.com
    feature_flags:
      source: flags/staging.flagd.yaml

workloads is explicit. Onebox must not silently give every daemon or job access to a new runtime dependency. Changing the source path, delivery mode, selected workloads, flagd version, or any other declaration shape is deploy-bound. Changing only the contents of the already-deployed source is eligible for the independent flag-update operation.

Runtime contract

• Run one pinned, application-scoped flagd container on the existing service network, outside the application release's Compose lifecycle so application rollback cannot remove it.
• Use the official ghcr.io/open-feature/flagd image, resolve the selected tag to an immutable digest through the existing image-selection path, and never use latest.
• Keep the flagd evaluation, sync, OFREP, and management ports private. The first version exposes no public route and supports server-side workloads only.
• Start flagd against one immutable, repository-backed flag generation. It has no writable application data volume and no backup policy: the repository plus retained generations are the recovery source.
• Project the standard flagd RPC connection facts needed by the selected workloads (FLAGD_HOST, FLAGD_PORT, and resolver configuration), without putting application flags or evaluation context in environment variables.
• Use RPC evaluation first. File and in-process resolver delivery remain future modes because Onebox cannot prove that every application process observed a hot-replaced file or sync event.
• Applications remain responsible for passing a safe default to every OpenFeature evaluation. OpenFeature requires evaluation failures to return the supplied default rather than terminate application execution.

Validation and compatibility

• Vendor the exact supported flagd JSON Schema into the binary; validation must not download a remote schema.
• Accept JSON and YAML only when they normalize to the supported flagd document contract.
• Validate the source locally before contacting the host and report typed path-aware errors.
• Include the normalized document digest in preview, deploy inputs, plans, and runtime evidence.
• Treat flag keys and values as ordinary, reviewable configuration, not secrets. Documentation must explicitly say not to put credentials in flag definitions and not to use a feature flag as an authorization decision.
• Pin and test an explicit compatibility tuple: flagd image version, flagd schema version, evaluation protocol, and the connection variables Onebox projects. Upstream is pre-1.0, so accepting whatever a future tag or remote schema happens to do is not a compatibility strategy.

Independent flag-update lifecycle

Ship an independent plan/apply operation rather than a blind push. Exact command spelling can be settled with the CLI design, but the lifecycle must provide the equivalent of:

ob flags list
ob flags plan --out flags-plan.json
ob approve --plan flags-plan.json --out flags-approval.json
ob flags apply --plan flags-plan.json --approval flags-approval.json
ob flags rollback

The plan must bind:

• target and host-owner state;
• current flag generation and normalized digest;
• proposed normalized digest and exact semantic diff;
• deployed declaration graph (source path, delivery mode, selected workloads, runtime identity);
• pinned flagd image digest;
• operation graph, expiry, risk, reversibility, and approval class.

The semantic diff should show flag additions/removals and changes to enabled state, variants, default variant, targeting, and metadata. These values are not redacted by default.

Apply must:

1. revalidate the plan and current host/declaration state before mutation;
2. acquire the existing host lock and fence;
3. stage the exact planned document under a fresh opaque generation;
4. checkpoint the old and candidate generations durably;
5. replace/start flagd against the candidate's immutable file;
6. wait for flagd readiness;
7. commit the generation pointer and journal success only after readiness;
8. restore the old generation and runtime on failure; and
9. retain enough prior generations for explicit rollback and safe cleanup.

Use fresh-process readiness as the activation evidence. flagd's readiness endpoint returns 412 until every configured sync source has completed an initial successful sync, then 200. Its documented readiness does not regress after a later source failure, so updating a symlink under a long-running process and checking /readyz would falsely claim evidence. Generation-bound replacement avoids that claim. A short evaluation interruption may occur during replacement; the contract must state that clients can transiently receive their code defaults. Zero-downtime flagd replacement is a later capability, not an implicit promise.

Interrupted apply must resume or recover from the durable checkpoint. It must never report success when the selected generation, running container arguments, and journal disagree.

Deploy, status, destroy, and eject

• ob deploy stages and reconciles the initial/current flag generation before starting consumers that declare it.
• The normal deploy plan shows the flagd runtime and flag document diff. An unchanged, healthy flagd runtime is retained.
• A flag-document-only change reconciles flagd without replacing application workloads; changing endpoint/declaration wiring follows ordinary workload revision rules.
• ob status reports declaration state, active generation, normalized digest, selected image digest, container health, and divergence without printing evaluation context.
• ob doctor reports missing/unhealthy flagd and a source/runtime generation mismatch with a named resolving command.
• ob destroy removes the flagd container and generated state at the same ownership boundary as other application-scoped runtime. Retention/destruction of the authored source remains the repository owner's concern.
• ob eject must include the flagd runtime, the active non-secret flag document, network wiring, and selected-workload connection configuration so the ejected system remains operable without Onebox.

Current workaround

Applications can run flagd as a user-owned daemon workload or connect an OpenFeature SDK to a commercial provider. In both cases Onebox sees only an arbitrary container or endpoint. It cannot validate the flag document, include flag changes in a plan, distinguish declaration changes from value changes, observe the active generation, recover an interrupted update, or roll back flag configuration independently of the application.

Environment variables are simpler but require a workload replacement and cannot express typed variants, targeting rules, or deterministic fractional rollout.

Scope and safety

This does not widen Onebox beyond one application on one host and does not introduce Consul or another cluster/control-plane dependency. flagd remains private to the application's existing Docker network and is re-creatable from repository state.

It does add a new application-scoped mutation independent of deployment. That mutation must use the canonical operation service and existing plan/approval, owner, lock, fence, journal, checkpoint, structured-output, and typed-error contracts. It must not be implemented as an unplanned file copy or a CLI-only path.

Initial non-goals:

• installing or modifying application OpenFeature SDK code;
• a feature-flag dashboard or mutable server-side database;
• experimentation analytics, exposure-event storage, or statistical analysis;
• user/cohort storage;
• public browser/mobile evaluation;
• external/commercial provider management;
• multiple flag sources or flag-set merge precedence;
• remote HTTP, object-storage, gRPC, or Kubernetes sync sources;
• file/in-process resolver delivery;
• zero-downtime flagd replacement;
• treating flags as secrets, authentication, authorization, or entitlement enforcement.

Due diligence

Checked 2026-09-04:

• OpenFeature is the vendor-neutral evaluation API. Its provider abstraction is expressly designed to separate application calls from a concrete flag system.
• The OpenFeature evaluation contract requires a typed default and requires abnormal evaluation to return that default rather than throw.
• flagd is the OpenFeature project's focused evaluator. It supports RPC, in-process, and file resolver patterns; local files and several remote sync sources; targeting; deterministic fractional rollout; and OpenTelemetry-compatible signals.
• The provider specification standardizes FLAGD_HOST, FLAGD_PORT, resolver selection, TLS settings, and lifecycle/change notifications.
• The monitoring contract exposes liveness, readiness, and metrics. Readiness proves the first successful sync but intentionally stays ready afterward, which is why this proposal does not use it as evidence for an in-place hot reload.
• The file-sync guidance recommends atomic symbolic-link replacement, but the absence of a Onebox-observable per-consumer acknowledgement makes that insufficient for a first evidence-backed file-delivery contract.
• OpenFeature spec, flagd, and flagd-schemas are active, Apache-2.0 projects under the OpenFeature organization with a shared security policy. At filing, the current published lines are still pre-1.0 (OpenFeature spec v0.9.0, flagd v0.16.2, flagd schema/protobuf components likewise pre-1.0), so Onebox must own a pinned and tested compatibility envelope.
• The local issue tracker has no existing OpenFeature, flagd, or feature-flags proposal.

Implementation map

The existing seams are suitable, but this is a cross-cutting capability:

• internal/app/types.go, defaults, loader, cross-field validation, environment resolution, canonical origins, file existence checks, JSON Schema, and conformance corpus for the new declaration.
• Generated project-field documentation and the capability inventory; do not advertise the feature until the executable path ships.
• Names and renderers for the app-scoped flagd directory, container, immutable generation file, current pointer, and Compose document.
• Workload runtime generation for selected connection projection without conflating flags with env_files or managed-service credentials.
• Artifact/config hashing so flag content and declaration changes affect preview and deploy plans deterministically.
• A new canonical operation kind, artifact binding, plan type, risk/reversibility classification, service methods, adapter commands, JSON/NDJSON records, typed failures, and guidance.
• Engine reconciliation, fresh-instance readiness probe, checkpoint/recovery, rollback, retention, status/drift, destroy, and eject.
• Unit tests at each state boundary plus a remote E2E proving deploy, independent update, failed candidate recovery, interrupted resume, rollback, reboot survival, and destroy.

The secret-generation implementation is a useful model for declaration-graph equality, opaque generations, checkpoints, selective lifecycle, and recovery, but flag documents are non-secret and should receive exact semantic diffs rather than secret-value suppression.

Acceptance criteria

• A valid top-level declaration and environment-specific source replacement are represented in canonical output with origins; unknown fields, paths, workload names, delivery modes, and invalid flag documents fail locally with typed errors.
• The supported flagd schema and compatibility tuple are vendored/pinned and tested; validation performs no network access.
• A normal deploy plan includes the exact flagd image digest, normalized flag digest, runtime artifact, selected workloads, and change classification.
• Deploy starts/reconciles a private app-scoped flagd runtime before selected consumers and retains an unchanged healthy runtime.
• Selected workloads receive the documented standard flagd RPC connection contract; unselected workloads and daemons receive nothing implicitly.
• Authored environment values that claim a projected flagd connection name are refused rather than silently shadowing the owned endpoint.
• No flagd port is published or routed by default.
• A flag-document-only deploy or independent apply does not replace application workloads.
• Independent flag changes require a sealed, expiring plan and applicable approval; apply rejects target, owner, declaration, image, active-generation, or source drift.
• Plans show deterministic semantic flag diffs and never include runtime evaluation context.
• Apply stages an immutable generation, checkpoints before switching, starts a fresh flagd process, and commits only after initial-sync readiness.
• Candidate failure restores the previous generation/runtime; interruption is resumable or recoverable and cannot leave success evidence for mixed state.
• Explicit rollback, status/drift, doctor, destroy, and eject have tested contracts.
• Application evaluation failure semantics and transient-default behavior during flagd replacement are documented and tested in the E2E fixture.
• Structured human/JSON/NDJSON output, error registries, command policy, generated CLI reference, project reference, capabilities page, and examples are updated together.
• CI covers the pinned flagd image on the supported server architectures and the release gate does not depend on latest or a remote schema.

Design gates before implementation

1. Confirm the exact first supported OpenFeature/flagd compatibility tuple and provider languages used by the E2E fixture.
2. Confirm how the runner probes a private, potentially shell-less flagd image without introducing an unpinned helper dependency.
3. Decide whether standard FLAGD_* variables are projected directly or through an explicit workload mapping while preserving Onebox's environment-variable naming policy.
4. Specify the checkpoint phases and recovery truth table before adding the operation kind.
5. Specify the maximum accepted flag document size, targeting-expression limits, and resource caps before treating authored rules as safe runtime input.