From 4e85a2ffcf8b3fc68314fa0879b9f4278395ae9e Mon Sep 17 00:00:00 2001 From: cooper Date: Wed, 23 Sep 2026 04:13:13 -0700 Subject: [PATCH] docs(adr): record agent API, project selection, .stack layout and deploy decisions ADRs 0004-0007 capture the decisions from the September 2026 codebase review so they outlive the conversation that produced them: Connect as the single agent API, per-request project selection, .stack/ as the one stackpanel directory with JSON machine data, and deploy backends as guarded options. Refs: stackpanel-thq, stackpanel-thq.18 --- docs/adr/0004-agent-api-on-connect.md | 164 ++++++++++++++++++ .../adr/0005-per-request-project-selection.md | 101 +++++++++++ docs/adr/0006-single-stack-directory.md | 123 +++++++++++++ docs/adr/0007-deploy-backends-are-options.md | 83 +++++++++ docs/adr/README.md | 4 + 5 files changed, 475 insertions(+) create mode 100644 docs/adr/0004-agent-api-on-connect.md create mode 100644 docs/adr/0005-per-request-project-selection.md create mode 100644 docs/adr/0006-single-stack-directory.md create mode 100644 docs/adr/0007-deploy-backends-are-options.md diff --git a/docs/adr/0004-agent-api-on-connect.md b/docs/adr/0004-agent-api-on-connect.md new file mode 100644 index 00000000..ec221f49 --- /dev/null +++ b/docs/adr/0004-agent-api-on-connect.md @@ -0,0 +1,164 @@ +# 0004 — The local agent API is Connect, organized as versioned domain services + +- **Status**: Accepted +- **Date**: 2026-09-23 +- **Related**: [ADR 0005](./0005-per-request-project-selection.md), + darkmatter org ADR-0003 (protobuf is the source of truth; Connect is the + default transport) + +## Context + +The Studio talks to the local Go agent (`stack agent`, port 9876) over two +transports at once: + +- **REST/JSON** — ~60 hand-written `/api/*` handlers, called through + `AgentHttpClient` (`apps/web/src/lib/agent.ts`) from 28 files. Most + local traffic uses it. +- **Connect-RPC** — one `AgentService` with 48 RPCs + (`packages/proto/proto/agent.proto`). Only 13 are reached by live + components (project, apps, variables, users, secrets, processes, Nix + config, shell status/rebuild, exec, installed packages, module outputs, + and `PatchNixData`, which is how the Studio saves config edits). The other + 35 duplicate REST endpoints and have no caller. + +Keeping both has cost correctness, not just code size. The September 2026 +review found the twins drifting apart: the Connect `ReadFile`/`WriteFile` +handlers skip the project-root containment the REST versions enforce; the +Connect handler is mounted without the project guard; search results are +sorted on one side only; a custom `protoc-gen-connect-handlers` plugin +generates entity getters that ignore request fields. The web side carries +71 exported hooks in `use-agent.ts` (37 unused), and demo fixtures exist +twice, once per wire shape. + +A separate tRPC API (`packages/api`) serves the cloud features (waitlist, +GitHub, hosted state). tRPC gets its types from the client importing the +server's TypeScript router type, so it cannot serve a Go agent. The +realistic choice for the agent was always REST or Connect. + +The Studio is deployed continuously from Cloudflare while each user's agent +is pinned by their flake, so every Studio build talks to agents of many +versions. Wire compatibility has to be checkable, not hoped for. + +## Decision + +The agent exposes one API: **Connect**, defined in protobuf, organized as +small domain services under a versioned package. REST handlers are retired +as their callers move. tRPC remains the transport for the TypeScript cloud +API only. + +**Schema** + +- Service definitions are hand-written in `proto/stackpanel/agent/v1/`, one + file per domain: `AgentService` (`GetAgentInfo` with version and + capabilities, `Pair`), `ProjectService`, `ConfigService`, + `SecretService`, `ProcessService`, `ShellService`, `CommandService`, + `PackageService`, `ModuleService`, `HealthService`, `DeployService`, + and `EventService`. +- Config types are generated from the proto-nix schemas into + `proto/stackpanel/config/v1/`. The generator owns the mapping between + proto field names and kebab-case Nix attribute names, so a `FieldMask` + path is also a `config.nix` attribute path. +- Messages are typed end to end: no JSON strings or `google.protobuf.Struct` + payloads. Singleton settings change through `UpdateConfig` with a + `FieldMask`; keyed collections (apps, variables, users, envs) change + through per-entry `Upsert`/`Delete` RPCs. +- Side-effect-free reads set `idempotency_level = NO_SIDE_EFFECTS` so + clients may use HTTP GET. +- `buf lint` and `buf breaking` (against `main`) run in CI, and generated + code is committed with a `buf generate` drift check, per org ADR-0003. + Field numbers are never reused. + +**Server** + +- Cross-cutting concerns live in one interceptor chain applied to every + service: authentication (`Authorization: Bearer` only; `Pair` is + exempt), project resolution ([ADR 0005](./0005-per-request-project-selection.md)), + protovalidate (`connectrpc.com/validate`), panic recovery + (`connect.WithRecover`), and logging. Handlers contain domain logic only. +- Each domain lives in `internal/agent//`, which owns its logic and + a thin Connect adapter. `internal/agent/server` only composes the mux, + the interceptors, and the per-project runtime registry. +- Long-running or live work uses server streaming (logs, shell rebuilds, + commands, deploys, health runs). Browsers support only unary and + server-streaming calls, so no client or bidirectional streams. The + browser reaches the agent over plain `http://localhost`, which means + HTTP/1.1 and roughly six connections per origin, so all live updates are + multiplexed onto a single `EventService.Watch` stream. +- Failures use Connect error codes with typed details (for example a + `NixEvalError` with file, line, and message) instead of + `{success: false}` envelopes. +- `grpcreflect` and `grpchealth` are served so `buf curl` and `grpcurl` can + discover the API. + +**Clients** + +- The Studio creates one transport in `AgentProvider`, with interceptors + that add the auth and project headers, and uses Connect-Query for every + read and write. The demo mode implements the services in memory with + `createRouterTransport` and one set of typed fixtures. +- `packages/agent-client` is the published client (generated code plus the + transport factory) for the Studio, the TUI, scripts, and third parties. +- The Studio calls `GetAgentInfo` on connect and gates features on the + reported capabilities, showing "update your agent" rather than failing + on `Unimplemented`. +- The CLI keeps calling the Go domain packages directly, so it works + without a running agent, and shares that code with the handlers. + +## Consequences + +**Pros** + +- One contract for every client, checked for breaking changes in CI. The + version skew between the Cloudflare Studio and local agents becomes a + tooling problem instead of a runtime surprise. +- Cross-cutting behavior (auth, project, validation) cannot be forgotten on + one route, which is how the REST/Connect twins drifted. +- The web data layer becomes generated code: the hand-written hooks, the + 52 copy-pasted fetch blocks, and the duplicated demo fixtures go away. +- The typed `FieldMask` path removes the hand-written kebab/snake + conversions that caused saved Studio fields to revert. + +**Cons and risks** + +- Protobuf's type system is narrower than TypeScript's; sum types need + `oneof` and optionality needs care. +- The migration touches most Studio screens. It must move one area at a + time, deleting each REST route when its last caller moves, and never + keep a REST mirror "for compatibility". +- Contributors need to learn enough `buf` to change the schema safely. + +**Follow-ups** (tracked under stackpanel-thq.8) + +1. Delete the 35 unused `AgentService` RPCs, the unused `use-agent.ts` + hooks, and the tRPC `agent` router, which proxies to `localhost:9876` + from the web server and so cannot reach a user's agent from Cloudflare. +2. Stand up the v1 services, interceptors, and runtime registry beside the + current server. +3. Move the Studio one area at a time. +4. Delete the old `AgentService`, the REST routes, the SSE and WebSocket + endpoints, `protoc-gen-connect-handlers` and its committed binary, the + protobuf-ts output, and the unused JSON-schema output. + +## Alternatives considered + +- **REST as the single transport.** Carries most traffic today, but has no + schema, no generated clients, and no breaking-change tooling; the Go and + TypeScript types would stay hand-maintained on both sides. +- **Adopt the existing `AgentService` as is.** Its shape is the problem: a + 48-method service with generic entity getters/setters and a custom + handler generator. Migrating callers onto it would carry the drift over. +- **tRPC for the agent.** Not possible: tRPC's types come from a TypeScript + server router, and the agent is Go. (Org ADR-0003 also rejects + single-language RPC for services with polyglot consumers.) +- **Serve the Studio from the agent (same origin).** Would remove CORS, + pairing tokens across origins, browser local-network prompts, and version + skew, but gives up instant UI updates and the hosted cloud features. Not + adopted; the capability handshake handles skew instead. + +## References + +- stackpanel-thq (simplification epic), stackpanel-thq.8 (Agent API v1), + stackpanel-0h5 (public agent API), stackpanel-382 (demo fixtures from + proto-nix examples) +- `packages/proto/proto/agent.proto`, `apps/stackpanel-go/internal/agent/server/`, + `apps/web/src/lib/{agent.ts,use-agent.ts}`, `packages/api/src/routers/agent.ts` diff --git a/docs/adr/0005-per-request-project-selection.md b/docs/adr/0005-per-request-project-selection.md new file mode 100644 index 00000000..c83552b1 --- /dev/null +++ b/docs/adr/0005-per-request-project-selection.md @@ -0,0 +1,101 @@ +# 0005 — Agent requests select their project per request + +- **Status**: Accepted +- **Date**: 2026-09-23 +- **Related**: [ADR 0004](./0004-agent-api-on-connect.md) + +## Context + +One agent process serves every client on the machine: Studio tabs, the TUI, +scripts, and (with the planned global service, stackpanel-f1n) the whole +user session. Today it holds a single project: + +- `s.config.ProjectRoot` is chosen at startup (flag or env, then the working + directory, then the saved "current project" in + `~/.config/stackpanel/stackpanel.yaml`). +- `/api/project/open` overwrites that field (`project_handlers.go:173`) + with no lock while roughly 99 other sites read it, then + `reinitializeExecutor()` (`server.go:559`) swaps the singletons: executor, + data store, shell manager, and flake watcher. +- Every client therefore shares one current project. Two Studio tabs, or + the Studio and the CLI, on different projects switch each other. + +A per-request design has existed since the initial import (2026-01-19): +`project_context.go` resolves the project from an `X-Stackpanel-Project` +header, then a query parameter, then the default and current projects. It +was never mounted on a route. The surrounding code assumed it worked: the +web app sends the header, CORS allows it, and `stack project` help and the +docs tell users to send it. + +stackpanel-7hn asked for exactly this ("support multiple projects at once +... by passing the project in every request"). The stackpanel-f1n spec +(`docs/superpowers/specs/2026-06-15-global-agent-service.md`) instead kept +a single current project with open/switch endpoints and listed replacing +them as a non-goal. The two plans conflicted. + +## Decision + +Each agent request names its project. The agent keeps no global +"current project". + +- A Connect interceptor ([ADR 0004](./0004-agent-api-on-connect.md)) + resolves the project from the `Stackpanel-Project` request header. When + the header is absent it falls back to the registry's default project and + then its current project, so interactive CLI use stays convenient. +- The interceptor attaches that project's **runtime** to the request + context. A runtime owns everything project-scoped: the command executor, + the data store, the shell manager, the flake watcher, caches, and an + `os.Root` opened on the project directory. File access goes through that + root, so escaping the project is impossible by construction. +- Runtimes live in a registry keyed by project ID. They start on first use + and are shut down after an idle period. +- Handlers read the runtime from the context; nothing reads a global + project field. +- Event streams are scoped to a project. +- The project registry in `~/.config/stackpanel/stackpanel.yaml` stays the + source of known projects. "Open" becomes "register or select", not + "replace the agent's global state". + +This supersedes the single-current-project model in the stackpanel-f1n +spec; that spec's non-goal is updated accordingly. + +## Consequences + +**Pros** + +- Clients stop interfering with each other; one global agent can serve + several projects at once, which the global service needs. +- The data race on `s.config.ProjectRoot` and the goroutine started on every + project switch disappear with the mutable field. +- Path containment is structural (`os.Root`) instead of a helper each file + handler has to remember. + +**Cons and risks** + +- Memory grows with the number of active projects; idle eviction must be + tuned. +- Caches keyed by project must not leak across runtimes (today a + process-wide config cache survives project switches). + +**Follow-ups** (tracked under stackpanel-thq.8.1) + +- Until the interceptor lands, delete `project_context.go` and correct the + CLI help and docs that advertise a header the agent ignores. The + resolution logic is small enough to rewrite as the interceptor. + +## Alternatives considered + +- **Keep one current project with open/switch (f1n spec as written).** + Simple, but a global agent shared by several clients makes interference + the normal case, and the switch path is racy. +- **Mount the existing `project_context.go` middleware.** It only resolves + a path; every handler still uses per-agent singletons bound to one + project, so it would not deliver multi-project behavior. +- **One agent process per project.** Rejected by stackpanel-f1n: it + multiplies services, ports, and pairing state. + +## References + +- stackpanel-thq.8.1, stackpanel-7hn, stackpanel-f1n +- `apps/stackpanel-go/internal/agent/server/{project_context.go,project_handlers.go,server.go}` +- `apps/stackpanel-go/cmd/cli/project.go`, `apps/docs/content/docs/cli/project/` diff --git a/docs/adr/0006-single-stack-directory.md b/docs/adr/0006-single-stack-directory.md new file mode 100644 index 00000000..421a2e5c --- /dev/null +++ b/docs/adr/0006-single-stack-directory.md @@ -0,0 +1,123 @@ +# 0006 — `.stack/` is the single stackpanel directory, with one config file and JSON machine data + +- **Status**: Accepted +- **Date**: 2026-09-23 +- **Related**: [ADR 0004](./0004-agent-api-on-connect.md) + +## Context + +Stackpanel is aimed at developers who are not Nix natives. For them the +most important property of the project directory is that every setting has +one obvious home, and that the file the Studio edits is the file Nix reads. + +The September 2026 review found about ten places a config value can live, +added one after another without the previous one being removed: + +| Location | State | +| --- | --- | +| `.stack/config.nix` | Canonical; edited by humans and by the Studio (syntax-tree patching) | +| Split imports (`import ./config.apps.nix`) | Supported; one agent write path inlines them back | +| `.stack/config/` (haumea tree) | Optional; shallow-overrides `config.nix`; needs haumea in the user's flake; its map-entity list is hand-synced with Go and has drifted | +| `.stack/data/.nix` | Go reads and writes it when present; Nix reads only `data/packages.nix`, so Studio edits can land in files Nix never reads | +| `.stack/data.nix` | Scaffolded by the `default`, `minimal`, and `haumea` templates, documented as "merged with config.nix"; nothing reads it | +| `.stack/_internal.nix` | The loader's first choice; nothing writes it | +| `.stack/config.local.nix` | Personal overrides (gitignored) | +| `.stack/modules/` | Full NixOS-module escape hatch | +| `.stackpanel/{config.nix,_internal.nix,config/}` | Legacy directory name, still probed (~100 references across Go, Nix, and TypeScript) | + +Runtime files were split between `state/` and `profile/` (plus `keys/`, +`gen/`, `bin/`, `.token`, `secrets/state/`, `secrets/bin/`), taking 14 +`.gitignore` lines; the `state/`/`profile/` split already broke the TUI +certificate check. Machine-synced data (GitHub collaborators, the SSH-age +cache) was written as Nix source by Go, which is where several quoting and +formatting bugs lived, and one writer and its readers disagreed about the +directory. + +## Decision + +`.stack/` holds every stackpanel source of truth and all stackpanel state, +in one layout: + +``` +.stack/ + config.nix the one config file: plain data, edited by humans and the Studio + config.local.nix personal overrides (gitignored) + modules/ Nix escape hatch; the agent never writes here + data/ machine-synced, committed, never hand-edited (JSON) + secrets/ SOPS-encrypted values and recipients + machines/ deploy targets + gen/ generated files (gitignored) + state/ everything machine-local (gitignored): keys, profile, binaries, tokens, caches +``` + +1. **`config.nix` is Nix, but plain data.** No `let` blocks or logic; logic + moves to `modules/`. It reads like JSON with `=` and `;`, and the + Studio's syntax-tree edits stay safe. Splitting it with plain + `import ./.nix` is allowed and understood by the patcher. +2. **Machine data is JSON under `data/`.** It is lockfile-like: synced from + outside and committed so Nix evaluation stays pure. Go writes JSON; Nix + reads it with `builtins.fromJSON`. Go no longer emits Nix source for + data. Studio-installed packages become an entry in `config.nix`. +3. **Two ignored directories:** `gen/` for anything regenerable, `state/` + for anything machine-local. What is safe to delete is obvious. +4. **The project root is the nearest directory containing + `.stack/config.nix`.** The `.stackpanel-root` marker goes away, and the + four root-finding implementations become one rule. +5. **Removed:** the `.stackpanel/` fallbacks, `_internal.nix`, `data.nix` + (from templates and the proto-nix schema options), Go's + `data/.nix` fallback, and the haumea tree layout. +6. **Files outside `.stack/` are generated projections.** Some files must + live where their tool looks: `.envrc`, `turbo.json`, the workspace + packages under `packages/gen/`, per-app integration outputs such as + `apps/api/fly.toml`. Each is fully regenerated, carries a header + pointing back to its `.stack/` source, and is recorded in the CLI's file + ledger so `stack doctor` can list and clean them. +7. **One cutover, no compatibility window.** Go, Nix, TypeScript, + templates, and docs change together. `stack doctor` fails with exact + move instructions if it finds a legacy path, instead of silently + supporting it. + +## Consequences + +**Pros** + +- One home per value; the Studio writes exactly what Nix reads. +- Removes a class of bugs: Nix source emitted by Go for data, a key encoding + shared by hand between Go and Nix, and writers and readers using different + directories. +- Smaller loader and writer in both Go and Nix; no extra flake input for + users. + +**Cons and risks** + +- Projects using the haumea tree or the legacy directory must migrate; the + doctor check makes that explicit. +- Dropping the tree layout gives up per-entry files. If merge conflicts in + `config.nix` become a problem, split it with imports rather than bringing + back a second layout. + +**Follow-ups** (tracked under stackpanel-thq.9) + +- Decide whether `.stack/gen/` is committed (AGENTS.md says yes, + `.gitignore` says no) and make the docs match. + +## Alternatives considered + +- **Keep the haumea tree as an option.** Solves per-entry diffs and merge + conflicts, but it is a second layout both Go and Nix must support, adds a + flake input, has surprising precedence (tree wins, shallowly), and its + hand-synced key encoding had already drifted. +- **A separate agent-written overlay (`data.nix`).** Tried already: two + files per setting with precedence rules, and it fell out of use without + being removed. +- **Keep machine data as Nix files.** Requires Go to emit Nix source safely, + which is where the quoting and formatter bugs came from. +- **Keep a compatibility window for `.stackpanel/`.** Rejected: silent + fallbacks are how the current ten locations accumulated. + +## References + +- stackpanel-thq.9 (consolidation), stackpanel-thq.7 (writer/reader + directory mismatch) +- `nix/flake/load-config.nix`, `apps/stackpanel-go/pkg/nixdata/{paths.go,entities.go,store.go}`, + `nix/stackpanel/db/lib/proto.nix`, `nix/flake/templates/` diff --git a/docs/adr/0007-deploy-backends-are-options.md b/docs/adr/0007-deploy-backends-are-options.md new file mode 100644 index 00000000..f9c754f4 --- /dev/null +++ b/docs/adr/0007-deploy-backends-are-options.md @@ -0,0 +1,83 @@ +# 0007 — Deploy backends are options; collisions are guarded and internals shared + +- **Status**: Accepted +- **Date**: 2026-09-23 + +## Context + +Stackpanel supports several ways to deploy: Alchemy (Cloudflare, AWS), SST, +Colmena/NixOS, containers (Fly and others), and per-app Alchemy scripts. +The September 2026 review asked whether these should collapse into one +deploy path. They should not: stackpanel's purpose is to give projects a +choice of backend. + +The review did find problems that come from how the options are built, not +from having them: + +- **Broken options.** The Nix-generated Alchemy infra modules import 25 + paths from `@stackpanel/infra` that do not exist; local deploy wrappers + (`just deploy`, `scripts/deploy/*`) run a file deleted months ago; the + CI NixOS deploy targets a flake whose `colmenaHive` is empty. +- **Copy-pasted internals.** The same AMI lookup, EC2 provisioning, and + container field declarations exist in several backends and have drifted. +- **Collisions.** SST and `aws-secrets` both create + `${projectName}-secrets-role` and the KMS alias `${projectName}-secrets`; + enabling both (as this repository does) gives two tools ownership of the + same AWS resources. +- **Ambiguous entry points.** "deploy" means four different things across + the root `package.json`, the Justfile, turbo, and per-app scripts. + +## Decision + +Deploy backends remain independent, user-selectable options. Each option +must meet these rules: + +1. **It works or it is removed.** A backend that cannot run end to end is + fixed or deleted; it is not left discoverable. +2. **Shared mechanics live in one place.** Logic needed by more than one + backend (AMI lookup, instance provisioning, container schema) is + implemented once and called by each backend. +3. **Options that would own the same resources cannot be enabled + together.** A Nix assertion fails evaluation with a message naming both + options and the contested resource. +4. **Entry points are named per backend** (for example `deploy:alchemy`, + `deploy:colmena`). A bare `deploy` either dispatches to the configured + backend or does not exist. +5. **Two definitions may not claim one external name** (a DNS record, a + hostname, a cloud resource name). Stackpanel's own services follow the + same rule: `api.stackpanel.com` gets one owner. + +## Consequences + +**Pros** + +- Users keep the choice of backend, which is the product's point. +- Future reviews stop proposing "one deploy path"; the rules above give a + concrete bar instead. +- Collisions fail at evaluation time instead of in a cloud account. + +**Cons** + +- Supporting several backends is permanent maintenance cost; shared + internals and the "works or removed" rule keep it bounded. + +**Follow-ups** (tracked under stackpanel-thq.13) + +- Fix or remove the Nix-generated Alchemy modules, the stale deploy + wrappers, and the CI NixOS flake reference; add the SST/aws-secrets + assertion; extract the shared AMI lookup and EC2 provisioning; rename + entry points per backend. + +## Alternatives considered + +- **Consolidate on one deploy path (per-app Alchemy).** Simpler to + maintain, but contradicts stackpanel's purpose of supporting the backend + a project already uses. +- **Keep options as they are.** Leaves broken backends discoverable and + resource collisions possible. + +## References + +- stackpanel-thq.13, stackpanel-rz0 +- `nix/stackpanel/integrations/{infra,sst,deployment,containers}/`, + `Justfile`, `scripts/deploy/`, `.github/workflows/deploy-*.y*ml` diff --git a/docs/adr/README.md b/docs/adr/README.md index 71f168c6..18a0d371 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -28,3 +28,7 @@ rather than editing history. - [0001 — Runtime secrets are decrypted via `@gen/env`, not forwarded as Worker env vars](./0001-runtime-secrets-via-gen-env-loader.md) — *Superseded by 0003* - [0002 — Database migrations are applied programmatically at app startup, not via `drizzle-kit push`](./0002-runtime-startup-migrations.md) - [0003 — Build-time env injection with `effect/Config` for typed redacted access](./0003-build-time-env-injection-with-effect-config.md) +- [0004 — The local agent API is Connect, organized as versioned domain services](./0004-agent-api-on-connect.md) +- [0005 — Agent requests select their project per request](./0005-per-request-project-selection.md) +- [0006 — `.stack/` is the single stackpanel directory, with one config file and JSON machine data](./0006-single-stack-directory.md) +- [0007 — Deploy backends are options; collisions are guarded and internals shared](./0007-deploy-backends-are-options.md)