From 9168e553c4a83c5fd6faca3c8f9a9b880699b6eb Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Thu, 24 Sep 2026 08:07:44 +0200 Subject: [PATCH 1/2] docs: the owner runs updates; native has no update UI FTW's own work is the EMS and the Energy Planner. The owner operates the host and runs a few documented steps, by hand or through their own automation or agent. ftw update must run unattended, fall back on its own and take a verified full backup before a change to stored data. The same steps are API calls. The native web UI shows the version and the release notice only. ADR 0007 gains decisions 10-14 and adjusts 2-4, 6 and 9: settling period before commit, offline launcher rollback, no rollback point or UI controls on native, the ftw command set, schema-changing updates, install-time files and one native layout for installer, Pi image and migration. The roadmap gets a row with the evidence. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01MuerPFZFG88kgu8sWVHeq7 --- VISION.md | 26 ++++-- docs/adr/0007-self-updating-binary.md | 123 +++++++++++++++++++++----- docs/roadmap.md | 1 + 3 files changed, 121 insertions(+), 29 deletions(-) diff --git a/VISION.md b/VISION.md index a82d8d2c..9c67ab47 100644 --- a/VISION.md +++ b/VISION.md @@ -219,12 +219,26 @@ already works. They should be able to follow a guide, take a backup and apply an update. systemd and a container are both valid ways to run it. The project documents those ways. -The update itself stays small. On a native install, Core downloads the next -verified release and swaps to it. On a container install, the owner pulls the -new image. Either way the owner can see the step and can wrap it. A privileged -sidecar whose job is to update Core is not part of the product. The machinery -around that sidecar is retired as sites leave it. The same ADR records the -shape. +FTW's own work is the EMS and the Energy Planner. The owner operates the +machine it runs on: the service manager, when to update, backups kept off the +box and logs. They can do that by hand or through their own automation or +agent. The project gives them a few documented steps and does not take over +running their host. + +Those steps are short commands on the machine, and the same operations are +available over the API. Each one runs without questions, shows its progress +and ends with a clear result, so it can be wrapped. The web UI shows the +running version, whether a newer release exists and the command that installs +it. On a native install it does not update, roll back or restore. + +The update itself stays small and must be robust enough to run unattended. On +a native install, Core downloads the next verified release and swaps to it. A +new release that does not stay up falls back to the previous one without +operator action. A release that changes stored data first makes a verified +full backup. On a container install, the owner pulls the new image. A +privileged sidecar whose job is to update Core is not part of the product. The +machinery around that sidecar is retired as sites leave it. The same ADR +records the shape. A hosted service, in the spirit of the Sourceful Blixt gateway, is the later path for a household that wants FTW to just work and does not want to operate diff --git a/docs/adr/0007-self-updating-binary.md b/docs/adr/0007-self-updating-binary.md index ca83bd1d..19c74f13 100644 --- a/docs/adr/0007-self-updating-binary.md +++ b/docs/adr/0007-self-updating-binary.md @@ -1,6 +1,8 @@ # ADR 0007: Core updates itself as a static binary -- Status: accepted as direction on 2026-09-23; rollout pending +- Status: accepted as direction on 2026-09-23; amended on 2026-09-24 so the + owner runs updates from the command line or API and native installs have + no update UI (decisions 2–4, 6, 9 and 10–14); rollout pending - Date: 2026-09-18 - Issue: [#1308](https://github.com/srcfl/ftw/issues/1308) - Also decides the version scheme, tracked in @@ -101,29 +103,38 @@ to it. No native update needs a Docker socket or Docker engine.** `current`; else if `next` exists, it renames `next` to `trial` and starts `trial`; else it starts `current`. A Core started from `trial` must reach readiness (API up, state open, drivers loaded) within a bounded time and - then commits by renaming `trial` to `current` and the old `current` to - `previous`. If it crashes or times out, it exits and systemd restarts the - launcher, which falls back. Each rename is atomic on one file system, but + stay ready for a short settling period. It then commits by renaming + `trial` to `current` and the old `current` to `previous`. If it crashes + or times out before that, it exits and systemd restarts the launcher, + which falls back. Each rename is atomic on one file system, but the series is not one atomic operation. The launcher needs durable intent and recovery for a crash between renames. -3. **Rollback is the same swap in reverse.** The UI offers `previous` only - when that binary can read the current state schema. Core renames it to - `next` and exits; the launcher runs it as a trial. The rollback point stays - what #1302 made it: `state.db` and config, taken before every Core update, - never a copy of history. A full data restore is an offline operation. +3. **Rollback is the same swap in reverse.** `ftw rollback` and its API call + offer `previous` only when that binary can read the current state schema. + Core renames it to `next` and exits; the launcher runs it as a trial. When + `current` cannot start at all, the launcher selects `previous` offline + under the same schema rule. A native update keeps current data in place and + takes no separate rollback point; binary rollback keeps that data, and a + release that changes the state schema is covered by decision 13. A full + data restore is an offline operation. -4. **Restart is an exit.** The unit has `Restart=always`. "Restart" in the UI - makes Core shut down cleanly and exit; systemd starts the launcher, which - starts `current`. No image, tag or file is consulted. +4. **Restart is an exit.** The unit has `Restart=always`. When Core needs a + restart, for example after a settings change, it shuts down cleanly and + exits; systemd starts the launcher, which starts `current`. No image, tag + or file is consulted. 5. **No privilege for updates.** The install root belongs to the `ftw` user. Downloading, unpacking, renaming and exiting need nothing else. The host keeps patching itself with `unattended-upgrades` on the Pi image. -6. **The Pi image runs the binary.** [`deploy/ftw.service`](../../deploy/ftw.service) - becomes the shipped unit. The image drops the Docker engine and Compose. - Mosquitto comes from `apt`, on the same host port as today. +6. **The Pi image runs the native install.** It uses the same installer, + launcher and unit as any other host, + [`deploy/ftw-native.service`](../../deploy/ftw-native.service). The image + drops the Docker engine and Compose. Mosquitto comes from `apt`, on the + same host port as today. The monthly image built from `master` still + installs Docker 2.x with the sidecar; it moves to native early, so new + Pi users stop arriving on the old line. 7. **New Docker packaging becomes a plain image.** The Compose file has one Core service and Mosquitto, no sidecar and no `update-ipc` volume. The @@ -141,7 +152,58 @@ to it. No native update needs a Docker socket or Docker engine.** or left support.** A 3.5 version check is not enough: 1.x and 2.x boxes stay on their line until the new installer is proven. Remove each old path after checking the box inventory and its recovery need. Keep the legacy - state-schema marker while any supported reader still needs it. + state-schema marker while any supported reader still needs it. Code on + `master` that only an installed 1.x, 2.x or 3.x box would run is not part + of their migration: those boxes run their installed binaries, and a 2.x + repair builds from its own branch. `master` keeps what the migration and + its way back need. + +10. **The owner operates the host.** FTW's own work is the EMS and the + Energy Planner. The service manager, when to update, copies of backups + off the box and logs belong to the owner, by hand or through their own + automation or agent. The project documents the steps; it does not run + the owner's host. + +11. **`ftw` is the operator command.** The installer puts `ftw` on `PATH`. + It talks only to the local Core's HTTP API, needs no root, asks no + questions and never starts Core: + + - `ftw status`: version, channel, published release, the last update's + result and health, plus where to look next, such as `journalctl -u ftw` + - `ftw update [--channel beta|stable] [--backup-dir DIR]`: install the + next release on the saved channel; already current exits 0 + - `ftw rollback`: return to `previous` under decision 3 + - `ftw backup [--output-dir DIR]`: make, verify and optionally copy a + full backup + - `ftw support`: write the redacted support file + - `ftw help` + + Each command prints its phases and elapsed time, bounds every request and + exits non-zero only when its step failed. Every step is also an API call, + so a script or agent can wrap either. Starting, stopping and logs stay + with systemd. + +12. **Native installs have no update UI.** The web UI shows the running + version and channel, whether a newer release is published, the command + that installs it and the last update's result. It has no update, + rollback, restore, channel, snapshot or backup controls, and the setup + wizard offers no update. Full backups come from `ftw backup` or the API. + +13. **A release that changes the state schema is still one `ftw update`.** + Core first makes and verifies a full backup of the current data, then + installs the release; `--backup-dir` also copies that backup off the + box. The way back across that step is an offline restore of the backup + with the previous release. Until that path exists and is tested, the + native release workflow must refuse a release whose state schema + differs from the one before it. + +14. **Install-time files are part of the release contract.** The installer + writes the launcher, the unit and `ftw`; self-update replaces only + `releases/`. These files must be complete before the first native + user. A release that needs a newer launcher says so and refuses to + prepare, rather than failing its trial. The installer can refresh the + files on an existing native box. Installation, migration and the Pi + image produce one layout: `/opt/ftw` with `ftw-native.service`. ## Versions @@ -190,11 +252,14 @@ anything may change, and that is the true state of FTW. - **The capability handshake and the six-hour readiness budget.** Both existed to protect Core from an older sidecar. There is no sidecar. - **The Compose and `.env` pin.** The launcher reads a directory, not a file. +- **Update, rollback, snapshot and backup controls in the web UI on native.** + The owner, or their agent, runs `ftw` or the API. The UI keeps the version + and the notice that a release exists. +- **The pre-update rollback point on native.** Nothing on native could restore + it, and a schema step now takes a full backup instead (decision 13). ## What is kept, and why -- **The rollback point.** `state.db` and config before every Core update, in - the #1302 form. It is what makes a schema step reversible. - **The readiness gate.** A trial that does not become ready does not become `current`. The existing `/api/status` check is the signal. - **Immutable tags and the release checker.** `selfupdate` keeps resolving @@ -214,7 +279,10 @@ anything may change, and that is the true state of FTW. the table above. - **Added:** download and verify, the slot swap and commit, a launcher script and its tests, and installer support for the binary layout. Rough size: - a few hundred lines of Go and under a hundred of shell. + a few hundred lines of Go and under a hundred of shell. The `ftw` command + is a thin HTTP client beside them. +- **Removed from the native UI:** the Updates dialog, its component, channel, + snapshot and backup sections, and the setup wizard's update offer. - **Existing Docker boxes.** The same guided installer accepts 1.x, 2.x and 3.x without an intermediate update. It first creates and verifies a full backup kept off the box. It preserves config, history, site identity and @@ -247,8 +315,8 @@ anything may change, and that is the true state of FTW. today, but it keeps Compose, `.env`, the project-path coupling and the Docker engine on the Pi. Rejected. - **Debian package and `apt`, the evcc model.** Good for servers and easy to - add later by packaging the same tarball. It gives no UI rollback and no - health-gated commit, so it does not replace the slot model. Deferred. + add later by packaging the same tarball. It gives no health-gated commit + and no automatic fallback, so it does not replace the slot model. Deferred. - **Calendar versions (`2026.9.x`).** They sort after 3.x with the existing comparator and claim nothing about maturity, so they could land on the Docker line today. They drop the signal a minor carries in 0.x: that a @@ -263,10 +331,19 @@ anything may change, and that is the true state of FTW. - Update and rollback on the home box (Raspberry Pi 4, 7.5 GB history) with timings for download, swap and readiness. -- An induced crash during a trial falls back to `current` with no operator - action, and the UI reports it. +- An induced crash during a trial, and one just after readiness, falls back + to `current` with no operator action, and `ftw status` reports it. +- `ftw update` on a box whose new Core takes longer than five minutes to + become ready keeps reporting progress and does not report a failure while + the trial is inside its deadline. +- A release that changes the state schema installs with one `ftw update`, + and the way back from it is tested. +- A script that runs `ftw update` unattended gets the same result and exit + code as a person at the terminal. - Disk use after ten updates stays at the retained slots. - The Docker-to-binary installer path on a box with data in `~/ftw/data`. +- A freshly flashed Pi image comes up as the same native layout and passes + these checks. - A verified full backup copied off the box before cutover, plus a tested restore or return to the old installation after a failed cutover. - Checks that history, site identity, device goals and live device state diff --git a/docs/roadmap.md b/docs/roadmap.md index d56f937a..8bf307a4 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -167,6 +167,7 @@ or reopen closed issues. | Useful analysis and fair savings | Keep enough provenance to explain plans and outcomes. Main savings target compares with ordinary self-consumption on the same installation. | Actual cost reconciles with measured import/export and prices. Specify EV behaviour, initial and final stored-energy accounting, efficiency and coverage. Show missing and negative results. Label the current no-PV/no-battery comparison as total site value until replacement is verified. | | External automation and agent access | Give authorized clients structured analysis data, schedule/goal changes and proposed-plan submission. Temporary external control expires; durable goals persist. Support local access and secure cloud MCP access. | Paired Core/client contract tests cover permissions, expiry, replay, rejection, revocation and reconnect. An agent can trace a request through to measured outcome. Prove local fallback when the caller disappears. Reuse the session/relay where suitable and verify that relay and escrow remain blind. Cloud MCP is a target, not a claim of a shipped endpoint. | | Less configuration, reliable operation | Every normal setting serves a user need. Keep expert controls discoverable. Installation, updates, backup and recovery remain part of the finished experience. | Audit settings and feature use before removal. Test migration of stored choices so hidden settings cannot keep directing behaviour. Verify restart, upgrade and restore on a target box and review affected UI flows. | +| Updates the owner runs | The owner operates the host; FTW supplies the steps. `ftw update` runs unattended, falls back on its own when a new release does not stay up, and makes a verified full backup before a change to stored data. The same steps are API calls, so owners and their agents can wrap them. On native, the web UI shows the version and the release notice only. The installer, the Pi image and the Docker migration produce one native layout. | The evidence list in [ADR 0007](adr/0007-self-updating-binary.md#evidence-required-before-rollout), on the home box and one other site: update and rollback timings, a crash during and just after the trial, a slow first start, a state-schema change and its way back, an unattended run, disk use after ten updates, a freshly flashed Pi image, and a Docker 2.x or 3.x box moved to native with a tested return to its old installation. | Safety is part of each row. Core remains the only dispatch authority, every plan is untrusted input, stale required site-meter data stops dispatch, and From e4ab278ac1f5a567ef7700ac6be3b783521c8822 Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Thu, 24 Sep 2026 08:30:39 +0200 Subject: [PATCH 2/2] docs: a failed schema-changing trial restores the backup first The previous Core cannot safely reopen migrated state, so decision 2's automatic fallback must not start it on that data. The launcher restores the verified backup before it starts previous, and the rollout evidence covers that case. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01MuerPFZFG88kgu8sWVHeq7 --- docs/adr/0007-self-updating-binary.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/adr/0007-self-updating-binary.md b/docs/adr/0007-self-updating-binary.md index 19c74f13..d92f8d4b 100644 --- a/docs/adr/0007-self-updating-binary.md +++ b/docs/adr/0007-self-updating-binary.md @@ -192,10 +192,14 @@ to it. No native update needs a Docker socket or Docker engine.** 13. **A release that changes the state schema is still one `ftw update`.** Core first makes and verifies a full backup of the current data, then installs the release; `--backup-dir` also copies that backup off the - box. The way back across that step is an offline restore of the backup - with the previous release. Until that path exists and is tested, the - native release workflow must refuse a release whose state schema - differs from the one before it. + box. The previous Core cannot safely reopen migrated data, so the + automatic fallback of decision 2 must not start it on that data: when + such a trial fails, the launcher restores the verified backup before it + starts `previous`, without operator action, and `ftw status` reports + both steps. After the release has committed, going back is an offline + restore of that backup with the previous release. Until this path + exists and is tested, the native release workflow must refuse a + release whose state schema differs from the one before it. 14. **Install-time files are part of the release contract.** The installer writes the launcher, the unit and `ftw`; self-update replaces only @@ -336,8 +340,9 @@ anything may change, and that is the true state of FTW. - `ftw update` on a box whose new Core takes longer than five minutes to become ready keeps reporting progress and does not report a failure while the trial is inside its deadline. -- A release that changes the state schema installs with one `ftw update`, - and the way back from it is tested. +- A release that changes the state schema installs with one `ftw update`. + A trial of it that fails after migrating restores the backup and starts + the previous Core on its own data, and the offline way back is tested. - A script that runs `ftw update` unattended gets the same result and exit code as a person at the terminal. - Disk use after ten updates stays at the retained slots.