From ecf0473e45f664f0e78efb80124424e498f3256e Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 14 Sep 2026 10:55:28 +1000 Subject: [PATCH 1/2] fix(governance): lock the six sizing fields and explain the sizing directory The upgrade classifier in dfe-infra refuses to move these fields without --migrate, and this policy is the write-side refusal. The README says what the resolver commits under sizing/ and why it is committed rather than re-derived. --- governance/policies/sizing-locks.yaml | 27 +++++++++++++++++++++++++++ sizing/README.md | 21 +++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100644 governance/policies/sizing-locks.yaml create mode 100644 sizing/README.md diff --git a/governance/policies/sizing-locks.yaml b/governance/policies/sizing-locks.yaml new file mode 100644 index 0000000..494fce1 --- /dev/null +++ b/governance/policies/sizing-locks.yaml @@ -0,0 +1,27 @@ +name: sizing-locks +description: >- + The six fields the upgrade classifier (dfe-ops upgrade) refuses to move + without an explicit --migrate: each one either strands the data already + written under the old value, or costs nothing less than a cluster rebuild to + change. The reason behind each field is dfe-infra's sizing/sizing.yaml + locked: section; this file is the mechanism that refuses the write. Holders + of helmvars:override can still make a deliberate exception. +protected: + # A keyed topic cannot be repartitioned without breaking hash(key) % + # partitions, so locking the dial inputs blocks the resolver from ever + # shrinking the derived count. + - "infravars:*:kafka.sizing.*" + # local and cached-object put parts in different places, so changing + # either chart's model is a data migration, not a values edit. + - "infravars:*:kafka.storageModel" + - "infravars:*:clickhouse.storageModel" + # Standard to Express is a cluster replacement plus a client cutover, + # not an update. + - "infravars:*:kafka.external.msk.*" + # Combined to separate controllers re-forms the metadata quorum. + - "infravars:*:kafka.controllerPool.enabled" + # The cloud a deployment lives in is not a field; it is a new deployment. + - "infravars:*:cloud" + # az_count has no chart key yet -- terraform/modules/kubernetes-cluster/ + # aws/vpc.tf hardcodes three AZs, so nothing is locked here until it + # becomes a dial field. diff --git a/sizing/README.md b/sizing/README.md new file mode 100644 index 0000000..02e95b7 --- /dev/null +++ b/sizing/README.md @@ -0,0 +1,21 @@ +# sizing/ -- the resolver's committed output for this deployment + +`sizing/resolved.yaml` lands here from a run of dfe-infra's sizing resolver: +the core requirements derived from the ingest estimate, the target-cloud +overlay, the provenance of every ratio it used, the price, the ceiling, and +the six fields `governance/policies/sizing-locks.yaml` refuses to move +without `--migrate`. The values fragments the resolver emits alongside it +merge into `infra/kafka.yaml` and `infra/clickhouse-cluster.yaml`; the tofu +inputs land in `infra/dial.auto.tfvars.json` and +`infra/sizing.auto.tfvars.json`. + +`infra/dial.auto.tfvars.json` and `infra/sizing.auto.tfvars.json` are +committed for the same reason: OpenTofu reads them at apply time, so the node +counts and shapes a re-size produces are visible in the diff before anything +is provisioned, not discovered afterwards from what actually got created. + +All of it is committed, on purpose: an upgrade or a re-size is then a diff +against what is already here, not a re-derivation from scratch, and the +sizing-locks policy has something concrete to compare a proposed change +against. Re-run the resolver and commit what it writes rather than hand-editing +this file. From 4e6e0285b5c2048f733c78e2796e0b07f41c841f Mon Sep 17 00:00:00 2001 From: Derek Date: Sat, 3 Oct 2026 06:15:28 +1000 Subject: [PATCH 2/2] fix(governance): lock only the sizing keys a chart carries storageModel was already held by storage-layout.yaml, and kafka.external.msk.* named the MSK bootstrap Job's settings rather than the broker type, which is an OpenTofu input along with the AZ count. The README now says what resolved.yaml actually records and where the resolver's other outputs go. --- governance/policies/sizing-locks.yaml | 26 +++++--------------------- sizing/README.md | 22 +++++----------------- 2 files changed, 10 insertions(+), 38 deletions(-) diff --git a/governance/policies/sizing-locks.yaml b/governance/policies/sizing-locks.yaml index 494fce1..2fd40e1 100644 --- a/governance/policies/sizing-locks.yaml +++ b/governance/policies/sizing-locks.yaml @@ -1,27 +1,11 @@ name: sizing-locks description: >- - The six fields the upgrade classifier (dfe-ops upgrade) refuses to move - without an explicit --migrate: each one either strands the data already - written under the old value, or costs nothing less than a cluster rebuild to - change. The reason behind each field is dfe-infra's sizing/sizing.yaml - locked: section; this file is the mechanism that refuses the write. Holders - of helmvars:override can still make a deliberate exception. + The sizing fields dfe-infra's resolver refuses to move without --migrate (the locked: section of its sizing/sizing.yaml), held here as protected vars wherever a chart key carries one. Holders of helmvars:override can still make a deliberate exception. protected: - # A keyed topic cannot be repartitioned without breaking hash(key) % - # partitions, so locking the dial inputs blocks the resolver from ever - # shrinking the derived count. + # The partition count derives from these, and a keyed topic cannot be repartitioned without breaking hash(key) % partitions. - "infravars:*:kafka.sizing.*" - # local and cached-object put parts in different places, so changing - # either chart's model is a data migration, not a values edit. - - "infravars:*:kafka.storageModel" - - "infravars:*:clickhouse.storageModel" - # Standard to Express is a cluster replacement plus a client cutover, - # not an update. - - "infravars:*:kafka.external.msk.*" - # Combined to separate controllers re-forms the metadata quorum. + # Combined to separate KRaft controllers re-forms the metadata quorum. - "infravars:*:kafka.controllerPool.enabled" - # The cloud a deployment lives in is not a field; it is a new deployment. + # A deployment moved to another cloud is a new deployment. - "infravars:*:cloud" - # az_count has no chart key yet -- terraform/modules/kubernetes-cluster/ - # aws/vpc.tf hardcodes three AZs, so nothing is locked here until it - # becomes a dial field. + # storageModel is held by storage-layout.yaml, and the MSK broker type and AZ count are OpenTofu inputs with no chart key. diff --git a/sizing/README.md b/sizing/README.md index 02e95b7..5567e3c 100644 --- a/sizing/README.md +++ b/sizing/README.md @@ -1,21 +1,9 @@ # sizing/ -- the resolver's committed output for this deployment -`sizing/resolved.yaml` lands here from a run of dfe-infra's sizing resolver: -the core requirements derived from the ingest estimate, the target-cloud -overlay, the provenance of every ratio it used, the price, the ceiling, and -the six fields `governance/policies/sizing-locks.yaml` refuses to move -without `--migrate`. The values fragments the resolver emits alongside it -merge into `infra/kafka.yaml` and `infra/clickhouse-cluster.yaml`; the tofu -inputs land in `infra/dial.auto.tfvars.json` and -`infra/sizing.auto.tfvars.json`. +`sizing/resolved.yaml` is written by dfe-infra's sizing resolver (`scripts/resolve_sizing.py`) and committed here. It records the tier, focus, cloud, region and ingest estimate the deployment was sized from, its compute cost bucket wherever cloud shapes were resolved, and a `locked:` section holding the current value of each field the resolver refuses to move without `--migrate`: partition count, storage model, MSK broker type, KRaft controller mode, cloud and AZ count. -`infra/dial.auto.tfvars.json` and `infra/sizing.auto.tfvars.json` are -committed for the same reason: OpenTofu reads them at apply time, so the node -counts and shapes a re-size produces are visible in the diff before anything -is provisioned, not discovered afterwards from what actually got created. +It is committed on purpose. `dfe-ops upgrade plan --deploy --dial deployment.yaml --live` re-resolves and diffs the result against this file, so a re-size is checked against what is already deployed rather than derived from scratch. Re-run the resolver and commit what it writes rather than hand-editing this file. -All of it is committed, on purpose: an upgrade or a re-size is then a diff -against what is already here, not a re-derivation from scratch, and the -sizing-locks policy has something concrete to compare a proposed change -against. Re-run the resolver and commit what it writes rather than hand-editing -this file. +The same run writes `sizing/.values.yaml` and `sizing/.report.md`. The `kafka` and `clickhouse` blocks of the values file take effect once they are in `infra/kafka.yaml` and `infra/clickhouse-cluster.yaml`, and the report says what was sized, from which ratio, at which confidence and in which cost bucket. The OpenTofu inputs, `sizing.auto.tfvars.json`, belong beside the OpenTofu root module that reads them. + +`governance/policies/sizing-locks.yaml` is the engine's side of the same locks: it refuses an API edit to the chart keys behind them unless the caller holds `helmvars:override`, and `storage-layout.yaml` does the same for the storage model. The MSK broker type and the AZ count are OpenTofu inputs with no chart key, so only the resolver guards those. Neither policy governs a direct git commit.