Skip to content

SCF 2026.2: rename-resilient sheet/column resolution - #3

Merged
sc2ben merged 2 commits into
mainfrom
fix/scf-2026-2-tab-renames
Aug 4, 2026
Merged

sc2ben merged 2 commits into
mainfrom
fix/scf-2026-2-tab-renames

Conversation

@tsunaminoai

Copy link
Copy Markdown
Collaborator

Closes #1. Closes #2.

Upstream renamed the Authoritative Sources tab to Focal Documents and shortened the compensating-controls SCF Control # header to SCF #, breaking the weekly autopilot twice. The gate is fail-fast, so only the tab rename ever reached the filed report — reconciliation in the run artifact was empty.

I measured the blast radius against the real 2026.2 workbook rather than inferring it. Those two renames were the entire code-level break.

Rename resilience

  • SHEET_PATTERNS maps each logical kind to an ordered tuple of accepted names (current first, historical after), so one parser reads every supported release. Sheet.col() gets the same via alts=.
  • Tab names now match with norm_key — the normalization column headers already got — so invisible whitespace drift no longer summons a human.
  • overrides.json gains a resolution section (sheets[] / columns[]), so a rename found in production is a JSON edit, not a release. Operator aliases are tried only after the built-in patterns miss and raise a resolution-alias WARN so they get folded into the parser later; a built-in pattern always wins over an alias.

Latent bugs fixed

These would all have surfaced immediately after the rename fix:

  • Stale catalog. The version-named catalog was never pruned on a bump, and find_catalog takes the first sorted scf-*.json — a leftover scf-2026-1-1.json would have shadowed scf-2026-2.json and published the previous release's data, failing the hermetic gate confusingly with no issue filed.
  • Duplicate issues. The summon step filed a new issue every cron run instead of commenting on the open one (hence SCF 2026.2: validation gate failed #1 and SCF 2026.2: validation gate failed #2), and a failing hermetic gate filed nothing at all. Both fixed.
  • Misleading diagnostics. _structural_failure labelled every ColumnResolveError as sheet-resolve (every such message contains the word "sheet"), str() on these KeyError subclasses emitted the quoted repr — the escaped mess visible in SCF 2026.2: validation gate failed #1 — and the report's workbook_sha256 was hardcoded empty.
  • validate_overrides_file was dead code; it now runs as part of validate.

The MON-03.3 evidence-request typo persists at 2026.2, so its override is no longer version-scoped — autopilot stays hands-off until upstream fixes it, at which point the existing override-unused WARN surfaces it.

Dataset

1,534 controls across 34 families (new QTS, Quantum Security) · 252 crosswalk frameworks · 5,956 AOs · 316 ERLs · 1,371 compensating controls · 42 threats (new MT-28).

Reconciliation is better than 2026.1.1: all 252 source rows match exact, where 2026.1.1 needed fuzzy matching and produced 5 warnings — upstream added explicit year suffixes to framework headers. Crosswalk api_ids come from the Sources sheet FDI column, not header text, so no slugs were renamed by that churn.

Two fields drift heavily but legitimately, verified against the raw sheets rather than assumed: threats on 99.9% of controls (MT-28 maps to nearly everything) and maturity on 77% (prose edits).

Verification

  • nix fmt clean, nix flake check -L green.
  • Gate on 2026.2: {'fail': 0, 'warn': 3, 'resolved': 1, 'ok': True} — the 3 warns are the non-framework MCR/DSR columns.
  • Parsing 2026.1.1 still reproduces the committed 2026.1.1 dataset byte-for-byte — the aliases are additive, not a cutover.
  • Parse twice → byte-identical.
  • Integration tests pass against both real workbooks; ground-truth counts are now a version-keyed table. New fixture defects cover the rename, the header change, whitespace drift, and the operator-alias path.
  • Oracle equivalence stays frozen at 2026.1.1 — the hackIDLE archive has no 2026.2 to compare against.

Follow-up (not in this PR)

praxis/scripts/generate-facts.py:50 hardcodes scf-2026-1-1.json. The catalog filename carries the version, so it is now scf-2026-2.json. Nothing breaks today — praxis, policypress and grc are all pinned to the 2026.1.1 tag — but praxis needs de-hardcoding before it bumps. Noted in CLAUDE.md.

After merge: dispatch Update SCF with tag 2026.2 to confirm autopilot completes unattended.

Upstream renamed the "Authoritative Sources" tab to "Focal Documents" and
shortened the compensating-controls "SCF Control #" header to "SCF #",
breaking the weekly autopilot twice (issues #1, #2). The validation gate is
fail-fast, so only the tab rename ever reached the filed report.

Resolution is now alias-capable rather than single-pattern:

- SHEET_PATTERNS maps each logical kind to an ordered tuple of accepted
  names (current first, historical after), so one parser reads every
  supported release. Sheet.col() takes the same treatment via `alts`.
- Tab names match with norm_key, the normalization column headers already
  got, so invisible whitespace drift no longer summons a human.
- overrides.json gains a `resolution` section carrying sheets[]/columns[]
  fallbacks, so a rename found in production is a JSON edit rather than a
  release. Operator aliases are tried only after the built-in patterns miss
  and raise a resolution-alias WARN, so they get folded into the parser
  later; a built-in pattern always wins over an alias.

Three latent bugs that would have surfaced immediately after the rename fix:

- The version-named catalog was never pruned on a bump. find_catalog takes
  the first sorted scf-*.json, so a leftover scf-2026-1-1.json would have
  shadowed scf-2026-2.json and published the previous release's data.
- The summon-a-human step filed a new issue every cron run instead of
  commenting on the open one (hence two identical issues), and a failing
  hermetic gate filed nothing at all.
- _structural_failure labelled every ColumnResolveError as sheet-resolve
  (every such message contains the word "sheet"), str() on these KeyError
  subclasses emitted the quoted repr, and the report's workbook sha256 was
  hardcoded empty.

validate_overrides_file was dead code; it now runs as part of `validate`.

The MON-03.3 evidence-request typo persists at 2026.2, so its override is no
longer version-scoped — autopilot stays hands-off until upstream fixes it.

Dataset: 1,534 controls across 34 families (new QTS, Quantum Security), 252
crosswalk frameworks, 5,956 AOs, 316 ERLs, 1,371 compensating controls, 42
threats (new MT-28). All 252 source rows now reconcile exact, where 2026.1.1
needed fuzzy matching and produced 5 warnings.

Parsing the 2026.1.1 workbook still reproduces the committed 2026.1.1
dataset byte-for-byte: the aliases are additive, not a cutover.
The new catalog-prune test called `parse` without --report, whose default
is the repo's validation/validation-report.json — so running the suite
overwrote the committed release report with fixture output.

Also regenerate the 2026.2 report with the workbook at the relative path
autopilot uses, instead of a local absolute path. Same sha256, and data/
is byte-identical.
@sc2ben sc2ben added bug Something isn't working documentation Improvements or additions to documentation labels Aug 4, 2026
@sc2ben sc2ben self-assigned this Aug 4, 2026
@sc2ben
sc2ben merged commit 924ae98 into main Aug 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SCF 2026.2: validation gate failed SCF 2026.2: validation gate failed

2 participants