docs(backlog): record that the live-API drift detector is invisible from the file it guards (BACKLOG #1450) - #886
docs(backlog): record that the live-API drift detector is invisible from the file it guards (BACKLOG #1450)#886wshallwshall wants to merge 2 commits into
Conversation
…rom the file it guards (BACKLOG #1450) The detector exists. scripts/ci/check_required_contexts_drift.py reads the public branches endpoint and set-compares it to the checked-in file, as the job `the required-contexts file matches the server`. Neither .github/required-contexts.txt nor tests/test_required_contexts.py names it, so two readers in one night read those files and concluded the drift was undetectable. One of them briefed this item as a structural impossibility. Measured 2026-09-04, each reading with a positive control: server 13 contexts on both endpoints, file 14, pin 14. The drift job carries no continue-on-error, is not a required context, and is paths-filtered on pull_request, so a server-side change reaches only its 07:00 UTC cron on main, which notifies nobody. Recommends AGAINST a live-API arm in the pytest suite: that suite sits inside a required leg and runs offline and on fork PRs, so a fail-closed network call there would red every PR on an API blip. The gap is routing, not detection. The reconciliation itself is PR 884; this Builder stood down on it after the collision registry flagged the overlap. No file in that diff is touched here. The one thing 884 leaves stale is named in the item. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
MANAGER NOTE, per the Manager playbook section 6: I commissioned this diff, so this is a record of what I checked and what I judged, not an independent review. NO WHAT I CHECKED ON THE DIFF:
ON THE COUNTS, because the repository rule says never to write the required-context count into a document and this item does state numbers. My judgment is that these are admissible, and the reasoning is that they are dated MEASUREMENTS rather than standing CLAIMS: each sits in a table beside the command that produced it, so a later reader can re-run it instead of trusting it. The rule exists because a bare count in prose goes stale silently and is then quoted as current. A row reading "server 13, by An item whose entire subject is a drift between two counts cannot avoid naming them. What it can avoid is asserting one as the present state, and this does not. Worth noting the same discipline applied elsewhere in this item: the finding about PR 884's stale header is deliberately written as "lines 18 to 24" rather than as a count, with the author's own note that it is "named rather than numbered so it does not need re-measuring". That is the correct instinct and I would rather record it than let it pass unremarked. A peer retracted a review label earlier today over exactly this defect in its live form: a pull request that wrote the count into workflow comments, where it was both standing and already stale. |
…main stale (BACKLOG #1450) The item said the guard's paths-filtered pull_request trigger means it does not run on an ordinary PR, so a server-side change is invisible to every merge path. The first half is right and the conclusion is wrong. A PR that touches those paths DOES fire it. Measured on PR 884: run created 2026-09-04T23:55:21Z, event=pull_request. What actually stops it reporting the drift is the checkout. The job runs actions/checkout with no ref:, so on a pull_request event it reads the file as that PR has it and compares against the live server. 884 compares 13 against 13 and passes. The run validates a fix; it cannot report that main is stale. "No pull request triggers it" is falsifiable and false. "No pull-request run can see the drift on main" is true and stronger. Caught by the Lander on review of the ledger entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One commit, one file: the
docs/BACKLOG.mdrow for BACKLOG #1450. Nothing else is touched.What changed, and what deliberately did not
I was briefed to reconcile
.github/required-contexts.txtand its pin to the server, and to file an item recording that the guard "is structurally incapable of catching a server-side change". Both halves turned out to be already-owned or wrong, and this PR is what survived the measurement.The reconciliation is PR 884, opened at 23:55:18Z while I was mid-edit. The collision registry flagged it, I read the branch, and I stood down. 884 removes the retired context, moves the pin 14 to 13, and also fixes
docs/CI.md,tests/negative_controls.toml,tests/test_merge_gate_controls.py,tests/test_security_posture.pyandcodeql.yml— six files I never touched. Shipping a competing edit to the same lines of a file 60 open PRs already conflict over would have cost the Lander a hand-merge to gain nothing. No file in 884's diff appears in this PR. I had made those edits; I reverted them.The item's premise was wrong, and the corrected version is the more interesting finding.
scripts/ci/check_required_contexts_drift.pyexists, reads the live server over the public/branches/{branch}endpoint, set-compares rather than count-compares, fails closed, and runs as the jobthe required-contexts file matches the server. A live-API guard is built. What is true is that neither.github/required-contexts.txtnortests/test_required_contexts.pynames it — so a reader asking "what would catch the server moving?" reads both and correctly concludes: nothing. That happened to two readers in one night, one of whom wrote my brief. That is the defect #1450 records.Measurements, each with a positive control
Taken 2026-09-04 between 23:24Z and 23:56Z. A broken instrument returns the same empty output as a clean repo, so every count below has a control beside it.
gh api .../branches/main/protectioncontextsa reviewer has read thisgh api .../branches/main(public) contextsgit show origin/main:.github/required-contexts.txtgrep -c "CI gate"= 4 (non-zero, so the read is live); reviewer line = 2 occurrencestests/test_required_contexts.pypinassert len(contexts) == 14continue-on-errorinrequired-workflow-state.ymlgrep -c runs-on= 2. It goes red on a findingsuccessat 23:25:00Z, read per job, not per run. Later runs queued with no conclusionThat 23:25:00Z green bounds the retirement: the server still held 14 then and held 13 by 23:52Z. The change landed inside that half hour.
I used
MSYS_NO_PATHCONV=1on everygit show origin/main:..., per the trap in my brief.Ruling on the live-API test arm: do not build it
I was asked to assess whether a pytest arm could read the live API and red on drift. Recommendation is against, on three grounds, all recorded in the item.
tests/test_required_contexts.pyruns inside a required test leg. The drift script fails closed by design — right for a scheduled advisory job, wrong for a required one. A transient API failure would red a required check on every open PR at once, for a reason unrelated to any of them.The gap is routing, not detection. Three things sit between a server-side change and a reader: the job blocks nothing, its
pull_request:trigger is paths-filtered so it does not run on an ordinary PR at all, and what remains is a 07:00 UTC cron onmainwhose red notifies nobody — the same shape as #1402 and #1406. The cheap fix is naming the drift job in the two files that hide it. Promoting it to required is an owner decision and I do not recommend it without reconsidering the fail-closed behaviour first.One thing PR 884 leaves stale, named so nobody re-measures it
.github/required-contexts.txtlines 18-24 still read "THE LIST BELOW IS SET-EQUAL TO THE SERVER -- fourteen contexts ... read from the API at 2026-08-31 20:57 CDT". 884 edits only the block near line 202, so on that branch the file will name thirteen contexts under a header asserting fourteen. Set-equality is the single assertion this file exists to carry. It wants the 2026-09-04 date and the thirteen-context reading, and the fix is that paragraph alone. I did not put it here because it cannot land coherently ahead of 884's removal of the context line. It is written into #1450 so it survives 884's author being gone.Checks
Interpreter provenance, proving I read my own worktree and not the primary:
pytest tests/test_required_contexts.pypytest tests/test_backlog_status_check.pyscripts/docs/backlog_status_check.pyscripts/docs/backlog_citation_check.pyscripts/docs/dangling_citation_check.pyruff check .ruff format --check .U+1F6A7(the sanctioned_OPENbanner) andU+00B7(house-style middle dot)Skipped: the full suite, per my brief — many live sessions and
tests/test_connscale_smoke.pyhard-codes a 24-port block (#1014). No pytest output was piped toheadortailbefore exit.Legs only a hosted runner reports, which no Builder sees: the three
testmatrix legs,CI gate's roll-up, and every security scanner. This is a Markdown-only change to one file, so I expect none of them to be load-bearing here, but they still need reading after my process exits.Assumptions, stated because I could not ask
enforce_adminssites alone, as that file's own decision directs.reviewedlabel or arm auto-merge.BACKLOG #1450 is claimed at
scripts/coord/claim.ps1 -Take 1450; release it on merge.🤖 Generated with Claude Code