Skip to content

Report the capture window from feed clocks and pass channel and mode to the eval hook (#98) - #99

Merged
thossullivan merged 4 commits into
mainfrom
feat/capture-window-98
Sep 10, 2026
Merged

thossullivan merged 4 commits into
mainfrom
feat/capture-window-98

Conversation

@thossullivan

Copy link
Copy Markdown
Owner

Closes #98.

model-eol owns the deadline and the eval-hook contract, not migration verification. This is the thin slice from the 2026-09-09 design review, recorded in docs/DESIGN_EVAL_BOUNDARY.md.

What changes

  • Capture window from feed clocks. Every retiring, retired, or scheduled finding gets an in-memory capture window: the applied clock's shutdown as the last day the old model answers, plus every other clock (publisher or distribution row) with a shutdown date after today and a status other than retired. Rows without a date are never listed. The human check line appends [still answers via aws-bedrock until 2026-10-14] when such a clock exists. A tentative or earliest floor renders as at least until.
  • The check --json document does not carry the field. Default output must keep validating against the pinned 0.5.2 schemas (test/fixture/schemas-0.5.2), and the fixture repo already has five findings with alternatives, so no "emit only when informative" rule keeps that pin green. Adding capture to the 0.1 report shape is a separate contract decision, noted in the design doc. SPEC permits additive fields; the PR 91 precedent kept default output byte-compatible. Your call, not mine.
  • Bot bodies. PRs and issues gain a ## Capture window section from the same computation. Distribution rows now join the group digest (order-independent) so a row change refreshes the body, and metadata records capture_expires, the first date on which the section changes, so an expired alternative refreshes the body even though the digest is date-independent. The evaluate and publish phases keep binding the same eval results manifest.
  • Eval hook. MODEL_EOL_VIA (the plan's channel, empty for the publisher API) and MODEL_EOL_EVAL_MODE (evaluate; capture reserved, refused today). Documented in README, docs/DESIGN_BOT.md, and the bot-config schema.
  • Code spans render literally. markdownCode entity-escaped underscores, and code spans show entities literally, so custom_hub rendered as custom_hub. It now drops backticks and line breaks and keeps everything else. Pre-existing; found by review.
  • Digest ordering. sortedJsonDigest used localeCompare, which returns zero for canonically equivalent Unicode strings. It now compares code units.

Operational note

Existing bot PRs and issues will refresh once after this ships, because their recorded feed_digest no longer matches the new digest input.

Verification

  • npm test green at every commit.
  • Cross-repo UAT sweep: ok 861 / retired 659 / retiring 43 / scheduled 1935, identical to the baseline after PR 96; the same four repos fail closed.
  • Acceptance: a repo referencing claude-sonnet-4-20250514 shows RETIRED 2026-06-15 ... [still answers via aws-bedrock until 2026-10-14] on the direct scan and RETIRES 2026-10-14 (35 days) [via aws-bedrock] under --via aws-bedrock, both from the feed.
  • Codex adversarial review: pass 1 found 3, pass 2 found 2, pass 3 found 3; all eight closed with the reviewer's inputs as regression tests. Pass 4 is in flight and will be pushed as a follow-up commit if it finds anything.

Implemented by Codex (gpt-5.6-luna) from a Claude brief; Claude reverted the check-schema change, moved the capture tests to findingFromRef, wrote the design note, and closed the review findings.

…ode to the eval hook (#98)

model-eol owns the deadline and the eval-hook contract, not migration
verification. This adds the thin slice decided in issue 98.

Findings gain an in-memory capture window computed from feed dates only:
the applied clock's shutdown as the last day the old model answers, plus
every other clock (publisher or distribution row) with a shutdown date
after today and a status other than retired. Rows without a date are
never listed. The human check line appends "[still answers via
aws-bedrock until 2026-10-14]" to retiring and retired findings when
such a clock exists. The check JSON document does not carry the field:
the 0.1 report shape published by 0.5.2 is kept byte-compatible for
default output, and adding a field there is a separate contract
decision recorded in docs/DESIGN_EVAL_BOUNDARY.md.

The bot adds a "## Capture window" section to PR and issue bodies from
the same computation and passes MODEL_EOL_VIA (the plan's channel, empty
for the publisher API) and MODEL_EOL_EVAL_MODE (evaluate) to the eval
hook. The hook refuses any mode other than evaluate; capture is reserved
and documented, not implemented.

Implemented by Codex (gpt-5.6-luna) from a Claude brief. Claude removed
the check-schema change Codex made after it broke the pinned 0.5.2
contract test, moved the capture tests to findingFromRef, wrote the
design note, and tightened the docs. Full npm test green.
…erally

Review pass 1 found three gaps in the capture window work.

The capture section reads distribution rows that are not plan items, so
a row changing status, date, or precision left an open PR or issue at
skip-unchanged with a stale "still answers via" line. The group digest
now folds in the entry's shutdown, precision, and every distribution
row's via, shutdown, status, and precision. Existing bot PRs and issues
will refresh once after this ships because their recorded digest no
longer matches.

markdownCode escaped underscores and other characters to HTML entities,
and code spans show entities literally, so a channel or model ID such as
custom_hub rendered as custom_hub. Code spans now drop backticks and
line breaks and keep everything else as typed. A pre-existing test that
asserted the corrupted form now asserts the literal one.

The final capture bullet told the reader to capture "before that date"
even when no clock in the feed still answered. It is now conditional.

Claude implemented these directly from the reviewer's inputs; each is a
regression test. Full npm test green.
…ated clocks

Review pass 2 left two P2 residuals. Distribution rows joined the group
digest in feed order, so a feed that only reordered rows between the
evaluate and publish phases made the publish job refuse the eval results
manifest on a digest mismatch; the rows now pass through the sorted item
digest. The closing capture bullet said no clock still answered whenever
the alternatives list was empty, but an active distribution without a
date is unknown, not closed; the bullet now says the feed lists no dated
clock. Both inputs are regression tests.
… unit

Review pass 3 found that a date-independent digest let an issue or PR keep
"still answers via aws-bedrock until <date>" after that date passed. The
bot metadata now records capture_expires, the first date on which the
capture section changes, and skip-unchanged requires that date to still
be in the future. The digest stays date-independent, so the evaluate and
publish phases keep binding the same eval results manifest.

sortedJsonDigest ordered items with localeCompare, which returns zero for
canonically equivalent Unicode strings, so reversing two such rows changed
the digest. It now compares code units.

The README and the design note said a floor was the last capture day;
they now say a tentative or earliest floor is "at least until".

Regression tests cover PR and issue expiry on the expiry date, the
expiry metadata field, and locale-equal and case-different digest order.
@thossullivan
thossullivan merged commit 7931c77 into main Sep 10, 2026
6 checks passed
@thossullivan
thossullivan deleted the feat/capture-window-98 branch September 10, 2026 03:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Capture window from feeds and two eval-hook env vars; verification stays out of scope

1 participant