Skip to content

verify(trace): promote 105 verification artifacts from EXECUTED gate evidence (TRACE-P02) - #365

Merged
avrabe merged 8 commits into
mainfrom
verify/trace-p02-promote-executed
Sep 11, 2026
Merged

avrabe merged 8 commits into
mainfrom
verify/trace-p02-promote-executed

Conversation

@avrabe

@avrabe avrabe commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

The right side of the V was almost entirely unpromoted — 22 of 247 requirements verified, with the evidence already existing and simply unbound. This binds the part a gate run can actually justify.

Method: executed evidence, not a verifies link

A full sweep ran 323 steps, 0 FAIL. An artifact qualifies only if it had ≥1 step that genuinely executed and none that failed. Skipped steps don't count — bench-only, enforced-by-kani-gate and skip-no-steps are all exclusions, because "the gate did not run it" is not evidence that it passes.

artifacts in the sweep 162
≥1 executed step, zero failures 130 → promoted
only skipped steps 32 → left unpromoted, deliberately
any FAIL 0

Net 105 promoted here (25 were already verified). Verification artifacts now 130 verified / 123 approved / 34 implemented.

A third category TRACE-P02 didn't anticipate — found by doing it

The mechanical sweep initially promoted six artifacts whose own text says they must not be verified:

artifact why it must stay implemented
FV-FALCON-OCI-002 "stays implemented until the fuse is run"
FV-FALCON-OCI-003 wasm.directory criteria (b) and (c) not met
FV-FALCON-OCI-004 kill-criterion owned by gale/meld, unrun
FV-FALCON-OCI-005 ditto
FV-FALCON-PERF-002 criterion (d), the PX4 head-to-head, not done (#362)
FV-FALCON-REL-001 failure injection not run

All six have passing executed steps. The gate result is real. It is also not sufficient, because each names a criterion the gate cannot check — an external kill-criterion, a live-page fetch, a comparison nobody has run.

TRACE-P02 anticipated two populations: gate-runnable (promote) and bench-only (waive). This is a third: gate-passing and self-blocked. Promoting them would have asserted exactly the outcomes those criteria exist to test — the error v1.133 avoided, and which jess's meld run then vindicated by rejecting.

All six reverted. The rule this establishes: an artifact naming an undischarged criterion is excluded from mechanical promotion no matter how green its steps are. Gate evidence is necessary, never sufficient.

Deliberately not promoted

Code-free, per the two-commit rule. rivet validate PASS, exit 0.

🤖 Generated with Claude Code

…evidence (TRACE-P02)

The right side of the V was almost entirely unpromoted: 22 of 247 requirements
`verified`, with the evidence already existing and simply unbound. This binds
the part that a gate run can actually justify.

METHOD — executed evidence, not a `verifies` link. A full sweep
(`--filter '(has-tag "falcon")'`) ran 323 steps with 0 FAIL. An artifact
qualifies only if it had >=1 step that genuinely EXECUTED and none that failed.
Skipped steps do not count: bench-only, `enforced-by-kani-gate` and
`skip-no-steps` are all exclusions, because "the gate did not run it" is not
evidence that it passes.

  artifacts in the sweep              162
  >=1 EXECUTED step, zero failures    130   <- promoted
  only skipped steps                   32   <- left unpromoted, deliberately
  any FAIL                              0

Net: 105 promoted here (the other 25 were already `verified`).
Verification artifacts: 130 verified / 123 approved / 34 implemented.

--- A THIRD CATEGORY TRACE-P02 DID NOT ANTICIPATE, FOUND BY DOING IT ---

The mechanical sweep initially promoted SIX artifacts whose own text says they
must NOT be verified:

  FV-FALCON-OCI-002   "stays `implemented` until the fuse is run"
  FV-FALCON-OCI-003   wasm.directory criteria (b) and (c) NOT MET
  FV-FALCON-OCI-004   kill-criterion owned by gale/meld, unrun
  FV-FALCON-OCI-005   ditto
  FV-FALCON-PERF-002  criterion (d), the PX4 head-to-head, not done (#362)
  FV-FALCON-REL-001   failure injection not run

All six have PASSING executed steps. The gate result is real. It is also NOT
SUFFICIENT, because each names a criterion the gate cannot check — an external
kill-criterion, a live-page fetch, a comparison nobody has run.

TRACE-P02 anticipated two populations: gate-runnable (promote) and bench-only
(waive). This is a third: **gate-passing AND self-blocked**. Promoting them
would have asserted precisely the outcomes those criteria exist to test — the
error v1.133 avoided and that jess's meld run then vindicated by REJECTING.

All six reverted to `implemented`. The rule this establishes: an artifact that
names an undischarged criterion is excluded from mechanical promotion no matter
how green its steps are. Gate evidence is necessary, never sufficient.

WHAT IS DELIBERATELY NOT PROMOTED:
  - 32 artifacts whose steps only skipped (bench-only / kani-enforced)
  - the 6 self-blocked above
  - 45 artifacts with NO steps at all — separately tracked; they cannot be
    evidence as they stand and need steps written or an honest demotion
  - requirements themselves — this promotes VERIFIERS; requirement promotion
    follows once their verifiers are all `verified`

Code-free, per the two-commit rule. rivet validate: PASS, exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe
avrabe enabled auto-merge (squash) August 26, 2026 04:36
@temper-pulseengine

Copy link
Copy Markdown
Contributor

Automated review for PR #365

pulseengine/relay:verify/trace-p02-promote-executed → pulseengine/relay:main

Verdict: 💬 Comment

Summary: The code snippet provided appears to be a list of verification artifacts for a software project, specifically related to the FALCON (Flight Automation and Control Library) HAL (Hardware Abstraction Layer). Each artifact has an ID, type, title, status, and description. The status of each artifact is updated from 'approved' to 'verified'. This suggests that the verification process has been complete

Findings: 0 mechanical (rivet) · 1 from local AI model.

Findings (1):

  1. artifacts/verification/FV-FALCON-NID-002.yaml:14
    status: approved
    
    The status of each artifact is updated from 'approved' to 'verified'. This suggests that the verification process has been complete.

Generated by a local AI model and post-validated against a strict JSON contract. Each finding includes the verbatim line being criticised — verify by reading the file at the cited location.

Reviewed at 295652a

avrabe added a commit that referenced this pull request Sep 10, 2026
…eads as a regression (#383)

PR #365's verification gate failed on one step out of 163:

    [ FAIL (rc=1)] FV-FALCON-WORLD-001: scripts/falcon-hello-demo.sh
      [falcon-hello-demo] vehicle sent 0 heartbeat(s)
      [falcon-hello-demo] FAIL: expected at least 8 heartbeats, got 0
      error: bind 127.0.0.1:14701: Address already in use (os error 98)

Nothing was wrong with the code. The demo hardcoded the 14700/14701 pair, the
gate runners are SHARED, and two jobs on one runner fight for the same socket.

That is the worst shape a flake can take on a REQUIRED check: it fails loudly,
names a plausible-sounding functional symptom ("got 0 heartbeats"), and the
actual cause is one line further down. #365 is a 105-artifact promotion PR, so
the obvious reading was that a promotion had broken something.

FIX: probe for a free port pair instead of assuming one. An explicit
FALCON_HELLO_PORT_BASE still wins, so a bench operator can pin it, and if 64
probes somehow all fail the script falls back to the historical default rather
than erroring — a demo that cannot find a port should still try, not add a
second failure mode.

MEASURED, both directions, three concurrent runs:

    forced to the old fixed base   2 of 3 FAIL
                                   "Address already in use" x3
                                   "expected at least 8 heartbeats, got 0" x2
    with the probe                 3 of 3 PASS, 14 heartbeats each

The negative case is the real bug reproduced on purpose, not a synthetic stand-in
— #365's exact error text, from the exact condition.

Refs #365, #345


Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@avrabe
avrabe disabled auto-merge September 10, 2026 21:16
@avrabe
avrabe enabled auto-merge (squash) September 10, 2026 21:56
@avrabe
avrabe merged commit 519d531 into main Sep 11, 2026
11 checks passed
@avrabe
avrabe deleted the verify/trace-p02-promote-executed branch September 11, 2026 05:56
avrabe added a commit that referenced this pull request Sep 16, 2026
…it in an agent's head (#423)

THE ANSWER TO "do we have a cron loop driving this" was no, and the cost is
visible in the loop's own state file. Its `pending_gates` still said "TAG
falcon-v1.136.0" after falcon-v1.138.0 had shipped. Its watermark sat at
2026-09-10 while twelve new issues were filed. Every gate it was tracking
(#381, #383, #343, #365, #386) was resolved by hand, not by the loop.

A session cron would not have fixed that: those jobs live only as long as the
session. The half that MUST be durable is the half that lives in the repo, so
that is what this adds.

scripts/release-readiness.rs answers one question from the tree: what is the
next release waiting on? Readiness is the query the release plan already
defines — every artifact scoped to the release must be `verified` or `accepted`.
`implemented` deliberately counts as blocking, because the two-commit rule makes
promotion the separate evidence-bearing step, so an `implemented` artifact is
exactly one whose evidence has not been shown.

TWO WRONG DEFAULTS, both caught by running it rather than reasoning about it:

  "lowest release with something incomplete" selected falcon-v0.1.0 — shipped
  long ago, still carrying artifacts left at `implemented`. A shipped release's
  stale statuses are traceability debt, not work the next release waits on.

  "lowest UNTAGGED release" then selected falcon-v1.113.0, which was never
  tagged under that number at all. The next release is the lowest scope AHEAD
  of the latest tag.

That second wrong answer turned out to be worth keeping as a finding: four
scopes exist that were never tagged under their own number —
falcon-v1.113.0, v1.132.0, v1.136.0 and v1.137.0. The last two were folded into
the v1.138.0 cut deliberately; the first two are older drift. The report now
lists them separately, because a release scope going quietly missing is exactly
the failure this tool exists to make visible.

An unparseable artifact file warns and continues rather than silently shrinking
the scope — the empty-scope-passes shape this repo keeps finding.

WHAT IT DOES NOT DO. It never tags, merges, closes or promotes. Cutting a
release carries a signature and a partner consuming the output, and the release
process names tagging an explicit stop-and-ask. This makes that decision
well-informed; it does not make it automatic. The nightly job also never fails
on "not ready" — that is the normal state for most of a cycle, and a job that is
red by default stops being read.

Current output: falcon-v1.139.0, 0 of 7 artifacts done, all `proposed`.


Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant