Skip to content

DRV-138: own the schema provenance stamp and check - #34

Merged
svc-finitelabs[bot] merged 3 commits into
mainfrom
agent/DRV-138-proto-provenance
Sep 22, 2026
Merged

svc-finitelabs[bot] merged 3 commits into
mainfrom
agent/DRV-138-proto-provenance

Conversation

@svc-finitelabs

@svc-finitelabs svc-finitelabs Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Promotes the proto-schema provenance mechanism out of control4-esphome and into this repo, so every consumer that generates a schema from the library gets it and the prover travels with the release it proves.

Part of DRV-138. Prompted by control4-esphome#131 (DRV-128), which added tools/proto-provenance to the driver repo; that PR is reworked to consume this instead of shipping its own copy.

Why it belongs here

  • It reads local VERSION = "..." from the vendored runtime. That is this repo's convention, injected by git describe in our own build target. There is no ESPHome in it.
  • It stamps over the banner tools/gen_lua_proto_schema emits. Those two strings had to agree across repos and now sit in one.
  • tools/ already carries gen_* and check_*, including check_schema_refs.lua.
  • It enforces an invariant that is ours: the schema, the runtime protobuf.lua that decodes against it, and now the prover all come from one release.

What the tool does

tools/proto-provenance stamp <schema> <runtime> [key=value ...]
tools/proto-provenance check <schema> <runtime> [key=value ...]

stamp writes a header; check asserts three things against files already in the tree.

  1. header generator equals VERSION in the vendored runtime
  2. every caller-supplied key=value equals the header line of that name
  3. header body-sha256 equals a fresh hash of everything below the boundary

No protoc, no Python, no stylua, no network. POSIX shell. A consumer gets the
tool by checking out this repo at the release tag its vendored protobuf.lua
came from, which needs no toolchain, and runs it against its own tree.

Comparison 1 is the matched-pair invariant, carried over from a0780c1c on #131: the generator that emits the schema and the runtime that decodes it ship from one release, and how a field is represented is co-designed with how it is read back. It is not a staleness check with false positives.

What changed in the move

Consumer fields are now opaque. The signature went from three fixed arguments to key=value pairs the tool records and compares without interpreting. control4-esphome passes esphome=2026.8.2; this repo does not know what ESPHome is. generator and body-sha256 are reserved.

An unsupplied field is an error, not a pass. check compares the set of keys recorded in the header against the set supplied on the command line, both directions. Without that, a consumer that dropped esphome= from its Makefile would silently retire comparison 2 while still reporting green. This also closes a header-injection path: a -- protoc: 29.3 line added above the boundary is not covered by the body hash, but it now fails the key-set comparison.

The header format is unchanged. Verified two ways against control4-esphome's committed proto_schema.lua:

  • this tool validates the already-stamped file, exit 0, same output line as the embedded copy
  • re-stamping with each tool produces byte-identical files, and both are byte-identical to the committed bytes

So the consumer rework needs no re-stamp.

Controls

The 13 from #131 move here as this repo's own test of the tool, plus 14 more. make check-provenance, 27 total, each perturbing one thing and restoring before the next.

group controls
the recorded pair clean tree; body edited mid-file; the message names reformatting as a cause; header generator moved back a release; runtime moved forward; consumer field bumped without restamping
loud failures boundary removed; body-sha256 removed; duplicate boundary; runtime VERSION=dev; runtime VERSION not a release tag
hash boundary header-only edit above it stays green; first line below it edited fails; last line of the file edited fails; everything restored
opaque fields recorded field left unsupplied; field supplied that is not recorded; field line injected into the header; reserved key supplied; key supplied twice; empty value; not key=value; zero-field round trip; two-field round trip checked in the other order
idempotence re-stamping leaves the body byte-identical

The hash-boundary group is asserted from both sides on purpose. An always-red or always-green boundary is the failure mode that would make the whole check theatre.

The suite is mutation-tested. Making each comparison vacuous in turn fails exactly the controls that cover it and no others:

mutation controls that go red
comparison 1 always true 2 (generator moved back, runtime moved forward)
key-set check always true 3 (unsupplied, extra, injected)
body-hash check always true 4 (mid-file, reformatting message, first line, last line)
consumer-field compare always true 1 (field bumped)

Without that, 27 green controls would be evidence about the harness rather than about the tool.

Verification

  • make check exit 0, with check-provenance in the chain (make -n check shows the recipe)
  • make test exit 0, 8/8 modules across both math modes
  • controls 27/27 with the tool run under dash as well as sh, because CI's /bin/sh is dash
  • a deliberately broken tool turns make check-provenance red, exit 2, so the target is not decorative

One local note that is not about this branch: make lint fails on this machine because Homebrew's stock lua is 5.5 and luacheck 1.2.0 cannot load under it. Pristine main fails identically, so it is environmental. Re-run under a 5.4 rocks tree it is 0 warnings / 0 errors, which is what CI's pinned 5.4 will do.

Not in this PR

tools/check_option_order in #33 lands at the same Makefile anchor this originally used, so check-provenance is grouped with the other check-* targets instead. git merge-tree against #33's head is conflict-free in both orders.

Releasing is a separate step: control4-esphome cannot vendor this until a tag carries it.

A generated schema is marked "Do not edit manually" and nothing asserted it, so
hand edits and generator drift both landed silently. control4-esphome#131 fixed
that with a provenance header, but put the prover in the driver repo. The script
is consumer-agnostic in shape and reads this repo's own VERSION convention, so it
belongs with the generator that emits the banner it stamps over.

It also enforces an invariant that is this library's to own: the schema, the
runtime protobuf.lua that decodes against it, and now the prover all come from
one release. A consumer vendors the script alongside the runtime, so the two
versions travel together.

Generalized from three fixed arguments to opaque key=value fields. The tool
records and compares whatever a caller passes without interpreting it;
control4-esphome passes esphome=<version>. A field recorded in the header but
not supplied on the command line now fails rather than going uncompared, so a
consumer cannot silently retire a comparison by dropping it from a Makefile.

The header format is unchanged: this tool re-stamps control4-esphome's committed
proto_schema.lua to the same bytes the embedded copy produces, and validates the
already-stamped file without a re-stamp.

The 13 positive controls from #131 move here as the library's own test, plus 14
covering the opaque-field handling and the loud-failure paths for an untagged or
non-tag runtime VERSION. Wired into `make check` as check-provenance: it needs no
venv, no protoc and no network, the same footing a consumer's check runs on.
Its previous slot, immediately above the stylua targets, is the same insertion
point lua-protobuf#33 uses for check-option-order, so the two conflicted on an
anchor rather than on anything semantic.
The paragraph told consumers to vendor `tools/proto-provenance` alongside
`protobuf.lua` and run their own copy. The tool stays here, and a consumer
reaches it by checking out this repo at the release tag its vendored
`protobuf.lua` came from, so the prover and the runtime it asserts against
stay a matched pair.

That also makes "no lua-protobuf checkout" false: the checkout is how the
consumer gets the tool. What `check` actually avoids is regenerating, and
the toolchain and network a regeneration would need.
@svc-finitelabs
svc-finitelabs Bot merged commit 906d4b8 into main Sep 22, 2026
8 checks passed
@svc-finitelabs
svc-finitelabs Bot deleted the agent/DRV-138-proto-provenance branch September 22, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant