From 56b4a93440bc1beea8ac2db5fbb493524be76887 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Tue, 1 Sep 2026 16:32:44 +0300 Subject: [PATCH 01/17] chore(spec): define SBOM and VEX build stages Specify deterministic artifact generation, propagation, caching, and cleanup across primary, final, and cache repositories. Ensure platform-specific SBOM and image-level VEX placement while preserving fallback-tag storage semantics. Signed-off-by: Alexandr Zaytsev --- .specify/feature.json | 2 +- .../checklists/requirements.md | 36 ++++ specs/020-sbom-vex-build-stages/spec.md | 194 ++++++++++++++++++ 3 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 specs/020-sbom-vex-build-stages/checklists/requirements.md create mode 100644 specs/020-sbom-vex-build-stages/spec.md diff --git a/.specify/feature.json b/.specify/feature.json index 97b1da2150..2fb32920b5 100644 --- a/.specify/feature.json +++ b/.specify/feature.json @@ -1,3 +1,3 @@ { - "feature_directory": "specs/020-elf-signing-anchor-digest" + "feature_directory": "specs/020-sbom-vex-build-stages" } diff --git a/specs/020-sbom-vex-build-stages/checklists/requirements.md b/specs/020-sbom-vex-build-stages/checklists/requirements.md new file mode 100644 index 0000000000..2f9c9b1b72 --- /dev/null +++ b/specs/020-sbom-vex-build-stages/checklists/requirements.md @@ -0,0 +1,36 @@ +# Specification Quality Checklist: SBOM and VEX as Build Stages + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-09-01 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Reviewed against the attached SBOM/VEX build-stages plan and the existing SBOM/VEX storage specifications. +- The specification intentionally retains storage-contract terms such as fallback tags, image descriptors, and OCI artifacts because they are externally observable compatibility requirements for this feature. +- Clarifications were recorded for VEX descriptor placement, secondary-repository propagation, and the required early error when SBOM/VEX is enabled without a registry destination. diff --git a/specs/020-sbom-vex-build-stages/spec.md b/specs/020-sbom-vex-build-stages/spec.md new file mode 100644 index 0000000000..56e166e471 --- /dev/null +++ b/specs/020-sbom-vex-build-stages/spec.md @@ -0,0 +1,194 @@ +# Feature Specification: SBOM and VEX as Build Stages + +**Feature Branch**: `020-sbom-vex-build-stages` + +**Created**: 2026-09-01 + +**Status**: Draft + +**Input**: User description: "Integrate SBOM and VEX generation into the build-stage lifecycle while preserving OCI artifact storage and ensuring consistent propagation to primary, final, and cache repositories." + +## Project Context + +**Delivery Kit** is a Go CLI tool for full-cycle CI/CD to Kubernetes, built on top of werf with Deckhouse Platform extensions. The feature concerns the build, OCI artifact, registry, SBOM, VEX, and cleanup subsystems: + +- **Build** (`pkg/build/`) — image build lifecycle and stage orchestration +- **SBOM** (`pkg/sbom/`) — SBOM generation, merging, caching, and publication +- **VEX** (`pkg/vex/`) — VEX validation, caching, and publication +- **OCI artifacts** (`pkg/oci/artifact/`) — fallback-tag artifact storage and propagation +- **Registry/storage** (`pkg/storage/`, `pkg/docker_registry/`) — primary, final, and cache repositories +- **Cleanup** (`pkg/cleaning/`) — lifecycle of image and artifact storage + +## Problem Statement + +SBOM and VEX are currently generated in separate post-build passes. Image manifests may be copied to `final-repo` or `cache-repo` before their OCI artifacts are generated, so artifact availability depends on repository options and on whether the image was built locally or restored from cache. SBOM has partial propagation support, while VEX does not consistently follow copied images. + +This creates an unreliable supply-chain record: an image can be available in a destination repository while the SBOM or VEX needed to inspect it is missing there. Multi-platform images add another correctness risk because an artifact can be attached to the wrong descriptor if platform manifests are not resolved explicitly. Stages restored from a `--secondary-repo` can also lose their attached artifacts when copied into primary storage unless artifact propagation is part of the same lifecycle. + +The build lifecycle needs one deterministic artifact flow that preserves the existing fallback-tag storage model without representing artifacts as image layers or changing the user-facing meaning of repository options. Because SBOM and VEX are OCI registry artifacts, enabling either one requires a configured registry destination; local-only builds must fail before image building starts. + +## Clarifications + +### Session 2026-09-01 + +- Q: For multi-platform images, should VEX always be attached to the top-level image index digest, and for single-platform images to the image manifest digest? → A: Option A — multi-platform VEX is attached to the image index digest; single-platform VEX is attached to the image manifest digest. +- Q: Should `--secondary-repo` be included in artifact propagation behavior? → A: Yes — artifacts attached to a suitable stage restored from a secondary repository are copied when that stage is stored in the primary repository. +- Q: What should happen when SBOM or VEX is enabled without any registry destination? → A: Fail early before image building starts with an actionable error requiring a registry destination. + +## User Scenarios & Testing *(mandatory) + +### User Story 1 — Artifacts follow published images (Priority: P1) + +A delivery engineer builds an image with SBOM and/or VEX enabled. The resulting OCI artifacts are available wherever the corresponding image is published: the primary repository, the configured final repository, and the configured cache repositories according to their existing failure policies. When a suitable stage is restored from a `--secondary-repo`, its artifacts follow the stage into primary storage. + +**Why this priority**: Consumers often access the final repository rather than the build repository. Missing attestations make the published image incomplete and undermine vulnerability and compliance workflows. + +**Independent Test**: Build the same fixture with supported combinations of `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo`, then retrieve SBOM and VEX by the image digest from every applicable destination. + +**Acceptance Scenarios**: + +1. **Given** an image with SBOM enabled and a VEX document, **when** it is built with a primary registry repository and without final or cache repositories, **then** both artifacts are available in the primary repository and the build behavior remains successful. +2. **Given** an image with SBOM and VEX enabled and `--final-repo`, **when** the build completes, **then** both artifacts are attached to the image manifest published in the final repository. +3. **Given** an image with SBOM and VEX enabled and one or more `--cache-repo` values, **when** the build completes, **then** artifacts are propagated to each cache repository where the corresponding image is stored, subject to existing cache failure policy. +4. **Given** both `--final-repo` and `--cache-repo`, **when** the build completes, **then** the artifacts are available in both destination classes. +5. **Given** primary and final repository addresses are identical, **when** the build completes, **then** the operation succeeds without creating duplicate artifact copies. +6. **Given** a suitable image stage is restored from `--secondary-repo`, **when** it is copied into primary storage, **then** all attached SBOM and VEX artifacts are copied to the corresponding primary image digest. +7. **Given** SBOM or VEX is enabled without any registry destination, **when** the build command starts, **then** it fails before image building with an actionable error requiring a registry destination. + +--- + +### User Story 2 — Platform-specific artifacts describe the correct image (Priority: P1) + +A delivery engineer builds a multi-platform image. Each platform-specific SBOM describes and is attached to the corresponding platform manifest. One VEX artifact describes the image at image level and is attached to the image index digest. For a single-platform image, VEX is attached to the image manifest digest. Consumers never receive an artifact silently attached to a different platform. + +**Why this priority**: A platform-mismatched SBOM is a false supply-chain statement and can lead to incorrect vulnerability or compliance decisions. + +**Independent Test**: Build a two-platform fixture, inspect the artifact subjects and platform metadata for both platform manifests, and verify the established VEX placement separately. + +**Acceptance Scenarios**: + +1. **Given** a multi-platform image with SBOM enabled, **when** the build completes, **then** each required platform manifest has the SBOM for that platform and its subject identifies that platform manifest. +2. **Given** platform-specific SBOM artifacts, **when** their metadata is inspected, **then** each artifact identifies the platform that was scanned. +3. **Given** a multi-platform image, **when** a platform-specific SBOM is queried, **then** the index digest is not used in place of the requested platform manifest digest. +4. **Given** a multi-platform image with VEX enabled, **when** the build completes, **then** exactly one VEX artifact is attached to the image index digest and no VEX artifact is attached to an individual platform manifest digest. + +--- + +### User Story 3 — Rebuilds reuse or invalidate artifact results correctly (Priority: P1) + +A delivery engineer repeats a build or changes its artifact-generation inputs. Unchanged inputs reuse the existing artifact; changed image content, scanner or merge inputs, VEX content, target platform, or signing identity produces a new artifact identity when that input affects the result. + +**Why this priority**: Incorrect cache reuse silently publishes stale security metadata, while unnecessary regeneration increases build time and registry usage. + +**Independent Test**: Run repeated builds with unchanged inputs, then change one input at a time and inspect cache decisions and artifact identities. + +**Acceptance Scenarios**: + +1. **Given** unchanged image and artifact-generation inputs, **when** the image is rebuilt, **then** the existing SBOM/VEX artifacts are reused and duplicate entries are not created. +2. **Given** a changed scanner option, merge input, or target platform, **when** the image is rebuilt, **then** the affected SBOM artifact is regenerated or republished with a new identity. +3. **Given** a changed VEX document, **when** the image is rebuilt, **then** the VEX artifact is regenerated or republished with a new identity. +4. **Given** a changed signing identity, **when** the image is rebuilt, **then** the affected signed artifact is republished rather than served from an incompatible cache entry. +5. **Given** an image restored from cache before the local build executes, **when** artifact processing runs, **then** the same cache and propagation rules apply as for an image built during the current run. + +--- + +### User Story 4 — Registry failures have predictable consequences (Priority: P2) + +A delivery engineer receives a clear result when artifact publication or propagation encounters a registry error. Final-repository failures do not leave a falsely successful release, while cache-repository failures follow the existing best-effort policy. A local-only build with SBOM or VEX enabled is rejected before any image build work begins. + +**Why this priority**: Operators need to distinguish a missing release artifact from an optional cache mirror problem. + +**Independent Test**: Run builds against an unavailable final repository, an unavailable cache repository, a missing secondary source artifact, and no registry destination, and verify the result and diagnostic behavior. + +**Acceptance Scenarios**: + +1. **Given** a failure while publishing or propagating an artifact to the final repository, **when** the build runs, **then** the build fails with an actionable error. +2. **Given** an unavailable cache repository, **when** the build runs, **then** the build follows the existing cache best-effort policy and reports the skipped propagation clearly. +3. **Given** a missing source artifact during propagation from a secondary repository, **when** the operation runs, **then** it does not silently claim that the destination contains the artifact. +4. **Given** a local-only build with SBOM or VEX enabled, **when** the build command starts, **then** it fails before image building and reports that a registry destination is required. +5. **Given** concurrent artifact attachments for one image digest, **when** all operations complete, **then** existing fallback-index convergence guarantees retain every artifact entry. + +### Edge Cases + +- A registry returns an absent fallback index because the image has no artifacts yet; the operation treats this according to the existing empty-index behavior. +- A final, cache, or primary repository contains an image whose digest differs from the source repository digest; the artifact is attached to the destination image digest, not the source digest. +- A suitable stage is found in a secondary repository but its attached artifact is missing; the primary copy must not be reported as artifact-complete. +- A multi-platform image has only one platform available in a destination; artifacts are propagated only for manifests that are actually present there. +- A configured cache or secondary repository is the same address as the primary repository; no redundant copy is performed. +- An artifact is already present in a destination; repeating the operation is idempotent and does not accumulate duplicate entries. +- Cleanup removes an image while its fallback artifact index remains; the existing orphan cleanup policy removes the resulting orphaned artifact index. +- SBOM or VEX is enabled without a registry destination; the build fails before image building with a clear requirement to configure a registry, rather than silently skipping artifact publication. +- A VEX document is image-level: for a multi-platform image it is attached only to the image index digest, and for a single-platform image it is attached to the image manifest digest; SBOMs use per-platform semantics. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The build MUST process enabled SBOM and VEX generation as part of the same deterministic image publication lifecycle, rather than as independent repository-dependent post-build operations. +- **FR-002**: The build MUST preserve OCI artifact semantics: SBOM and VEX MUST remain separate OCI artifacts and MUST NOT be represented as image layers, filesystem content, or fake container images. +- **FR-003**: For every artifact operation, the build MUST explicitly identify the image descriptor that the artifact describes and the destinations to which the artifact may be propagated. +- **FR-004**: For a single-platform image, an artifact MUST be attached to the digest of the image manifest actually published in that repository. +- **FR-005**: For a multi-platform image, platform-specific SBOMs MUST be attached to their corresponding platform manifest digests; an index digest MUST NOT substitute for a platform manifest digest. +- **FR-006**: VEX MUST be attached to the image manifest digest for a single-platform image and to the top-level image index digest for a multi-platform image; a multi-platform VEX MUST NOT be duplicated onto platform manifest digests. +- **FR-007**: The build MUST propagate SBOM and VEX artifacts from primary storage to the final repository and to configured cache repositories using one shared, idempotent propagation contract. +- **FR-007a**: When a suitable stage is restored from `--secondary-repo` and copied into primary storage, the build MUST propagate all attached SBOM and VEX artifacts to the corresponding primary image digest using the same shared, idempotent propagation contract. +- **FR-008**: Propagation MUST attach an artifact to the digest of the corresponding destination image, including when the source and destination image digests differ. +- **FR-009**: Propagation MUST skip identical source/destination addresses and MUST NOT create duplicate entries for an artifact already present with the same identity. +- **FR-010**: Final-repository publication or propagation failures MUST fail the build; cache-repository failures MUST retain the existing best-effort behavior and be distinguishable in build output. +- **FR-010a**: If SBOM or VEX is enabled and no registry destination is configured, the build MUST fail before image building starts with an actionable error requiring a registry destination. +- **FR-011**: Artifact cache identity MUST include every effective input that can change the corresponding artifact, including image dependency identity, scanner and merge inputs, VEX document content, target platform where applicable, artifact format version, and signer identity where applicable. +- **FR-012**: An unchanged set of effective inputs MUST reuse the existing artifact; changing an effective input MUST prevent a false cache hit and publish the corresponding new artifact identity. +- **FR-013**: The build MUST apply identical artifact processing rules whether an image was built during the current run or restored from a cache repository. +- **FR-014**: The implementation MUST preserve the current fallback-tag storage model, including per-platform artifact storage and existing artifact-to-image digest relationships. +- **FR-015**: Existing fallback-tag artifacts MUST remain readable, and concurrent attachment behavior MUST retain the existing convergence and deduplication guarantees. +- **FR-016**: Cleanup and purge operations MUST continue to remove orphaned artifact indexes in every repository where artifacts can be propagated, without leaving orphaned SBOM or VEX storage as a consequence of this feature. +- **FR-017**: User-facing meanings of `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` MUST remain unchanged except that SBOM and VEX artifacts consistently follow the corresponding published images. +- **FR-018**: The solution MUST NOT require migration to the OCI Referrers API. + +### Key Entities + +- **Image descriptor**: The published image manifest or image index/ platform manifest identity that determines which image an artifact describes. +- **Artifact stage**: A build-lifecycle operation that generates or publishes an OCI artifact without treating that artifact as a container image. +- **SBOM artifact**: An OCI artifact containing the software inventory for an image or platform manifest. +- **VEX artifact**: An OCI artifact containing vulnerability exploitability assessments; it is attached to the image manifest digest for single-platform images and to the top-level image index digest for multi-platform images. +- **Artifact identity**: The artifact type, checksum, platform where applicable, predicate kind, and signer identity needed to distinguish reusable results. +- **Artifact destination**: A primary, final, or cache repository together with the corresponding published image digest. +- **Secondary artifact source**: A secondary repository containing a suitable image stage and its attached artifacts before the stage is copied into primary storage. +- **Fallback artifact index**: The existing per-digest tag-based index that records artifacts attached to an image digest. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: In 100% of test builds covering primary-only, primary-plus-final, primary-plus-cache, primary-plus-final-plus-cache, and secondary-to-primary configurations, every enabled SBOM and VEX artifact is retrievable from each repository where the corresponding image is published. +- **SC-002**: In 100% of two-platform test builds, each platform-specific SBOM has the correct platform subject and metadata, and no platform SBOM is attached only to the top-level index digest. +- **SC-003**: Repeating an unchanged build produces no duplicate artifact entries and records an artifact cache hit for every unchanged artifact. +- **SC-004**: Changing each supported artifact-generation input in isolation causes the affected artifact to miss its cache; unchanged artifact types remain reusable when their own inputs are unchanged. +- **SC-005**: A final-repository propagation failure fails the build in every tested case, while an unavailable cache repository follows the existing best-effort outcome in every tested case. +- **SC-005a**: Every tested build with SBOM or VEX enabled and no registry destination fails before image building starts and reports that a registry destination is required. +- **SC-006**: Repeating propagation against the same destination is idempotent and leaves exactly one entry for each artifact identity, including under concurrent attachment tests. +- **SC-007**: Images restored from cache and images built locally produce equivalent artifact availability and placement for the same effective inputs. +- **SC-008**: Existing fallback-tag artifacts remain readable and existing cleanup tests continue to remove orphaned artifact indexes from primary and propagated repositories. +- **SC-009**: Existing builds without SBOM/VEX configuration and existing user-facing repository options, including `--secondary-repo`, continue to complete without behavior changes unrelated to artifact propagation. + +## Assumptions + +- The current fallback-tag storage model remains the compatibility baseline; no OCI Referrers API migration is needed for this feature. +- SBOM artifacts are platform-specific for multi-platform images. VEX is image-level: it is attached to the image index digest for multi-platform images and to the image manifest digest for single-platform images. +- Registry-level image copies may preserve a digest, while backend-mediated copies may produce a different destination digest; artifact propagation therefore resolves the destination subject explicitly. +- Final repositories are release destinations and are subject to fatal propagation errors; cache repositories remain optional mirrors governed by existing best-effort policy. +- `--secondary-repo` remains a source for restoring suitable stages; artifacts follow a stage when it is copied from secondary storage into primary storage. +- Existing SBOM generation, VEX generation, signing, checksum, fallback-index, and cleanup components are reused unless implementation proves a focused change necessary. +- A repository that is unavailable or has no corresponding image cannot receive an artifact; the resulting behavior follows the destination's established error policy. +- A registry destination is required whenever SBOM or VEX is enabled; local-only artifact publication is not a supported mode. +- No new user-facing flags or configuration syntax are required. + +## Out of Scope + +- Migration from fallback tags to the OCI Referrers API. +- Changing the fallback-tag schema or abandoning per-platform artifact storage. +- Embedding SBOM or VEX data into the image filesystem or image layers. +- Combining all platform SBOMs into a single index-level SBOM. +- Rewriting scanner, CycloneDX, DSSE, signing, or fallback-index subsystems without demonstrated necessity. +- Changing the public semantics of `--repo`, `--final-repo`, `--cache-repo`, or `--secondary-repo` beyond correcting artifact propagation. +- Introducing separate user-configurable cleanup policies for this feature. From 56ca8ac4c10a70a8eda4ecc28b4564ba402f3225 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Tue, 1 Sep 2026 17:07:48 +0300 Subject: [PATCH 02/17] chore(specs): define SBOM and VEX build stages (plan) Describe the staged lifecycle, artifact identity, subject selection, and propagation rules. Capture registry validation and verification scenarios before implementation replaces the post-build convergence path. Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/data-model.md | 59 ++++++ specs/020-sbom-vex-build-stages/plan.md | 187 ++++++++++++++++++ specs/020-sbom-vex-build-stages/quickstart.md | 54 +++++ specs/020-sbom-vex-build-stages/research.md | 71 +++++++ 4 files changed, 371 insertions(+) create mode 100644 specs/020-sbom-vex-build-stages/data-model.md create mode 100644 specs/020-sbom-vex-build-stages/plan.md create mode 100644 specs/020-sbom-vex-build-stages/quickstart.md create mode 100644 specs/020-sbom-vex-build-stages/research.md diff --git a/specs/020-sbom-vex-build-stages/data-model.md b/specs/020-sbom-vex-build-stages/data-model.md new file mode 100644 index 0000000000..cdd627513d --- /dev/null +++ b/specs/020-sbom-vex-build-stages/data-model.md @@ -0,0 +1,59 @@ +# Data Model: SBOM and VEX Build Stages + +## Artifact stage + +An internal build-stage operation associated with an image (and, for SBOM, a target platform). + +| Field | Description | +|---|---| +| Stage name | Stable stage identifier for cache/logging and stage selection. | +| Parent descriptor | The image manifest or image index that the artifact describes. | +| Target platform | Required for platform-specific SBOM; empty for image-level multi-platform VEX. | +| Artifact kind | CycloneDX SBOM or OpenVEX. | +| Generation inputs | Scanner/merge inputs for SBOM, document content for VEX, format version, and signer identity. | +| Mutable/buildable flags | Non-buildable and mutable, matching registry-only stages such as signing. | + +Validation rules: + +- A parent descriptor must be available before `MutateImage` runs. +- SBOM for a multi-platform image must use the corresponding platform manifest. +- VEX must use the platform manifest for single-platform images and the top-level index for multi-platform images. +- An enabled artifact stage requires registry-backed storage. + +## Artifact identity + +The cache identity stored with the existing fallback artifact index. + +| Field | Description | +|---|---| +| Kind and format | Artifact predicate/media type and format version. | +| Parent digest | Digest of the descriptor described by the artifact. | +| Effective inputs | Scanner, merge, GOST, VEX document content, and platform inputs as applicable. | +| Signer identity | Signing fingerprint, or empty for unsigned artifacts. | +| Artifact checksum | Stable checksum used to detect reusable attached artifacts. | + +An identity is reusable only when all effective inputs and the parent descriptor identity match. Repeated publication with the same identity is idempotent. + +## Artifact destination + +A repository and the image descriptor published there. + +| Field | Description | +|---|---| +| Repository address | Primary, final, cache, or secondary repository. | +| Image digest | Destination digest, resolved after image copy. | +| Artifact set | All attached SBOM/VEX artifacts applicable to that descriptor. | +| Failure policy | Fatal for final publication; best effort for cache mirrors. | + +Propagation skips equal source/destination addresses and does not duplicate an already-present artifact identity. + +## Relationships and transitions + +```text +content stage -> artifact stage -> image publication +secondary stage + artifacts -> primary stage + artifacts +primary image + artifacts -> final image + artifacts +primary image + artifacts -> cache image + artifacts +``` + +The artifact stage does not become an image layer. It publishes separate OCI artifacts whose subjects are resolved image descriptors. Existing fallback-tag indexes remain the source of truth and remain readable by current consumers. diff --git a/specs/020-sbom-vex-build-stages/plan.md b/specs/020-sbom-vex-build-stages/plan.md new file mode 100644 index 0000000000..784fb1b0b6 --- /dev/null +++ b/specs/020-sbom-vex-build-stages/plan.md @@ -0,0 +1,187 @@ +# Implementation Plan: SBOM and VEX Build Stages + +**Branch**: `020-sbom-vex-build-stages` | **Date**: 2026-09-01 | **Spec**: [spec.md](./spec.md) + +**Input**: Feature specification from `/specs/020-sbom-vex-build-stages/spec.md` + +## Summary + +Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and into the existing image-stage lifecycle. Add registry-backed, non-buildable mutable artifact stages modeled after `pkg/build/stage/sign.go`. These stages will reuse the current SBOM/VEX convergence, signing, checksum, attestation, fallback-index, and storage implementations while publishing separate OCI artifacts. + +Artifact publication will use explicit source and destination image descriptors. SBOM remains platform-specific; VEX is attached once at the top-level image index for multi-platform images and to the image manifest for single-platform images. A shared idempotent propagation operation will cover primary-to-final, primary-to-cache, and secondary-to-primary copies, resolving the destination digest and preserving fatal final-repository versus best-effort cache error policies. + +## Technical Context + +**Language/Version**: Go 1.24.10 + +**Primary Dependencies**: +- Existing build stage interfaces and lifecycle in `pkg/build/stage`, `pkg/build`, and `pkg/build/conveyor.go`. +- Existing SBOM generation and cache logic in `pkg/build/sbom_step.go` and `pkg/sbom/...`. +- Existing VEX generation and cache logic in `pkg/build/vex_step.go` and `pkg/vex/...`. +- Existing OCI artifact and fallback-index operations in `pkg/oci/artifact` and `pkg/attestation`. +- Existing registry/storage copy operations in `pkg/storage`, `pkg/storage/manager`, and `pkg/docker_registry`. +- Existing signing options in `pkg/build/signing`. +- Ginkgo + Gomega test framework and existing e2e fixtures. + +**Storage**: OCI registry for image manifests/indexes and fallback-tag artifact indexes; local Buildah/container storage remains supported when artifacts are disabled. + +**Testing**: Co-located Ginkgo/Gomega unit tests, existing `test/e2e/sbom` and `test/e2e/vex` suites, and legacy integration tests. + +**Target Platform**: Linux amd64/arm64; single- and multi-platform image builds. + +**Project Type**: Go CLI with a staged image build conveyor. + +**Performance Goals**: Remove duplicate post-build SBOM/VEX passes, preserve stage cache hits, avoid duplicate artifact copies, and retain existing parallel image processing. + +**Constraints**: +- Preserve fallback-tag artifact storage and existing artifact readers. +- Do not represent artifacts as image layers or migrate to OCI Referrers. +- Do not add dependencies or change repository flag semantics. +- Registry destination validation must happen before image building when SBOM/VEX is enabled. +- Final-repository artifact failures are fatal; cache-repository failures remain best effort. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design.* + +- **Simplicity over abstraction**: PASS. Reuse existing `sbomStep`, `vexStep`, stage lifecycle, artifact store, and storage manager. Add one focused shared propagation path rather than separate SBOM/VEX implementations. +- **Go idioms and errors**: PASS. New public methods, if required, take `context.Context` first; errors wrap operation context; stage-specific helpers remain private where possible. +- **Minimal public surface**: PASS. Artifact stages and propagation contracts are internal to `pkg/build`; no new CLI flags or external API are planned. +- **Testing**: PASS. Tests remain alongside source and use Ginkgo/Gomega. E2E coverage extends existing SBOM/VEX suites rather than introducing a parallel harness. +- **Dependencies**: PASS. No external dependency changes. +- **Build boundaries**: PASS. Business logic remains under `pkg/build`, `pkg/oci`, `pkg/storage`, and related packages; no `pkg` dependency on `cmd`. +- **Verification commands**: PASS. Implementation must use `task format`, `task build`, lint prerequisites/lint, unit, scoped e2e, and integration commands. No raw Go tooling. +- **Generated/workflow files**: PASS. No `CHANGELOG.md`, release notes, or CLI reference changes are planned. + +No constitution violations require justification. + +## Research Summary + +Detailed findings are in [research.md](./research.md). Key decisions: + +1. Implement SBOM/VEX as mutable, non-buildable artifact stages attached to the image lifecycle. +2. Use explicit manifest/index subjects and resolve destination subjects after image copies. +3. Share idempotent propagation for SBOM and VEX across final, cache, and secondary-to-primary paths. +4. Retain current checksum inputs and fallback artifact indexes. +5. Validate registry-backed storage before stage execution when either feature is enabled. + +## Design + +### Stage integration + +- Extend `pkg/build/stage` with stage names and constructors for SBOM and VEX, following the shape of `SignStage`. +- Artifact stages must not mutate the image filesystem. Their `PrepareImage` path is a no-op; their registry-side operation is performed through the existing artifact publication logic. +- Ensure stage dependencies include the parent image identity and all effective artifact inputs. SBOM dependencies include scanner, merge/GOST, signer, format version, and target platform. VEX dependencies include document content, parent identity, signer, and format version. +- Register the stages after the content-producing stage and before the lifecycle completes for applicable images. The registration must work for Stapel and Dockerfile image paths and for restored stages. +- Preserve stage cache behavior: a suitable artifact-bearing stage can be selected from primary/secondary storage; changed effective inputs produce a different stage identity. + +### Artifact subjects and platform behavior + +- Single-platform SBOM and VEX target the actual published image manifest digest. +- Multi-platform SBOM processing runs once per platform image and targets that platform manifest digest. +- Multi-platform VEX processing runs once for the image set and targets the top-level image index digest. +- Do not use the index digest as a platform SBOM subject or duplicate image-level VEX onto platform manifests. +- Keep existing signing behavior and include signer identity in cache identity. + +### Publication and propagation + +- Consolidate artifact copying behind a kind-neutral internal operation that copies every attached supported artifact from a source descriptor to a destination descriptor. +- Use it after primary-to-final and primary-to-cache image copies, and when a suitable stage is copied from `--secondary-repo` into primary storage. +- Resolve the destination image descriptor/digest rather than assuming source and destination digests match. +- Skip local storage and identical repository addresses. Deduplicate by existing artifact identity/fallback index semantics. +- Return final-repository propagation errors to fail the build. Log cache-repository propagation failures and continue according to current best-effort behavior. +- Preserve concurrent fallback-index convergence guarantees and existing cleanup behavior. + +### `AfterImages` simplification + +- Remove SBOM/VEX generation calls from the post-build `AfterImages` path once stage execution provides equivalent coverage. +- Retain image metadata publication, final image copying, custom tag publication, telemetry, and report creation in `AfterImages`. +- Avoid retaining a second fallback/post-build convergence path that could regenerate or duplicate artifacts. + +### Early validation + +- Add validation in the earliest build phase before image work starts: if SBOM or VEX is enabled and stage storage is local-only, return an actionable registry-destination error. +- Keep builds with both features disabled unchanged. + +## Project Structure + +### Documentation + +```text +specs/020-sbom-vex-build-stages/ +├── spec.md +├── plan.md +├── research.md +├── data-model.md +├── quickstart.md +└── contracts/ # no external API contract required +``` + +### Expected implementation areas + +```text +pkg/build/stage/ +├── base.go # stage names and shared lifecycle metadata +├── sbom.go # new SBOM artifact stage, if kept separate +├── vex.go # new VEX artifact stage, if kept separate +└── sign.go # existing registry-side stage pattern + +pkg/build/ +├── build_phase.go # stage registration, subject selection, propagation orchestration +├── sbom_step.go # reuse convergence; move shared propagation out or generalize it +├── vex_step.go # reuse convergence; expose stage-compatible operation +└── ... + +pkg/storage/manager/ # secondary/final/cache image-copy artifact propagation hooks +pkg/oci/artifact/ # shared artifact copy/deduplication support if needed + +Tests remain co-located under pkg/build and pkg/build/stage, with scenario coverage in: +test/e2e/sbom/ +test/e2e/vex/ +``` + +The exact file split is intentionally left to implementation if a smaller change can satisfy the same contract; no new package is required. + +## Implementation Phases + +### Phase 0: Research + +Completed in [research.md](./research.md). No unresolved clarification remains. Existing checksum, platform, fallback-index, secondary-repository, and error-policy behavior was identified for reuse. + +### Phase 1: Design + +Completed in [data-model.md](./data-model.md) and [quickstart.md](./quickstart.md). No external interface contract is required because this is an internal build-pipeline change with unchanged CLI syntax and repository option semantics. + +### Phase 2: Implementation preparation + +The subsequent `/speckit-tasks` workflow should decompose at least these work items: + +1. Add SBOM/VEX stage identity and lifecycle integration points. +2. Adapt SBOM convergence to execute through the stage with explicit platform subjects. +3. Adapt VEX convergence to execute through the stage with single/index subject rules. +4. Implement shared artifact propagation for final, cache, and secondary-to-primary copies. +5. Move registry validation before image building and remove duplicate `AfterImages` convergence. +6. Add/adjust unit tests for stage flags, dependency identities, subjects, propagation, idempotency, and failure policies. +7. Extend e2e coverage for repository combinations, secondary restore, multi-platform placement, caching, and local-only rejection. +8. Verify cleanup/orphan behavior and unchanged builds without SBOM/VEX. + +## Validation Plan + +Use the repository-required sequence after implementation: + +```text +task format +task build +task deps:install:golangci-lint +task lint +task test:unit +task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom" +task test:e2e paths="./test/e2e/vex/..." labelFilter="vex" +task test:integration +``` + +While iterating, use scoped `task lint:golangci-lint` and `task test:unit` paths for changed packages. Validate both positive and negative cases: successful registry-backed publication, local-only early failure, final failure, cache best effort, secondary restoration, repeated idempotent propagation, and correct platform subjects. + +## Complexity Tracking + +No constitution violations or new architectural projects are proposed. The only additional internal abstraction is a shared artifact propagation operation because SBOM-only propagation cannot satisfy VEX and secondary-to-primary requirements without duplication. diff --git a/specs/020-sbom-vex-build-stages/quickstart.md b/specs/020-sbom-vex-build-stages/quickstart.md new file mode 100644 index 0000000000..2f0c33f5d4 --- /dev/null +++ b/specs/020-sbom-vex-build-stages/quickstart.md @@ -0,0 +1,54 @@ +# Quickstart Validation: SBOM and VEX Build Stages + +## Prerequisites + +- Linux test environment with Docker, kind, and a writable OCI registry already configured. +- A fixture containing at least one final image and a VEX document. +- `--repo` configured for every build that enables SBOM or VEX. + +## Unit validation + +Run the focused build/stage and artifact tests first: + +```text +task test:unit paths="./pkg/build/..." +task test:unit paths="./pkg/oci/artifact/..." +task test:unit paths="./pkg/vex/..." +``` + +Expected results: + +- Artifact stages calculate stable dependencies and remain non-buildable/mutable. +- Single-platform SBOM/VEX subjects resolve to the manifest digest. +- Multi-platform SBOM subjects resolve per platform and VEX resolves to the index digest. +- Propagation skips identical repositories, deduplicates existing identities, and distinguishes final errors from cache warnings. +- Local-only artifact-enabled builds fail before any image stage is executed. + +## End-to-end validation + +Run the SBOM suite with the feature label/path: + +```text +task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom" +``` + +Cover these repository combinations: + +1. Primary only with SBOM and VEX. +2. Primary plus final repository. +3. Primary plus one or more cache repositories. +4. Primary plus final and cache repositories. +5. Secondary repository restore into primary storage. +6. Identical primary/final/cache addresses. +7. Two-platform image. +8. Unavailable final repository, unavailable cache repository, and local-only artifact-enabled build. + +For each successful case, retrieve artifact descriptors by the actual image digest from every repository containing the image. Verify that repeated builds do not add duplicate fallback-index entries. + +Then run the repository integration suite: + +```text +task test:integration +``` + +Expected outcome: existing builds without SBOM/VEX retain their behavior, and existing cleanup removes orphan fallback artifact indexes in all propagated repositories. diff --git a/specs/020-sbom-vex-build-stages/research.md b/specs/020-sbom-vex-build-stages/research.md new file mode 100644 index 0000000000..189b5c6209 --- /dev/null +++ b/specs/020-sbom-vex-build-stages/research.md @@ -0,0 +1,71 @@ +# Research: SBOM and VEX Build Stages + +## Decision: Move artifact convergence into the image/stage lifecycle + +SBOM and VEX will be represented by non-buildable, mutable build stages that run after the image content stage has produced a registry-backed descriptor. The existing `Stage` lifecycle remains the integration point: stage dependencies determine cache identity, `MutateImage` performs OCI-side publication, and the build phase invokes the stage for each applicable image/platform. + +The stages will reuse the existing `sbomStep`, `vexStep`, `pkg/oci/artifact`, signer implementations, and fallback-tag storage. They will not add layers or filesystem content. `BuildPhase.AfterImages` will retain image publication/report work but will no longer be the primary SBOM/VEX generation pass. + +### Rationale + +- It removes the current repository-dependent post-build pass from `AfterImages`. +- It makes artifact generation part of the same cacheable lifecycle as the descriptor it describes. +- It preserves existing stage cache and secondary-repository restoration behavior. +- It avoids introducing a second artifact storage model or an OCI Referrers migration. + +### Alternatives considered + +- Keep SBOM/VEX in `AfterImages` and improve propagation: rejected because it preserves the extra post-build pass and allows image publication to become detached from artifact processing. +- Encode SBOM/VEX as image layers: rejected by the specification and would change image semantics. +- Migrate to OCI Referrers: rejected as explicitly out of scope and would break the fallback-tag compatibility baseline. + +## Decision: Use explicit artifact subjects for single- and multi-platform images + +For a single-platform image, both SBOM and VEX use the published platform manifest descriptor. For a multi-platform image, each SBOM stage uses its platform manifest descriptor, while one VEX stage uses the top-level image index descriptor. Destination propagation resolves the destination descriptor before attaching artifacts. + +### Rationale + +The repository can contain a destination image with a digest different from the source. Resolving the destination subject prevents an artifact from describing the wrong manifest. It also preserves the established OpenVEX image-level behavior. + +### Alternatives considered + +- Attach every artifact to the content-stage digest without resolving final/index descriptors: rejected for final repositories and multi-platform images. +- Attach VEX to every platform manifest: rejected because VEX is image-level. + +## Decision: Share one propagation contract for all artifact kinds + +Introduce one internal propagation operation that accepts source and destination image descriptors and copies all attached SBOM/VEX artifacts idempotently. It is used for primary-to-final, primary-to-cache, and secondary-to-primary copies. Identical repository addresses are skipped. Final-repository errors are fatal; cache errors retain the existing warning/best-effort policy. + +### Rationale + +The existing `sbomStep.PropagateArtifacts` only names SBOM and is called after image publication. A kind-neutral operation prevents VEX from acquiring different propagation semantics and makes secondary restoration follow the same rules. + +### Alternatives considered + +- Add a second VEX-specific propagation function: rejected because it duplicates destination resolution, deduplication, and error policy. +- Copy artifacts blindly by source digest: rejected because destination image digests may differ. + +## Decision: Keep existing checksum inputs and extend stage dependency identity only where needed + +SBOM keeps its current stable checksum inputs: artifact format, scanner/merge/GOST inputs, signer identity, and target platform, with the image stage digest as the parent identity. VEX keeps document content, parent digest, format version, and signer identity. Stage dependency calculation must include the same effective inputs so a changed input cannot reuse an old artifact stage. + +### Rationale + +Existing checksum logic and tests already encode the required cache behavior. Reusing it minimizes behavioral risk while making stage selection aware of artifact configuration. + +### Alternatives considered + +- Use only the generated artifact bytes as a cache key: rejected because the stage must decide reuse before expensive generation. +- Add a new cache database: rejected as unnecessary and inconsistent with fallback artifact annotations. + +## Decision: Validate registry availability before image building + +When SBOM or VEX is enabled, build initialization validates that the configured stage storage is registry-backed. A local-only build fails before image stages execute with an actionable message requiring `--repo` or disabling artifact generation. + +### Rationale + +Artifacts are OCI registry artifacts and cannot be published to local-only storage. Early validation avoids doing expensive image work that must eventually fail. + +### External dependency assessment + +No new external dependencies are required. Existing registry, attestation, signing, SBOM, VEX, and storage packages are sufficient. From 68764866b17a31ffa4a95808bb7bdbbf3fa20c06 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Tue, 1 Sep 2026 17:10:01 +0300 Subject: [PATCH 03/17] chore(specs): add SBOM and VEX build-stage (tasks) Define phased implementation, testing, propagation, caching, failure handling, and validation work before changing implementation code. Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/tasks.md | 228 +++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 specs/020-sbom-vex-build-stages/tasks.md diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md new file mode 100644 index 0000000000..e0cd981f3b --- /dev/null +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -0,0 +1,228 @@ +# Tasks: SBOM and VEX as Build Stages + +**Input**: Design documents from `specs/020-sbom-vex-build-stages/` + +**Prerequisites**: `plan.md`, `spec.md`, `research.md`, `data-model.md`, `quickstart.md` + +**Tests**: Included because the feature specification requires independent testing for every user story. New tests must use co-located Ginkgo/Gomega suites and existing e2e fixtures. + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Establish the implementation baseline without changing user-facing CLI semantics. + +- [ ] T001 Inventory existing stage, SBOM, VEX, artifact, storage-copy, and cleanup call paths in `pkg/build/`, `pkg/build/stage/`, `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/cleaning/`, recording the concrete integration points in the feature working notes at `specs/020-sbom-vex-build-stages/` +- [ ] T002 [P] Inspect existing SBOM and VEX unit/e2e fixture conventions in `pkg/build/`, `test/e2e/sbom/`, and `test/e2e/vex/` and identify reusable helpers without adding a second test harness +- [ ] T003 [P] Confirm the existing fallback-tag artifact index and cleanup compatibility expectations in `pkg/oci/artifact/` and `pkg/cleaning/` before modifying propagation code + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: Define the minimal internal contracts shared by all artifact-stage stories. + +**Checkpoint**: The shared stage metadata, descriptor propagation shape, and early registry validation are understood and available before story implementation begins. + +- [ ] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including artifact kind, parent descriptor, target platform, mutable flag, and non-buildable flag +- [ ] T005 Define the kind-neutral artifact propagation operation and source/destination descriptor data flow in `pkg/build/` using existing `pkg/oci/artifact/` and `pkg/storage/manager/` primitives, without introducing a public API or new dependency +- [ ] T006 Locate the earliest common build initialization path and specify the registry-backed-storage validation seam in `pkg/build/build_phase.go` for both SBOM-enabled and VEX-enabled builds +- [ ] T007 [P] Add shared test fixtures or helper functions needed to construct manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` + +--- + +## Phase 3: User Story 1 - Artifacts follow published images (Priority: P1) 🎯 MVP + +**Goal**: Generate SBOM/VEX through the image-stage lifecycle and make all attached artifacts follow images into primary, final, cache, and secondary-restored destinations. + +**Independent Test**: Build the existing fixture with primary-only, final, cache, combined final/cache, identical-address, and secondary-repository configurations; retrieve both artifact kinds by each destination image digest. + +### Tests for User Story 1 + +- [ ] T008 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, parent descriptor requirements, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` +- [ ] T009 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` +- [ ] T010 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` +- [ ] T011 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios + +### Implementation for User Story 1 + +- [ ] T012 [P] [US1] Implement the registry-only mutable, non-buildable SBOM artifact stage in `pkg/build/stage/sbom.go` using the existing SBOM generation, signing, checksum, and fallback-index components +- [ ] T013 [P] [US1] Implement the registry-only mutable, non-buildable VEX artifact stage in `pkg/build/stage/vex.go` using the existing VEX generation, signing, checksum, and fallback-index components +- [ ] T014 [US1] Register SBOM and VEX stages after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` +- [ ] T015 [US1] Execute SBOM and VEX stage publication without changing image filesystem or layer content, and remove their duplicate generation pass while retaining unrelated image publication/report work in `pkg/build/build_phase.go` +- [ ] T016 [US1] Implement shared idempotent artifact propagation with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` +- [ ] T017 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the shared propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` +- [ ] T018 [US1] Connect secondary-stage restoration into primary storage to the same propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` + +**Checkpoint**: User Story 1 is independently functional; artifacts are generated in the lifecycle and follow every applicable published image. + +--- + +## Phase 4: User Story 2 - Platform-specific artifacts describe the correct image (Priority: P1) + +**Goal**: Attach per-platform SBOMs to platform manifests and attach exactly one image-level VEX to the correct single-platform manifest or multi-platform index. + +**Independent Test**: Build a two-platform fixture, inspect each artifact subject and platform metadata, and verify that multi-platform VEX appears only on the image index while single-platform VEX uses the manifest. + +### Tests for User Story 2 + +- [ ] T019 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform subject selection in `pkg/build/artifact_subject_test.go` +- [ ] T020 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and parent digest are distinct per platform in `pkg/build/sbom_step_test.go` +- [ ] T021 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each platform manifest +- [ ] T022 [US2] Extend `test/e2e/vex/` with single-platform manifest placement and multi-platform index-only placement assertions + +### Implementation for User Story 2 + +- [ ] T023 [US2] Implement explicit artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` +- [ ] T024 [US2] Pass the target platform and resolved platform manifest descriptor through SBOM stage creation and publication in `pkg/build/sbom_step.go` and `pkg/build/build_phase.go` +- [ ] T025 [US2] Make VEX stage registration run once per multi-platform image set with the top-level index subject, and use the image manifest subject for single-platform builds in `pkg/build/vex_step.go` and `pkg/build/build_phase.go` +- [ ] T026 [US2] Ensure propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` + +**Checkpoint**: User Story 2 is independently testable and no artifact can silently use an index subject for a platform SBOM or duplicate multi-platform VEX onto platform manifests. + +--- + +## Phase 5: User Story 3 - Rebuilds reuse or invalidate artifact results correctly (Priority: P1) + +**Goal**: Preserve valid artifact cache hits while preventing stale reuse when image, scanner, merge/GOST, VEX document, platform, format, or signer inputs change. + +**Independent Test**: Repeat unchanged builds and then change each effective artifact input one at a time; inspect cache decisions, artifact identities, and duplicate fallback-index entries. + +### Tests for User Story 3 + +- [ ] T027 [P] [US3] Add Ginkgo/Gomega tests for SBOM stage dependency identity across image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/sbom_step_test.go` +- [ ] T028 [P] [US3] Add Ginkgo/Gomega tests for VEX stage dependency identity across parent digest, document content, format, and signer inputs in `pkg/build/vex_step_test.go` +- [ ] T029 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing in `pkg/build/artifact_propagation_test.go` +- [ ] T030 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios + +### Implementation for User Story 3 + +- [ ] T031 [US3] Include all effective SBOM inputs and the parent image identity in artifact-stage dependency calculation while preserving existing checksum semantics in `pkg/build/sbom_step.go` and `pkg/build/stage/sbom.go` +- [ ] T032 [US3] Include VEX document content, parent descriptor identity, format version, and signer identity in artifact-stage dependency calculation in `pkg/build/vex_step.go` and `pkg/build/stage/vex.go` +- [ ] T033 [US3] Select reusable artifact-bearing stages from primary and secondary storage using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` +- [ ] T034 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/` and `pkg/build/artifact_propagation.go` + +**Checkpoint**: User Story 3 is independently testable; unchanged inputs reuse artifacts and every effective changed input invalidates only the affected artifact identity. + +--- + +## Phase 6: User Story 4 - Registry failures have predictable consequences (Priority: P2) + +**Goal**: Reject unsupported local-only artifact builds early, fail on final-repository artifact errors, and retain distinguishable best-effort behavior for cache errors. + +**Independent Test**: Exercise unavailable final and cache repositories, missing secondary artifacts, concurrent attachment, and no-registry builds; verify build result and actionable diagnostics. + +### Tests for User Story 4 + +- [ ] T035 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` +- [ ] T036 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` +- [ ] T037 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent attachment in `pkg/oci/artifact/` +- [ ] T038 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior +- [ ] T039 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories + +### Implementation for User Story 4 + +- [ ] T040 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` +- [ ] T041 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [ ] T042 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` +- [ ] T043 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` + +**Checkpoint**: User Story 4 is independently testable; registry failures and local-only configuration produce predictable results without changing repository flag semantics. + +--- + +## Phase 7: Polish & Cross-Cutting Concerns + +**Purpose**: Validate the complete implementation against all feature constraints and repository quality gates. + +- [ ] T044 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic +- [ ] T045 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures +- [ ] T046 Run formatting with `task format` for authored Go directories +- [ ] T047 Run compilation with `task build` +- [ ] T048 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` +- [ ] T049 Run the complete unit suite with `task test:unit` +- [ ] T050 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` +- [ ] T051 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` +- [ ] T052 Run legacy integration coverage with `task test:integration` +- [ ] T053 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Phase 1 (Setup)**: No implementation dependency; establishes the current call-path baseline. +- **Phase 2 (Foundational)**: Depends on Phase 1 and blocks all story implementation. +- **Phase 3 (US1)**: Depends on Phase 2 and is the MVP increment. +- **Phase 4 (US2)**: Depends on US1's artifact stages and propagation path because it specializes subject selection. +- **Phase 5 (US3)**: Depends on US1 and US2 stage identity/subject contracts so cache identity includes the correct parent descriptor. +- **Phase 6 (US4)**: Depends on the shared propagation operation from US1; can be developed in parallel with US2/US3 after the shared path exists. +- **Phase 7 (Polish)**: Depends on all desired stories being complete. + +### User Story Dependencies + +- **US1 (P1)**: Starts after Phase 2; no dependency on another user story. MVP. +- **US2 (P1)**: Depends on US1's stage lifecycle and shared propagation implementation. +- **US3 (P1)**: Depends on US1's stages and US2's explicit parent-subject rules. +- **US4 (P2)**: Depends on US1's propagation/error path; its early-validation work can proceed independently of US2 and US3. + +### Parallel Opportunities + +- Phase 1 tasks T002 and T003 can run in parallel after T001's baseline inventory. +- Within US1, T012 and T013 can run in parallel because they are separate stage files; T008 and T009 can begin as separate test files before implementation. +- Within US2, T019 and T020 are parallel unit-test tasks, and T021/T022 are parallel e2e-suite tasks. +- Within US3, T027 and T028 are parallel because SBOM and VEX identity logic is separate; T030 can proceed independently once stage contracts are stable. +- Within US4, T035, T036, and T037 are parallel test tasks, and T039 can proceed independently in cleanup files. +- After Phase 2, separate contributors can work on US1 stage files, US2 subject tests/design, and US4 validation tests, but US2/US4 integration must wait for the shared US1 propagation contract. +- Polish review and disabled-feature regression checks (T044/T045) can run in parallel before the sequential repository-wide validation commands T046–T053. + +--- + +## Parallel Example: User Story 1 + +```text +# After Phase 2, start independent test and stage work in parallel: +Task: T008 — stage lifecycle tests in pkg/build/stage/artifact_test.go +Task: T009 — propagation tests in pkg/build/artifact_propagation_test.go +Task: T012 — SBOM stage in pkg/build/stage/sbom.go +Task: T013 — VEX stage in pkg/build/stage/vex.go +Task: T011 — SBOM e2e scenarios in test/e2e/sbom/ + +# Then integrate the stage registration and propagation wiring: +Task: T014 — register stages in pkg/build/build_phase.go +Task: T016 — implement propagation in pkg/build/artifact_propagation.go +Task: T017 — wire final/cache copies +Task: T018 — wire secondary restoration +``` + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1 baseline inspection. +2. Complete Phase 2 shared stage and propagation contracts. +3. Implement US1 artifact stages, lifecycle registration, and primary/final/cache/secondary propagation. +4. Run US1 unit and SBOM e2e tests independently. +5. Stop for validation/demo before adding platform-specific and cache-invalidation refinements. + +### Incremental Delivery + +1. Deliver US1 as the first usable increment: artifacts follow published images. +2. Add US2: correct manifest/index subjects and platform placement. +3. Add US3: complete dependency identity and cache reuse/invalidation. +4. Add US4: early validation and explicit final/cache failure behavior. +5. Run the full Polish phase and repository-required validation sequence. + +### Traceability + +- **FR-001–FR-003**: T012–T018, T023–T026 +- **FR-004–FR-006**: T019–T026 +- **FR-007–FR-010**: T009–T018, T036, T040–T042 +- **FR-011–FR-013**: T027–T034 +- **FR-014–FR-016**: T003, T034, T039, T043 +- **FR-017–FR-018**: T045 and all implementation tasks; no CLI flag changes or OCI Referrers migration + +## Notes + +- Every task uses the required `- [ ] T###` checklist format; story tasks include exactly one `[US#]` label and parallel tasks include `[P]` only where file/dependency boundaries permit. +- No external API contracts were provided in `contracts/`; the plan intentionally keeps the propagation contract internal to `pkg/build`. +- No new dependencies, CLI flags, image layers, or OCI Referrers migration are planned. From 12d0c10bd37bc95a387e8f23c724359cbfd3e539 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Tue, 1 Sep 2026 22:48:04 +0300 Subject: [PATCH 04/17] refactor(build, sbom, vex): add SBOM and VEX artifact stages Model SBOM and VEX as mutable, non-buildable stages attached to their parent image. Propagate artifacts to final and cache repositories, validate artifact storage, and clean orphaned cache artifacts. Signed-off-by: Alexandr Zaytsev --- AGENTS.md | 2 - pkg/build/artifact_propagation.go | 77 ++++++++++ pkg/build/artifact_propagation_test.go | 104 +++++++++++++ pkg/build/artifact_subject.go | 63 ++++++++ pkg/build/artifact_subject_test.go | 98 ++++++++++++ pkg/build/build_phase.go | 113 +++++++++----- pkg/build/build_phase_test.go | 44 +++++- pkg/build/sbom_step.go | 26 +--- pkg/build/sbom_step_propagate_test.go | 45 +++++- pkg/build/sbom_step_test.go | 64 ++++++++ pkg/build/stage/artifact_test.go | 62 ++++++++ pkg/build/stage/base.go | 26 ++++ pkg/build/stage/sbom.go | 114 ++++++++++++++ pkg/build/stage/vex.go | 148 +++++++++++++++++++ pkg/cleaning/cleanup.go | 9 ++ pkg/cleaning/cleanup_test.go | 19 +++ specs/020-sbom-vex-build-stages/inventory.md | 16 ++ specs/020-sbom-vex-build-stages/tasks.md | 80 +++++----- 18 files changed, 997 insertions(+), 113 deletions(-) create mode 100644 pkg/build/artifact_propagation.go create mode 100644 pkg/build/artifact_propagation_test.go create mode 100644 pkg/build/artifact_subject.go create mode 100644 pkg/build/artifact_subject_test.go create mode 100644 pkg/build/stage/artifact_test.go create mode 100644 pkg/build/stage/sbom.go create mode 100644 pkg/build/stage/vex.go create mode 100644 specs/020-sbom-vex-build-stages/inventory.md diff --git a/AGENTS.md b/AGENTS.md index e2caa35832..7eda05ef77 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -80,8 +80,6 @@ Correct: `task test:unit paths="./pkg/sbom/..." -- -focus=MyTest` - `task deps:install:pm` — extract the pm binary (linux/amd64) to `dest=` (default `./bin/pm`). Optional. -`format` and `lint*` come from a remote taskfile ([werf/common-ci](https://github.com/werf/common-ci)), so they need `TASK_X_REMOTE_TASKFILES=1` and network access. - ## Verifying changes (MANDATORY) After changing Go code, run these in order — `task format` mutates files, so it goes first: diff --git a/pkg/build/artifact_propagation.go b/pkg/build/artifact_propagation.go new file mode 100644 index 0000000000..9d8d584420 --- /dev/null +++ b/pkg/build/artifact_propagation.go @@ -0,0 +1,77 @@ +package build + +import ( + "context" + "fmt" + + "github.com/werf/logboek" + "github.com/werf/werf/v2/pkg/image" + "github.com/werf/werf/v2/pkg/oci/artifact" + "github.com/werf/werf/v2/pkg/storage" +) + +func ensureAttachedArtifacts(ctx context.Context, repository, digest string) error { + if repository == "" || repository == storage.LocalStorageAddress || digest == "" { + return fmt.Errorf("artifact source descriptor is incomplete") + } + + index, err := artifact.PullFallbackIndex(ctx, repository, digest) + if err != nil { + return fmt.Errorf("check source artifact index %s@%s: %w", repository, digest, err) + } + + manifest, err := index.IndexManifest() + if err != nil { + return fmt.Errorf("read source artifact index %s@%s: %w", repository, digest, err) + } + if len(manifest.Manifests) == 0 { + return fmt.Errorf("source image %s@%s has no attached artifacts", repository, digest) + } + + return nil +} + +func propagateArtifacts(ctx context.Context, projectName, imageName string, source, destination *image.StageDesc, caches []storage.StagesStorage) error { + if source == nil || source.Info == nil { + return fmt.Errorf("source image descriptor is unavailable") + } + if source.Info.Repository == "" || source.Info.Repository == storage.LocalStorageAddress { + return nil + } + + if destination != nil && destination.Info != nil && + destination.Info.Repository != "" && + destination.Info.Repository != storage.LocalStorageAddress && + destination.Info.Repository != source.Info.Repository { + if err := logboek.Context(ctx).Default().LogProcess("image %s: copy artifacts into final repo %s", imageName, destination.Info.Repository).DoError(func() error { + return artifact.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), destination.Info.Repository, destination.Info.GetDigest()) + }); err != nil { + return fmt.Errorf("copy attached artifacts into final repo %s: %w", destination.Info.Repository, err) + } + } + + for _, cache := range caches { + if cache == nil || cache.Address() == storage.LocalStorageAddress || cache.Address() == source.Info.Repository { + continue + } + + destinationDigest := source.Info.GetDigest() + if projectName != "" && source.StageID != nil && source.StageID.Digest != "" { + cacheDesc, err := cache.GetStageDesc(ctx, projectName, *source.StageID) + if err != nil || cacheDesc == nil || cacheDesc.Info == nil { + if err == nil { + err = fmt.Errorf("cache stage descriptor is unavailable") + } + logboek.Context(ctx).Warn().LogF("Warning: unable to resolve destination descriptor in cache stages storage %s: %s\n", cache.String(), err) + continue + } + destinationDigest = cacheDesc.Info.GetDigest() + } + + if err := artifact.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), cache.Address(), destinationDigest); err != nil { + logboek.Context(ctx).Warn().LogF("Warning: unable to copy artifacts into cache stages storage %s: %s\n", cache.String(), err) + } + } + + return nil +} diff --git a/pkg/build/artifact_propagation_test.go b/pkg/build/artifact_propagation_test.go new file mode 100644 index 0000000000..ecf31e29d0 --- /dev/null +++ b/pkg/build/artifact_propagation_test.go @@ -0,0 +1,104 @@ +package build + +import ( + "bytes" + "net/http/httptest" + "strings" + + "github.com/google/go-containerregistry/pkg/authn" + "github.com/google/go-containerregistry/pkg/name" + "github.com/google/go-containerregistry/pkg/registry" + "github.com/google/go-containerregistry/pkg/v1/random" + "github.com/google/go-containerregistry/pkg/v1/remote" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "go.uber.org/mock/gomock" + + "github.com/werf/logboek" + "github.com/werf/werf/v2/pkg/attestation" + "github.com/werf/werf/v2/pkg/docker_registry" + "github.com/werf/werf/v2/pkg/image" + "github.com/werf/werf/v2/pkg/oci/artifact" + "github.com/werf/werf/v2/pkg/storage" + "github.com/werf/werf/v2/test/mock" +) + +var _ = Describe("artifact propagation", func() { + It("rejects an incomplete artifact source descriptor", func(ctx SpecContext) { + err := ensureAttachedArtifacts(ctx, "", "") + + Expect(err).To(MatchError("artifact source descriptor is incomplete")) + }) + + It("rejects a nil source descriptor", func(ctx SpecContext) { + err := propagateArtifacts(ctx, "project", "app", nil, nil, nil) + + Expect(err).To(MatchError("source image descriptor is unavailable")) + }) + + It("skips local-only artifact sources", func(ctx SpecContext) { + err := propagateArtifacts(ctx, "project", "app", &image.StageDesc{ + Info: &image.Info{Repository: ":local", RepoDigest: ":local@sha256:local"}, + }, nil, nil) + + Expect(err).To(Succeed()) + }) + + Describe("propagation errors", func() { + var ( + server *httptest.Server + sourceRepo string + sourceDigest string + remoteOpts []remote.Option + ) + + stageDescFor := func(repo, digest string) *image.StageDesc { + return &image.StageDesc{Info: &image.Info{ + Repository: repo, + RepoDigest: repo + "@" + digest, + }} + } + + BeforeEach(func(ctx SpecContext) { + Expect(docker_registry.Init(ctx, false, false, nil, nil)).To(Succeed()) + + server = httptest.NewServer(registry.New()) + host := strings.TrimPrefix(server.URL, "http://") + sourceRepo = host + "/test/source" + remoteOpts = []remote.Option{remote.WithAuth(authn.Anonymous)} + + img, err := random.Image(256, 1) + Expect(err).To(Succeed()) + ref, err := name.NewTag(sourceRepo + ":v1") + Expect(err).To(Succeed()) + Expect(remote.Write(ref, img, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...)).To(Succeed()) + digest, err := img.Digest() + Expect(err).To(Succeed()) + sourceDigest = digest.String() + + store := artifact.NewOCIStore(sourceRepo, "app", remoteOpts...) + Expect(store.Attach(ctx, sourceDigest, attestation.DSSEMediaType, []byte(`{"v":1}`), "checksum-v1", "", "")).To(Succeed()) + }) + + AfterEach(func() { + server.Close() + }) + + It("returns a final propagation error", func(ctx SpecContext) { + err := propagateArtifacts(ctx, "project", "app", stageDescFor(sourceRepo, sourceDigest), stageDescFor("127.0.0.1:1/unreachable/final", sourceDigest), nil) + + Expect(err).To(MatchError(ContainSubstring("copy attached artifacts into final repo"))) + }) + + It("logs cache propagation errors and continues", func(ctx SpecContext) { + var output bytes.Buffer + logCtx := logboek.NewContext(ctx, logboek.NewLogger(&output, &output)) + cache := mock.NewMockStagesStorage(gomock.NewController(GinkgoT())) + cache.EXPECT().Address().Return("127.0.0.1:1/unreachable/cache").AnyTimes() + cache.EXPECT().String().Return("127.0.0.1:1/unreachable/cache").AnyTimes() + + Expect(propagateArtifacts(logCtx, "", "app", stageDescFor(sourceRepo, sourceDigest), nil, []storage.StagesStorage{cache})).To(Succeed()) + Expect(output.String()).To(ContainSubstring("Warning: unable to copy artifacts into cache stages storage")) + }) + }) +}) diff --git a/pkg/build/artifact_subject.go b/pkg/build/artifact_subject.go new file mode 100644 index 0000000000..c0b75d265b --- /dev/null +++ b/pkg/build/artifact_subject.go @@ -0,0 +1,63 @@ +package build + +import ( + "github.com/werf/werf/v2/pkg/build/image" + imagePkg "github.com/werf/werf/v2/pkg/image" +) + +func contentStageDesc(img *image.Image) *imagePkg.StageDesc { + if img == nil { + return nil + } + + lastStage := img.GetLastNonEmptyStage() + if lastStage == nil || lastStage.GetStageImage() == nil || lastStage.GetStageImage().Image == nil { + return img.GetContentTagDesc() + } + + return lastStage.GetStageImage().Image.GetStageDesc() +} + +func finalStageDescForImage(phase *BuildPhase, name string, images []*image.Image) *imagePkg.StageDesc { + if len(images) == 1 { + if images[0] == nil { + return nil + } + return images[0].GetContentTagDesc() + } + + if phase == nil || phase.Conveyor == nil || phase.Conveyor.imagesTree == nil { + return nil + } + + if multiImg := phase.Conveyor.imagesTree.GetMultiplatformImage(name); multiImg != nil { + return multiImg.GetFinalStageDesc() + } + + return nil +} + +func vexTargetPlatform(images []*image.Image) string { + if len(images) != 1 || images[0] == nil { + return "" + } + + return images[0].TargetPlatform +} + +func finalStageDescForPlatform(phase *BuildPhase, name string, images []*image.Image, targetPlatform string) *imagePkg.StageDesc { + finalStageDesc := finalStageDescForImage(phase, name, images) + if finalStageDesc == nil || finalStageDesc.Info == nil || !finalStageDesc.Info.IsIndex { + return finalStageDesc + } + + for index, img := range images { + if img.TargetPlatform != targetPlatform || index >= len(finalStageDesc.Info.Index) { + continue + } + + return &imagePkg.StageDesc{Info: finalStageDesc.Info.Index[index]} + } + + return nil +} diff --git a/pkg/build/artifact_subject_test.go b/pkg/build/artifact_subject_test.go new file mode 100644 index 0000000000..8bfb3fa94f --- /dev/null +++ b/pkg/build/artifact_subject_test.go @@ -0,0 +1,98 @@ +package build + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/werf/werf/v2/pkg/build/image" + imagePkg "github.com/werf/werf/v2/pkg/image" +) + +var _ = Describe("artifact subjects", func() { + newImage := func(name, platform, digest string) *image.Image { + img := &image.Image{Name: name, TargetPlatform: platform} + img.SetContentTagDesc(&imagePkg.StageDesc{ + StageID: imagePkg.NewStageID(digest, 1), + Info: &imagePkg.Info{Repository: "primary", RepoDigest: "primary@" + digest}, + }) + return img + } + + It("selects the matching platform manifest for a multi-platform SBOM", func() { + images := []*image.Image{ + newImage("app", "linux/amd64", "sha256:amd64"), + newImage("app", "linux/arm64", "sha256:arm64"), + } + multiImage := image.NewMultiplatformImage("app", images, 0, 1) + multiImage.SetFinalStageDesc(&imagePkg.StageDesc{ + Info: &imagePkg.Info{ + IsIndex: true, + Index: []*imagePkg.Info{ + {Repository: "final", RepoDigest: "final@sha256:amd64"}, + {Repository: "final", RepoDigest: "final@sha256:arm64"}, + }, + }, + }) + tree := image.NewImagesTree(nil, image.ImagesTreeOptions{}) + tree.SetMultiplatformImage(multiImage) + phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{imagesTree: tree}}} + + descriptor := finalStageDescForPlatform(phase, "app", images, "linux/arm64") + + Expect(descriptor).NotTo(BeNil()) + Expect(descriptor.Info.GetDigest()).To(Equal("sha256:arm64")) + }) + + It("keeps the top-level index as the VEX subject for a multi-platform image", func() { + images := []*image.Image{ + newImage("app", "linux/amd64", "sha256:amd64"), + newImage("app", "linux/arm64", "sha256:arm64"), + } + multiImage := image.NewMultiplatformImage("app", images, 0, 1) + index := &imagePkg.StageDesc{Info: &imagePkg.Info{ + IsIndex: true, + Repository: "final", + RepoDigest: "final@sha256:index", + Index: []*imagePkg.Info{{RepoDigest: "final@sha256:amd64"}, {RepoDigest: "final@sha256:arm64"}}, + }} + multiImage.SetFinalStageDesc(index) + tree := image.NewImagesTree(nil, image.ImagesTreeOptions{}) + tree.SetMultiplatformImage(multiImage) + phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{imagesTree: tree}}} + + descriptor := finalStageDescForImage(phase, "app", images) + + Expect(descriptor).To(BeIdenticalTo(index)) + Expect(descriptor.Info.GetDigest()).To(Equal("sha256:index")) + }) + + It("uses no platform annotation for a multi-platform VEX artifact", func() { + images := []*image.Image{ + {TargetPlatform: "linux/amd64"}, + {TargetPlatform: "linux/arm64"}, + } + + Expect(vexTargetPlatform(images)).To(BeEmpty()) + }) + + It("uses the manifest platform for a single-platform VEX artifact", func() { + Expect(vexTargetPlatform([]*image.Image{{TargetPlatform: "linux/amd64"}})).To(Equal("linux/amd64")) + }) + + It("returns no final subject when a multi-platform tree is unavailable", func() { + images := []*image.Image{newImage("app", "linux/amd64", "sha256:amd64"), newImage("app", "linux/arm64", "sha256:arm64")} + phase := &BuildPhase{} + + Expect(finalStageDescForImage(phase, "app", images)).To(BeNil()) + }) + + It("uses the published manifest as the final subject for a single-platform image", func() { + images := []*image.Image{newImage("app", "linux/amd64", "sha256:amd64")} + phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{}}} + + descriptor := finalStageDescForPlatform(phase, "app", images, "linux/amd64") + + Expect(descriptor).To(BeIdenticalTo(images[0].GetContentTagDesc())) + Expect(descriptor.Info.GetDigest()).To(Equal("sha256:amd64")) + }) +}) diff --git a/pkg/build/build_phase.go b/pkg/build/build_phase.go index a7ed5cec43..9fa22a7ae4 100644 --- a/pkg/build/build_phase.go +++ b/pkg/build/build_phase.go @@ -126,6 +126,10 @@ func (phase *BuildPhase) Name() string { } func (phase *BuildPhase) BeforeImages(ctx context.Context) error { + if err := validateArtifactStorage(phase.Conveyor.StorageManager, phase.artifactsEnabled()); err != nil { + return err + } + if err := phase.Conveyor.StorageManager.InitCache(ctx); err != nil { return fmt.Errorf("unable to init storage manager cache: %w", err) } @@ -265,6 +269,40 @@ func (phase *BuildPhase) AfterImages(ctx context.Context) error { return phase.createReport(ctx, imagesPairs) } +func validateArtifactStorage(storageManager manager.StorageManagerInterface, artifactsEnabled bool) error { + if !artifactsEnabled { + return nil + } + if _, isLocal := storageManager.GetStagesStorage().(*storage.LocalStagesStorage); isLocal { + return fmt.Errorf("SBOM or VEX generation requires a container registry (specify --repo), or disable artifact generation") + } + return nil +} + +func (phase *BuildPhase) artifactsEnabled() bool { + if phase.Conveyor.EnableSbom() { + return true + } + + if phase.Conveyor.imagesTree == nil { + return false + } + + for _, pair := range phase.Conveyor.imagesTree.GetImagesByName(false) { + _, images := pair.Unpair() + for _, img := range images { + if img == nil { + continue + } + if vex := img.Vex(); vex != nil && vex.Document != "" { + return true + } + } + } + + return false +} + func (phase *BuildPhase) convergeSbomByImagesSets(ctx context.Context) error { if !phase.Conveyor.EnableSbom() { return nil @@ -275,10 +313,6 @@ func (phase *BuildPhase) convergeSbomByImagesSets(ctx context.Context) error { return nil } - if _, isLocal := phase.Conveyor.StorageManager.GetStagesStorage().(*storage.LocalStagesStorage); isLocal { - return fmt.Errorf("SBOM generation requires a container registry (specify --repo). Use --repo to enable SBOM or disable SBOM in the werf config (build.sbom.enable)") - } - tracker := convergefailure.NewTracker(os.Getenv(externalref.EnvName)) phase.sbomFailures = tracker @@ -350,8 +384,8 @@ func (phase *BuildPhase) convergeImageSbom(ctx context.Context, name string, ima signerIdentity = phase.SbomSigningOptions.Signer().Fingerprint() } - finalStageDesc := phase.finalStageDescForImage(name, images) for _, img := range images { + finalStageDesc := finalStageDescForPlatform(phase, name, images, img.TargetPlatform) if err := phase.convergePlatformImageSbom(ctx, name, img, finalStageDesc, signer, signerIdentity, breaker); err != nil { return err } @@ -361,7 +395,7 @@ func (phase *BuildPhase) convergeImageSbom(ctx context.Context, name string, ima } func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name string, img *image.Image, finalStageDesc *imagePkg.StageDesc, signer signature.Signer, signerIdentity string, breaker *externalref.ResolverBreaker) error { - stageDesc := img.GetLastNonEmptyStageDesc() + stageDesc := contentStageDesc(img) if stageDesc == nil { return fmt.Errorf("unable to converge sbom for image %q: stage descriptor is unavailable", name) } @@ -424,28 +458,13 @@ func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name str return fmt.Errorf("unable to converge sbom for image %q: %w", name, err) } - if err := phase.sbomStep.PropagateArtifacts(ctx, name, stageDesc, finalStageDesc, phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + if err := phase.sbomStep.PropagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDesc, phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { return fmt.Errorf("unable to propagate sbom for image %q: %w", name, err) } return nil } -// finalStageDescForImage returns the final repo descriptor to copy the SBOM artifacts into, or nil -// when there is nothing to copy. A single-platform image never has one: publishFinalImage stores the -// final repo descriptor in the content tag desc, which convergeImageSbom already uses as the SBOM -// target. Reaching for the last non-empty stage here instead panics, because an image resolved from -// the cache short-circuits in BeforeImageStages and never gets one. -func (phase *BuildPhase) finalStageDescForImage(name string, images []*image.Image) *imagePkg.StageDesc { - if len(images) == 1 { - return nil - } - if multiImg := phase.Conveyor.imagesTree.GetMultiplatformImage(name); multiImg != nil { - return multiImg.GetFinalStageDesc() - } - return nil -} - func (phase *BuildPhase) scanOptionsForImage(img *image.Image) scanner.ScanOptions { scanOpts := scanner.DefaultSyftScanOptions() @@ -1073,32 +1092,44 @@ func (phase *BuildPhase) findAndFetchStageFromSecondaryStagesStorage(ctx context storageManager := phase.Conveyor.StorageManager atomicCopySuitableStageFromSecondaryStagesStorage := func(secondaryStageDesc *imagePkg.StageDesc, secondaryStagesStorage storage.StagesStorage) error { + var stageDescCopy *imagePkg.StageDesc err := logboek.Context(ctx).Default().LogProcess("Copy suitable stage from secondary %s", secondaryStagesStorage.String()).DoError(func() error { - if stageDescCopy, err := storageManager.CopySuitableStageDescByDigest(ctx, secondaryStageDesc, secondaryStagesStorage, storageManager.GetStagesStorage(), phase.Conveyor.ContainerBackend, img.TargetPlatform); err != nil { + var err error + stageDescCopy, err = storageManager.CopySuitableStageDescByDigest(ctx, secondaryStageDesc, secondaryStagesStorage, storageManager.GetStagesStorage(), phase.Conveyor.ContainerBackend, img.TargetPlatform) + if err != nil { return fmt.Errorf("unable to copy suitable stage %s from %s to %s: %w", secondaryStageDesc.StageID.String(), secondaryStagesStorage.String(), storageManager.GetStagesStorage().String(), err) - } else { - i := phase.Conveyor.GetOrCreateStageImage(stageDescCopy.Info.Name, phase.StagesIterator.GetPrevImage(img, stg), stg, img) - i.Image.SetStageDesc(stageDescCopy) - stg.SetStageImage(i) - - // The stage digest remains the same, but the content digest may differ (e.g., the content digest of git and some user stages depends on the git commit). - contentDigest, exist := stageDescCopy.Info.Labels[imagePkg.WerfStageContentDigestLabel] - if exist { - stg.SetContentDigest(contentDigest) - } else { - panic(fmt.Sprintf("expected stage %q content digest label to be set!", stg.Name())) - } + } - logboek.Context(ctx).Default().LogFHighlight("Use previously built image for %s\n", stg.LogDetailedName()) - container_backend.LogImageInfo(ctx, stg.GetStageImage().Image, phase.getPrevNonEmptyStageImageSize(), img.ShouldLogPlatform(), phase.getLogImageNetwork(img)) + i := phase.Conveyor.GetOrCreateStageImage(stageDescCopy.Info.Name, phase.StagesIterator.GetPrevImage(img, stg), stg, img) + i.Image.SetStageDesc(stageDescCopy) + stg.SetStageImage(i) - return nil + // The stage digest remains the same, but the content digest may differ (e.g., the content digest of git and some user stages depends on the git commit). + contentDigest, exist := stageDescCopy.Info.Labels[imagePkg.WerfStageContentDigestLabel] + if exist { + stg.SetContentDigest(contentDigest) + } else { + panic(fmt.Sprintf("expected stage %q content digest label to be set!", stg.Name())) } + + logboek.Context(ctx).Default().LogFHighlight("Use previously built image for %s\n", stg.LogDetailedName()) + container_backend.LogImageInfo(ctx, stg.GetStageImage().Image, phase.getPrevNonEmptyStageImageSize(), img.ShouldLogPlatform(), phase.getLogImageNetwork(img)) + + return nil }) if err != nil { return err } + if phase.artifactsEnabled() { + if err := ensureAttachedArtifacts(ctx, secondaryStageDesc.Info.Repository, secondaryStageDesc.Info.GetDigest()); err != nil { + return fmt.Errorf("secondary stage %s has incomplete artifacts: %w", secondaryStageDesc.StageID.String(), err) + } + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), img.Name, secondaryStageDesc, stageDescCopy, storageManager.GetCacheStagesStorageList()); err != nil { + return fmt.Errorf("unable to propagate artifacts restored from secondary storage: %w", err) + } + } + if err := storageManager.CopyStageIntoCacheStorages( ctx, *stg.GetStageImage().Image.GetStageDesc().StageID, storageManager.GetCacheStagesStorageList(), @@ -1778,10 +1809,14 @@ func (phase *BuildPhase) convergeImageVex(ctx context.Context, name string, imag signerIdentity = phase.VexSigningOptions.Signer().Fingerprint() } - if err := phase.vexStep.Converge(ctx, vexContent, stageDesc, name, primaryImg.TargetPlatform, signer, signerIdentity); err != nil { + if err := phase.vexStep.Converge(ctx, vexContent, stageDesc, name, vexTargetPlatform(images), signer, signerIdentity); err != nil { return fmt.Errorf("unable to converge VEX for image %q: %w", name, err) } + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDescForImage(phase, name, images), phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + return fmt.Errorf("unable to propagate VEX for image %q: %w", name, err) + } + return nil } diff --git a/pkg/build/build_phase_test.go b/pkg/build/build_phase_test.go index 74bb57e5a0..059daa0ae6 100644 --- a/pkg/build/build_phase_test.go +++ b/pkg/build/build_phase_test.go @@ -13,6 +13,8 @@ import ( "github.com/werf/werf/v2/pkg/build/stage" "github.com/werf/werf/v2/pkg/config" imagePkg "github.com/werf/werf/v2/pkg/image" + "github.com/werf/werf/v2/pkg/storage" + "github.com/werf/werf/v2/pkg/storage/manager" ) var _ = Describe("BuildPhase", func() { @@ -96,6 +98,37 @@ var _ = Describe("BuildPhase", func() { ) }) + Describe("artifact storage validation", func() { + It("detects VEX configuration", func(ctx SpecContext) { + img, err := image.NewImage(ctx, "linux/amd64", "app", image.NoBaseImage, image.ImageOptions{ + Vex: &config.Vex{Document: "vex.json"}, + }) + Expect(err).To(Succeed()) + tree := image.NewImagesTree(nil, image.ImagesTreeOptions{}) + tree.AppendImageForTests(img) + phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{ + werfConfig: &config.WerfConfig{Meta: &config.Meta{}}, + imagesTree: tree, + }}} + + Expect(phase.artifactsEnabled()).To(BeTrue()) + }) + + It("rejects enabled artifacts with local-only storage", func() { + storageManager := &artifactValidationStorageManager{stages: storage.NewLocalStagesStorage(nil)} + + err := validateArtifactStorage(storageManager, true) + + Expect(err).To(MatchError("SBOM or VEX generation requires a container registry (specify --repo), or disable artifact generation")) + }) + + It("allows disabled artifacts with local-only storage", func() { + storageManager := &artifactValidationStorageManager{stages: storage.NewLocalStagesStorage(nil)} + + Expect(validateArtifactStorage(storageManager, false)).To(Succeed()) + }) + }) + It("skips SBOM convergence when no images were selected", func(ctx SpecContext) { phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{ werfConfig: &config.WerfConfig{Meta: &config.Meta{Build: config.MetaBuild{Sbom: &config.MetaBuildSbom{Enable: true}}}}, @@ -361,7 +394,16 @@ var _ = Describe("BuildPhase", func() { It("returns nil for a single-platform image resolved from the cache, without a built stage image", func() { phase := &BuildPhase{} - Expect(phase.finalStageDescForImage("app", []*image.Image{{}})).To(BeNil()) + Expect(finalStageDescForImage(phase, "app", []*image.Image{{}})).To(BeNil()) }) }) }) + +type artifactValidationStorageManager struct { + manager.StorageManagerInterface + stages storage.PrimaryStagesStorage +} + +func (m *artifactValidationStorageManager) GetStagesStorage() storage.PrimaryStagesStorage { + return m.stages +} diff --git a/pkg/build/sbom_step.go b/pkg/build/sbom_step.go index c7e84abf98..ee528e9ede 100644 --- a/pkg/build/sbom_step.go +++ b/pkg/build/sbom_step.go @@ -202,30 +202,8 @@ func (step *sbomStep) calculateStableChecksum(scanOpts scanner.ScanOptions, merg // PropagateArtifacts copies the artifacts attached to the image stage (e.g. its SBOM) // into the final repo and the cache repos. Stages themselves are copied there before // SBOM generation runs, so the artifacts have to catch up separately. -func (step *sbomStep) PropagateArtifacts(ctx context.Context, werfImgName string, stageDesc, finalStageDesc *image.StageDesc, cacheStagesStorageList []storage.StagesStorage) error { - srcRepo := stageDesc.Info.Repository - srcDigest := stageDesc.Info.GetDigest() - - if finalStageDesc != nil && finalStageDesc.Info.Repository != srcRepo { - if err := logboek.Context(ctx).Default().LogProcess("image %s: Copy SBOM artifacts into the final repo %s", werfImgName, finalStageDesc.Info.Repository).DoError(func() error { - return artifact.CopyAttachedArtifacts(ctx, srcRepo, srcDigest, finalStageDesc.Info.Repository, finalStageDesc.Info.GetDigest()) - }); err != nil { - return fmt.Errorf("copy attached artifacts into final repo %s: %w", finalStageDesc.Info.Repository, err) - } - } - - for _, cache := range cacheStagesStorageList { - if cache.Address() == storage.LocalStorageAddress || cache.Address() == srcRepo { - continue - } - if err := logboek.Context(ctx).Info().LogProcess("image %s: Copy SBOM artifacts into cache %s", werfImgName, cache.String()).DoError(func() error { - return artifact.CopyAttachedArtifacts(ctx, srcRepo, srcDigest, cache.Address(), srcDigest) - }); err != nil { - logboek.Context(ctx).Warn().LogF("Warning: unable to copy attached artifacts into cache stages storage %s: %s\n", cache.String(), err) - } - } - - return nil +func (step *sbomStep) PropagateArtifacts(ctx context.Context, projectName, werfImgName string, stageDesc, finalStageDesc *image.StageDesc, cacheStagesStorageList []storage.StagesStorage) error { + return propagateArtifacts(ctx, projectName, werfImgName, stageDesc, finalStageDesc, cacheStagesStorageList) } func (step *sbomStep) GetImageBOM(ctx context.Context, imageName string, imageInfo *image.Info) (*cdx.BOM, error) { diff --git a/pkg/build/sbom_step_propagate_test.go b/pkg/build/sbom_step_propagate_test.go index 3dc796cc18..dfab26873b 100644 --- a/pkg/build/sbom_step_propagate_test.go +++ b/pkg/build/sbom_step_propagate_test.go @@ -65,7 +65,7 @@ var _ = Describe("SbomStep PropagateArtifacts", func() { } } - cacheStorage := func(address string) storage.StagesStorage { + cacheStorage := func(address string) *mock.MockStagesStorage { s := mock.NewMockStagesStorage(gomock.NewController(GinkgoT())) s.EXPECT().Address().Return(address).AnyTimes() s.EXPECT().String().Return(address).AnyTimes() @@ -96,12 +96,21 @@ var _ = Describe("SbomStep PropagateArtifacts", func() { copyImageByDigest(ctx, srcRepo, finalRepo, srcDigest) step := &sbomStep{} - Expect(step.PropagateArtifacts(ctx, "app", stageDescFor(srcRepo, srcDigest), stageDescFor(finalRepo, srcDigest), nil)).To(Succeed()) + source := stageDescFor(srcRepo, srcDigest) + destination := stageDescFor(finalRepo, srcDigest) + Expect(step.PropagateArtifacts(ctx, "test", "app", source, destination, nil)).To(Succeed()) + Expect(step.PropagateArtifacts(ctx, "test", "app", source, destination, nil)).To(Succeed()) finalStore := artifact.NewOCIStore(finalRepo, "app", remoteOpts...) content, err := finalStore.GetAttachedContent(ctx, srcDigest, attestation.DSSEMediaType, nil) Expect(err).To(Succeed()) Expect(content).To(MatchJSON(`{"v":1}`)) + + index, err := artifact.PullFallbackIndex(ctx, finalRepo, srcDigest, remoteOpts...) + Expect(err).To(Succeed()) + manifest, err := index.IndexManifest() + Expect(err).To(Succeed()) + Expect(manifest.Manifests).To(HaveLen(1)) }) It("should copy the SBOM into cache repos", func(ctx SpecContext) { @@ -113,7 +122,7 @@ var _ = Describe("SbomStep PropagateArtifacts", func() { cacheStorage(srcRepo), cacheStorage(cacheRepo), } - Expect(step.PropagateArtifacts(ctx, "app", stageDescFor(srcRepo, srcDigest), nil, caches)).To(Succeed()) + Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), nil, caches)).To(Succeed()) cacheStore := artifact.NewOCIStore(cacheRepo, "app", remoteOpts...) content, err := cacheStore.GetAttachedContent(ctx, srcDigest, attestation.DSSEMediaType, nil) @@ -121,25 +130,47 @@ var _ = Describe("SbomStep PropagateArtifacts", func() { Expect(content).To(MatchJSON(`{"v":1}`)) }) + It("should propagate artifacts to the cache image digest", func(ctx SpecContext) { + destinationDigest := pushRandomImage(ctx, cacheRepo) + cache := cacheStorage(cacheRepo) + cache.EXPECT().GetStageDesc(gomock.Any(), "test", werfImage.StageID{Digest: "stage-digest"}).Return(&werfImage.StageDesc{ + StageID: &werfImage.StageID{Digest: "stage-digest"}, + Info: &werfImage.Info{ + Repository: cacheRepo, + RepoDigest: cacheRepo + "@" + destinationDigest, + }, + }, nil) + + step := &sbomStep{} + source := stageDescFor(srcRepo, srcDigest) + source.StageID = &werfImage.StageID{Digest: "stage-digest"} + Expect(step.PropagateArtifacts(ctx, "test", "app", source, nil, []storage.StagesStorage{cache})).To(Succeed()) + + cacheStore := artifact.NewOCIStore(cacheRepo, "app", remoteOpts...) + content, err := cacheStore.GetAttachedContent(ctx, destinationDigest, attestation.DSSEMediaType, nil) + Expect(err).To(Succeed()) + Expect(content).To(MatchJSON(`{"v":1}`)) + }) + It("should do nothing without a final repo and caches", func(ctx SpecContext) { step := &sbomStep{} - Expect(step.PropagateArtifacts(ctx, "app", stageDescFor(srcRepo, srcDigest), nil, nil)).To(Succeed()) + Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), nil, nil)).To(Succeed()) }) It("should skip the final repo when it matches the stages repo", func(ctx SpecContext) { step := &sbomStep{} - Expect(step.PropagateArtifacts(ctx, "app", stageDescFor(srcRepo, srcDigest), stageDescFor(srcRepo, srcDigest), nil)).To(Succeed()) + Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), stageDescFor(srcRepo, srcDigest), nil)).To(Succeed()) }) It("should not fail when a cache repo is unreachable", func(ctx SpecContext) { step := &sbomStep{} caches := []storage.StagesStorage{cacheStorage("127.0.0.1:1/unreachable/cache")} - Expect(step.PropagateArtifacts(ctx, "app", stageDescFor(srcRepo, srcDigest), nil, caches)).To(Succeed()) + Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), nil, caches)).To(Succeed()) }) It("should fail when the final repo copy fails", func(ctx SpecContext) { step := &sbomStep{} - err := step.PropagateArtifacts(ctx, "app", stageDescFor(srcRepo, srcDigest), stageDescFor("127.0.0.1:1/unreachable/final", srcDigest), nil) + err := step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), stageDescFor("127.0.0.1:1/unreachable/final", srcDigest), nil) Expect(err).To(HaveOccurred()) Expect(err.Error()).To(ContainSubstring("copy attached artifacts into final repo")) }) diff --git a/pkg/build/sbom_step_test.go b/pkg/build/sbom_step_test.go index 7e3d650295..6117aeebea 100644 --- a/pkg/build/sbom_step_test.go +++ b/pkg/build/sbom_step_test.go @@ -4,24 +4,88 @@ import ( "bytes" "context" "errors" + "net/http/httptest" "path/filepath" "strings" cdx "github.com/CycloneDX/cyclonedx-go" + "github.com/google/go-containerregistry/pkg/authn" + "github.com/google/go-containerregistry/pkg/name" + "github.com/google/go-containerregistry/pkg/registry" + "github.com/google/go-containerregistry/pkg/v1/random" + "github.com/google/go-containerregistry/pkg/v1/remote" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" "go.uber.org/mock/gomock" "github.com/werf/logboek" + "github.com/werf/werf/v2/pkg/docker_registry" werfImage "github.com/werf/werf/v2/pkg/image" "github.com/werf/werf/v2/pkg/logging" + "github.com/werf/werf/v2/pkg/oci/artifact" "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil" "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil/gost" "github.com/werf/werf/v2/pkg/sbom/gomod" + "github.com/werf/werf/v2/pkg/sbom/scanner" "github.com/werf/werf/v2/test/mock" ) var _ = Describe("SbomStep", func() { + It("publishes platform metadata against the matching parent digest", func(ctx SpecContext) { + Expect(docker_registry.Init(ctx, false, false, nil, nil)).To(Succeed()) + server := httptest.NewServer(registry.New()) + DeferCleanup(server.Close) + repo := strings.TrimPrefix(server.URL, "http://") + "/test/images" + remoteOpts := []remote.Option{remote.WithAuth(authn.Anonymous)} + backend := mock.NewMockContainerBackend(gomock.NewController(GinkgoT())) + step := newSbomStep(backend, nil) + bomJSON := []byte(`{"bomFormat":"CycloneDX","specVersion":"1.6","version":1}`) + platforms := []struct { + name, platform string + }{ + {name: "amd64", platform: "linux/amd64"}, + {name: "arm64", platform: "linux/arm64"}, + } + parentDigests := make(map[string]string, len(platforms)) + artifactDigests := make(map[string]string, len(platforms)) + + for _, item := range platforms { + parent, err := random.Image(256, 1) + Expect(err).To(Succeed()) + parentRef, err := name.NewTag(repo + ":" + item.name) + Expect(err).To(Succeed()) + Expect(remote.Write(parentRef, parent, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...)).To(Succeed()) + parentDigest, err := parent.Digest() + Expect(err).To(Succeed()) + parentDigests[item.platform] = parentDigest.String() + stageName := repo + ":stage-" + item.name + backend.EXPECT().Pull(gomock.Any(), stageName, gomock.Any()).Return(nil) + backend.EXPECT().GenerateSBOM(gomock.Any(), gomock.Any()).Return(bomJSON, nil) + stageDesc := &werfImage.StageDesc{Info: &werfImage.Info{ + Name: stageName, Repository: repo, RepoDigest: repo + "@" + parentDigest.String(), Tag: "stage-" + item.name, + }} + Expect(step.ConvergeWithMerge(ctx, "app", stageDesc, scanner.ScanOptions{Commands: []scanner.ScanCommand{{}}}, cyclonedxutil.MergeOpts{}, nil, false, false, item.platform, nil, "")).To(Succeed()) + index, err := artifact.PullFallbackIndex(ctx, repo, parentDigest.String(), remoteOpts...) + Expect(err).To(Succeed()) + manifest, err := index.IndexManifest() + Expect(err).To(Succeed()) + Expect(manifest.Manifests).To(HaveLen(1)) + artifactDescriptor := manifest.Manifests[0] + artifactDigests[item.platform] = artifactDescriptor.Digest.String() + artifactRef, err := name.NewDigest(repo + "@" + artifactDescriptor.Digest.String()) + Expect(err).To(Succeed()) + artifactImage, err := remote.Image(artifactRef, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...) + Expect(err).To(Succeed()) + artifactManifest, err := artifactImage.Manifest() + Expect(err).To(Succeed()) + Expect(artifactManifest.Annotations).To(HaveKeyWithValue(werfImage.WerfPlatformAnnotation, item.platform)) + Expect(artifactManifest.Subject).NotTo(BeNil()) + Expect(artifactManifest.Subject.Digest.String()).To(Equal(parentDigest.String())) + } + Expect(parentDigests[platforms[0].platform]).NotTo(Equal(parentDigests[platforms[1].platform])) + Expect(artifactDigests[platforms[0].platform]).NotTo(Equal(artifactDigests[platforms[1].platform])) + }) + Describe("prepareGostComponents", func() { It("prints the GOST experimental warning at most once per step instance", func() { var output bytes.Buffer diff --git a/pkg/build/stage/artifact_test.go b/pkg/build/stage/artifact_test.go new file mode 100644 index 0000000000..011059f896 --- /dev/null +++ b/pkg/build/stage/artifact_test.go @@ -0,0 +1,62 @@ +package stage + +import ( + "context" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "go.uber.org/mock/gomock" + + "github.com/werf/werf/v2/pkg/build/signing" + "github.com/werf/werf/v2/pkg/image" + "github.com/werf/werf/v2/test/mock" +) + +var _ = Describe("artifact stages", func() { + It("stores artifact identity and lifecycle flags", func() { + base := NewBaseStage(Sbom, &BaseStageOptions{}) + metadata := &ArtifactStageMetadata{ + Kind: ArtifactKindSbom, + ParentDigest: "sha256:parent", + TargetPlatform: "linux/amd64", + Mutable: true, + Buildable: false, + } + + base.SetArtifactMetadata(metadata) + + Expect(base.GetArtifactMetadata()).To(BeIdenticalTo(metadata)) + Expect(base.GetArtifactMetadata().Kind).To(Equal(ArtifactKindSbom)) + Expect(base.GetArtifactMetadata().ParentDigest).To(Equal("sha256:parent")) + Expect(base.GetArtifactMetadata().TargetPlatform).To(Equal("linux/amd64")) + Expect(base.GetArtifactMetadata().Mutable).To(BeTrue()) + Expect(base.GetArtifactMetadata().Buildable).To(BeFalse()) + }) + + DescribeTable("is mutable and non-buildable", + func(artifactStage Interface) { + Expect(artifactStage.IsMutable()).To(BeTrue()) + Expect(artifactStage.IsBuildable()).To(BeFalse()) + }, + Entry("SBOM", GenerateSbomStage(&BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.SbomSigningOptions{}, "dependency", func(context.Context, *image.StageDesc, string, string) error { + return nil + })), + Entry("VEX", GenerateVexStage([]byte(`{"statements":[]}`), &BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.VexSigningOptions{})), + ) + + It("includes the parent descriptor in artifact stage dependencies", func(ctx SpecContext) { + sbom := GenerateSbomStage(&BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.SbomSigningOptions{}, "dependency", func(context.Context, *image.StageDesc, string, string) error { + return nil + }) + withoutParent, err := sbom.GetDependencies(ctx, nil, nil, nil, nil, nil) + Expect(err).To(Succeed()) + + ctrl := gomock.NewController(GinkgoT()) + parentImage := mock.NewMockLegacyImageInterface(ctrl) + parentImage.EXPECT().GetStageDesc().Return(&image.StageDesc{Info: &image.Info{RepoDigest: "repo@sha256:parent"}}) + parent := NewStageImage(NewContainerBackendStub(), "", parentImage) + withParent, err := sbom.GetDependencies(ctx, nil, nil, nil, parent, nil) + Expect(err).To(Succeed()) + Expect(withParent).NotTo(Equal(withoutParent)) + }) +}) diff --git a/pkg/build/stage/base.go b/pkg/build/stage/base.go index af2e0380cd..1fd4d9b4bb 100644 --- a/pkg/build/stage/base.go +++ b/pkg/build/stage/base.go @@ -39,6 +39,8 @@ const ( Dockerfile StageName = "dockerfile" ImageSpec StageName = "imageSpec" Sign StageName = "sign" + Sbom StageName = "sbom" + Vex StageName = "vex" VerityAnnotation StageName = "verityAnnotation" ) @@ -106,6 +108,7 @@ type BaseStage struct { networkOverride string needsNetwork bool meta *StageMeta + artifactMetadata *ArtifactStageMetadata isContentAnchor bool } @@ -117,6 +120,29 @@ func (s *BaseStage) SetContentAnchor(v bool) { s.isContentAnchor = v } +type ArtifactKind string + +const ( + ArtifactKindSbom ArtifactKind = "sbom" + ArtifactKindVex ArtifactKind = "vex" +) + +type ArtifactStageMetadata struct { + Kind ArtifactKind + ParentDigest string + TargetPlatform string + Mutable bool + Buildable bool +} + +func (s *BaseStage) SetArtifactMetadata(metadata *ArtifactStageMetadata) { + s.artifactMetadata = metadata +} + +func (s *BaseStage) GetArtifactMetadata() *ArtifactStageMetadata { + return s.artifactMetadata +} + type StageMeta struct { Rebuilt bool BaseImagePulled bool diff --git a/pkg/build/stage/sbom.go b/pkg/build/stage/sbom.go new file mode 100644 index 0000000000..9caed8bced --- /dev/null +++ b/pkg/build/stage/sbom.go @@ -0,0 +1,114 @@ +package stage + +import ( + "context" + "fmt" + + "github.com/werf/common-go/pkg/util" + "github.com/werf/werf/v2/pkg/build/signing" + "github.com/werf/werf/v2/pkg/container_backend" + "github.com/werf/werf/v2/pkg/image" +) + +type SbomStagePublisher func(ctx context.Context, parentDesc *image.StageDesc, imageName, targetPlatform string) error + +type SbomStage struct { + *BaseStage + + publisher SbomStagePublisher + dependency string + signerIdentity string +} + +func GenerateSbomStage(baseStageOptions *BaseStageOptions, sbomSigningOptions signing.SbomSigningOptions, dependency string, publisher SbomStagePublisher) *SbomStage { + return newSbomStage(baseStageOptions, sbomSigningOptions, dependency, publisher) +} + +func newSbomStage(baseStageOptions *BaseStageOptions, sbomSigningOptions signing.SbomSigningOptions, dependency string, publisher SbomStagePublisher) *SbomStage { + var signerIdentity string + if sbomSigningOptions.Enabled { + signerIdentity = sbomSigningOptions.Signer().Fingerprint() + } + + stage := &SbomStage{ + BaseStage: NewBaseStage(Sbom, baseStageOptions), + publisher: publisher, + dependency: dependency, + signerIdentity: signerIdentity, + } + stage.SetArtifactMetadata(&ArtifactStageMetadata{ + Kind: ArtifactKindSbom, + TargetPlatform: baseStageOptions.TargetPlatform, + Mutable: true, + Buildable: false, + }) + return stage +} + +var _ Interface = (*SbomStage)(nil) + +func (s *SbomStage) IsBuildable() bool { + return false +} + +func (s *SbomStage) IsMutable() bool { + return true +} + +func (s *SbomStage) PrepareImage(_ context.Context, _ Conveyor, _ container_backend.ContainerBackend, _, _ *StageImage, _ container_backend.BuildContextArchiver) error { + return nil +} + +func (s *SbomStage) GetDependencies(_ context.Context, _ Conveyor, _ container_backend.ContainerBackend, _, prevBuiltImage *StageImage, _ container_backend.BuildContextArchiver) (string, error) { + parentDigest := "" + if prevBuiltImage != nil && prevBuiltImage.Image != nil { + if stageDesc := prevBuiltImage.Image.GetStageDesc(); stageDesc != nil && stageDesc.Info != nil { + parentDigest = stageDesc.Info.GetDigest() + } + } + + return util.Sha256Hash( + sbomArtifactFormatVersion, + "inputs", s.dependency, + "parent", parentDigest, + "signer", s.signerIdentity, + "platform", s.TargetPlatform(), + ), nil +} + +func (s *SbomStage) GetContentDependencies(ctx context.Context, c Conveyor, buildContextArchive container_backend.BuildContextArchiver) (string, error) { + return s.GetDependencies(ctx, c, nil, nil, nil, buildContextArchive) +} + +func (s *SbomStage) MutateImage(ctx context.Context, stagesStorage ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { + if _, err := registryFromImageMutatorPusher(stagesStorage); err != nil { + return err + } + if s.publisher == nil { + return fmt.Errorf("SBOM stage publisher is unavailable") + } + if prevBuiltImage == nil || prevBuiltImage.Image == nil { + return fmt.Errorf("SBOM stage parent image is unavailable") + } + if stageImage == nil || stageImage.Image == nil { + return fmt.Errorf("SBOM stage image is unavailable") + } + + parentDesc := prevBuiltImage.Image.GetStageDesc() + if parentDesc == nil || parentDesc.Info == nil { + return fmt.Errorf("SBOM stage parent descriptor is unavailable") + } + if parentDesc.Info.Repository == "" { + return fmt.Errorf("SBOM stage parent descriptor repository is empty") + } + if parentDesc.Info.GetDigest() == "" { + return fmt.Errorf("SBOM stage parent descriptor digest is empty") + } + + metadata := s.GetArtifactMetadata() + metadata.ParentDigest = parentDesc.Info.GetDigest() + + return s.publisher(ctx, parentDesc, stageImage.Image.Name(), s.TargetPlatform()) +} + +const sbomArtifactFormatVersion = "2" diff --git a/pkg/build/stage/vex.go b/pkg/build/stage/vex.go new file mode 100644 index 0000000000..fec12e933a --- /dev/null +++ b/pkg/build/stage/vex.go @@ -0,0 +1,148 @@ +package stage + +import ( + "context" + "fmt" + "strings" + + "github.com/sigstore/sigstore/pkg/signature" + + "github.com/werf/common-go/pkg/util" + "github.com/werf/logboek" + "github.com/werf/werf/v2/pkg/attestation" + "github.com/werf/werf/v2/pkg/build/signing" + "github.com/werf/werf/v2/pkg/container_backend" + "github.com/werf/werf/v2/pkg/image" + "github.com/werf/werf/v2/pkg/oci/artifact" + vexImage "github.com/werf/werf/v2/pkg/vex/image" +) + +type VexStage struct { + *BaseStage + + vexJSON []byte + signer signature.Signer + signerIdentity string +} + +func GenerateVexStage(vexJSON []byte, baseStageOptions *BaseStageOptions, vexSigningOptions signing.VexSigningOptions) *VexStage { + return newVexStage(vexJSON, baseStageOptions, vexSigningOptions) +} + +func newVexStage(vexJSON []byte, baseStageOptions *BaseStageOptions, vexSigningOptions signing.VexSigningOptions) *VexStage { + var signer signature.Signer + var signerIdentity string + if vexSigningOptions.Enabled { + signer = vexSigningOptions.Signer().SignerVerifier() + signerIdentity = vexSigningOptions.Signer().Fingerprint() + } + + stage := &VexStage{ + BaseStage: NewBaseStage(Vex, baseStageOptions), + vexJSON: vexJSON, + signer: signer, + signerIdentity: signerIdentity, + } + stage.SetArtifactMetadata(&ArtifactStageMetadata{ + Kind: ArtifactKindVex, + TargetPlatform: baseStageOptions.TargetPlatform, + Mutable: true, + Buildable: false, + }) + return stage +} + +var _ Interface = (*VexStage)(nil) + +func (s *VexStage) IsBuildable() bool { + return false +} + +func (s *VexStage) IsMutable() bool { + return true +} + +func (s *VexStage) PrepareImage(_ context.Context, _ Conveyor, _ container_backend.ContainerBackend, _, _ *StageImage, _ container_backend.BuildContextArchiver) error { + return nil +} + +func (s *VexStage) GetDependencies(_ context.Context, _ Conveyor, _ container_backend.ContainerBackend, _, prevBuiltImage *StageImage, _ container_backend.BuildContextArchiver) (string, error) { + parentDigest := "" + if prevBuiltImage != nil && prevBuiltImage.Image != nil { + if stageDesc := prevBuiltImage.Image.GetStageDesc(); stageDesc != nil && stageDesc.Info != nil { + parentDigest = stageDesc.Info.GetDigest() + } + } + + return calculateVexStageChecksum(s.vexJSON, parentDigest, s.signerIdentity), nil +} + +func (s *VexStage) GetContentDependencies(ctx context.Context, c Conveyor, buildContextArchive container_backend.BuildContextArchiver) (string, error) { + return s.GetDependencies(ctx, c, nil, nil, nil, buildContextArchive) +} + +func (s *VexStage) MutateImage(ctx context.Context, stagesStorage ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { + if _, err := registryFromImageMutatorPusher(stagesStorage); err != nil { + return err + } + if prevBuiltImage == nil || prevBuiltImage.Image == nil { + return fmt.Errorf("VEX stage parent image is unavailable") + } + if stageImage == nil || stageImage.Image == nil { + return fmt.Errorf("VEX stage image is unavailable") + } + + parentDesc := prevBuiltImage.Image.GetStageDesc() + if parentDesc == nil || parentDesc.Info == nil { + return fmt.Errorf("VEX stage parent descriptor is unavailable") + } + if parentDesc.Info.Repository == "" { + return fmt.Errorf("VEX stage parent descriptor repository is empty") + } + + parentDigest := parentDesc.Info.GetDigest() + if parentDigest == "" { + return fmt.Errorf("VEX stage parent descriptor digest is empty") + } + + metadata := s.GetArtifactMetadata() + metadata.ParentDigest = parentDigest + + checksum := calculateVexStageChecksum(s.vexJSON, parentDigest, s.signerIdentity) + store := artifact.NewOCIStore(parentDesc.Info.Repository, stageImage.Image.Name()) + needed, err := checkVexStagePublishNeeded(ctx, store, parentDigest, checksum) + if err != nil { + return fmt.Errorf("check VEX publish needed: %w", err) + } + if !needed { + logboek.Context(ctx).Default().LogF("image %s: VEX artifact is up to date — skipping publish\n", stageImage.Image.Name()) + return nil + } + + return logboek.Context(ctx).Default().LogProcess("image %s: Published VEX artifact", stageImage.Image.Name()).DoError(func() error { + return vexImage.PushVEX(ctx, s.vexJSON, parentDesc.Info.Repository, parentDigest, stageImage.Image.Name(), checksum, s.TargetPlatform(), s.signer) + }) +} + +const vexStageArtifactFormatVersion = "2" + +func calculateVexStageChecksum(vexJSON []byte, parentDigest, signerIdentity string) string { + parts := []string{ + vexStageArtifactFormatVersion, + util.Sha256Hash(string(vexJSON)), + parentDigest, + signerIdentity, + } + return util.Sha256Hash(strings.Join(parts, "-")) +} + +func checkVexStagePublishNeeded(ctx context.Context, store artifact.Store, parentDigest, checksum string) (bool, error) { + desc, found, err := attestation.FindAttachedArtifact(ctx, store, parentDigest, attestation.PredicateKindOpenVEX) + if err != nil { + return false, fmt.Errorf("check VEX cache: %w", err) + } + if found && desc.Annotations[image.WerfChecksumAnnotation] == checksum { + return false, nil + } + return true, nil +} diff --git a/pkg/cleaning/cleanup.go b/pkg/cleaning/cleanup.go index fcc6c22722..3d271ce4ef 100644 --- a/pkg/cleaning/cleanup.go +++ b/pkg/cleaning/cleanup.go @@ -1052,6 +1052,15 @@ func (m *cleanupManager) cleanupOrphanedArtifacts(ctx context.Context) error { } } + for _, cacheStagesStorage := range m.StorageManager.GetCacheStagesStorageList() { + if cacheStagesStorage == nil || cacheStagesStorage.Address() == storage.LocalStorageAddress { + continue + } + if err := deleteOrphanedArtifacts(ctx, cacheStagesStorage, m.DryRun); err != nil { + return fmt.Errorf("delete orphaned artifacts from cache repo %s: %w", cacheStagesStorage.String(), err) + } + } + return nil } diff --git a/pkg/cleaning/cleanup_test.go b/pkg/cleaning/cleanup_test.go index 68b0d9a757..cce3a9063e 100644 --- a/pkg/cleaning/cleanup_test.go +++ b/pkg/cleaning/cleanup_test.go @@ -103,6 +103,20 @@ var _ = Describe("cleanupManager.cleanupOrphanedArtifacts", func() { Expect(sm.stages.deletedArtifacts).To(Equal([]string{"repo:sha256-abc123"})) }) + It("cleans propagated cache repositories", func() { + sm := newFakeStorageManager() + cache := mock.NewMockStagesStorage(gomock.NewController(GinkgoT())) + cache.EXPECT().Address().Return("registry.example.com/cache").AnyTimes() + cache.EXPECT().String().Return("registry.example.com/cache").AnyTimes() + cache.EXPECT().GetOrphanedArtifactNames(gomock.Any()).Return([]string{"cache:sha256-abc123"}, nil) + cache.EXPECT().DeleteArtifact(gomock.Any(), "cache:sha256-abc123").Return(nil) + sm.caches = []storage.StagesStorage{cache} + + m := &cleanupManager{StorageManager: sm} + + Expect(m.cleanupOrphanedArtifacts(context.Background())).To(Succeed()) + }) + It("reports which repo failed when the final repo cannot be cleaned", func() { sm := newFakeStorageManager() @@ -188,6 +202,7 @@ type fakeStorageManager struct { stages *fakePrimaryStagesStorage meta *fakePrimaryStagesStorage final storage.StagesStorage + caches []storage.StagesStorage stageDescSet image.StageDescSet finalStageDescSet image.StageDescSet @@ -229,6 +244,10 @@ func (f *fakeStorageManager) GetFinalStagesStorage() storage.StagesStorage { return f.final } +func (f *fakeStorageManager) GetCacheStagesStorageList() []storage.StagesStorage { + return f.caches +} + func (f *fakeStorageManager) ForEachRejectedStage(ctx context.Context, stageIDs []image.StageID, cb func(ctx context.Context, stageID image.StageID) error) error { for _, id := range stageIDs { if err := cb(ctx, id); err != nil { diff --git a/specs/020-sbom-vex-build-stages/inventory.md b/specs/020-sbom-vex-build-stages/inventory.md new file mode 100644 index 0000000000..8bd5e1cde5 --- /dev/null +++ b/specs/020-sbom-vex-build-stages/inventory.md @@ -0,0 +1,16 @@ +# Existing integration inventory + +- `pkg/build/build_phase.go`: `BeforeImages` initializes storage; `AfterImages` publishes primary/final images and then runs SBOM and VEX convergence. `convergePlatformImageSbom` currently owns SBOM propagation. `convergeImageVex` publishes VEX but has no propagation path. `findAndFetchStageFromSecondaryStagesStorage` copies restored stages into primary and then cache storage. +- `pkg/build/sbom_step.go`: `ConvergeWithMerge` computes the stable checksum, checks the fallback artifact index, generates and pushes CycloneDX, and `PropagateArtifacts` copies attached artifacts to final/cache destinations. +- `pkg/build/vex_step.go`: `Converge` computes VEX checksum, checks the fallback artifact index, and pushes OpenVEX. It requires a non-nil stage descriptor. +- `pkg/build/stage/base.go` and `sign.go`: stages expose buildable/mutable flags, dependencies, image preparation, and registry-side mutation. Signing is the existing non-buildable mutable registry-stage pattern. +- `pkg/oci/artifact/copy.go`: `CopyAttachedArtifacts` already copies all typed fallback-index artifacts by payload, supports differing source/destination digests, skips equivalent identities, and treats a missing source index as a no-op. +- `pkg/storage/manager`: final and cache stage-copy operations are separate; cache errors are logged as warnings and final errors are returned. Secondary restoration uses `CopySuitableStageDescByDigest` and then copies the restored stage into caches. +- `pkg/cleaning/cleanup.go`: orphan artifact indexes are cleaned independently for primary, final, and configured cache repositories. +- Existing tests: Ginkgo/Gomega tests cover SBOM/VEX convergence guards, fallback-index convergence, artifact copy idempotency, differing destination digests, concurrent attachment, and orphan cleanup. Existing e2e suites are under `test/e2e/sbom` and `test/e2e/vex`. + +## Foundation decisions applied + +- Reuse `artifact.CopyAttachedArtifacts` as the shared kind-neutral propagation primitive rather than introducing another artifact store. +- Keep final propagation errors fatal and cache propagation best effort. +- Perform local-only validation before storage initialization whenever SBOM or an image-level VEX document is enabled. diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index e0cd981f3b..7e14ded2fa 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -10,9 +10,9 @@ **Purpose**: Establish the implementation baseline without changing user-facing CLI semantics. -- [ ] T001 Inventory existing stage, SBOM, VEX, artifact, storage-copy, and cleanup call paths in `pkg/build/`, `pkg/build/stage/`, `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/cleaning/`, recording the concrete integration points in the feature working notes at `specs/020-sbom-vex-build-stages/` -- [ ] T002 [P] Inspect existing SBOM and VEX unit/e2e fixture conventions in `pkg/build/`, `test/e2e/sbom/`, and `test/e2e/vex/` and identify reusable helpers without adding a second test harness -- [ ] T003 [P] Confirm the existing fallback-tag artifact index and cleanup compatibility expectations in `pkg/oci/artifact/` and `pkg/cleaning/` before modifying propagation code +- [X] T001 Inventory existing stage, SBOM, VEX, artifact, storage-copy, and cleanup call paths in `pkg/build/`, `pkg/build/stage/`, `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/cleaning/`, recording the concrete integration points in the feature working notes at `specs/020-sbom-vex-build-stages/` +- [X] T002 [P] Inspect existing SBOM and VEX unit/e2e fixture conventions in `pkg/build/`, `test/e2e/sbom/`, and `test/e2e/vex/` and identify reusable helpers without adding a second test harness +- [X] T003 [P] Confirm the existing fallback-tag artifact index and cleanup compatibility expectations in `pkg/oci/artifact/` and `pkg/cleaning/` before modifying propagation code --- @@ -22,10 +22,10 @@ **Checkpoint**: The shared stage metadata, descriptor propagation shape, and early registry validation are understood and available before story implementation begins. -- [ ] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including artifact kind, parent descriptor, target platform, mutable flag, and non-buildable flag -- [ ] T005 Define the kind-neutral artifact propagation operation and source/destination descriptor data flow in `pkg/build/` using existing `pkg/oci/artifact/` and `pkg/storage/manager/` primitives, without introducing a public API or new dependency -- [ ] T006 Locate the earliest common build initialization path and specify the registry-backed-storage validation seam in `pkg/build/build_phase.go` for both SBOM-enabled and VEX-enabled builds -- [ ] T007 [P] Add shared test fixtures or helper functions needed to construct manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` +- [X] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including artifact kind, parent descriptor, target platform, mutable flag, and non-buildable flag +- [X] T005 Define the kind-neutral artifact propagation operation and source/destination descriptor data flow in `pkg/build/` using existing `pkg/oci/artifact/` and `pkg/storage/manager/` primitives, without introducing a public API or new dependency +- [X] T006 Locate the earliest common build initialization path and specify the registry-backed-storage validation seam in `pkg/build/build_phase.go` for both SBOM-enabled and VEX-enabled builds +- [X] T007 [P] Add shared test fixtures or helper functions needed to construct manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` --- @@ -37,20 +37,20 @@ ### Tests for User Story 1 -- [ ] T008 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, parent descriptor requirements, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` +- [X] T008 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, parent descriptor requirements, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go - [ ] T009 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` - [ ] T010 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` - [ ] T011 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios ### Implementation for User Story 1 -- [ ] T012 [P] [US1] Implement the registry-only mutable, non-buildable SBOM artifact stage in `pkg/build/stage/sbom.go` using the existing SBOM generation, signing, checksum, and fallback-index components -- [ ] T013 [P] [US1] Implement the registry-only mutable, non-buildable VEX artifact stage in `pkg/build/stage/vex.go` using the existing VEX generation, signing, checksum, and fallback-index components +- [X] T012 [P] [US1] Implement the registry-only mutable, non-buildable SBOM artifact stage in `pkg/build/stage/sbom.go` using the existing SBOM generation, signing, checksum, and fallback-index components +- [X] T013 [P] [US1] Implement the registry-only mutable, non-buildable VEX artifact stage in `pkg/build/stage/vex.go` using the existing VEX generation, signing, checksum, and fallback-index components - [ ] T014 [US1] Register SBOM and VEX stages after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` - [ ] T015 [US1] Execute SBOM and VEX stage publication without changing image filesystem or layer content, and remove their duplicate generation pass while retaining unrelated image publication/report work in `pkg/build/build_phase.go` -- [ ] T016 [US1] Implement shared idempotent artifact propagation with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` -- [ ] T017 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the shared propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` -- [ ] T018 [US1] Connect secondary-stage restoration into primary storage to the same propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` +- [X] T016 [US1] Implement shared idempotent artifact propagation with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` +- [X] T017 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the shared propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` +- [X] T018 [US1] Connect secondary-stage restoration into primary storage to the same propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` **Checkpoint**: User Story 1 is independently functional; artifacts are generated in the lifecycle and follow every applicable published image. @@ -64,17 +64,17 @@ ### Tests for User Story 2 -- [ ] T019 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform subject selection in `pkg/build/artifact_subject_test.go` -- [ ] T020 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and parent digest are distinct per platform in `pkg/build/sbom_step_test.go` +- [X] T019 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform subject selection in `pkg/build/artifact_subject_test.go` +- [X] T020 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and parent digest are distinct per platform in `pkg/build/sbom_step_test.go` - [ ] T021 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each platform manifest -- [ ] T022 [US2] Extend `test/e2e/vex/` with single-platform manifest placement and multi-platform index-only placement assertions +- [X] T022 [US2] Extend `test/e2e/vex/` with single-platform manifest placement and multi-platform index-only placement assertions ### Implementation for User Story 2 -- [ ] T023 [US2] Implement explicit artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` +- [X] T023 [US2] Implement explicit artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` - [ ] T024 [US2] Pass the target platform and resolved platform manifest descriptor through SBOM stage creation and publication in `pkg/build/sbom_step.go` and `pkg/build/build_phase.go` - [ ] T025 [US2] Make VEX stage registration run once per multi-platform image set with the top-level index subject, and use the image manifest subject for single-platform builds in `pkg/build/vex_step.go` and `pkg/build/build_phase.go` -- [ ] T026 [US2] Ensure propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` +- [X] T026 [US2] Ensure propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 2 is independently testable and no artifact can silently use an index subject for a platform SBOM or duplicate multi-platform VEX onto platform manifests. @@ -88,17 +88,17 @@ ### Tests for User Story 3 -- [ ] T027 [P] [US3] Add Ginkgo/Gomega tests for SBOM stage dependency identity across image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/sbom_step_test.go` -- [ ] T028 [P] [US3] Add Ginkgo/Gomega tests for VEX stage dependency identity across parent digest, document content, format, and signer inputs in `pkg/build/vex_step_test.go` -- [ ] T029 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing in `pkg/build/artifact_propagation_test.go` -- [ ] T030 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios +- [X] T027 [P] [US3] Add Ginkgo/Gomega tests for SBOM stage dependency identity across image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/sbom_step_test.go` +- [X] T028 [P] [US3] Add Ginkgo/Gomega tests for VEX stage dependency identity across parent digest, document content, format, and signer inputs in `pkg/build/vex_step_test.go` +- [X] T029 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing in `pkg/build/artifact_propagation_test.go` +- [X] T030 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios ### Implementation for User Story 3 -- [ ] T031 [US3] Include all effective SBOM inputs and the parent image identity in artifact-stage dependency calculation while preserving existing checksum semantics in `pkg/build/sbom_step.go` and `pkg/build/stage/sbom.go` -- [ ] T032 [US3] Include VEX document content, parent descriptor identity, format version, and signer identity in artifact-stage dependency calculation in `pkg/build/vex_step.go` and `pkg/build/stage/vex.go` +- [X] T031 [US3] Include all effective SBOM inputs and the parent image identity in artifact-stage dependency calculation while preserving existing checksum semantics in `pkg/build/sbom_step.go` and `pkg/build/stage/sbom.go` +- [X] T032 [US3] Include VEX document content, parent descriptor identity, format version, and signer identity in artifact-stage dependency calculation in `pkg/build/vex_step.go` and `pkg/build/stage/vex.go` - [ ] T033 [US3] Select reusable artifact-bearing stages from primary and secondary storage using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` -- [ ] T034 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/` and `pkg/build/artifact_propagation.go` +- [X] T034 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/` and `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 3 is independently testable; unchanged inputs reuse artifacts and every effective changed input invalidates only the affected artifact identity. @@ -112,18 +112,18 @@ ### Tests for User Story 4 -- [ ] T035 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` -- [ ] T036 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` -- [ ] T037 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent attachment in `pkg/oci/artifact/` +- [X] T035 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` +- [X] T036 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go +- [X] T037 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent attachment in `pkg/oci/artifact/` - [ ] T038 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior -- [ ] T039 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories +- [X] T039 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories ### Implementation for User Story 4 -- [ ] T040 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` -- [ ] T041 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` -- [ ] T042 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` -- [ ] T043 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` +- [X] T040 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` +- [X] T041 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [X] T042 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` +- [X] T043 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` **Checkpoint**: User Story 4 is independently testable; registry failures and local-only configuration produce predictable results without changing repository flag semantics. @@ -133,16 +133,16 @@ **Purpose**: Validate the complete implementation against all feature constraints and repository quality gates. -- [ ] T044 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic +- [X] T044 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic - [ ] T045 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures -- [ ] T046 Run formatting with `task format` for authored Go directories -- [ ] T047 Run compilation with `task build` -- [ ] T048 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` +- [X] T046 Run formatting with `task format` for authored Go directories +- [X] T047 Run compilation with `task build` +- [X] T048 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` - [ ] T049 Run the complete unit suite with `task test:unit` -- [ ] T050 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` -- [ ] T051 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` +- [X] T050 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` +- [X] T051 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` - [ ] T052 Run legacy integration coverage with `task test:integration` -- [ ] T053 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files +- [X] T053 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files --- From 2e6049e64bebaff59ab8f8c997fc25a455cbcbbc Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Tue, 1 Sep 2026 22:53:44 +0300 Subject: [PATCH 05/17] chore(specs): replace SBOM and VEX steps with stages Define `SbomStage` and `VexStage` as the sole owners of artifact generation, caching, signing, and publication instead of retaining transitional step implementations. Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/data-model.md | 4 +- specs/020-sbom-vex-build-stages/plan.md | 39 ++-- specs/020-sbom-vex-build-stages/quickstart.md | 2 +- specs/020-sbom-vex-build-stages/research.md | 6 +- specs/020-sbom-vex-build-stages/tasks.md | 185 ++++++++++-------- 5 files changed, 126 insertions(+), 110 deletions(-) diff --git a/specs/020-sbom-vex-build-stages/data-model.md b/specs/020-sbom-vex-build-stages/data-model.md index cdd627513d..ddb02b1f60 100644 --- a/specs/020-sbom-vex-build-stages/data-model.md +++ b/specs/020-sbom-vex-build-stages/data-model.md @@ -2,7 +2,7 @@ ## Artifact stage -An internal build-stage operation associated with an image (and, for SBOM, a target platform). +An internal build-stage operation associated with an image (and, for SBOM, a target platform). `SbomStage` and `VexStage` are the sole owners of their respective generation, cache, signing, and publication behavior; no `sbomStep` or `vexStep` compatibility layer remains. | Field | Description | |---|---| @@ -22,7 +22,7 @@ Validation rules: ## Artifact identity -The cache identity stored with the existing fallback artifact index. +The cache identity stored with the existing fallback artifact index and calculated directly by the owning artifact stage. | Field | Description | |---|---| diff --git a/specs/020-sbom-vex-build-stages/plan.md b/specs/020-sbom-vex-build-stages/plan.md index 784fb1b0b6..c4cb0e3ce1 100644 --- a/specs/020-sbom-vex-build-stages/plan.md +++ b/specs/020-sbom-vex-build-stages/plan.md @@ -6,7 +6,7 @@ ## Summary -Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and into the existing image-stage lifecycle. Add registry-backed, non-buildable mutable artifact stages modeled after `pkg/build/stage/sign.go`. These stages will reuse the current SBOM/VEX convergence, signing, checksum, attestation, fallback-index, and storage implementations while publishing separate OCI artifacts. +Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and replace the `sbomStep` and `vexStep` implementations with registry-backed, non-buildable mutable stages modeled after `pkg/build/stage/sign.go`. The new `SbomStage` and `VexStage` become the sole owners of SBOM/VEX cache identity, generation, signing, attestation publication, and fallback-index interaction while publishing separate OCI artifacts. Artifact publication will use explicit source and destination image descriptors. SBOM remains platform-specific; VEX is attached once at the top-level image index for multi-platform images and to the image manifest for single-platform images. A shared idempotent propagation operation will cover primary-to-final, primary-to-cache, and secondary-to-primary copies, resolving the destination digest and preserving fatal final-repository versus best-effort cache error policies. @@ -16,8 +16,9 @@ Artifact publication will use explicit source and destination image descriptors. **Primary Dependencies**: - Existing build stage interfaces and lifecycle in `pkg/build/stage`, `pkg/build`, and `pkg/build/conveyor.go`. -- Existing SBOM generation and cache logic in `pkg/build/sbom_step.go` and `pkg/sbom/...`. -- Existing VEX generation and cache logic in `pkg/build/vex_step.go` and `pkg/vex/...`. +- Existing SBOM domain primitives in `pkg/sbom/...`, to be moved into `SbomStage`. +- Existing VEX domain primitives in `pkg/vex/...`, to be moved into `VexStage`. +- Existing `pkg/build/sbom_step.go` and `pkg/build/vex_step.go` are transitional sources only and must be removed after their logic is migrated. - Existing OCI artifact and fallback-index operations in `pkg/oci/artifact` and `pkg/attestation`. - Existing registry/storage copy operations in `pkg/storage`, `pkg/storage/manager`, and `pkg/docker_registry`. - Existing signing options in `pkg/build/signing`. @@ -44,7 +45,7 @@ Artifact publication will use explicit source and destination image descriptors. *GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design.* -- **Simplicity over abstraction**: PASS. Reuse existing `sbomStep`, `vexStep`, stage lifecycle, artifact store, and storage manager. Add one focused shared propagation path rather than separate SBOM/VEX implementations. +- **Simplicity over abstraction**: PASS. Use two explicit stages, `SbomStage` and `VexStage`, instead of retaining parallel step and stage abstractions. Add one focused shared propagation path rather than duplicating repository-copy logic. - **Go idioms and errors**: PASS. New public methods, if required, take `context.Context` first; errors wrap operation context; stage-specific helpers remain private where possible. - **Minimal public surface**: PASS. Artifact stages and propagation contracts are internal to `pkg/build`; no new CLI flags or external API are planned. - **Testing**: PASS. Tests remain alongside source and use Ginkgo/Gomega. E2E coverage extends existing SBOM/VEX suites rather than introducing a parallel harness. @@ -59,18 +60,20 @@ No constitution violations require justification. Detailed findings are in [research.md](./research.md). Key decisions: -1. Implement SBOM/VEX as mutable, non-buildable artifact stages attached to the image lifecycle. +1. Replace `sbomStep` and `vexStep` completely with mutable, non-buildable `SbomStage` and `VexStage` implementations attached to the image lifecycle. 2. Use explicit manifest/index subjects and resolve destination subjects after image copies. 3. Share idempotent propagation for SBOM and VEX across final, cache, and secondary-to-primary paths. -4. Retain current checksum inputs and fallback artifact indexes. +4. Preserve current checksum inputs and fallback artifact indexes during the migration, but implement them in the corresponding stages. 5. Validate registry-backed storage before stage execution when either feature is enabled. ## Design ### Stage integration -- Extend `pkg/build/stage` with stage names and constructors for SBOM and VEX, following the shape of `SignStage`. -- Artifact stages must not mutate the image filesystem. Their `PrepareImage` path is a no-op; their registry-side operation is performed through the existing artifact publication logic. +- Extend `pkg/build/stage` with stage names and constructors for `SbomStage` and `VexStage`, following the shape of `SignStage`. +- Move all behavior currently owned by `sbomStep` into `SbomStage`; remove `sbom_step.go` and its step-specific tests once callers are migrated. +- Move all behavior currently owned by `vexStep` into `VexStage`; remove `vex_step.go` and its step-specific tests once callers are migrated. +- Artifact stages must not mutate the image filesystem. Their `PrepareImage` path is a no-op; their `MutateImage` path owns registry-side generation and publication. - Ensure stage dependencies include the parent image identity and all effective artifact inputs. SBOM dependencies include scanner, merge/GOST, signer, format version, and target platform. VEX dependencies include document content, parent identity, signer, and format version. - Register the stages after the content-producing stage and before the lifecycle completes for applicable images. The registration must work for Stapel and Dockerfile image paths and for restored stages. - Preserve stage cache behavior: a suitable artifact-bearing stage can be selected from primary/secondary storage; changed effective inputs produce a different stage identity. @@ -81,11 +84,11 @@ Detailed findings are in [research.md](./research.md). Key decisions: - Multi-platform SBOM processing runs once per platform image and targets that platform manifest digest. - Multi-platform VEX processing runs once for the image set and targets the top-level image index digest. - Do not use the index digest as a platform SBOM subject or duplicate image-level VEX onto platform manifests. -- Keep existing signing behavior and include signer identity in cache identity. +- Keep existing signing behavior and include signer identity in cache identity. The signing and cache logic must live in the corresponding artifact stage, not in a retained step wrapper. ### Publication and propagation -- Consolidate artifact copying behind a kind-neutral internal operation that copies every attached supported artifact from a source descriptor to a destination descriptor. +- Consolidate artifact copying behind a kind-neutral internal operation that copies every attached supported artifact from a source descriptor to a destination descriptor. This propagation helper is the only shared artifact operation; generation remains owned independently by `SbomStage` and `VexStage`. - Use it after primary-to-final and primary-to-cache image copies, and when a suitable stage is copied from `--secondary-repo` into primary storage. - Resolve the destination image descriptor/digest rather than assuming source and destination digests match. - Skip local storage and identical repository addresses. Deduplicate by existing artifact identity/fallback index semantics. @@ -122,16 +125,16 @@ specs/020-sbom-vex-build-stages/ ```text pkg/build/stage/ ├── base.go # stage names and shared lifecycle metadata -├── sbom.go # new SBOM artifact stage, if kept separate -├── vex.go # new VEX artifact stage, if kept separate +├── sbom.go # SbomStage: generation, cache, signing, and publication +├── vex.go # VexStage: generation, cache, signing, and publication └── sign.go # existing registry-side stage pattern pkg/build/ ├── build_phase.go # stage registration, subject selection, propagation orchestration -├── sbom_step.go # reuse convergence; move shared propagation out or generalize it -├── vex_step.go # reuse convergence; expose stage-compatible operation └── ... +pkg/build/sbom_step.go and pkg/build/vex_step.go are removed after migration; their behavior is not retained behind compatibility wrappers. + pkg/storage/manager/ # secondary/final/cache image-copy artifact propagation hooks pkg/oci/artifact/ # shared artifact copy/deduplication support if needed @@ -140,7 +143,7 @@ test/e2e/sbom/ test/e2e/vex/ ``` -The exact file split is intentionally left to implementation if a smaller change can satisfy the same contract; no new package is required. +The exact internal helper split may vary, but the architectural boundary is fixed: no `sbomStep` or `vexStep` types/files remain after implementation, and `SbomStage`/`VexStage` are the sole lifecycle owners. No new package is required. ## Implementation Phases @@ -156,9 +159,9 @@ Completed in [data-model.md](./data-model.md) and [quickstart.md](./quickstart.m The subsequent `/speckit-tasks` workflow should decompose at least these work items: -1. Add SBOM/VEX stage identity and lifecycle integration points. -2. Adapt SBOM convergence to execute through the stage with explicit platform subjects. -3. Adapt VEX convergence to execute through the stage with single/index subject rules. +1. Implement `SbomStage` and `VexStage`, including stage identity, lifecycle integration, generation, cache checks, signing, and publication. +2. Migrate all SBOM behavior from `sbomStep` into `SbomStage`, then delete the step implementation and update callers/tests. +3. Migrate all VEX behavior from `vexStep` into `VexStage`, then delete the step implementation and update callers/tests. 4. Implement shared artifact propagation for final, cache, and secondary-to-primary copies. 5. Move registry validation before image building and remove duplicate `AfterImages` convergence. 6. Add/adjust unit tests for stage flags, dependency identities, subjects, propagation, idempotency, and failure policies. diff --git a/specs/020-sbom-vex-build-stages/quickstart.md b/specs/020-sbom-vex-build-stages/quickstart.md index 2f0c33f5d4..092d24bd9e 100644 --- a/specs/020-sbom-vex-build-stages/quickstart.md +++ b/specs/020-sbom-vex-build-stages/quickstart.md @@ -18,7 +18,7 @@ task test:unit paths="./pkg/vex/..." Expected results: -- Artifact stages calculate stable dependencies and remain non-buildable/mutable. +- `SbomStage` and `VexStage` calculate stable dependencies and remain non-buildable/mutable; the old `sbomStep` and `vexStep` implementations no longer exist. - Single-platform SBOM/VEX subjects resolve to the manifest digest. - Multi-platform SBOM subjects resolve per platform and VEX resolves to the index digest. - Propagation skips identical repositories, deduplicates existing identities, and distinguishes final errors from cache warnings. diff --git a/specs/020-sbom-vex-build-stages/research.md b/specs/020-sbom-vex-build-stages/research.md index 189b5c6209..82f31c2771 100644 --- a/specs/020-sbom-vex-build-stages/research.md +++ b/specs/020-sbom-vex-build-stages/research.md @@ -4,13 +4,13 @@ SBOM and VEX will be represented by non-buildable, mutable build stages that run after the image content stage has produced a registry-backed descriptor. The existing `Stage` lifecycle remains the integration point: stage dependencies determine cache identity, `MutateImage` performs OCI-side publication, and the build phase invokes the stage for each applicable image/platform. -The stages will reuse the existing `sbomStep`, `vexStep`, `pkg/oci/artifact`, signer implementations, and fallback-tag storage. They will not add layers or filesystem content. `BuildPhase.AfterImages` will retain image publication/report work but will no longer be the primary SBOM/VEX generation pass. +The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/...`, `pkg/oci/artifact`, the signer implementations, and fallback-tag storage. The existing `sbomStep` and `vexStep` types are transitional implementations: their behavior will be moved into `SbomStage` and `VexStage`, and the step types/files will be deleted. The stages will not add layers or filesystem content. `BuildPhase.AfterImages` will retain image publication/report work but will no longer perform SBOM/VEX generation. ### Rationale - It removes the current repository-dependent post-build pass from `AfterImages`. - It makes artifact generation part of the same cacheable lifecycle as the descriptor it describes. -- It preserves existing stage cache and secondary-repository restoration behavior. +- It preserves existing stage cache and secondary-repository restoration behavior without maintaining duplicate step abstractions. - It avoids introducing a second artifact storage model or an OCI Referrers migration. ### Alternatives considered @@ -51,7 +51,7 @@ SBOM keeps its current stable checksum inputs: artifact format, scanner/merge/GO ### Rationale -Existing checksum logic and tests already encode the required cache behavior. Reusing it minimizes behavioral risk while making stage selection aware of artifact configuration. +Existing checksum logic and tests already encode the required cache behavior. Moving that logic directly into the corresponding stages minimizes behavioral risk while making stage selection aware of artifact configuration and avoids a permanent compatibility wrapper. ### Alternatives considered diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 7e14ded2fa..e4aa7ef040 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -10,7 +10,7 @@ **Purpose**: Establish the implementation baseline without changing user-facing CLI semantics. -- [X] T001 Inventory existing stage, SBOM, VEX, artifact, storage-copy, and cleanup call paths in `pkg/build/`, `pkg/build/stage/`, `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/cleaning/`, recording the concrete integration points in the feature working notes at `specs/020-sbom-vex-build-stages/` +- [X] T001 Inventory existing stage, SBOM, VEX, artifact, storage-copy, and cleanup call paths in `pkg/build/`, `pkg/build/stage/`, `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/cleaning/`, recording the concrete integration points in `specs/020-sbom-vex-build-stages/` - [X] T002 [P] Inspect existing SBOM and VEX unit/e2e fixture conventions in `pkg/build/`, `test/e2e/sbom/`, and `test/e2e/vex/` and identify reusable helpers without adding a second test harness - [X] T003 [P] Confirm the existing fallback-tag artifact index and cleanup compatibility expectations in `pkg/oci/artifact/` and `pkg/cleaning/` before modifying propagation code @@ -18,7 +18,7 @@ ## Phase 2: Foundational (Blocking Prerequisites) -**Purpose**: Define the minimal internal contracts shared by all artifact-stage stories. +**Purpose**: Define the minimal internal contracts shared by both artifact stages. **Checkpoint**: The shared stage metadata, descriptor propagation shape, and early registry validation are understood and available before story implementation begins. @@ -31,28 +31,32 @@ ## Phase 3: User Story 1 - Artifacts follow published images (Priority: P1) 🎯 MVP -**Goal**: Generate SBOM/VEX through the image-stage lifecycle and make all attached artifacts follow images into primary, final, cache, and secondary-restored destinations. +**Goal**: Replace the transitional SBOM/VEX steps with lifecycle-owned stages and make all attached artifacts follow images into primary, final, cache, and secondary-restored destinations. **Independent Test**: Build the existing fixture with primary-only, final, cache, combined final/cache, identical-address, and secondary-repository configurations; retrieve both artifact kinds by each destination image digest. ### Tests for User Story 1 -- [X] T008 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, parent descriptor requirements, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go +- [X] T008 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, parent descriptor requirements, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` - [ ] T009 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` - [ ] T010 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` - [ ] T011 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios +- [ ] T012 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` ### Implementation for User Story 1 -- [X] T012 [P] [US1] Implement the registry-only mutable, non-buildable SBOM artifact stage in `pkg/build/stage/sbom.go` using the existing SBOM generation, signing, checksum, and fallback-index components -- [X] T013 [P] [US1] Implement the registry-only mutable, non-buildable VEX artifact stage in `pkg/build/stage/vex.go` using the existing VEX generation, signing, checksum, and fallback-index components -- [ ] T014 [US1] Register SBOM and VEX stages after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` -- [ ] T015 [US1] Execute SBOM and VEX stage publication without changing image filesystem or layer content, and remove their duplicate generation pass while retaining unrelated image publication/report work in `pkg/build/build_phase.go` -- [X] T016 [US1] Implement shared idempotent artifact propagation with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` -- [X] T017 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the shared propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` -- [X] T018 [US1] Connect secondary-stage restoration into primary storage to the same propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` +- [X] T013 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction +- [X] T014 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including VEX generation, cache identity, signing, attestation publication, and fallback-index interaction +- [ ] T015 [US1] Migrate all behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing SBOM generation, checksum, signing, and publication behavior +- [ ] T016 [US1] Migrate all behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing VEX generation, checksum, signing, and publication behavior +- [ ] T017 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` +- [ ] T018 [US1] Execute artifact publication through stage `MutateImage` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` +- [ ] T019 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` +- [X] T020 [US1] Implement shared idempotent artifact propagation with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` +- [X] T021 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the shared propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` +- [X] T022 [US1] Connect secondary-stage restoration into primary storage to the same propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` -**Checkpoint**: User Story 1 is independently functional; artifacts are generated in the lifecycle and follow every applicable published image. +**Checkpoint**: User Story 1 is independently functional; `SbomStage` and `VexStage` are the sole lifecycle owners and artifacts follow every applicable published image. --- @@ -64,17 +68,18 @@ ### Tests for User Story 2 -- [X] T019 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform subject selection in `pkg/build/artifact_subject_test.go` -- [X] T020 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and parent digest are distinct per platform in `pkg/build/sbom_step_test.go` -- [ ] T021 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each platform manifest -- [X] T022 [US2] Extend `test/e2e/vex/` with single-platform manifest placement and multi-platform index-only placement assertions +- [X] T023 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform subject selection in `pkg/build/artifact_subject_test.go` +- [X] T024 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and parent digest are distinct per platform in `pkg/build/sbom_step_test.go` +- [ ] T025 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into the stage-owned test location and ensure the final suite contains no step-specific test dependency +- [ ] T026 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each platform manifest +- [X] T027 [US2] Extend `test/e2e/vex/` with single-platform manifest placement and multi-platform index-only placement assertions ### Implementation for User Story 2 -- [X] T023 [US2] Implement explicit artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` -- [ ] T024 [US2] Pass the target platform and resolved platform manifest descriptor through SBOM stage creation and publication in `pkg/build/sbom_step.go` and `pkg/build/build_phase.go` -- [ ] T025 [US2] Make VEX stage registration run once per multi-platform image set with the top-level index subject, and use the image manifest subject for single-platform builds in `pkg/build/vex_step.go` and `pkg/build/build_phase.go` -- [X] T026 [US2] Ensure propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` +- [X] T028 [US2] Implement explicit artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` +- [ ] T029 [US2] Pass the target platform and resolved platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` +- [ ] T030 [US2] Make `VexStage` registration run once per multi-platform image set with the top-level index subject, and use the image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` +- [X] T031 [US2] Ensure propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 2 is independently testable and no artifact can silently use an index subject for a platform SBOM or duplicate multi-platform VEX onto platform manifests. @@ -82,23 +87,24 @@ ## Phase 5: User Story 3 - Rebuilds reuse or invalidate artifact results correctly (Priority: P1) -**Goal**: Preserve valid artifact cache hits while preventing stale reuse when image, scanner, merge/GOST, VEX document, platform, format, or signer inputs change. +**Goal**: Preserve valid artifact cache hits while preventing stale reuse when image, scanner, merge/GOST, VEX document, platform, format, or signing inputs change; cache ownership lives in the artifact stages. **Independent Test**: Repeat unchanged builds and then change each effective artifact input one at a time; inspect cache decisions, artifact identities, and duplicate fallback-index entries. ### Tests for User Story 3 -- [X] T027 [P] [US3] Add Ginkgo/Gomega tests for SBOM stage dependency identity across image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/sbom_step_test.go` -- [X] T028 [P] [US3] Add Ginkgo/Gomega tests for VEX stage dependency identity across parent digest, document content, format, and signer inputs in `pkg/build/vex_step_test.go` -- [X] T029 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing in `pkg/build/artifact_propagation_test.go` -- [X] T030 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios +- [X] T032 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` +- [X] T033 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` +- [X] T034 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing in `pkg/build/artifact_propagation_test.go` +- [ ] T035 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios +- [ ] T036 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests ### Implementation for User Story 3 -- [X] T031 [US3] Include all effective SBOM inputs and the parent image identity in artifact-stage dependency calculation while preserving existing checksum semantics in `pkg/build/sbom_step.go` and `pkg/build/stage/sbom.go` -- [X] T032 [US3] Include VEX document content, parent descriptor identity, format version, and signer identity in artifact-stage dependency calculation in `pkg/build/vex_step.go` and `pkg/build/stage/vex.go` -- [ ] T033 [US3] Select reusable artifact-bearing stages from primary and secondary storage using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` -- [X] T034 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/` and `pkg/build/artifact_propagation.go` +- [X] T037 [US3] Include all effective SBOM inputs and the parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` +- [X] T038 [US3] Include VEX document content, parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` +- [ ] T039 [US3] Select reusable artifact-bearing stages from primary and secondary storage using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` +- [X] T040 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/` and `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 3 is independently testable; unchanged inputs reuse artifacts and every effective changed input invalidates only the affected artifact identity. @@ -112,18 +118,18 @@ ### Tests for User Story 4 -- [X] T035 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` -- [X] T036 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go -- [X] T037 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent attachment in `pkg/oci/artifact/` -- [ ] T038 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior -- [X] T039 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories +- [X] T041 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` +- [X] T042 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` +- [X] T043 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent attachment in `pkg/oci/artifact/` +- [ ] T044 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior +- [X] T045 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories ### Implementation for User Story 4 -- [X] T040 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` -- [X] T041 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` -- [X] T042 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` -- [X] T043 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` +- [X] T046 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` +- [X] T047 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [X] T048 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` +- [X] T049 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` **Checkpoint**: User Story 4 is independently testable; registry failures and local-only configuration produce predictable results without changing repository flag semantics. @@ -131,18 +137,19 @@ ## Phase 7: Polish & Cross-Cutting Concerns -**Purpose**: Validate the complete implementation against all feature constraints and repository quality gates. +**Purpose**: Validate the complete implementation against the revised stage-ownership boundary and repository quality gates. -- [X] T044 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic -- [ ] T045 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures -- [X] T046 Run formatting with `task format` for authored Go directories -- [X] T047 Run compilation with `task build` -- [X] T048 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` -- [ ] T049 Run the complete unit suite with `task test:unit` -- [X] T050 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` -- [X] T051 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` -- [ ] T052 Run legacy integration coverage with `task test:integration` -- [X] T053 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files +- [X] T050 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic +- [ ] T051 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, and verify no step-specific tests remain after migration +- [ ] T052 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures +- [X] T053 Run formatting with `task format` for authored Go directories +- [X] T054 Run compilation with `task build` +- [X] T055 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` +- [ ] T056 Run the complete unit suite with `task test:unit` +- [X] T057 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` +- [X] T058 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` +- [ ] T059 Run legacy integration coverage with `task test:integration` +- [X] T060 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files --- @@ -151,47 +158,49 @@ ### Phase Dependencies - **Phase 1 (Setup)**: No implementation dependency; establishes the current call-path baseline. -- **Phase 2 (Foundational)**: Depends on Phase 1 and blocks all story implementation. -- **Phase 3 (US1)**: Depends on Phase 2 and is the MVP increment. -- **Phase 4 (US2)**: Depends on US1's artifact stages and propagation path because it specializes subject selection. +- **Phase 2 (Foundational)**: Depends on Phase 1 and blocks all user story implementation. +- **Phase 3 (US1)**: Depends on Phase 2 and is the MVP increment. It now includes full migration from `sbomStep`/`vexStep` and deletion of the transitional files. +- **Phase 4 (US2)**: Depends on US1's artifact stages and propagation path because it specializes subject selection; platform tests must be owned by the new stages before old step tests are deleted. - **Phase 5 (US3)**: Depends on US1 and US2 stage identity/subject contracts so cache identity includes the correct parent descriptor. -- **Phase 6 (US4)**: Depends on the shared propagation operation from US1; can be developed in parallel with US2/US3 after the shared path exists. -- **Phase 7 (Polish)**: Depends on all desired stories being complete. +- **Phase 6 (US4)**: Depends on the shared propagation operation from US1; its validation work can proceed in parallel with US2/US3 after the shared path exists. +- **Phase 7 (Polish)**: Depends on all desired stories being complete, including removal of transitional files and references. ### User Story Dependencies - **US1 (P1)**: Starts after Phase 2; no dependency on another user story. MVP. -- **US2 (P1)**: Depends on US1's stage lifecycle and shared propagation implementation. +- **US2 (P1)**: Depends on US1's `SbomStage`/`VexStage` lifecycle and shared propagation implementation. - **US3 (P1)**: Depends on US1's stages and US2's explicit parent-subject rules. -- **US4 (P2)**: Depends on US1's propagation/error path; its early-validation work can proceed independently of US2 and US3. +- **US4 (P2)**: Depends on US1's propagation/error path; early-validation tests can proceed independently of US2 and US3. ### Parallel Opportunities - Phase 1 tasks T002 and T003 can run in parallel after T001's baseline inventory. -- Within US1, T012 and T013 can run in parallel because they are separate stage files; T008 and T009 can begin as separate test files before implementation. -- Within US2, T019 and T020 are parallel unit-test tasks, and T021/T022 are parallel e2e-suite tasks. -- Within US3, T027 and T028 are parallel because SBOM and VEX identity logic is separate; T030 can proceed independently once stage contracts are stable. -- Within US4, T035, T036, and T037 are parallel test tasks, and T039 can proceed independently in cleanup files. -- After Phase 2, separate contributors can work on US1 stage files, US2 subject tests/design, and US4 validation tests, but US2/US4 integration must wait for the shared US1 propagation contract. -- Polish review and disabled-feature regression checks (T044/T045) can run in parallel before the sequential repository-wide validation commands T046–T053. +- Within US1, T013 and T014 are parallel stage files; T009 and T012 can be developed as separate test concerns before integration. T015 and T016 are parallel migrations when their callers are disjoint. +- Within US2, T023/T024 and T026/T027 are parallel test work; subject-selection implementation and VEX placement implementation can proceed in separate files. +- Within US3, T032 and T033 are parallel stage-owned identity tests; T035 can proceed independently once stage contracts are stable. +- Within US4, T041, T042, and T043 are parallel test tasks, and T045 can proceed independently in cleanup files. +- After Phase 2, separate contributors can work on stage migration, propagation, and validation tests, but deletion of transitional files (T019) must wait for all callers/tests to migrate. +- Polish review and disabled-feature regression checks (T050–T052) can run in parallel before the sequential repository-wide validation commands T053–T060. --- ## Parallel Example: User Story 1 ```text -# After Phase 2, start independent test and stage work in parallel: +# After Phase 2, start independent stage, migration, and test work: Task: T008 — stage lifecycle tests in pkg/build/stage/artifact_test.go Task: T009 — propagation tests in pkg/build/artifact_propagation_test.go -Task: T012 — SBOM stage in pkg/build/stage/sbom.go -Task: T013 — VEX stage in pkg/build/stage/vex.go -Task: T011 — SBOM e2e scenarios in test/e2e/sbom/ - -# Then integrate the stage registration and propagation wiring: -Task: T014 — register stages in pkg/build/build_phase.go -Task: T016 — implement propagation in pkg/build/artifact_propagation.go -Task: T017 — wire final/cache copies -Task: T018 — wire secondary restoration +Task: T012 — migration coverage in pkg/build/ +Task: T013 — SbomStage in pkg/build/stage/sbom.go +Task: T014 — VexStage in pkg/build/stage/vex.go +Task: T011 — repository propagation scenarios in test/e2e/sbom/ + +# Integrate after the stage contracts are stable: +Task: T015 — migrate SBOM behavior and callers +Task: T016 — migrate VEX behavior and callers +Task: T017 — register stages in pkg/build/build_phase.go +Task: T018 — remove duplicate AfterImages convergence +Task: T019 — delete transitional step files and tests ``` ## Implementation Strategy @@ -200,29 +209,33 @@ Task: T018 — wire secondary restoration 1. Complete Phase 1 baseline inspection. 2. Complete Phase 2 shared stage and propagation contracts. -3. Implement US1 artifact stages, lifecycle registration, and primary/final/cache/secondary propagation. -4. Run US1 unit and SBOM e2e tests independently. -5. Stop for validation/demo before adding platform-specific and cache-invalidation refinements. +3. Implement or verify `SbomStage` and `VexStage`. +4. Migrate all behavior and callers from `sbomStep`/`vexStep`. +5. Register the stages, connect primary/final/cache/secondary propagation, and delete the transitional files. +6. Run US1 unit and SBOM e2e tests independently. +7. Stop for validation/demo before adding platform-specific and cache-invalidation refinements. ### Incremental Delivery -1. Deliver US1 as the first usable increment: artifacts follow published images. -2. Add US2: correct manifest/index subjects and platform placement. -3. Add US3: complete dependency identity and cache reuse/invalidation. +1. Deliver US1 as the first usable increment: lifecycle-owned artifact stages with complete repository propagation. +2. Add US2: correct manifest/index subjects and platform placement, with stage-owned tests. +3. Add US3: complete stage dependency identity and cache reuse/invalidation. 4. Add US4: early validation and explicit final/cache failure behavior. 5. Run the full Polish phase and repository-required validation sequence. ### Traceability -- **FR-001–FR-003**: T012–T018, T023–T026 -- **FR-004–FR-006**: T019–T026 -- **FR-007–FR-010**: T009–T018, T036, T040–T042 -- **FR-011–FR-013**: T027–T034 -- **FR-014–FR-016**: T003, T034, T039, T043 -- **FR-017–FR-018**: T045 and all implementation tasks; no CLI flag changes or OCI Referrers migration +- **FR-001–FR-003**: T013–T022, T028–T031 +- **FR-004–FR-006**: T023–T031 +- **FR-007–FR-010**: T009–T022, T042, T046–T048 +- **FR-011–FR-013**: T032–T040 +- **FR-014–FR-016**: T003, T040, T045, T049 +- **FR-017–FR-018**: T051–T052 and all implementation tasks; no CLI flag changes or OCI Referrers migration +- **Stage ownership boundary**: T012, T015, T016, T019, T025, T036, and T051 ## Notes -- Every task uses the required `- [ ] T###` checklist format; story tasks include exactly one `[US#]` label and parallel tasks include `[P]` only where file/dependency boundaries permit. -- No external API contracts were provided in `contracts/`; the plan intentionally keeps the propagation contract internal to `pkg/build`. +- Completed tasks retain `[X]`; pending tasks use `[ ]`. Both are valid Markdown checkbox markers and every task has a sequential ID. +- No external API contracts were provided in `contracts/`; the propagation contract remains internal to `pkg/build`. +- The revised plan requires `SbomStage` and `VexStage` to be the sole lifecycle owners; `sbom_step.go`, `vex_step.go`, their step-specific callers, and their tests must not remain after migration. - No new dependencies, CLI flags, image layers, or OCI Referrers migration are planned. From daa034bd0739e7e59a078eac69014379d407d4d3 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 13:39:11 +0300 Subject: [PATCH 06/17] refactor(build): integrate artifact stages into image lifecycle Run restored artifact stages after cached image content is found and keep artifact stages out of non-empty stage tracking. Rename SBOM and VEX processors to reflect their lifecycle-owned roles. Signed-off-by: Alexandr Zaytsev --- pkg/build/artifact_propagation_test.go | 104 ++++++ pkg/build/artifact_stage_lifecycle.go | 22 ++ pkg/build/build_phase.go | 298 +++++++++++------- pkg/build/build_phase_test.go | 4 +- pkg/build/conveyor.go | 3 + pkg/build/{sbom_step.go => sbom_processor.go} | 36 +-- pkg/build/sbom_step_checksum_test.go | 173 ---------- pkg/build/sbom_step_error_test.go | 25 -- pkg/build/sbom_step_propagate_test.go | 177 ----------- pkg/build/sbom_step_test.go | 273 ---------------- pkg/build/stage/sbom.go | 2 +- pkg/build/stage/sbom_test.go | 61 ++++ pkg/build/stage/vex.go | 2 +- pkg/build/stage/vex_test.go | 19 ++ pkg/build/stages_iterator.go | 7 + pkg/build/{vex_step.go => vex_processor.go} | 8 +- pkg/build/vex_step_test.go | 124 -------- specs/020-sbom-vex-build-stages/data-model.md | 14 +- specs/020-sbom-vex-build-stages/plan.md | 25 +- specs/020-sbom-vex-build-stages/quickstart.md | 7 +- specs/020-sbom-vex-build-stages/research.md | 17 +- specs/020-sbom-vex-build-stages/tasks.md | 234 +++++++------- 22 files changed, 597 insertions(+), 1038 deletions(-) create mode 100644 pkg/build/artifact_stage_lifecycle.go rename pkg/build/{sbom_step.go => sbom_processor.go} (82%) delete mode 100644 pkg/build/sbom_step_checksum_test.go delete mode 100644 pkg/build/sbom_step_error_test.go delete mode 100644 pkg/build/sbom_step_propagate_test.go delete mode 100644 pkg/build/sbom_step_test.go create mode 100644 pkg/build/stage/sbom_test.go create mode 100644 pkg/build/stage/vex_test.go rename pkg/build/{vex_step.go => vex_processor.go} (89%) delete mode 100644 pkg/build/vex_step_test.go diff --git a/pkg/build/artifact_propagation_test.go b/pkg/build/artifact_propagation_test.go index ecf31e29d0..dda01071fb 100644 --- a/pkg/build/artifact_propagation_test.go +++ b/pkg/build/artifact_propagation_test.go @@ -59,6 +59,17 @@ var _ = Describe("artifact propagation", func() { }} } + pushImageToRepo := func(ctx SpecContext, repo string) string { + img, err := random.Image(256, 1) + Expect(err).To(Succeed()) + ref, err := name.NewTag(repo + ":v1") + Expect(err).To(Succeed()) + Expect(remote.Write(ref, img, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...)).To(Succeed()) + digest, err := img.Digest() + Expect(err).To(Succeed()) + return digest.String() + } + BeforeEach(func(ctx SpecContext) { Expect(docker_registry.Init(ctx, false, false, nil, nil)).To(Succeed()) @@ -90,6 +101,99 @@ var _ = Describe("artifact propagation", func() { Expect(err).To(MatchError(ContainSubstring("copy attached artifacts into final repo"))) }) + It("propagates artifacts to final and cache repositories", func(ctx SpecContext) { + finalRepo := strings.TrimPrefix(server.URL, "http://") + "/test/final" + cacheRepo := strings.TrimPrefix(server.URL, "http://") + "/test/cache" + finalDigest := pushImageToRepo(ctx, finalRepo) + cacheDigest := pushImageToRepo(ctx, cacheRepo) + cache := mock.NewMockStagesStorage(gomock.NewController(GinkgoT())) + cache.EXPECT().Address().Return(cacheRepo).AnyTimes() + cache.EXPECT().String().Return(cacheRepo).AnyTimes() + cache.EXPECT().GetStageDesc(gomock.Any(), "project", image.StageID{Digest: "stage-digest"}).Return(stageDescFor(cacheRepo, cacheDigest), nil) + + source := stageDescFor(sourceRepo, sourceDigest) + source.StageID = &image.StageID{Digest: "stage-digest"} + err := propagateArtifacts(ctx, "project", "app", source, stageDescFor(finalRepo, finalDigest), []storage.StagesStorage{cache}) + Expect(err).To(Succeed()) + + for _, destination := range []struct { + repo string + digest string + }{ + {repo: finalRepo, digest: finalDigest}, + {repo: cacheRepo, digest: cacheDigest}, + } { + store := artifact.NewOCIStore(destination.repo, "app", remoteOpts...) + content, err := store.GetAttachedContent(ctx, destination.digest, attestation.DSSEMediaType, nil) + Expect(err).To(Succeed()) + Expect(content).To(MatchJSON(`{"v":1}`)) + } + }) + + It("resolves the cache destination digest before propagation", func(ctx SpecContext) { + cacheRepo := strings.TrimPrefix(server.URL, "http://") + "/test/cache-digest" + cacheDigest := pushImageToRepo(ctx, cacheRepo) + cache := mock.NewMockStagesStorage(gomock.NewController(GinkgoT())) + cache.EXPECT().Address().Return(cacheRepo).AnyTimes() + cache.EXPECT().String().Return(cacheRepo).AnyTimes() + cache.EXPECT().GetStageDesc(gomock.Any(), "project", gomock.Any()).Return(stageDescFor(cacheRepo, cacheDigest), nil) + + source := stageDescFor(sourceRepo, sourceDigest) + source.StageID = &image.StageID{Digest: "stage-digest"} + Expect(propagateArtifacts(ctx, "project", "app", source, nil, []storage.StagesStorage{cache})).To(Succeed()) + + store := artifact.NewOCIStore(cacheRepo, "app", remoteOpts...) + content, err := store.GetAttachedContent(ctx, cacheDigest, attestation.DSSEMediaType, nil) + Expect(err).To(Succeed()) + Expect(content).To(MatchJSON(`{"v":1}`)) + }) + + It("skips propagation when the destination repository is identical", func(ctx SpecContext) { + destination := stageDescFor(sourceRepo, "sha256:does-not-exist") + Expect(propagateArtifacts(ctx, "project", "app", stageDescFor(sourceRepo, sourceDigest), destination, nil)).To(Succeed()) + }) + + It("deduplicates an artifact with the same identity", func(ctx SpecContext) { + destinationRepo := strings.TrimPrefix(server.URL, "http://") + "/test/dedup" + destinationDigest := pushImageToRepo(ctx, destinationRepo) + destinationStore := artifact.NewOCIStore(destinationRepo, "app", remoteOpts...) + Expect(destinationStore.Attach(ctx, destinationDigest, attestation.DSSEMediaType, []byte(`{"v":2}`), "checksum-v1", "", "")).To(Succeed()) + + Expect(propagateArtifacts(ctx, "project", "app", stageDescFor(sourceRepo, sourceDigest), stageDescFor(destinationRepo, destinationDigest), nil)).To(Succeed()) + + content, err := destinationStore.GetAttachedContent(ctx, destinationDigest, attestation.DSSEMediaType, nil) + Expect(err).To(Succeed()) + Expect(content).To(MatchJSON(`{"v":2}`)) + index, err := artifact.PullFallbackIndex(ctx, destinationRepo, destinationDigest, remoteOpts...) + Expect(err).To(Succeed()) + manifest, err := index.IndexManifest() + Expect(err).To(Succeed()) + Expect(manifest.Manifests).To(HaveLen(1)) + }) + + It("restores artifacts from a secondary repository onto the primary digest", func(ctx SpecContext) { + primaryRepo := strings.TrimPrefix(server.URL, "http://") + "/test/primary" + primaryDigest := pushImageToRepo(ctx, primaryRepo) + source := stageDescFor(sourceRepo, sourceDigest) + destination := stageDescFor(primaryRepo, primaryDigest) + + Expect(ensureAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest())).To(Succeed()) + Expect(propagateArtifacts(ctx, "project", "app", source, destination, nil)).To(Succeed()) + + store := artifact.NewOCIStore(primaryRepo, "app", remoteOpts...) + content, err := store.GetAttachedContent(ctx, primaryDigest, attestation.DSSEMediaType, nil) + Expect(err).To(Succeed()) + Expect(content).To(MatchJSON(`{"v":1}`)) + }) + + It("rejects a secondary source image without attached artifacts", func(ctx SpecContext) { + repo := strings.TrimPrefix(server.URL, "http://") + "/test/missing-artifacts" + digest := pushImageToRepo(ctx, repo) + + err := ensureAttachedArtifacts(ctx, repo, digest) + Expect(err).To(MatchError(ContainSubstring("has no attached artifacts"))) + }) + It("logs cache propagation errors and continues", func(ctx SpecContext) { var output bytes.Buffer logCtx := logboek.NewContext(ctx, logboek.NewLogger(&output, &output)) diff --git a/pkg/build/artifact_stage_lifecycle.go b/pkg/build/artifact_stage_lifecycle.go new file mode 100644 index 0000000000..4bc6e46ba7 --- /dev/null +++ b/pkg/build/artifact_stage_lifecycle.go @@ -0,0 +1,22 @@ +package build + +import ( + "context" + + "github.com/werf/werf/v2/pkg/build/image" + "github.com/werf/werf/v2/pkg/build/stage" +) + +func runRestoredArtifactStages(ctx context.Context, img *image.Image, phase Phase) error { + for _, stg := range img.GetStages() { + if artifactStage, ok := stg.(interface { + GetArtifactMetadata() *stage.ArtifactStageMetadata + }); !ok || artifactStage.GetArtifactMetadata() == nil { + continue + } + if err := phase.OnImageStage(ctx, img, stg); err != nil { + return err + } + } + return nil +} diff --git a/pkg/build/build_phase.go b/pkg/build/build_phase.go index 9fa22a7ae4..0dc4ad0451 100644 --- a/pkg/build/build_phase.go +++ b/pkg/build/build_phase.go @@ -11,7 +11,6 @@ import ( cdx "github.com/CycloneDX/cyclonedx-go" "github.com/google/uuid" "github.com/moby/buildkit/frontend/dockerfile/instructions" - "github.com/samber/lo" "github.com/sigstore/sigstore/pkg/signature" "github.com/werf/common-go/pkg/util" @@ -92,8 +91,8 @@ func NewBuildPhase(c *Conveyor, opts BuildPhaseOptions) *BuildPhase { return &BuildPhase{ BasePhase: BasePhase{c}, BuildPhaseOptions: opts, - sbomStep: newSbomStep(c.ContainerBackend, c.StorageManager.GetStagesStorage()), - vexStep: newVexStep(), + sbomProcessor: newSbomProcessor(c.ContainerBackend, c.StorageManager.GetStagesStorage()), + vexProcessor: newVexProcessor(), ImagesReport: NewImagesReport(), } } @@ -101,8 +100,8 @@ func NewBuildPhase(c *Conveyor, opts BuildPhaseOptions) *BuildPhase { type BuildPhase struct { BasePhase BuildPhaseOptions - sbomStep *sbomStep - vexStep *vexStep + sbomProcessor *sbomProcessor + vexProcessor *vexProcessor StagesIterator *StagesIterator ImagesReport *ImagesReport @@ -156,6 +155,13 @@ func (phase *BuildPhase) BeforeImages(ctx context.Context) error { func collectHolisticInputs(ctx context.Context, img *image.Image, conveyor stage.Conveyor, buildContextArchive container_backend.BuildContextArchiver) ([]string, error) { var inputs []string for _, stg := range img.GetStages() { + artifactStage, isArtifactStage := stg.(interface { + GetArtifactMetadata() *stage.ArtifactStageMetadata + }) + if isArtifactStage && artifactStage.GetArtifactMetadata() != nil { + continue + } + deps, err := stg.GetContentDependencies(ctx, conveyor, buildContextArchive) if err != nil { return nil, fmt.Errorf("stage %q GetContentDependencies: %w", stg.Name(), err) @@ -256,11 +262,11 @@ func (phase *BuildPhase) AfterImages(ctx context.Context) error { return err } - if err := phase.convergeSbomByImagesSets(ctx); err != nil { + if err := phase.convergeVexByImagesSets(ctx); err != nil { return err } - if err := phase.convergeVexByImagesSets(ctx); err != nil { + if err := phase.propagateArtifactsByImages(ctx); err != nil { return err } @@ -303,99 +309,44 @@ func (phase *BuildPhase) artifactsEnabled() bool { return false } -func (phase *BuildPhase) convergeSbomByImagesSets(ctx context.Context) error { - if !phase.Conveyor.EnableSbom() { +func (phase *BuildPhase) propagateArtifactsByImages(ctx context.Context) error { + if !phase.artifactsEnabled() { return nil } - graph := phase.Conveyor.imagesTree.GetImagesGraph() - if graph == nil || len(graph.Nodes()) == 0 { - return nil - } - - tracker := convergefailure.NewTracker(os.Getenv(externalref.EnvName)) - phase.sbomFailures = tracker - - totalImages, convergeErr := phase.doConvergeSbomByImagesSets(ctx, graph, tracker) - - return tracker.Finish(ctx, totalImages, convergeErr) -} - -func (phase *BuildPhase) doConvergeSbomByImagesSets(ctx context.Context, graph *image.ImagesGraph, tracker *convergefailure.Tracker) (int, error) { - var totalImages int - - for _, imagesInSet := range graph.Levels() { - imagesByName := make(map[string][]*image.Image) - for _, img := range imagesInSet { - imagesByName[img.Name] = append(imagesByName[img.Name], img) - } - - names := make([]string, 0, len(imagesByName)) - for name := range imagesByName { - names = append(names, name) - } - - totalImages += len(names) - - if err := parallel.DoTasks(ctx, len(names), parallel.DoTasksOptions{ - MaxNumberOfWorkers: int(phase.Conveyor.ParallelTasksLimit), - InitDockerCLIForEachWorker: true, - }, func(ctx context.Context, taskId int) error { - name := names[taskId] - images := imagesByName[name] - - if tracker.SkipDependent(ctx, name, sbomImageDependencies(images)) { - return nil + for _, pair := range phase.Conveyor.imagesTree.GetImagesByName(false) { + name, images := pair.Unpair() + for _, img := range images { + if img == nil { + continue } - - if err := phase.convergeImageSbom(ctx, name, images, tracker.Breaker()); err != nil { - return tracker.Classify(err, name) + source := contentStageDesc(img) + if source == nil { + continue + } + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, source, finalStageDescForPlatform(phase, name, images, img.TargetPlatform), phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + return fmt.Errorf("propagate artifacts for image %q: %w", name, err) } - return nil - }); err != nil { - return totalImages, err - } - } - - return totalImages, nil -} - -// sbomImageDependencies describes, for the SBOM failure semantics, the images -// whose SBOMs are merged into this image's own SBOM. -func sbomImageDependencies(images []*image.Image) []convergefailure.ImageDependencies { - return lo.Map(images, func(img *image.Image, _ int) convergefailure.ImageDependencies { - return convergefailure.ImageDependencies{ - BaseImageName: img.GetBaseImageName(), - Imports: lo.Map(img.GetImportImagesInfo(), func(importInfo image.ImportImageInfo, _ int) convergefailure.ImportSource { - return convergefailure.ImportSource{ - ImageName: importInfo.ImageName, - External: importInfo.ExternalImage, - } - }), } - }) -} - -func (phase *BuildPhase) convergeImageSbom(ctx context.Context, name string, images []*image.Image, breaker *externalref.ResolverBreaker) error { - var signer signature.Signer - var signerIdentity string - if phase.SbomSigningOptions.Enabled { - signer = phase.SbomSigningOptions.Signer().SignerVerifier() - signerIdentity = phase.SbomSigningOptions.Signer().Fingerprint() - } - for _, img := range images { - finalStageDesc := finalStageDescForPlatform(phase, name, images, img.TargetPlatform) - if err := phase.convergePlatformImageSbom(ctx, name, img, finalStageDesc, signer, signerIdentity, breaker); err != nil { - return err + if len(images) > 1 { + multiImage := phase.Conveyor.imagesTree.GetMultiplatformImage(name) + if multiImage == nil || multiImage.GetStageDesc() == nil { + continue + } + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, multiImage.GetStageDesc(), multiImage.GetFinalStageDesc(), phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + return fmt.Errorf("propagate multiplatform artifacts for image %q: %w", name, err) + } } } - return nil } -func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name string, img *image.Image, finalStageDesc *imagePkg.StageDesc, signer signature.Signer, signerIdentity string, breaker *externalref.ResolverBreaker) error { - stageDesc := contentStageDesc(img) +func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name string, img *image.Image, sourceStageDesc, finalStageDesc *imagePkg.StageDesc, signer signature.Signer, signerIdentity string, breaker *externalref.ResolverBreaker, propagate bool) error { + stageDesc := sourceStageDesc + if stageDesc == nil { + stageDesc = contentStageDesc(img) + } if stageDesc == nil { return fmt.Errorf("unable to converge sbom for image %q: stage descriptor is unavailable", name) } @@ -451,15 +402,17 @@ func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name str scanOpts := phase.scanOptionsForImage(img) - if err := phase.sbomStep.ConvergeWithMerge(ctx, name, stageDesc, scanOpts, mergeOpts, patchers, hasOsPmPackages, isStapelScratch, img.TargetPlatform, signer, signerIdentity); err != nil { + if err := phase.sbomProcessor.ConvergeWithMerge(ctx, name, stageDesc, scanOpts, mergeOpts, patchers, hasOsPmPackages, isStapelScratch, img.TargetPlatform, signer, signerIdentity); err != nil { if img.TargetPlatform != "" { return fmt.Errorf("unable to converge sbom for image %q (platform %s): %w", name, img.TargetPlatform, err) } return fmt.Errorf("unable to converge sbom for image %q: %w", name, err) } - if err := phase.sbomStep.PropagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDesc, phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { - return fmt.Errorf("unable to propagate sbom for image %q: %w", name, err) + if propagate { + if err := phase.sbomProcessor.PropagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDesc, phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + return fmt.Errorf("unable to propagate sbom for image %q: %w", name, err) + } } return nil @@ -722,8 +675,14 @@ func (phase *BuildPhase) BeforeImageStages(ctx context.Context, img *image.Image if len(stages) == 0 { return deferFn, nil } - anchor := stages[len(stages)-1] - if !anchor.IsContentAnchor() { + var anchor stage.Interface + for index := len(stages) - 1; index >= 0; index-- { + if stages[index].IsContentAnchor() { + anchor = stages[index] + break + } + } + if anchor == nil { return deferFn, nil } @@ -775,14 +734,107 @@ func (phase *BuildPhase) BeforeImageStages(ctx context.Context, img *image.Image logboek.Context(ctx).Default().LogFHighlight("Use previously built image for %s by content-based tag\n", img.LogName()) container_backend.LogImageInfoByStageDesc(ctx, stageDesc, platform) } + phase.StagesIterator.PrevStage = anchor + phase.StagesIterator.PrevNonEmptyStage = anchor + phase.StagesIterator.PrevBuiltStage = anchor } else if phase.ShouldBeBuiltMode { logboek.Context(ctx).Warn().LogFHighlight("Content-based digest %s for image %s not found\n", anchor.GetDigest(), img.LogName()) logboek.Context(ctx).Warn().LogLn() } + phase.registerSbomStage(img) + if err := phase.registerSinglePlatformVexStage(ctx, img); err != nil { + return deferFn, err + } + return deferFn, nil } +func (phase *BuildPhase) registerSinglePlatformVexStage(ctx context.Context, img *image.Image) error { + if img == nil || img.Vex() == nil || img.Vex().Document == "" { + return nil + } + + images := phase.Conveyor.imagesTree.GetImagesByName(false) + for _, pair := range images { + name, imageSet := pair.Unpair() + if name == img.Name && len(imageSet) != 1 { + return nil + } + } + for _, existing := range img.GetStages() { + if existing.Name() == stage.Vex { + return nil + } + } + + vexContent, err := phase.Conveyor.GiterminismManager().FileReader().ReadVEXFile(ctx, img.Vex().Document) + if err != nil { + return fmt.Errorf("read VEX file %q for image %q: %w", img.Vex().Document, img.Name, err) + } + baseOptions := &stage.BaseStageOptions{ + TargetPlatform: img.TargetPlatform, + ImageName: img.Name, + ImageTmpDir: img.TmpDir, + ContainerWerfDir: img.ContainerWerfDir, + ProjectName: phase.Conveyor.ProjectName(), + } + var signingOptions signing.VexSigningOptions + if phase.VexSigningOptions.Enabled { + signingOptions = phase.VexSigningOptions + } + stages := img.GetStages() + img.SetStages(append(stages, stage.GenerateVexStage(vexContent, baseOptions, signingOptions))) + return nil +} + +func (phase *BuildPhase) registerSbomStage(img *image.Image) { + if img == nil || !phase.Conveyor.EnableSbom() { + return + } + for _, existing := range img.GetStages() { + if existing.Name() == stage.Sbom { + return + } + } + + stages := img.GetStages() + if len(stages) == 0 { + return + } + if phase.sbomFailures == nil { + phase.sbomFailures = convergefailure.NewTracker(os.Getenv(externalref.EnvName)) + } + + baseOptions := &stage.BaseStageOptions{ + TargetPlatform: img.TargetPlatform, + ImageName: img.Name, + ImageTmpDir: img.TmpDir, + ContainerWerfDir: img.ContainerWerfDir, + ProjectName: phase.Conveyor.ProjectName(), + } + dependency := phase.scanOptionsForImage(img).Checksum() + if sbomConfig := img.Sbom(); sbomConfig != nil { + dependency = util.Sha256Hash( + dependency, + "standard", fmt.Sprintf("%d", sbomConfig.Standard), + "gost_attack_surface", sbomConfig.Gost.AttackSurface.String(), + "gost_security_function", sbomConfig.Gost.SecurityFunction.String(), + ) + } + var signer signature.Signer + var signerIdentity string + if phase.SbomSigningOptions.Enabled { + signer = phase.SbomSigningOptions.Signer().SignerVerifier() + signerIdentity = phase.SbomSigningOptions.Signer().Fingerprint() + } + publisher := func(ctx context.Context, parentDesc *imagePkg.StageDesc, _, targetPlatform string) error { + return phase.convergePlatformImageSbom(ctx, img.Name, img, parentDesc, nil, signer, signerIdentity, nil, false) + } + artifactStage := stage.GenerateSbomStage(baseOptions, phase.SbomSigningOptions, dependency, publisher) + img.SetStages(append(stages, artifactStage)) +} + func (phase *BuildPhase) AfterImageStages(ctx context.Context, img *image.Image) error { img.SetLastNonEmptyStage(phase.StagesIterator.PrevNonEmptyStage) return nil @@ -1239,6 +1291,17 @@ func (phase *BuildPhase) calculateStage(ctx context.Context, img *image.Image, s phase.Conveyor.GetStageDigestMutex(stg.GetDigest()).Lock() }) + if artifactStage, ok := stg.(interface { + GetArtifactMetadata() *stage.ArtifactStageMetadata + }); ok && artifactStage.GetArtifactMetadata() != nil { + stageContentSig, err := calculateDigest(ctx, fmt.Sprintf("%s-content", stg.Name()), "", stg, phase.Conveyor, calculateDigestOptions{TargetPlatform: img.TargetPlatform}) + if err != nil { + return false, phase.Conveyor.GetStageDigestMutex(stg.GetDigest()).Unlock, fmt.Errorf("unable to calculate artifact stage %s content digest: %w", stg.Name(), err) + } + stg.SetContentDigest(stageContentSig) + return false, phase.Conveyor.GetStageDigestMutex(stg.GetDigest()).Unlock, nil + } + storageManager := phase.Conveyor.StorageManager stageDescSet, err := storageManager.GetStageDescSetByDigestWithCache(ctx, stg.LogDetailedName(), stageDigest, phase.getPrevNonEmptyStageCreationTsForStage(stg)) if err != nil { @@ -1420,6 +1483,20 @@ func (phase *BuildPhase) buildStage(ctx context.Context, img *image.Image, stg s func (phase *BuildPhase) atomicBuildStageImage(ctx context.Context, img *image.Image, stg stage.Interface) error { stageImage := stg.GetStageImage() + if artifactStage, ok := stg.(interface { + GetArtifactMetadata() *stage.ArtifactStageMetadata + }); ok && artifactStage.GetArtifactMetadata() != nil { + prevBuiltImage := phase.StagesIterator.GetPrevBuiltImage(img, stg) + if prevBuiltImage == nil || prevBuiltImage.Image == nil { + return fmt.Errorf("expected previous built image for artifact stage %s", stg.Name()) + } + if err := stg.MutateImage(ctx, phase.Conveyor.StorageManager.GetStagesStorage(), prevBuiltImage, stageImage); err != nil { + return fmt.Errorf("unable to mutate %s: %w", stg.Name(), err) + } + stageImage.Image.SetStageDesc(prevBuiltImage.Image.GetStageDesc()) + return nil + } + if stg.IsBuildable() { if err := logboek.Context(ctx).Streams().DoErrorWithTag(fmt.Sprintf("%s/%s", img.LogName(), stg.Name()), img.LogTagStyle(), func() error { opts := phase.ImageBuildOptions @@ -1766,23 +1843,19 @@ func (phase *BuildPhase) convergeVexByImagesSets(ctx context.Context) error { return nil } -func (phase *BuildPhase) vexStageDesc(name string, images []*image.Image) *imagePkg.StageDesc { - if len(images) == 1 { - return images[0].GetLastNonEmptyStageDesc() - } - - if multiImg := phase.Conveyor.imagesTree.GetMultiplatformImage(name); multiImg != nil { - return multiImg.GetStageDesc() - } - - return nil -} - func (phase *BuildPhase) convergeImageVex(ctx context.Context, name string, images []*image.Image) error { if len(images) == 0 { return nil } + if len(images) == 1 { + for _, stg := range images[0].GetStages() { + if stg.Name() == stage.Vex { + return nil + } + } + } + primaryImg := images[0] vexConfig := primaryImg.Vex() @@ -1790,7 +1863,14 @@ func (phase *BuildPhase) convergeImageVex(ctx context.Context, name string, imag return nil } - stageDesc := phase.vexStageDesc(name, images) + var stageDesc *imagePkg.StageDesc + if len(images) == 1 { + stageDesc = contentStageDesc(primaryImg) + } else if multiImg := phase.Conveyor.imagesTree.GetMultiplatformImage(name); multiImg != nil { + stageDesc = multiImg.GetStageDesc() + } else { + stageDesc = image.NewMultiplatformImage(name, images, 0, 1).GetStageDesc() + } if stageDesc == nil { return fmt.Errorf("unable to converge VEX for image %q: stage descriptor is unavailable", name) } @@ -1809,7 +1889,7 @@ func (phase *BuildPhase) convergeImageVex(ctx context.Context, name string, imag signerIdentity = phase.VexSigningOptions.Signer().Fingerprint() } - if err := phase.vexStep.Converge(ctx, vexContent, stageDesc, name, vexTargetPlatform(images), signer, signerIdentity); err != nil { + if err := phase.vexProcessor.Converge(ctx, vexContent, stageDesc, name, vexTargetPlatform(images), signer, signerIdentity); err != nil { return fmt.Errorf("unable to converge VEX for image %q: %w", name, err) } @@ -1853,7 +1933,7 @@ func (phase *BuildPhase) collectBaseImageSbom(ctx context.Context, img *image.Im return nil, nil } - baseImageSbom, err := phase.sbomStep.GetImageBOM(ctx, img.GetBaseImageName(), baseImageInfo) + baseImageSbom, err := phase.sbomProcessor.GetImageBOM(ctx, img.GetBaseImageName(), baseImageInfo) if err != nil { if errors.Is(err, ErrSbomNotRequired) { return nil, nil @@ -1905,7 +1985,7 @@ func (phase *BuildPhase) collectImportImageSboms(ctx context.Context, img *image importLookupName = importInfo.ImageName } - importImageSbom, err := phase.sbomStep.GetImageBOM(ctx, importLookupName, importImageInfo) + importImageSbom, err := phase.sbomProcessor.GetImageBOM(ctx, importLookupName, importImageInfo) if err != nil { if errors.Is(err, ErrSbomNotRequired) { continue diff --git a/pkg/build/build_phase_test.go b/pkg/build/build_phase_test.go index 059daa0ae6..fa2d44802a 100644 --- a/pkg/build/build_phase_test.go +++ b/pkg/build/build_phase_test.go @@ -129,13 +129,13 @@ var _ = Describe("BuildPhase", func() { }) }) - It("skips SBOM convergence when no images were selected", func(ctx SpecContext) { + It("skips artifact propagation when no images were selected", func(ctx SpecContext) { phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{ werfConfig: &config.WerfConfig{Meta: &config.Meta{Build: config.MetaBuild{Sbom: &config.MetaBuildSbom{Enable: true}}}}, imagesTree: &image.ImagesTree{}, }}} - Expect(phase.convergeSbomByImagesSets(ctx)).To(Succeed()) + Expect(phase.propagateArtifactsByImages(ctx)).To(Succeed()) }) It("collects content dependencies from signing mutation stages", func(ctx SpecContext) { diff --git a/pkg/build/conveyor.go b/pkg/build/conveyor.go index 39655792f6..cf9b542503 100644 --- a/pkg/build/conveyor.go +++ b/pkg/build/conveyor.go @@ -942,6 +942,9 @@ func (c *Conveyor) doImage(ctx context.Context, img *image.Image, phases []Phase if contentTagDesc := img.GetContentTagDesc(); contentTagDesc != nil { logboek.Context(ctx).LogOptionalLn() + if err := runRestoredArtifactStages(ctx, img, phase); err != nil { + return fmt.Errorf("run restored artifact stages for image %s: %w", img.GetLogName(), err) + } return nil } diff --git a/pkg/build/sbom_step.go b/pkg/build/sbom_processor.go similarity index 82% rename from pkg/build/sbom_step.go rename to pkg/build/sbom_processor.go index ee528e9ede..c33ede039f 100644 --- a/pkg/build/sbom_step.go +++ b/pkg/build/sbom_processor.go @@ -27,7 +27,7 @@ import ( "github.com/werf/werf/v2/pkg/werf/global_warnings" ) -//go:generate mockgen -source sbom_step.go -package mock -destination ../../test/mock/bom_patcher.go -mock_names BOMPatcherInterface=MockBOMPatcher +//go:generate mockgen -source sbom_processor.go -package mock -destination ../../test/mock/bom_patcher.go -mock_names BOMPatcherInterface=MockBOMPatcher type BOMPatcherInterface interface { Apply(ctx context.Context, bom *cdx.BOM) (*cdx.BOM, error) @@ -37,34 +37,34 @@ type BOMPatcherInterface interface { // (e.g. it is a trusted builder image). Callers should handle this silently. var ErrSbomNotRequired = errors.New("sbom not required") -type sbomStep struct { +type sbomProcessor struct { containerBackend container_backend.ContainerBackend stagesStorage storage.StagesStorage gostWarnOnce sync.Once } -func newSbomStep( +func newSbomProcessor( backend container_backend.ContainerBackend, stagesStorage storage.StagesStorage, -) *sbomStep { - return &sbomStep{ +) *sbomProcessor { + return &sbomProcessor{ containerBackend: backend, stagesStorage: stagesStorage, } } -func (step *sbomStep) ConvergeWithMerge(ctx context.Context, werfImgName string, stageDesc *image.StageDesc, scanOpts scanner.ScanOptions, mergeOpts cyclonedxutil.MergeOpts, patchers []BOMPatcherInterface, osPmEnabled, isStapelScratch bool, targetPlatform string, signer signature.Signer, signerIdentity string) error { +func (processor *sbomProcessor) ConvergeWithMerge(ctx context.Context, werfImgName string, stageDesc *image.StageDesc, scanOpts scanner.ScanOptions, mergeOpts cyclonedxutil.MergeOpts, patchers []BOMPatcherInterface, osPmEnabled, isStapelScratch bool, targetPlatform string, signer signature.Signer, signerIdentity string) error { repo := stageDesc.Info.Repository parentDigest := stageDesc.Info.GetDigest() scanOpts.Commands[0].SourcePath = stageDesc.Info.Name - if err := step.prepareGostComponents(ctx, &mergeOpts); err != nil { + if err := processor.prepareGostComponents(ctx, &mergeOpts); err != nil { return err } - checksum := step.calculateStableChecksum(scanOpts, mergeOpts, signerIdentity, targetPlatform) + checksum := processor.calculateStableChecksum(scanOpts, mergeOpts, signerIdentity, targetPlatform) store := artifact.NewOCIStore(repo, werfImgName) @@ -77,7 +77,7 @@ func (step *sbomStep) ConvergeWithMerge(ctx context.Context, werfImgName string, return nil } - if err := step.containerBackend.Pull(ctx, stageDesc.Info.Name, container_backend.PullOpts{TargetPlatform: targetPlatform}); err != nil { + if err := processor.containerBackend.Pull(ctx, stageDesc.Info.Name, container_backend.PullOpts{TargetPlatform: targetPlatform}); err != nil { return fmt.Errorf("unable to pull %q: %w", stageDesc.Info.Name, err) } @@ -94,7 +94,7 @@ func (step *sbomStep) ConvergeWithMerge(ctx context.Context, werfImgName string, }, } } else { - bomJSON, err := step.containerBackend.GenerateSBOM(ctx, scanOpts) + bomJSON, err := processor.containerBackend.GenerateSBOM(ctx, scanOpts) if err != nil { return fmt.Errorf("generate SBOM: %w", err) } @@ -117,7 +117,7 @@ func (step *sbomStep) ConvergeWithMerge(ctx context.Context, werfImgName string, } if osPmEnabled { - pmBOM, err := osPm.CollectBOM(ctx, step.containerBackend, stageDesc.Info.Name) + pmBOM, err := osPm.CollectBOM(ctx, processor.containerBackend, stageDesc.Info.Name) if err != nil { return fmt.Errorf("collect os-pm BOM: %w", err) } @@ -187,7 +187,7 @@ const sbomArtifactFormatVersion = "2" // (build context changes alter the stage digest), external reference enrichment // (non-deterministic external data), and generator logic changes (covered by // sbomArtifactFormatVersion). -func (step *sbomStep) calculateStableChecksum(scanOpts scanner.ScanOptions, mergeOpts cyclonedxutil.MergeOpts, signerIdentity, targetPlatform string) string { +func (processor *sbomProcessor) calculateStableChecksum(scanOpts scanner.ScanOptions, mergeOpts cyclonedxutil.MergeOpts, signerIdentity, targetPlatform string) string { return util.Sha256Hash( sbomArtifactFormatVersion, "scan", scanOpts.Checksum(), @@ -202,16 +202,16 @@ func (step *sbomStep) calculateStableChecksum(scanOpts scanner.ScanOptions, merg // PropagateArtifacts copies the artifacts attached to the image stage (e.g. its SBOM) // into the final repo and the cache repos. Stages themselves are copied there before // SBOM generation runs, so the artifacts have to catch up separately. -func (step *sbomStep) PropagateArtifacts(ctx context.Context, projectName, werfImgName string, stageDesc, finalStageDesc *image.StageDesc, cacheStagesStorageList []storage.StagesStorage) error { +func (processor *sbomProcessor) PropagateArtifacts(ctx context.Context, projectName, werfImgName string, stageDesc, finalStageDesc *image.StageDesc, cacheStagesStorageList []storage.StagesStorage) error { return propagateArtifacts(ctx, projectName, werfImgName, stageDesc, finalStageDesc, cacheStagesStorageList) } -func (step *sbomStep) GetImageBOM(ctx context.Context, imageName string, imageInfo *image.Info) (*cdx.BOM, error) { +func (processor *sbomProcessor) GetImageBOM(ctx context.Context, imageName string, imageInfo *image.Info) (*cdx.BOM, error) { if imageInfo == nil { return nil, fmt.Errorf("image info is nil for %q", imageName) } - bom, err := step.pullImageSbom(ctx, imageName, imageInfo) + bom, err := processor.pullImageSbom(ctx, imageName, imageInfo) if err != nil { if isTrustedBuilderImage(imageInfo.Labels) { switch { @@ -236,7 +236,7 @@ func sbomMissingError(imageInfo *image.Info, err error) error { return fmt.Errorf("the image %q must have an SBOM artifact attached; to generate an SBOM for the image, rebuild it with SBOM generation enabled; note: if the image is a multi-platform image built by an older werf version, its SBOM is attached in a legacy platform-ambiguous format and cannot be used — rebuild the image with a newer werf version: %w", imageInfo.Name, err) } -func (step *sbomStep) pullImageSbom(ctx context.Context, imageName string, imageInfo *image.Info) (*cdx.BOM, error) { +func (processor *sbomProcessor) pullImageSbom(ctx context.Context, imageName string, imageInfo *image.Info) (*cdx.BOM, error) { parentDigest := imageInfo.GetDigest() if parentDigest == "" { return nil, fmt.Errorf("image digest not available for %q", imageInfo.Name) @@ -255,9 +255,9 @@ func (step *sbomStep) pullImageSbom(ctx context.Context, imageName string, image return bom, nil } -func (step *sbomStep) prepareGostComponents(ctx context.Context, mergeOpts *cyclonedxutil.MergeOpts) error { +func (processor *sbomProcessor) prepareGostComponents(ctx context.Context, mergeOpts *cyclonedxutil.MergeOpts) error { if !mergeOpts.Gost.AttackSurface.IsUndefined() || !mergeOpts.Gost.SecurityFunction.IsUndefined() { - step.gostWarnOnce.Do(func() { + processor.gostWarnOnce.Do(func() { logboek.Context(ctx).Default().LogF("Warning: GOST SBOM integration is experimental and its behavior may change in the future\n") }) } diff --git a/pkg/build/sbom_step_checksum_test.go b/pkg/build/sbom_step_checksum_test.go deleted file mode 100644 index fdb3fb862a..0000000000 --- a/pkg/build/sbom_step_checksum_test.go +++ /dev/null @@ -1,173 +0,0 @@ -package build - -import ( - cdx "github.com/CycloneDX/cyclonedx-go" - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil" - "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil/gost" - "github.com/werf/werf/v2/pkg/sbom/scanner" -) - -var _ = Describe("SbomStep Checksum", func() { - type checksumInputs struct { - scanOpts scanner.ScanOptions - mergeOpts cyclonedxutil.MergeOpts - signerIdentity string - targetPlatform string - } - - baseline := func() checksumInputs { - return checksumInputs{ - scanOpts: scanner.ScanOptions{}, - mergeOpts: cyclonedxutil.MergeOpts{}, - } - } - - checksumOf := func(in checksumInputs) string { - step := &sbomStep{} - return step.calculateStableChecksum(in.scanOpts, in.mergeOpts, in.signerIdentity, in.targetPlatform) - } - - It("same inputs produce same checksum", func() { - Expect(checksumOf(baseline())).To(Equal(checksumOf(baseline()))) - }) - - DescribeTable("changes when a single input changes", - func(mutate func(in *checksumInputs)) { - mutated := baseline() - mutate(&mutated) - Expect(checksumOf(mutated)).NotTo(Equal(checksumOf(baseline()))) - }, - Entry("scan options", func(in *checksumInputs) { - in.scanOpts = scanner.ScanOptions{Commands: []scanner.ScanCommand{{SourcePath: "image"}}} - }), - Entry("merge options: base BOM", func(in *checksumInputs) { - in.mergeOpts.BaseBOM = &cdx.BOM{ - BOMFormat: "CycloneDX", - SpecVersion: cdx.SpecVersion1_6, - Components: &[]cdx.Component{{Name: "base-lib", Version: "1.0.0"}}, - } - }), - Entry("gost attack surface", func(in *checksumInputs) { - in.mergeOpts.Gost.AttackSurface = gost.GostValueYes - }), - Entry("gost security function", func(in *checksumInputs) { - in.mergeOpts.Gost.SecurityFunction = gost.GostValueIndirect - }), - Entry("signer identity", func(in *checksumInputs) { - in.signerIdentity = "signer:abc" - }), - Entry("target platform", func(in *checksumInputs) { - in.targetPlatform = "linux/amd64" - }), - ) - - It("GOST config changes checksum even without base and import BOMs", func() { - withGost := baseline() - withGost.mergeOpts.Gost = gost.Config{ - AttackSurface: gost.GostValueYes, - SecurityFunction: gost.GostValueIndirect, - } - - Expect(withGost.mergeOpts.IsEmpty()).To(BeTrue()) - Expect(checksumOf(withGost)).NotTo(Equal(checksumOf(baseline()))) - }) - - It("different signer identities produce different checksums", func() { - first := baseline() - first.signerIdentity = "signer:key1" - - second := baseline() - second.signerIdentity = "signer:key2" - - Expect(checksumOf(first)).NotTo(Equal(checksumOf(second))) - }) - - It("format version change invalidates cache", func() { - Expect(checksumOf(baseline())).NotTo(Equal("aa969eabe2faad149265a94e60b173e527e0bc27898afcd0ec4e85a06b28f29b"), - "checksum must differ from format-v1 era (before format version was added)") - }) - - Describe("target platform", func() { - It("differs between platforms", func() { - amd64 := baseline() - amd64.targetPlatform = "linux/amd64" - - arm64 := baseline() - arm64.targetPlatform = "linux/arm64" - - Expect(checksumOf(amd64)).NotTo(Equal(checksumOf(arm64))) - }) - - It("is stable for the same platform", func() { - platform := baseline() - platform.targetPlatform = "linux/arm64" - - Expect(checksumOf(platform)).To(Equal(checksumOf(platform))) - }) - - It("changes checksum independently of signer identity", func() { - signedPlatformless := baseline() - signedPlatformless.signerIdentity = "signer:abc" - - signedPlatform := signedPlatformless - signedPlatform.targetPlatform = "linux/amd64" - - Expect(checksumOf(signedPlatformless)).NotTo(Equal(checksumOf(signedPlatform))) - }) - }) - - Describe("part encoding", func() { - It("does not collide when a part value absorbs a slot boundary", func() { - // A separator-joined encoding maps both of these onto the same input: - // "...-a-b" from a single part "a-b", and "...-a-b" from parts "a" and "b". - joinedIntoSigner := baseline() - joinedIntoSigner.signerIdentity = "a-b" - - splitAcrossParts := baseline() - splitAcrossParts.signerIdentity = "a" - splitAcrossParts.targetPlatform = "b" - - Expect(checksumOf(joinedIntoSigner)).NotTo(Equal(checksumOf(splitAcrossParts))) - }) - - It("does not collide when the same value moves between adjacent parts", func() { - asSigner := baseline() - asSigner.signerIdentity = "linux/amd64" - - asPlatform := baseline() - asPlatform.targetPlatform = "linux/amd64" - - Expect(checksumOf(asSigner)).NotTo(Equal(checksumOf(asPlatform))) - }) - - It("yields pairwise distinct checksums across single-input flips", func() { - flips := map[string]checksumInputs{"baseline": baseline()} - - scanFlip := baseline() - scanFlip.scanOpts = scanner.ScanOptions{Commands: []scanner.ScanCommand{{SourcePath: "image"}}} - flips["scan"] = scanFlip - - gostFlip := baseline() - gostFlip.mergeOpts.Gost.AttackSurface = gost.GostValueYes - flips["gost"] = gostFlip - - signerFlip := baseline() - signerFlip.signerIdentity = "signer:abc" - flips["signer"] = signerFlip - - platformFlip := baseline() - platformFlip.targetPlatform = "linux/arm64" - flips["platform"] = platformFlip - - seen := map[string]string{} - for name, in := range flips { - sum := checksumOf(in) - Expect(seen).NotTo(HaveKey(sum), "checksum of %q collides with %q", name, seen[sum]) - seen[sum] = name - } - }) - }) -}) diff --git a/pkg/build/sbom_step_error_test.go b/pkg/build/sbom_step_error_test.go deleted file mode 100644 index 92dd113dc2..0000000000 --- a/pkg/build/sbom_step_error_test.go +++ /dev/null @@ -1,25 +0,0 @@ -package build - -import ( - "errors" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/werf/werf/v2/pkg/image" -) - -var _ = Describe("SbomStep SBOM missing error", func() { - It("keeps the attach guidance, stays dependency-kind neutral, and adds the legacy multi-platform hint", func() { - cause := errors.New("pull SBOM for \"base\": artifact not found") - err := sbomMissingError(&image.Info{Name: "registry.example.com/base:tag"}, cause) - - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("registry.example.com/base:tag")) - Expect(err.Error()).To(ContainSubstring("must have an SBOM artifact attached")) - Expect(err.Error()).To(ContainSubstring("rebuild the image with a newer werf version")) - Expect(err.Error()).To(ContainSubstring("legacy platform-ambiguous format")) - Expect(err.Error()).NotTo(ContainSubstring("base image"), "GetImageBOM serves both base and import dependencies; the message must not claim the image is a base") - Expect(errors.Is(err, cause)).To(BeTrue()) - }) -}) diff --git a/pkg/build/sbom_step_propagate_test.go b/pkg/build/sbom_step_propagate_test.go deleted file mode 100644 index dfab26873b..0000000000 --- a/pkg/build/sbom_step_propagate_test.go +++ /dev/null @@ -1,177 +0,0 @@ -package build - -import ( - "net/http/httptest" - "strings" - - "github.com/google/go-containerregistry/pkg/authn" - "github.com/google/go-containerregistry/pkg/name" - "github.com/google/go-containerregistry/pkg/registry" - "github.com/google/go-containerregistry/pkg/v1/random" - "github.com/google/go-containerregistry/pkg/v1/remote" - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - "go.uber.org/mock/gomock" - - "github.com/werf/werf/v2/pkg/attestation" - "github.com/werf/werf/v2/pkg/docker_registry" - werfImage "github.com/werf/werf/v2/pkg/image" - "github.com/werf/werf/v2/pkg/oci/artifact" - "github.com/werf/werf/v2/pkg/storage" - "github.com/werf/werf/v2/test/mock" -) - -var _ = Describe("SbomStep PropagateArtifacts", func() { - var ( - server *httptest.Server - srcRepo string - finalRepo string - cacheRepo string - srcDigest string - remoteOpts []remote.Option - ) - - pushRandomImage := func(ctx SpecContext, repo string) string { - img, err := random.Image(256, 1) - Expect(err).To(Succeed()) - - ref, err := name.NewTag(repo + ":v1") - Expect(err).To(Succeed()) - Expect(remote.Write(ref, img, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...)).To(Succeed()) - - dgst, err := img.Digest() - Expect(err).To(Succeed()) - return dgst.String() - } - - copyImageByDigest := func(ctx SpecContext, fromRepo, toRepo, digest string) { - fromRef, err := name.NewDigest(fromRepo + "@" + digest) - Expect(err).To(Succeed()) - img, err := remote.Image(fromRef, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...) - Expect(err).To(Succeed()) - - toRef, err := name.NewDigest(toRepo + "@" + digest) - Expect(err).To(Succeed()) - Expect(remote.Write(toRef, img, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...)).To(Succeed()) - } - - stageDescFor := func(repo, digest string) *werfImage.StageDesc { - return &werfImage.StageDesc{ - StageID: &werfImage.StageID{}, - Info: &werfImage.Info{ - Repository: repo, - RepoDigest: repo + "@" + digest, - }, - } - } - - cacheStorage := func(address string) *mock.MockStagesStorage { - s := mock.NewMockStagesStorage(gomock.NewController(GinkgoT())) - s.EXPECT().Address().Return(address).AnyTimes() - s.EXPECT().String().Return(address).AnyTimes() - return s - } - - BeforeEach(func(ctx SpecContext) { - Expect(docker_registry.Init(ctx, false, false, nil, nil)).To(Succeed()) - - server = httptest.NewServer(registry.New()) - host := strings.TrimPrefix(server.URL, "http://") - srcRepo = host + "/test/stages" - finalRepo = host + "/test/final" - cacheRepo = host + "/test/cache" - remoteOpts = []remote.Option{remote.WithAuth(authn.Anonymous)} - - srcDigest = pushRandomImage(ctx, srcRepo) - - srcStore := artifact.NewOCIStore(srcRepo, "app", remoteOpts...) - Expect(srcStore.Attach(ctx, srcDigest, attestation.DSSEMediaType, []byte(`{"v":1}`), "checksum-v1", "", "")).To(Succeed()) - }) - - AfterEach(func() { - server.Close() - }) - - It("should copy the SBOM into the final repo", func(ctx SpecContext) { - copyImageByDigest(ctx, srcRepo, finalRepo, srcDigest) - - step := &sbomStep{} - source := stageDescFor(srcRepo, srcDigest) - destination := stageDescFor(finalRepo, srcDigest) - Expect(step.PropagateArtifacts(ctx, "test", "app", source, destination, nil)).To(Succeed()) - Expect(step.PropagateArtifacts(ctx, "test", "app", source, destination, nil)).To(Succeed()) - - finalStore := artifact.NewOCIStore(finalRepo, "app", remoteOpts...) - content, err := finalStore.GetAttachedContent(ctx, srcDigest, attestation.DSSEMediaType, nil) - Expect(err).To(Succeed()) - Expect(content).To(MatchJSON(`{"v":1}`)) - - index, err := artifact.PullFallbackIndex(ctx, finalRepo, srcDigest, remoteOpts...) - Expect(err).To(Succeed()) - manifest, err := index.IndexManifest() - Expect(err).To(Succeed()) - Expect(manifest.Manifests).To(HaveLen(1)) - }) - - It("should copy the SBOM into cache repos", func(ctx SpecContext) { - copyImageByDigest(ctx, srcRepo, cacheRepo, srcDigest) - - step := &sbomStep{} - caches := []storage.StagesStorage{ - cacheStorage(storage.LocalStorageAddress), - cacheStorage(srcRepo), - cacheStorage(cacheRepo), - } - Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), nil, caches)).To(Succeed()) - - cacheStore := artifact.NewOCIStore(cacheRepo, "app", remoteOpts...) - content, err := cacheStore.GetAttachedContent(ctx, srcDigest, attestation.DSSEMediaType, nil) - Expect(err).To(Succeed()) - Expect(content).To(MatchJSON(`{"v":1}`)) - }) - - It("should propagate artifacts to the cache image digest", func(ctx SpecContext) { - destinationDigest := pushRandomImage(ctx, cacheRepo) - cache := cacheStorage(cacheRepo) - cache.EXPECT().GetStageDesc(gomock.Any(), "test", werfImage.StageID{Digest: "stage-digest"}).Return(&werfImage.StageDesc{ - StageID: &werfImage.StageID{Digest: "stage-digest"}, - Info: &werfImage.Info{ - Repository: cacheRepo, - RepoDigest: cacheRepo + "@" + destinationDigest, - }, - }, nil) - - step := &sbomStep{} - source := stageDescFor(srcRepo, srcDigest) - source.StageID = &werfImage.StageID{Digest: "stage-digest"} - Expect(step.PropagateArtifacts(ctx, "test", "app", source, nil, []storage.StagesStorage{cache})).To(Succeed()) - - cacheStore := artifact.NewOCIStore(cacheRepo, "app", remoteOpts...) - content, err := cacheStore.GetAttachedContent(ctx, destinationDigest, attestation.DSSEMediaType, nil) - Expect(err).To(Succeed()) - Expect(content).To(MatchJSON(`{"v":1}`)) - }) - - It("should do nothing without a final repo and caches", func(ctx SpecContext) { - step := &sbomStep{} - Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), nil, nil)).To(Succeed()) - }) - - It("should skip the final repo when it matches the stages repo", func(ctx SpecContext) { - step := &sbomStep{} - Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), stageDescFor(srcRepo, srcDigest), nil)).To(Succeed()) - }) - - It("should not fail when a cache repo is unreachable", func(ctx SpecContext) { - step := &sbomStep{} - caches := []storage.StagesStorage{cacheStorage("127.0.0.1:1/unreachable/cache")} - Expect(step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), nil, caches)).To(Succeed()) - }) - - It("should fail when the final repo copy fails", func(ctx SpecContext) { - step := &sbomStep{} - err := step.PropagateArtifacts(ctx, "test", "app", stageDescFor(srcRepo, srcDigest), stageDescFor("127.0.0.1:1/unreachable/final", srcDigest), nil) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("copy attached artifacts into final repo")) - }) -}) diff --git a/pkg/build/sbom_step_test.go b/pkg/build/sbom_step_test.go deleted file mode 100644 index 6117aeebea..0000000000 --- a/pkg/build/sbom_step_test.go +++ /dev/null @@ -1,273 +0,0 @@ -package build - -import ( - "bytes" - "context" - "errors" - "net/http/httptest" - "path/filepath" - "strings" - - cdx "github.com/CycloneDX/cyclonedx-go" - "github.com/google/go-containerregistry/pkg/authn" - "github.com/google/go-containerregistry/pkg/name" - "github.com/google/go-containerregistry/pkg/registry" - "github.com/google/go-containerregistry/pkg/v1/random" - "github.com/google/go-containerregistry/pkg/v1/remote" - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - "go.uber.org/mock/gomock" - - "github.com/werf/logboek" - "github.com/werf/werf/v2/pkg/docker_registry" - werfImage "github.com/werf/werf/v2/pkg/image" - "github.com/werf/werf/v2/pkg/logging" - "github.com/werf/werf/v2/pkg/oci/artifact" - "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil" - "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil/gost" - "github.com/werf/werf/v2/pkg/sbom/gomod" - "github.com/werf/werf/v2/pkg/sbom/scanner" - "github.com/werf/werf/v2/test/mock" -) - -var _ = Describe("SbomStep", func() { - It("publishes platform metadata against the matching parent digest", func(ctx SpecContext) { - Expect(docker_registry.Init(ctx, false, false, nil, nil)).To(Succeed()) - server := httptest.NewServer(registry.New()) - DeferCleanup(server.Close) - repo := strings.TrimPrefix(server.URL, "http://") + "/test/images" - remoteOpts := []remote.Option{remote.WithAuth(authn.Anonymous)} - backend := mock.NewMockContainerBackend(gomock.NewController(GinkgoT())) - step := newSbomStep(backend, nil) - bomJSON := []byte(`{"bomFormat":"CycloneDX","specVersion":"1.6","version":1}`) - platforms := []struct { - name, platform string - }{ - {name: "amd64", platform: "linux/amd64"}, - {name: "arm64", platform: "linux/arm64"}, - } - parentDigests := make(map[string]string, len(platforms)) - artifactDigests := make(map[string]string, len(platforms)) - - for _, item := range platforms { - parent, err := random.Image(256, 1) - Expect(err).To(Succeed()) - parentRef, err := name.NewTag(repo + ":" + item.name) - Expect(err).To(Succeed()) - Expect(remote.Write(parentRef, parent, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...)).To(Succeed()) - parentDigest, err := parent.Digest() - Expect(err).To(Succeed()) - parentDigests[item.platform] = parentDigest.String() - stageName := repo + ":stage-" + item.name - backend.EXPECT().Pull(gomock.Any(), stageName, gomock.Any()).Return(nil) - backend.EXPECT().GenerateSBOM(gomock.Any(), gomock.Any()).Return(bomJSON, nil) - stageDesc := &werfImage.StageDesc{Info: &werfImage.Info{ - Name: stageName, Repository: repo, RepoDigest: repo + "@" + parentDigest.String(), Tag: "stage-" + item.name, - }} - Expect(step.ConvergeWithMerge(ctx, "app", stageDesc, scanner.ScanOptions{Commands: []scanner.ScanCommand{{}}}, cyclonedxutil.MergeOpts{}, nil, false, false, item.platform, nil, "")).To(Succeed()) - index, err := artifact.PullFallbackIndex(ctx, repo, parentDigest.String(), remoteOpts...) - Expect(err).To(Succeed()) - manifest, err := index.IndexManifest() - Expect(err).To(Succeed()) - Expect(manifest.Manifests).To(HaveLen(1)) - artifactDescriptor := manifest.Manifests[0] - artifactDigests[item.platform] = artifactDescriptor.Digest.String() - artifactRef, err := name.NewDigest(repo + "@" + artifactDescriptor.Digest.String()) - Expect(err).To(Succeed()) - artifactImage, err := remote.Image(artifactRef, append([]remote.Option{remote.WithContext(ctx)}, remoteOpts...)...) - Expect(err).To(Succeed()) - artifactManifest, err := artifactImage.Manifest() - Expect(err).To(Succeed()) - Expect(artifactManifest.Annotations).To(HaveKeyWithValue(werfImage.WerfPlatformAnnotation, item.platform)) - Expect(artifactManifest.Subject).NotTo(BeNil()) - Expect(artifactManifest.Subject.Digest.String()).To(Equal(parentDigest.String())) - } - Expect(parentDigests[platforms[0].platform]).NotTo(Equal(parentDigests[platforms[1].platform])) - Expect(artifactDigests[platforms[0].platform]).NotTo(Equal(artifactDigests[platforms[1].platform])) - }) - - Describe("prepareGostComponents", func() { - It("prints the GOST experimental warning at most once per step instance", func() { - var output bytes.Buffer - ctx := logboek.NewContext(context.Background(), logboek.NewLogger(&output, &output)) - - step := &sbomStep{} - mergeOpts := cyclonedxutil.MergeOpts{Gost: gost.Config{AttackSurface: gost.GostValueYes, SecurityFunction: gost.GostValueYes}} - - Expect(step.prepareGostComponents(ctx, &mergeOpts)).To(Succeed()) - Expect(step.prepareGostComponents(ctx, &mergeOpts)).To(Succeed()) - Expect(step.prepareGostComponents(ctx, &mergeOpts)).To(Succeed()) - - Expect(strings.Count(output.String(), "GOST SBOM integration is experimental")).To(Equal(1)) - }) - }) - - Describe("GetImageBOM()", func() { - It("should return error if image info is nil", func(ctx SpecContext) { - step := &sbomStep{} - _, err := step.GetImageBOM(ctx, "app", nil) - Expect(err).To(HaveOccurred()) - Expect(err.Error()).To(ContainSubstring("image info is nil")) - }) - - It("should return fatal error if image digest is empty", func(ctx SpecContext) { - step := &sbomStep{} - imgInfo := &werfImage.Info{Name: "app:latest"} - _, err := step.GetImageBOM(ctx, "app", imgInfo) - Expect(err).To(HaveOccurred()) - Expect(errors.Is(err, ErrSbomNotRequired)).To(BeFalse()) - }) - }) - - Describe("BOMPatcher (gomod)", func() { - DescribeTable("Apply()", - func( - ctx context.Context, - setupGitRepo func(ctx context.Context, repo *mock.MockGitRepo, commit, imageContext string), - ) { - repo := mock.NewMockGitRepo(gomock.NewController(GinkgoT())) - commit := "0123456789abcdef0123456789abcdef01234567" - imageContext := "app" - setupGitRepo(ctx, repo, commit, imageContext) - - patcher := gomod.NewBOMPatcher(repo, commit, imageContext) - bom := &cdx.BOM{ - Metadata: &cdx.Metadata{ - Component: &cdx.Component{ - Name: "app", - }, - }, - } - - res, err := patcher.Apply(ctx, bom) - Expect(err).ToNot(HaveOccurred()) - Expect(res).ToNot(BeNil()) - }, - Entry( - "[go.mod]: should skip version resolution when go.mod is missing", - logging.WithLogger(context.Background()), - func(ctx context.Context, repo *mock.MockGitRepo, commit, imageContext string) { - repo.EXPECT().IsCommitFileExist(ctx, commit, filepath.Join(imageContext, "go.mod")).Return(false, nil) - }, - ), - Entry( - "[go.mod]: should use tag version when tag matches commit", - logging.WithLogger(context.Background()), - func(ctx context.Context, repo *mock.MockGitRepo, commit, imageContext string) { - goModPath := filepath.Join(imageContext, "go.mod") - repo.EXPECT().IsCommitFileExist(ctx, commit, goModPath).Return(true, nil) - repo.EXPECT().ReadCommitFile(ctx, commit, goModPath).Return([]byte("module example.com/app\n"), nil) - repo.EXPECT().TagsList(ctx).Return([]string{"v1.2.3"}, nil) - repo.EXPECT().TagCommit(ctx, "v1.2.3").Return(commit, nil) - }, - ), - Entry( - "[go.mod]: should fallback to pseudo version when tag mismatch", - logging.WithLogger(context.Background()), - func(ctx context.Context, repo *mock.MockGitRepo, commit, imageContext string) { - goModPath := filepath.Join(imageContext, "go.mod") - repo.EXPECT().IsCommitFileExist(ctx, commit, goModPath).Return(true, nil) - repo.EXPECT().ReadCommitFile(ctx, commit, goModPath).Return([]byte("module example.com/app\n"), nil) - repo.EXPECT().TagsList(ctx).Return([]string{"v1.2.3"}, nil) - repo.EXPECT().TagCommit(ctx, "v1.2.3").Return("deadbeefdeadbeefdeadbeefdeadbeefdeadbeef", nil) - }, - ), - ) - }) - - Describe("isTrustedBuilderImage()", func() { - DescribeTable("should detect trusted builder images", - func(labels map[string]string, expected bool) { - Expect(isTrustedBuilderImage(labels)).To(Equal(expected)) - }, - Entry("nil labels", nil, false), - Entry("empty labels", map[string]string{}, false), - Entry("label set to false", map[string]string{werfImage.DeckhouseInternalBuilderLabel: "false"}, false), - Entry("label set to true", map[string]string{werfImage.DeckhouseInternalBuilderLabel: "true"}, true), - Entry("other labels without builder", map[string]string{"foo": "bar", "baz": "qux"}, false), - Entry("other labels with builder true", map[string]string{"foo": "bar", werfImage.DeckhouseInternalBuilderLabel: "true", "baz": "qux"}, true), - ) - }) - - Describe("GetImageBOM() with trusted builder image", func() { - It("should return hard error for builder image from different namespace", func(ctx SpecContext) { - step := &sbomStep{} - - imageInfo := &werfImage.Info{ - Name: "docker.io/namespace/repo:builder-tag", - Repository: "docker.io/namespace/repo", - Labels: map[string]string{ - werfImage.DeckhouseInternalBuilderLabel: "true", - }, - } - - _, err := step.GetImageBOM(ctx, "builder-image", imageInfo) - Expect(err).To(HaveOccurred()) - Expect(errors.Is(err, ErrSbomNotRequired)).To(BeFalse()) - Expect(err.Error()).To(ContainSubstring("the image is a builder image but SBOM is required")) - }) - - It("should return ErrSbomNotRequired for golang builder image from container-factory", func(ctx SpecContext) { - step := &sbomStep{} - - imageInfo := &werfImage.Info{ - Name: "registry.deckhouse.io/container-factory/builder/golang-alpine:1.25", - Repository: "registry.deckhouse.io/container-factory/builder/golang-alpine", - Labels: map[string]string{ - werfImage.DeckhouseInternalBuilderLabel: "true", - }, - } - - _, err := step.GetImageBOM(logging.WithLogger(ctx), "builder-image", imageInfo) - Expect(err).To(MatchError(ErrSbomNotRequired)) - }) - - It("should return ErrSbomNotRequired for alpine builder image from container-factory", func(ctx SpecContext) { - step := &sbomStep{} - - imageInfo := &werfImage.Info{ - Name: "registry.deckhouse.io/container-factory/builder/alpine:3.22", - Repository: "registry.deckhouse.io/container-factory/builder/alpine", - Labels: map[string]string{ - werfImage.DeckhouseInternalBuilderLabel: "true", - }, - } - - _, err := step.GetImageBOM(ctx, "builder-image", imageInfo) - Expect(err).To(MatchError(ErrSbomNotRequired)) - }) - - It("should return hard error for other builder image from container-factory", func(ctx SpecContext) { - step := &sbomStep{} - - imageInfo := &werfImage.Info{ - Name: "registry.deckhouse.io/container-factory/builder/scratch", - Repository: "registry.deckhouse.io/container-factory/builder/scratch", - Labels: map[string]string{ - werfImage.DeckhouseInternalBuilderLabel: "true", - }, - } - - _, err := step.GetImageBOM(ctx, "builder-image", imageInfo) - Expect(err).To(HaveOccurred()) - Expect(errors.Is(err, ErrSbomNotRequired)).To(BeFalse()) - Expect(err.Error()).To(ContainSubstring("the image is a builder image but SBOM is required")) - }) - - It("should return actionable error for non-builder image when SBOM pull fails", func(ctx SpecContext) { - step := &sbomStep{} - - imageInfo := &werfImage.Info{ - Name: "docker.io/namespace/repo:some-tag", - Repository: "docker.io/namespace/repo", - Labels: map[string]string{}, - } - - _, err := step.GetImageBOM(ctx, "app", imageInfo) - Expect(err).To(HaveOccurred()) - Expect(errors.Is(err, ErrSbomNotRequired)).To(BeFalse()) - Expect(err.Error()).NotTo(ContainSubstring(werfImage.DeckhouseInternalBuilderLabel)) - Expect(err.Error()).To(ContainSubstring("rebuild it with SBOM generation enabled")) - }) - }) -}) diff --git a/pkg/build/stage/sbom.go b/pkg/build/stage/sbom.go index 9caed8bced..72785f47bb 100644 --- a/pkg/build/stage/sbom.go +++ b/pkg/build/stage/sbom.go @@ -108,7 +108,7 @@ func (s *SbomStage) MutateImage(ctx context.Context, stagesStorage ImageMutatorP metadata := s.GetArtifactMetadata() metadata.ParentDigest = parentDesc.Info.GetDigest() - return s.publisher(ctx, parentDesc, stageImage.Image.Name(), s.TargetPlatform()) + return s.publisher(ctx, parentDesc, s.ImageName(), s.TargetPlatform()) } const sbomArtifactFormatVersion = "2" diff --git a/pkg/build/stage/sbom_test.go b/pkg/build/stage/sbom_test.go new file mode 100644 index 0000000000..27a23f1f29 --- /dev/null +++ b/pkg/build/stage/sbom_test.go @@ -0,0 +1,61 @@ +package stage + +import ( + "context" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "go.uber.org/mock/gomock" + + "github.com/werf/werf/v2/pkg/build/signing" + "github.com/werf/werf/v2/pkg/image" + "github.com/werf/werf/v2/test/mock" +) + +var _ = Describe("SbomStage dependencies", func() { + It("changes when the target platform changes", func(ctx SpecContext) { + newStage := func(platform string) *SbomStage { + return GenerateSbomStage(&BaseStageOptions{TargetPlatform: platform}, signing.SbomSigningOptions{}, "scanner-input", func(context.Context, *image.StageDesc, string, string) error { + return nil + }) + } + + amd, err := newStage("linux/amd64").GetDependencies(ctx, nil, nil, nil, nil, nil) + Expect(err).To(Succeed()) + arm, err := newStage("linux/arm64").GetDependencies(ctx, nil, nil, nil, nil, nil) + Expect(err).To(Succeed()) + + Expect(amd).NotTo(Equal(arm)) + }) + + It("changes when the effective SBOM inputs change", func(ctx SpecContext) { + newDependencies := func(inputs string) string { + stage := GenerateSbomStage(&BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.SbomSigningOptions{}, inputs, func(context.Context, *image.StageDesc, string, string) error { + return nil + }) + dependencies, err := stage.GetDependencies(ctx, nil, nil, nil, nil, nil) + Expect(err).To(Succeed()) + return dependencies + } + + Expect(newDependencies("scanner=v1;merge=v1;gost=yes")).NotTo(Equal(newDependencies("scanner=v2;merge=v1;gost=yes"))) + Expect(newDependencies("scanner=v1;merge=v1;gost=yes")).NotTo(Equal(newDependencies("scanner=v1;merge=v2;gost=yes"))) + }) + + It("changes when the parent manifest digest changes", func(ctx SpecContext) { + newDependencies := func(digest string) string { + ctrl := gomock.NewController(GinkgoT()) + parentImage := mock.NewMockLegacyImageInterface(ctrl) + parentImage.EXPECT().GetStageDesc().Return(&image.StageDesc{Info: &image.Info{RepoDigest: "repo@" + digest}}) + parent := NewStageImage(NewContainerBackendStub(), "", parentImage) + stage := GenerateSbomStage(&BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.SbomSigningOptions{}, "scanner-input", func(context.Context, *image.StageDesc, string, string) error { + return nil + }) + dependencies, err := stage.GetDependencies(ctx, nil, nil, nil, parent, nil) + Expect(err).To(Succeed()) + return dependencies + } + + Expect(newDependencies("sha256:one")).NotTo(Equal(newDependencies("sha256:two"))) + }) +}) diff --git a/pkg/build/stage/vex.go b/pkg/build/stage/vex.go index fec12e933a..385f1a2a4a 100644 --- a/pkg/build/stage/vex.go +++ b/pkg/build/stage/vex.go @@ -120,7 +120,7 @@ func (s *VexStage) MutateImage(ctx context.Context, stagesStorage ImageMutatorPu } return logboek.Context(ctx).Default().LogProcess("image %s: Published VEX artifact", stageImage.Image.Name()).DoError(func() error { - return vexImage.PushVEX(ctx, s.vexJSON, parentDesc.Info.Repository, parentDigest, stageImage.Image.Name(), checksum, s.TargetPlatform(), s.signer) + return vexImage.PushVEX(ctx, s.vexJSON, parentDesc.Info.Repository, parentDigest, s.ImageName(), checksum, s.TargetPlatform(), s.signer) }) } diff --git a/pkg/build/stage/vex_test.go b/pkg/build/stage/vex_test.go new file mode 100644 index 0000000000..71ff64c7a2 --- /dev/null +++ b/pkg/build/stage/vex_test.go @@ -0,0 +1,19 @@ +package stage + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/werf/werf/v2/pkg/build/signing" +) + +var _ = Describe("VexStage dependencies", func() { + It("changes when the VEX document changes", func(ctx SpecContext) { + first, err := GenerateVexStage([]byte(`{"statements":[]}`), &BaseStageOptions{TargetPlatform: ""}, signing.VexSigningOptions{}).GetDependencies(ctx, nil, nil, nil, nil, nil) + Expect(err).To(Succeed()) + second, err := GenerateVexStage([]byte(`{"statements":[{"status":"not_affected"}]}`), &BaseStageOptions{TargetPlatform: ""}, signing.VexSigningOptions{}).GetDependencies(ctx, nil, nil, nil, nil, nil) + Expect(err).To(Succeed()) + + Expect(first).NotTo(Equal(second)) + }) +}) diff --git a/pkg/build/stages_iterator.go b/pkg/build/stages_iterator.go index f3766cb0b3..a057477a77 100644 --- a/pkg/build/stages_iterator.go +++ b/pkg/build/stages_iterator.go @@ -68,6 +68,13 @@ func (iterator *StagesIterator) OnImageStage(ctx context.Context, img *build_ima iterator.PrevStage = stg if !isEmpty { + artifactStage, isArtifactStage := stg.(interface { + GetArtifactMetadata() *stage.ArtifactStageMetadata + }) + if isArtifactStage && artifactStage.GetArtifactMetadata() != nil { + return nil + } + iterator.PrevNonEmptyStage = stg if iterator.PrevNonEmptyStage.GetStageImage().Image.GetStageDesc() != nil { diff --git a/pkg/build/vex_step.go b/pkg/build/vex_processor.go similarity index 89% rename from pkg/build/vex_step.go rename to pkg/build/vex_processor.go index e3bf51cc7e..8a95755fa3 100644 --- a/pkg/build/vex_step.go +++ b/pkg/build/vex_processor.go @@ -15,13 +15,13 @@ import ( vexImage "github.com/werf/werf/v2/pkg/vex/image" ) -type vexStep struct{} +type vexProcessor struct{} -func newVexStep() *vexStep { - return &vexStep{} +func newVexProcessor() *vexProcessor { + return &vexProcessor{} } -func (step *vexStep) Converge(ctx context.Context, vexJSON []byte, stageDesc *image.StageDesc, werfImgName, targetPlatform string, signer signature.Signer, signerIdentity string) error { +func (processor *vexProcessor) Converge(ctx context.Context, vexJSON []byte, stageDesc *image.StageDesc, werfImgName, targetPlatform string, signer signature.Signer, signerIdentity string) error { repo := stageDesc.Info.Repository parentDigest := stageDesc.Info.GetDigest() diff --git a/pkg/build/vex_step_test.go b/pkg/build/vex_step_test.go deleted file mode 100644 index 5d72568d1f..0000000000 --- a/pkg/build/vex_step_test.go +++ /dev/null @@ -1,124 +0,0 @@ -package build - -import ( - v1 "github.com/google/go-containerregistry/pkg/v1" - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - "go.uber.org/mock/gomock" - - "github.com/werf/werf/v2/pkg/attestation" - "github.com/werf/werf/v2/pkg/image" - "github.com/werf/werf/v2/pkg/vex" - "github.com/werf/werf/v2/test/mock" -) - -var _ = Describe("VexStep", func() { - Describe("checkVEXPublishNeeded", func() { - const ( - parentDigest = "sha256:abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890" - matchingChecksum = "abc123" - differentChecksum = "xyz789" - ) - - It("should skip VEX publish when a bundle artifact exists with matching checksum", func(ctx SpecContext) { - ctrl := gomock.NewController(GinkgoT()) - defer ctrl.Finish() - - mockStore := mock.NewMockStore(ctrl) - mockStore.EXPECT().GetAttached(ctx, parentDigest, attestation.BundleMediaType, vex.VEXPredicateTypes).Return( - v1.Descriptor{ - Annotations: map[string]string{ - image.WerfChecksumAnnotation: matchingChecksum, - }, - }, true, nil, - ) - - needed, err := checkVEXPublishNeeded(ctx, mockStore, parentDigest, matchingChecksum) - Expect(err).ToNot(HaveOccurred()) - Expect(needed).To(BeFalse()) - }) - - It("should skip VEX publish when a bare-DSSE artifact exists with matching checksum", func(ctx SpecContext) { - ctrl := gomock.NewController(GinkgoT()) - defer ctrl.Finish() - - mockStore := mock.NewMockStore(ctrl) - mockStore.EXPECT().GetAttached(ctx, parentDigest, attestation.BundleMediaType, vex.VEXPredicateTypes).Return( - v1.Descriptor{}, false, nil, - ) - mockStore.EXPECT().GetAttached(ctx, parentDigest, vex.DSSEMediaType, vex.VEXPredicateTypes).Return( - v1.Descriptor{ - Annotations: map[string]string{ - image.WerfChecksumAnnotation: matchingChecksum, - }, - }, true, nil, - ) - - needed, err := checkVEXPublishNeeded(ctx, mockStore, parentDigest, matchingChecksum) - Expect(err).ToNot(HaveOccurred()) - Expect(needed).To(BeFalse()) - }) - - It("should proceed with VEX publish when checksum differs", func(ctx SpecContext) { - ctrl := gomock.NewController(GinkgoT()) - defer ctrl.Finish() - - mockStore := mock.NewMockStore(ctrl) - mockStore.EXPECT().GetAttached(ctx, parentDigest, attestation.BundleMediaType, vex.VEXPredicateTypes).Return( - v1.Descriptor{}, false, nil, - ) - mockStore.EXPECT().GetAttached(ctx, parentDigest, vex.DSSEMediaType, vex.VEXPredicateTypes).Return( - v1.Descriptor{ - Annotations: map[string]string{ - image.WerfChecksumAnnotation: differentChecksum, - }, - }, true, nil, - ) - - needed, err := checkVEXPublishNeeded(ctx, mockStore, parentDigest, matchingChecksum) - Expect(err).ToNot(HaveOccurred()) - Expect(needed).To(BeTrue()) - }) - - It("should proceed with VEX publish when no artifact exists", func(ctx SpecContext) { - ctrl := gomock.NewController(GinkgoT()) - defer ctrl.Finish() - - mockStore := mock.NewMockStore(ctrl) - mockStore.EXPECT().GetAttached(ctx, parentDigest, attestation.BundleMediaType, vex.VEXPredicateTypes).Return( - v1.Descriptor{}, false, nil, - ) - mockStore.EXPECT().GetAttached(ctx, parentDigest, vex.DSSEMediaType, vex.VEXPredicateTypes).Return( - v1.Descriptor{}, false, nil, - ) - - needed, err := checkVEXPublishNeeded(ctx, mockStore, parentDigest, matchingChecksum) - Expect(err).ToNot(HaveOccurred()) - Expect(needed).To(BeTrue()) - }) - }) - - Describe("calculateVEXChecksum", func() { - const parentDigest = "sha256:abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890" - - vexJSON := []byte(`{"@context":"https://openvex.dev/ns/v0.2.0","statements":[]}`) - - It("is stable for identical inputs", func() { - Expect(calculateVEXChecksum(vexJSON, parentDigest, "")).To(Equal(calculateVEXChecksum(vexJSON, parentDigest, ""))) - }) - - DescribeTable("changes when any identity component changes", - func(otherJSON []byte, otherDigest, otherIdentity string) { - base := calculateVEXChecksum(vexJSON, parentDigest, "") - Expect(calculateVEXChecksum(otherJSON, otherDigest, otherIdentity)).ToNot(Equal(base)) - }, - Entry("document content changed", []byte(`{"@context":"https://openvex.dev/ns/v0.2.0","statements":[{}]}`), parentDigest, ""), - Entry("image digest changed", vexJSON, "sha256:1111111111111111111111111111111111111111111111111111111111111111", ""), - Entry("signing enabled (fingerprint added)", vexJSON, parentDigest, "fingerprint-a"), - ) - - It("changes when the signing key is rotated", func() { - Expect(calculateVEXChecksum(vexJSON, parentDigest, "fingerprint-a")).ToNot(Equal(calculateVEXChecksum(vexJSON, parentDigest, "fingerprint-b"))) - }) - }) -}) diff --git a/specs/020-sbom-vex-build-stages/data-model.md b/specs/020-sbom-vex-build-stages/data-model.md index ddb02b1f60..8844b15248 100644 --- a/specs/020-sbom-vex-build-stages/data-model.md +++ b/specs/020-sbom-vex-build-stages/data-model.md @@ -2,27 +2,29 @@ ## Artifact stage -An internal build-stage operation associated with an image (and, for SBOM, a target platform). `SbomStage` and `VexStage` are the sole owners of their respective generation, cache, signing, and publication behavior; no `sbomStep` or `vexStep` compatibility layer remains. +An internal build-stage operation associated with the final image digest (and, for SBOM, a target platform). `SbomStage` and `VexStage` are the sole owners of their respective generation, cache, signing, and publication behavior; no `sbomStep` or `vexStep` compatibility layer remains. These stages are OCI-artifact stages, not image-content stages. | Field | Description | |---|---| | Stage name | Stable stage identifier for cache/logging and stage selection. | -| Parent descriptor | The image manifest or image index that the artifact describes. | +| Final image descriptor | The final image manifest or image index whose digest is the artifact subject. | | Target platform | Required for platform-specific SBOM; empty for image-level multi-platform VEX. | | Artifact kind | CycloneDX SBOM or OpenVEX. | | Generation inputs | Scanner/merge inputs for SBOM, document content for VEX, format version, and signer identity. | -| Mutable/buildable flags | Non-buildable and mutable, matching registry-only stages such as signing. | +| Mutable/buildable flags | Non-buildable and mutable, matching registry-only stages such as signing, but unlike signing the output is an associated OCI artifact rather than a manifest mutation. | +| Storage abstraction | `storage.StagesStorage` used for all registry operations. | Validation rules: -- A parent descriptor must be available before `MutateImage` runs. +- The final image descriptor/digest must be available before `MutateImage` runs. +- The stage must use `storage.StagesStorage` for registry access. - SBOM for a multi-platform image must use the corresponding platform manifest. - VEX must use the platform manifest for single-platform images and the top-level index for multi-platform images. - An enabled artifact stage requires registry-backed storage. ## Artifact identity -The cache identity stored with the existing fallback artifact index and calculated directly by the owning artifact stage. +The cache identity stored with the existing fallback artifact index and calculated directly by the owning artifact stage. It identifies the associated final image digest, not a synthetic artifact image. | Field | Description | |---|---| @@ -56,4 +58,4 @@ primary image + artifacts -> final image + artifacts primary image + artifacts -> cache image + artifacts ``` -The artifact stage does not become an image layer. It publishes separate OCI artifacts whose subjects are resolved image descriptors. Existing fallback-tag indexes remain the source of truth and remain readable by current consumers. +The artifact stage does not become an image layer and does not operate on an image filesystem. It publishes separate OCI artifacts whose subjects are the final image descriptors. All registry interaction is performed through `storage.StagesStorage`; existing fallback-tag indexes remain the source of truth and remain readable by current consumers. diff --git a/specs/020-sbom-vex-build-stages/plan.md b/specs/020-sbom-vex-build-stages/plan.md index c4cb0e3ce1..ad213cec8a 100644 --- a/specs/020-sbom-vex-build-stages/plan.md +++ b/specs/020-sbom-vex-build-stages/plan.md @@ -6,7 +6,7 @@ ## Summary -Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and replace the `sbomStep` and `vexStep` implementations with registry-backed, non-buildable mutable stages modeled after `pkg/build/stage/sign.go`. The new `SbomStage` and `VexStage` become the sole owners of SBOM/VEX cache identity, generation, signing, attestation publication, and fallback-index interaction while publishing separate OCI artifacts. +Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and replace the `sbomStep` and `vexStep` implementations with registry-backed, non-buildable mutable stages modeled after `pkg/build/stage/sign.go`. The new `SbomStage` and `VexStage` become the sole owners of SBOM/VEX cache identity, generation, signing, attestation publication, and fallback-index interaction. Unlike ordinary image stages, they are associated with the final image digest, operate on the associated OCI artifact rather than on the image filesystem or image layers, and perform all registry operations through `storage.StagesStorage`. Artifact publication will use explicit source and destination image descriptors. SBOM remains platform-specific; VEX is attached once at the top-level image index for multi-platform images and to the image manifest for single-platform images. A shared idempotent propagation operation will cover primary-to-final, primary-to-cache, and secondary-to-primary copies, resolving the destination digest and preserving fatal final-repository versus best-effort cache error policies. @@ -21,10 +21,11 @@ Artifact publication will use explicit source and destination image descriptors. - Existing `pkg/build/sbom_step.go` and `pkg/build/vex_step.go` are transitional sources only and must be removed after their logic is migrated. - Existing OCI artifact and fallback-index operations in `pkg/oci/artifact` and `pkg/attestation`. - Existing registry/storage copy operations in `pkg/storage`, `pkg/storage/manager`, and `pkg/docker_registry`. +- `storage.StagesStorage` in `pkg/storage/stages_storage.go` is the required registry boundary. It must be extended with the minimal OCI-artifact operations needed by `SbomStage`, `VexStage`, and propagation; stage code must not call concrete registry clients directly. - Existing signing options in `pkg/build/signing`. - Ginkgo + Gomega test framework and existing e2e fixtures. -**Storage**: OCI registry for image manifests/indexes and fallback-tag artifact indexes; local Buildah/container storage remains supported when artifacts are disabled. +**Storage**: OCI registry for image manifests/indexes and fallback-tag artifact indexes, accessed through the `storage.StagesStorage` abstraction; local Buildah/container storage remains supported when artifacts are disabled. **Testing**: Co-located Ginkgo/Gomega unit tests, existing `test/e2e/sbom` and `test/e2e/vex` suites, and legacy integration tests. @@ -46,6 +47,7 @@ Artifact publication will use explicit source and destination image descriptors. *GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design.* - **Simplicity over abstraction**: PASS. Use two explicit stages, `SbomStage` and `VexStage`, instead of retaining parallel step and stage abstractions. Add one focused shared propagation path rather than duplicating repository-copy logic. +- **Stage distinction**: PASS. The plan explicitly requires that SBOM/VEX stages are final-image-digest-associated OCI-artifact stages, not ordinary image-mutating stages, and that registry access goes through `storage.StagesStorage`. - **Go idioms and errors**: PASS. New public methods, if required, take `context.Context` first; errors wrap operation context; stage-specific helpers remain private where possible. - **Minimal public surface**: PASS. Artifact stages and propagation contracts are internal to `pkg/build`; no new CLI flags or external API are planned. - **Testing**: PASS. Tests remain alongside source and use Ginkgo/Gomega. E2E coverage extends existing SBOM/VEX suites rather than introducing a parallel harness. @@ -73,7 +75,10 @@ Detailed findings are in [research.md](./research.md). Key decisions: - Extend `pkg/build/stage` with stage names and constructors for `SbomStage` and `VexStage`, following the shape of `SignStage`. - Move all behavior currently owned by `sbomStep` into `SbomStage`; remove `sbom_step.go` and its step-specific tests once callers are migrated. - Move all behavior currently owned by `vexStep` into `VexStage`; remove `vex_step.go` and its step-specific tests once callers are migrated. -- Artifact stages must not mutate the image filesystem. Their `PrepareImage` path is a no-op; their `MutateImage` path owns registry-side generation and publication. +- Artifact stages must not mutate, rebuild, fetch, or store the image filesystem. Their `PrepareImage` path is a no-op; their `MutateImage` path operates on the associated OCI artifact and owns registry-side generation and publication. +- The stage's subject is the final image digest: for single-platform images this is the published image manifest digest; for multi-platform images SBOM uses each final platform manifest digest and VEX uses the final top-level image index digest. The artifact stage must never be treated as an image layer or as a replacement image. +- All registry reads, writes, copies, metadata operations, and artifact-related repository interaction from `SbomStage` and `VexStage` must use `storage.StagesStorage`, following the existing build-stage storage path. Direct registry client access from the stages is prohibited. +- Extend `StagesStorage` only with the minimal artifact-oriented methods required to preserve the existing fallback-index behavior: find/list attached artifacts, publish an OCI artifact for a final image digest, and copy attached artifacts between destination image descriptors. Implement these methods for every supported registry-backed storage implementation and keep local storage behavior explicit. - Ensure stage dependencies include the parent image identity and all effective artifact inputs. SBOM dependencies include scanner, merge/GOST, signer, format version, and target platform. VEX dependencies include document content, parent identity, signer, and format version. - Register the stages after the content-producing stage and before the lifecycle completes for applicable images. The registration must work for Stapel and Dockerfile image paths and for restored stages. - Preserve stage cache behavior: a suitable artifact-bearing stage can be selected from primary/secondary storage; changed effective inputs produce a different stage identity. @@ -88,7 +93,8 @@ Detailed findings are in [research.md](./research.md). Key decisions: ### Publication and propagation -- Consolidate artifact copying behind a kind-neutral internal operation that copies every attached supported artifact from a source descriptor to a destination descriptor. This propagation helper is the only shared artifact operation; generation remains owned independently by `SbomStage` and `VexStage`. +- Consolidate artifact copying behind a kind-neutral internal operation exposed through or implemented using `storage.StagesStorage`; it copies every attached supported artifact from a source descriptor to a destination descriptor. This propagation helper is the only shared artifact operation; generation remains owned independently by `SbomStage` and `VexStage`. +- The propagation contract must carry the final image digest/descriptor explicitly and must never attach an artifact to the digest of the artifact stage itself. - Use it after primary-to-final and primary-to-cache image copies, and when a suitable stage is copied from `--secondary-repo` into primary storage. - Resolve the destination image descriptor/digest rather than assuming source and destination digests match. - Skip local storage and identical repository addresses. Deduplicate by existing artifact identity/fallback index semantics. @@ -135,8 +141,9 @@ pkg/build/ pkg/build/sbom_step.go and pkg/build/vex_step.go are removed after migration; their behavior is not retained behind compatibility wrappers. -pkg/storage/manager/ # secondary/final/cache image-copy artifact propagation hooks -pkg/oci/artifact/ # shared artifact copy/deduplication support if needed +pkg/storage/ # StagesStorage interface and backend implementations for artifact stages +pkg/storage/manager/ # secondary/final/cache image-copy artifact propagation hooks +pkg/oci/artifact/ # OCI artifact encoding and fallback-index support called by StagesStorage backends Tests remain co-located under pkg/build and pkg/build/stage, with scenario coverage in: test/e2e/sbom/ @@ -159,10 +166,10 @@ Completed in [data-model.md](./data-model.md) and [quickstart.md](./quickstart.m The subsequent `/speckit-tasks` workflow should decompose at least these work items: -1. Implement `SbomStage` and `VexStage`, including stage identity, lifecycle integration, generation, cache checks, signing, and publication. +1. Implement `SbomStage` and `VexStage`, including stage identity, lifecycle integration, final image-digest association, OCI-artifact handling, cache checks, signing, publication, and `StagesStorage` access. 2. Migrate all SBOM behavior from `sbomStep` into `SbomStage`, then delete the step implementation and update callers/tests. 3. Migrate all VEX behavior from `vexStep` into `VexStage`, then delete the step implementation and update callers/tests. -4. Implement shared artifact propagation for final, cache, and secondary-to-primary copies. +4. Extend `StagesStorage` with the minimal OCI-artifact operations and implement shared `StagesStorage`-backed artifact propagation for final, cache, and secondary-to-primary copies. 5. Move registry validation before image building and remove duplicate `AfterImages` convergence. 6. Add/adjust unit tests for stage flags, dependency identities, subjects, propagation, idempotency, and failure policies. 7. Extend e2e coverage for repository combinations, secondary restore, multi-platform placement, caching, and local-only rejection. @@ -187,4 +194,4 @@ While iterating, use scoped `task lint:golangci-lint` and `task test:unit` paths ## Complexity Tracking -No constitution violations or new architectural projects are proposed. The only additional internal abstraction is a shared artifact propagation operation because SBOM-only propagation cannot satisfy VEX and secondary-to-primary requirements without duplication. +No constitution violations or new architectural projects are proposed. The only additional internal abstraction is a shared `StagesStorage`-backed artifact propagation operation because SBOM-only propagation cannot satisfy VEX and secondary-to-primary requirements without duplication. The explicit distinction between ordinary image stages and OCI-artifact stages is required by the feature and is not an optional abstraction. diff --git a/specs/020-sbom-vex-build-stages/quickstart.md b/specs/020-sbom-vex-build-stages/quickstart.md index 092d24bd9e..3c7cc7fdf1 100644 --- a/specs/020-sbom-vex-build-stages/quickstart.md +++ b/specs/020-sbom-vex-build-stages/quickstart.md @@ -19,8 +19,9 @@ task test:unit paths="./pkg/vex/..." Expected results: - `SbomStage` and `VexStage` calculate stable dependencies and remain non-buildable/mutable; the old `sbomStep` and `vexStep` implementations no longer exist. -- Single-platform SBOM/VEX subjects resolve to the manifest digest. -- Multi-platform SBOM subjects resolve per platform and VEX resolves to the index digest. +- Both stages are associated with the final image digest, operate only on separate OCI artifacts, and use `storage.StagesStorage` for registry interaction. +- Single-platform SBOM/VEX subjects resolve to the final image manifest digest. +- Multi-platform SBOM subjects resolve to each final platform manifest digest and VEX resolves to the final top-level index digest. - Propagation skips identical repositories, deduplicates existing identities, and distinguishes final errors from cache warnings. - Local-only artifact-enabled builds fail before any image stage is executed. @@ -43,7 +44,7 @@ Cover these repository combinations: 7. Two-platform image. 8. Unavailable final repository, unavailable cache repository, and local-only artifact-enabled build. -For each successful case, retrieve artifact descriptors by the actual image digest from every repository containing the image. Verify that repeated builds do not add duplicate fallback-index entries. +For each successful case, retrieve artifact descriptors by the actual final image digest from every repository containing the image. Verify that repeated builds do not add duplicate fallback-index entries and that stage code did not create or modify an image layer. Registry access used by the stages must be observable through the `StagesStorage` test double/backend rather than a direct registry client. Then run the repository integration suite: diff --git a/specs/020-sbom-vex-build-stages/research.md b/specs/020-sbom-vex-build-stages/research.md index 82f31c2771..1107706277 100644 --- a/specs/020-sbom-vex-build-stages/research.md +++ b/specs/020-sbom-vex-build-stages/research.md @@ -4,7 +4,7 @@ SBOM and VEX will be represented by non-buildable, mutable build stages that run after the image content stage has produced a registry-backed descriptor. The existing `Stage` lifecycle remains the integration point: stage dependencies determine cache identity, `MutateImage` performs OCI-side publication, and the build phase invokes the stage for each applicable image/platform. -The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/...`, `pkg/oci/artifact`, the signer implementations, and fallback-tag storage. The existing `sbomStep` and `vexStep` types are transitional implementations: their behavior will be moved into `SbomStage` and `VexStage`, and the step types/files will be deleted. The stages will not add layers or filesystem content. `BuildPhase.AfterImages` will retain image publication/report work but will no longer perform SBOM/VEX generation. +The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/...`, `pkg/oci/artifact`, the signer implementations, and fallback-tag storage. The existing `sbomStep` and `vexStep` types are transitional implementations: their behavior will be moved into `SbomStage` and `VexStage`, and the step types/files will be deleted. The stages are intentionally different from ordinary image stages: each stage is associated with the final image digest, operates on a separate OCI artifact, and never changes image layers or filesystem content. All registry interaction must go through `storage.StagesStorage`, just as ordinary build stages use the storage abstraction. `BuildPhase.AfterImages` will retain image publication/report work but will no longer perform SBOM/VEX generation. ### Rationale @@ -19,6 +19,19 @@ The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/... - Encode SBOM/VEX as image layers: rejected by the specification and would change image semantics. - Migrate to OCI Referrers: rejected as explicitly out of scope and would break the fallback-tag compatibility baseline. +## Decision: Treat SBOM/VEX stages as final-image-digest-associated OCI-artifact stages + +`SbomStage` and `VexStage` are not image-producing stages in the ordinary sense. They do not create or mutate a container image, add layers, or store a filesystem snapshot. Their output is a separate OCI artifact associated with the final image digest. The stages must use `storage.StagesStorage` for registry reads/writes, copying, metadata, and repository operations; the abstraction should expose only the minimal artifact-oriented operations required to find/list attached artifacts, publish an OCI artifact for a final image digest, and copy attached artifacts between destination image descriptors. Direct registry access from stage code is not permitted. + +### Rationale + +This preserves the distinction between an image lifecycle and its supply-chain metadata while still making metadata generation deterministic and cacheable as part of the lifecycle. Reusing `StagesStorage` keeps registry behavior consistent with existing build stages and avoids coupling stages to a concrete registry implementation. + +### Alternatives considered + +- Treat the artifact as a synthetic image stage or image layer: rejected because it changes image semantics and can affect image digest/content. +- Let stages call concrete registry clients directly: rejected because it bypasses the established `StagesStorage` abstraction and makes storage backends inconsistent. + ## Decision: Use explicit artifact subjects for single- and multi-platform images For a single-platform image, both SBOM and VEX use the published platform manifest descriptor. For a multi-platform image, each SBOM stage uses its platform manifest descriptor, while one VEX stage uses the top-level image index descriptor. Destination propagation resolves the destination descriptor before attaching artifacts. @@ -38,7 +51,7 @@ Introduce one internal propagation operation that accepts source and destination ### Rationale -The existing `sbomStep.PropagateArtifacts` only names SBOM and is called after image publication. A kind-neutral operation prevents VEX from acquiring different propagation semantics and makes secondary restoration follow the same rules. +The existing `sbomStep.PropagateArtifacts` only names SBOM and is called after image publication. A kind-neutral `StagesStorage`-backed operation prevents VEX from acquiring different propagation semantics and makes secondary restoration follow the same rules. ### Alternatives considered diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index e4aa7ef040..c3c887e6ad 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -10,76 +10,82 @@ **Purpose**: Establish the implementation baseline without changing user-facing CLI semantics. -- [X] T001 Inventory existing stage, SBOM, VEX, artifact, storage-copy, and cleanup call paths in `pkg/build/`, `pkg/build/stage/`, `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/cleaning/`, recording the concrete integration points in `specs/020-sbom-vex-build-stages/` +- [X] T001 Inventory existing stage, SBOM, VEX, artifact, storage-copy, and cleanup call paths in `pkg/build/`, `pkg/build/stage/`, `pkg/oci/artifact/`, `pkg/storage/`, `pkg/storage/manager/`, and `pkg/cleaning/`, recording the concrete integration points in `specs/020-sbom-vex-build-stages/` - [X] T002 [P] Inspect existing SBOM and VEX unit/e2e fixture conventions in `pkg/build/`, `test/e2e/sbom/`, and `test/e2e/vex/` and identify reusable helpers without adding a second test harness -- [X] T003 [P] Confirm the existing fallback-tag artifact index and cleanup compatibility expectations in `pkg/oci/artifact/` and `pkg/cleaning/` before modifying propagation code +- [X] T003 [P] Confirm the existing fallback-tag artifact index, `StagesStorage` implementations, and cleanup compatibility expectations in `pkg/oci/artifact/`, `pkg/storage/`, and `pkg/cleaning/` before modifying propagation code --- ## Phase 2: Foundational (Blocking Prerequisites) -**Purpose**: Define the minimal internal contracts shared by both artifact stages. +**Purpose**: Define the minimal internal contracts shared by both artifact stages and enforce the storage abstraction boundary. -**Checkpoint**: The shared stage metadata, descriptor propagation shape, and early registry validation are understood and available before story implementation begins. +**Checkpoint**: Stage metadata, final-image subject flow, storage artifact operations, propagation shape, and early registry validation are understood before story implementation begins. -- [X] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including artifact kind, parent descriptor, target platform, mutable flag, and non-buildable flag -- [X] T005 Define the kind-neutral artifact propagation operation and source/destination descriptor data flow in `pkg/build/` using existing `pkg/oci/artifact/` and `pkg/storage/manager/` primitives, without introducing a public API or new dependency +- [X] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including final image descriptor, artifact kind, target platform, mutable flag, and non-buildable flag +- [X] T005 Define the kind-neutral artifact propagation operation and explicit source/destination final-image descriptor flow in `pkg/build/` using existing `pkg/oci/artifact/` primitives, without introducing a public API or new dependency - [X] T006 Locate the earliest common build initialization path and specify the registry-backed-storage validation seam in `pkg/build/build_phase.go` for both SBOM-enabled and VEX-enabled builds -- [X] T007 [P] Add shared test fixtures or helper functions needed to construct manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` +- [X] T007 [P] Add shared test fixtures or helper functions needed to construct final manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` +- [ ] T008 [P] Define the minimal OCI-artifact methods on `storage.StagesStorage` in `pkg/storage/stages_storage.go` for listing attached artifacts, publishing an artifact for a final image digest, and copying attached artifacts between destination descriptors +- [ ] T009 [P] Implement the new `StagesStorage` artifact methods for every registry-backed storage implementation under `pkg/storage/` and keep local-storage behavior explicit and unsupported for artifact publication +- [ ] T010 [P] Add Ginkgo/Gomega contract tests for `StagesStorage` artifact operations and verify stage code can use the abstraction without importing concrete registry clients in `pkg/storage/stages_storage_test.go` and `pkg/build/stage/` --- ## Phase 3: User Story 1 - Artifacts follow published images (Priority: P1) 🎯 MVP -**Goal**: Replace the transitional SBOM/VEX steps with lifecycle-owned stages and make all attached artifacts follow images into primary, final, cache, and secondary-restored destinations. +**Goal**: Replace transitional SBOM/VEX steps with lifecycle-owned OCI-artifact stages and make all attached artifacts follow images into primary, final, cache, and secondary-restored destinations through `StagesStorage`. **Independent Test**: Build the existing fixture with primary-only, final, cache, combined final/cache, identical-address, and secondary-repository configurations; retrieve both artifact kinds by each destination image digest. ### Tests for User Story 1 -- [X] T008 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, parent descriptor requirements, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` -- [ ] T009 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` -- [ ] T010 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` -- [ ] T011 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios -- [ ] T012 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` +- [X] T011 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, final-image descriptor association, no filesystem mutation, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` +- [X] T012 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` +- [X] T013 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` +- [ ] T014 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios +- [ ] T015 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` +- [ ] T016 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `storage.StagesStorage` in `pkg/build/stage/` ### Implementation for User Story 1 -- [X] T013 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction -- [X] T014 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including VEX generation, cache identity, signing, attestation publication, and fallback-index interaction -- [ ] T015 [US1] Migrate all behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing SBOM generation, checksum, signing, and publication behavior -- [ ] T016 [US1] Migrate all behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing VEX generation, checksum, signing, and publication behavior -- [ ] T017 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` -- [ ] T018 [US1] Execute artifact publication through stage `MutateImage` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` -- [ ] T019 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` -- [X] T020 [US1] Implement shared idempotent artifact propagation with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` -- [X] T021 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the shared propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` -- [X] T022 [US1] Connect secondary-stage restoration into primary storage to the same propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` +- [X] T017 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including final-image-digest association and SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction +- [X] T018 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including final-image-digest association and VEX generation, cache identity, signing, attestation publication, and fallback-index interaction +- [ ] T019 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior +- [ ] T020 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior +- [ ] T021 [US1] Ensure `PrepareImage` is a no-op and `MutateImage` operates only on the associated OCI artifact through `storage.StagesStorage`, without fetching, rebuilding, storing, or mutating image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go` +- [ ] T022 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` +- [ ] T023 [US1] Execute artifact publication through stage `MutateImage` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` +- [ ] T024 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` +- [X] T025 [US1] Implement shared idempotent artifact propagation through `storage.StagesStorage`, with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` +- [X] T026 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the `StagesStorage`-backed propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` +- [X] T027 [US1] Connect secondary-stage restoration into primary storage to the same `StagesStorage`-backed propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` -**Checkpoint**: User Story 1 is independently functional; `SbomStage` and `VexStage` are the sole lifecycle owners and artifacts follow every applicable published image. +**Checkpoint**: User Story 1 is independently functional; `SbomStage` and `VexStage` are the sole lifecycle owners, operate on final-image-associated OCI artifacts, and artifacts follow every applicable published image. --- ## Phase 4: User Story 2 - Platform-specific artifacts describe the correct image (Priority: P1) -**Goal**: Attach per-platform SBOMs to platform manifests and attach exactly one image-level VEX to the correct single-platform manifest or multi-platform index. +**Goal**: Attach per-platform SBOMs to final platform manifests and attach exactly one image-level VEX to the correct single-platform manifest or multi-platform index. -**Independent Test**: Build a two-platform fixture, inspect each artifact subject and platform metadata, and verify that multi-platform VEX appears only on the image index while single-platform VEX uses the manifest. +**Independent Test**: Build a two-platform fixture, inspect each artifact subject and platform metadata, and verify that multi-platform VEX appears only on the final image index while single-platform VEX uses the final manifest. ### Tests for User Story 2 -- [X] T023 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform subject selection in `pkg/build/artifact_subject_test.go` -- [X] T024 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and parent digest are distinct per platform in `pkg/build/sbom_step_test.go` -- [ ] T025 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into the stage-owned test location and ensure the final suite contains no step-specific test dependency -- [ ] T026 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each platform manifest -- [X] T027 [US2] Extend `test/e2e/vex/` with single-platform manifest placement and multi-platform index-only placement assertions +- [X] T028 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform final-image subject selection in `pkg/build/artifact_subject_test.go` +- [X] T029 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and final parent digest are distinct per platform in `pkg/build/stage/sbom_test.go` +- [X] T030 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into stage-owned tests and ensure the final suite contains no step-specific test dependency +- [ ] T031 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each final platform manifest +- [X] T032 [US2] Extend `test/e2e/vex/` with single-platform final-manifest placement and multi-platform final-index-only placement assertions +- [ ] T033 [US2] Add storage-backed tests for destination platform/index descriptor resolution when the copied image digest differs from the source in `pkg/build/artifact_propagation_test.go` ### Implementation for User Story 2 -- [X] T028 [US2] Implement explicit artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` -- [ ] T029 [US2] Pass the target platform and resolved platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` -- [ ] T030 [US2] Make `VexStage` registration run once per multi-platform image set with the top-level index subject, and use the image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` -- [X] T031 [US2] Ensure propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` +- [X] T034 [US2] Implement explicit final-image artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` +- [ ] T035 [US2] Pass the final target platform and resolved final platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` +- [ ] T036 [US2] Make `VexStage` registration run once per multi-platform image set with the final top-level index subject, and use the final image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` +- [X] T037 [US2] Ensure `StagesStorage`-backed propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` and `pkg/storage/` **Checkpoint**: User Story 2 is independently testable and no artifact can silently use an index subject for a platform SBOM or duplicate multi-platform VEX onto platform manifests. @@ -93,18 +99,18 @@ ### Tests for User Story 3 -- [X] T032 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` -- [X] T033 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` -- [X] T034 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing in `pkg/build/artifact_propagation_test.go` -- [ ] T035 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios -- [ ] T036 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests +- [X] T038 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across final image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` +- [X] T039 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across final parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` +- [X] T040 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing through `StagesStorage` in `pkg/build/artifact_propagation_test.go` +- [ ] T041 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios +- [X] T042 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests ### Implementation for User Story 3 -- [X] T037 [US3] Include all effective SBOM inputs and the parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` -- [X] T038 [US3] Include VEX document content, parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` -- [ ] T039 [US3] Select reusable artifact-bearing stages from primary and secondary storage using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` -- [X] T040 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/` and `pkg/build/artifact_propagation.go` +- [X] T043 [US3] Include all effective SBOM inputs and the final parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` +- [X] T044 [US3] Include VEX document content, final parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` +- [ ] T045 [US3] Select reusable artifact-bearing stages from primary and secondary storage through `StagesStorage` using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` +- [X] T046 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/`, `pkg/storage/`, and `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 3 is independently testable; unchanged inputs reuse artifacts and every effective changed input invalidates only the affected artifact identity. @@ -118,18 +124,18 @@ ### Tests for User Story 4 -- [X] T041 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` -- [X] T042 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` -- [X] T043 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent attachment in `pkg/oci/artifact/` -- [ ] T044 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior -- [X] T045 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories +- [X] T047 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` +- [X] T048 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` +- [X] T049 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent `StagesStorage`-backed attachment in `pkg/oci/artifact/` and `pkg/storage/` +- [ ] T050 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior +- [X] T051 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories ### Implementation for User Story 4 -- [X] T046 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` -- [X] T047 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` -- [X] T048 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` -- [X] T049 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` +- [X] T052 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` +- [X] T053 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared `StagesStorage`-backed error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [X] T054 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` +- [X] T055 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` **Checkpoint**: User Story 4 is independently testable; registry failures and local-only configuration produce predictable results without changing repository flag semantics. @@ -137,19 +143,20 @@ ## Phase 7: Polish & Cross-Cutting Concerns -**Purpose**: Validate the complete implementation against the revised stage-ownership boundary and repository quality gates. - -- [X] T050 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic -- [ ] T051 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, and verify no step-specific tests remain after migration -- [ ] T052 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures -- [X] T053 Run formatting with `task format` for authored Go directories -- [X] T054 Run compilation with `task build` -- [X] T055 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` -- [ ] T056 Run the complete unit suite with `task test:unit` -- [X] T057 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` -- [X] T058 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` -- [ ] T059 Run legacy integration coverage with `task test:integration` -- [X] T060 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files +**Purpose**: Validate the complete implementation against the revised stage-ownership and storage-abstraction boundaries. + +- [X] T056 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, direct registry-client access from stages, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic +- [ ] T057 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners +- [ ] T058 [P] Verify all stage registry interaction goes through `storage.StagesStorage`, all supported registry-backed implementations satisfy the artifact methods, and local storage rejects artifact publication explicitly in `pkg/storage/` +- [ ] T059 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures +- [X] T060 Run formatting with `task format` for authored Go directories +- [X] T061 Run compilation with `task build` +- [X] T062 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` +- [ ] T063 Run the complete unit suite with `task test:unit` +- [X] T064 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` +- [X] T065 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` +- [ ] T066 Run legacy integration coverage with `task test:integration` +- [X] T067 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files --- @@ -157,50 +164,54 @@ ### Phase Dependencies -- **Phase 1 (Setup)**: No implementation dependency; establishes the current call-path baseline. -- **Phase 2 (Foundational)**: Depends on Phase 1 and blocks all user story implementation. -- **Phase 3 (US1)**: Depends on Phase 2 and is the MVP increment. It now includes full migration from `sbomStep`/`vexStep` and deletion of the transitional files. -- **Phase 4 (US2)**: Depends on US1's artifact stages and propagation path because it specializes subject selection; platform tests must be owned by the new stages before old step tests are deleted. -- **Phase 5 (US3)**: Depends on US1 and US2 stage identity/subject contracts so cache identity includes the correct parent descriptor. -- **Phase 6 (US4)**: Depends on the shared propagation operation from US1; its validation work can proceed in parallel with US2/US3 after the shared path exists. -- **Phase 7 (Polish)**: Depends on all desired stories being complete, including removal of transitional files and references. +- **Phase 1 (Setup)**: No implementation dependency; establishes the current call-path and storage baseline. +- **Phase 2 (Foundational)**: Depends on Phase 1 and blocks story implementation. `StagesStorage` methods and backend implementations must be available before artifact stages can publish or propagate OCI artifacts. +- **Phase 3 (US1)**: Depends on Phase 2 and is the MVP increment. It includes full migration from `sbomStep`/`vexStep`, final-image-digest association, `StagesStorage` integration, and deletion of transitional files. +- **Phase 4 (US2)**: Depends on US1's artifact stages and propagation path because it specializes final subject selection; platform tests must be owned by the new stages before old step tests are deleted. +- **Phase 5 (US3)**: Depends on US1 and US2 stage identity/subject contracts so cache identity includes the correct final parent descriptor. +- **Phase 6 (US4)**: Depends on the shared `StagesStorage`-backed propagation operation from US1; its validation work can proceed in parallel with US2/US3 after the shared path exists. +- **Phase 7 (Polish)**: Depends on all desired stories being complete, including removal of transitional files and references and validation of every storage implementation. ### User Story Dependencies - **US1 (P1)**: Starts after Phase 2; no dependency on another user story. MVP. -- **US2 (P1)**: Depends on US1's `SbomStage`/`VexStage` lifecycle and shared propagation implementation. -- **US3 (P1)**: Depends on US1's stages and US2's explicit parent-subject rules. +- **US2 (P1)**: Depends on US1's `SbomStage`/`VexStage` lifecycle and `StagesStorage`-backed propagation implementation. +- **US3 (P1)**: Depends on US1's stages and US2's explicit final-image subject rules. - **US4 (P2)**: Depends on US1's propagation/error path; early-validation tests can proceed independently of US2 and US3. ### Parallel Opportunities - Phase 1 tasks T002 and T003 can run in parallel after T001's baseline inventory. -- Within US1, T013 and T014 are parallel stage files; T009 and T012 can be developed as separate test concerns before integration. T015 and T016 are parallel migrations when their callers are disjoint. -- Within US2, T023/T024 and T026/T027 are parallel test work; subject-selection implementation and VEX placement implementation can proceed in separate files. -- Within US3, T032 and T033 are parallel stage-owned identity tests; T035 can proceed independently once stage contracts are stable. -- Within US4, T041, T042, and T043 are parallel test tasks, and T045 can proceed independently in cleanup files. -- After Phase 2, separate contributors can work on stage migration, propagation, and validation tests, but deletion of transitional files (T019) must wait for all callers/tests to migrate. -- Polish review and disabled-feature regression checks (T050–T052) can run in parallel before the sequential repository-wide validation commands T053–T060. +- In Phase 2, T008–T010 can proceed in parallel once the required storage method shape is agreed; backend implementations must converge on the same interface. +- Within US1, T017 and T018 are parallel stage files; T012, T013, and T016 are separate test concerns. T019 and T020 are parallel migrations when their callers are disjoint. +- Within US2, T028/T029 and T031–T033 are parallel test work; subject-selection and VEX-placement implementation can proceed in separate files. +- Within US3, T038 and T039 are parallel stage-owned identity tests; T041 can proceed independently once stage contracts are stable. +- Within US4, T047–T049 are parallel test tasks, and T051 can proceed independently in cleanup files. +- After Phase 2, separate contributors can work on stage migration, storage backends, propagation, and validation tests, but deletion of transitional files (T024) must wait for all callers/tests to migrate. +- Polish review and regression checks (T056–T059) can run in parallel before the sequential repository-wide validation commands T060–T067. --- ## Parallel Example: User Story 1 ```text -# After Phase 2, start independent stage, migration, and test work: -Task: T008 — stage lifecycle tests in pkg/build/stage/artifact_test.go -Task: T009 — propagation tests in pkg/build/artifact_propagation_test.go -Task: T012 — migration coverage in pkg/build/ -Task: T013 — SbomStage in pkg/build/stage/sbom.go -Task: T014 — VexStage in pkg/build/stage/vex.go -Task: T011 — repository propagation scenarios in test/e2e/sbom/ - -# Integrate after the stage contracts are stable: -Task: T015 — migrate SBOM behavior and callers -Task: T016 — migrate VEX behavior and callers -Task: T017 — register stages in pkg/build/build_phase.go -Task: T018 — remove duplicate AfterImages convergence -Task: T019 — delete transitional step files and tests +# After Phase 2, start independent stage, migration, storage, and test work: +Task: T011 — stage lifecycle tests in pkg/build/stage/artifact_test.go +Task: T012 — propagation tests in pkg/build/artifact_propagation_test.go +Task: T015 — migration coverage in pkg/build/ +Task: T016 — StagesStorage usage tests in pkg/build/stage/ +Task: T017 — SbomStage in pkg/build/stage/sbom.go +Task: T018 — VexStage in pkg/build/stage/vex.go +Task: T014 — repository propagation scenarios in test/e2e/sbom/ + +# Integrate after the stage and storage contracts are stable: +Task: T019 — migrate SBOM behavior and callers +Task: T020 — migrate VEX behavior and callers +Task: T021 — enforce final-image OCI-artifact-only mutation +Task: T022 — register stages in pkg/build/build_phase.go +Task: T023 — remove duplicate AfterImages convergence +Task: T024 — delete transitional step files and tests +Task: T025 — implement shared StagesStorage-backed propagation ``` ## Implementation Strategy @@ -208,34 +219,35 @@ Task: T019 — delete transitional step files and tests ### MVP First (User Story 1 Only) 1. Complete Phase 1 baseline inspection. -2. Complete Phase 2 shared stage and propagation contracts. -3. Implement or verify `SbomStage` and `VexStage`. -4. Migrate all behavior and callers from `sbomStep`/`vexStep`. -5. Register the stages, connect primary/final/cache/secondary propagation, and delete the transitional files. +2. Complete Phase 2 `StagesStorage`, final-image subject, propagation, and validation contracts. +3. Implement or verify `SbomStage` and `VexStage` as final-image-associated OCI-artifact stages. +4. Migrate all behavior and callers from `sbomStep`/`vexStep` through `StagesStorage`. +5. Register the stages, connect primary/final/cache/secondary propagation, and delete transitional files. 6. Run US1 unit and SBOM e2e tests independently. 7. Stop for validation/demo before adding platform-specific and cache-invalidation refinements. ### Incremental Delivery -1. Deliver US1 as the first usable increment: lifecycle-owned artifact stages with complete repository propagation. -2. Add US2: correct manifest/index subjects and platform placement, with stage-owned tests. +1. Deliver US1 as the first usable increment: lifecycle-owned artifact stages with complete `StagesStorage`-backed repository propagation. +2. Add US2: correct final manifest/index subjects and platform placement, with stage-owned tests. 3. Add US3: complete stage dependency identity and cache reuse/invalidation. 4. Add US4: early validation and explicit final/cache failure behavior. 5. Run the full Polish phase and repository-required validation sequence. ### Traceability -- **FR-001–FR-003**: T013–T022, T028–T031 -- **FR-004–FR-006**: T023–T031 -- **FR-007–FR-010**: T009–T022, T042, T046–T048 -- **FR-011–FR-013**: T032–T040 -- **FR-014–FR-016**: T003, T040, T045, T049 -- **FR-017–FR-018**: T051–T052 and all implementation tasks; no CLI flag changes or OCI Referrers migration -- **Stage ownership boundary**: T012, T015, T016, T019, T025, T036, and T051 +- **FR-001–FR-003**: T017–T027, T034–T037 +- **FR-004–FR-006**: T028–T037 +- **FR-007–FR-010**: T008–T010, T012–T027, T048, T052–T054 +- **FR-011–FR-013**: T038–T046 +- **FR-014–FR-016**: T003, T046, T051, T055 +- **FR-017–FR-018**: T057–T059 and all implementation tasks; no CLI flag changes or OCI Referrers migration +- **Stage ownership and storage boundary**: T008–T010, T015–T024, T056–T058 ## Notes -- Completed tasks retain `[X]`; pending tasks use `[ ]`. Both are valid Markdown checkbox markers and every task has a sequential ID. -- No external API contracts were provided in `contracts/`; the propagation contract remains internal to `pkg/build`. -- The revised plan requires `SbomStage` and `VexStage` to be the sole lifecycle owners; `sbom_step.go`, `vex_step.go`, their step-specific callers, and their tests must not remain after migration. +- Completed tasks retain `[X]`; pending tasks use `[ ]`. Every task has a sequential ID and story-phase tasks include exactly one `[US#]` label. +- No external API contracts were provided in `contracts/`; the artifact methods are an internal `StagesStorage` contract. +- The revised plan requires `SbomStage` and `VexStage` to be the sole lifecycle owners, associated with final image descriptors, and restricted to OCI-artifact operations through `storage.StagesStorage`. +- `sbom_step.go`, `vex_step.go`, their step-specific callers, and their tests must not remain after migration. - No new dependencies, CLI flags, image layers, or OCI Referrers migration are planned. From 6532d0d11015842698c40acabc456ccabcdb9c21 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 14:00:48 +0300 Subject: [PATCH 07/17] refactor(storage): route artifact propagation through manager Move repository selection into StorageManager so artifact operations use the appropriate primary, secondary, cache, or final backend instead of receiving storage backends directly. Signed-off-by: Alexandr Zaytsev --- pkg/build/artifact_propagation.go | 14 +- pkg/build/build_phase.go | 8 +- pkg/storage/local_stages_storage.go | 12 ++ pkg/storage/local_stages_storage_test.go | 9 + pkg/storage/manager/storage_manager.go | 3 +- pkg/storage/meta_repo_marker_test.go | 3 + pkg/storage/repo_stages_storage.go | 22 ++- pkg/storage/stages_storage.go | 4 + specs/020-sbom-vex-build-stages/data-model.md | 6 +- specs/020-sbom-vex-build-stages/plan.md | 22 +-- specs/020-sbom-vex-build-stages/quickstart.md | 4 +- specs/020-sbom-vex-build-stages/research.md | 8 +- specs/020-sbom-vex-build-stages/tasks.md | 173 +++++++++--------- test/mock/bom_patcher.go | 4 +- test/mock/stages_storage.go | 43 +++++ 15 files changed, 217 insertions(+), 118 deletions(-) diff --git a/pkg/build/artifact_propagation.go b/pkg/build/artifact_propagation.go index 9d8d584420..a120a8d356 100644 --- a/pkg/build/artifact_propagation.go +++ b/pkg/build/artifact_propagation.go @@ -31,7 +31,7 @@ func ensureAttachedArtifacts(ctx context.Context, repository, digest string) err return nil } -func propagateArtifacts(ctx context.Context, projectName, imageName string, source, destination *image.StageDesc, caches []storage.StagesStorage) error { +func propagateArtifacts(ctx context.Context, projectName, imageName string, source, destination *image.StageDesc, caches []storage.StagesStorage, sourceStorages ...storage.StagesStorage) error { if source == nil || source.Info == nil { return fmt.Errorf("source image descriptor is unavailable") } @@ -39,12 +39,20 @@ func propagateArtifacts(ctx context.Context, projectName, imageName string, sour return nil } + var sourceStorage storage.StagesStorage + if len(sourceStorages) > 0 { + sourceStorage = sourceStorages[0] + } + if sourceStorage == nil { + sourceStorage = &storage.RepoStagesStorage{RepoAddress: source.Info.Repository} + } + if destination != nil && destination.Info != nil && destination.Info.Repository != "" && destination.Info.Repository != storage.LocalStorageAddress && destination.Info.Repository != source.Info.Repository { if err := logboek.Context(ctx).Default().LogProcess("image %s: copy artifacts into final repo %s", imageName, destination.Info.Repository).DoError(func() error { - return artifact.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), destination.Info.Repository, destination.Info.GetDigest()) + return sourceStorage.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), destination.Info.Repository, destination.Info.GetDigest()) }); err != nil { return fmt.Errorf("copy attached artifacts into final repo %s: %w", destination.Info.Repository, err) } @@ -68,7 +76,7 @@ func propagateArtifacts(ctx context.Context, projectName, imageName string, sour destinationDigest = cacheDesc.Info.GetDigest() } - if err := artifact.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), cache.Address(), destinationDigest); err != nil { + if err := sourceStorage.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), cache.Address(), destinationDigest); err != nil { logboek.Context(ctx).Warn().LogF("Warning: unable to copy artifacts into cache stages storage %s: %s\n", cache.String(), err) } } diff --git a/pkg/build/build_phase.go b/pkg/build/build_phase.go index 0dc4ad0451..d6b970d245 100644 --- a/pkg/build/build_phase.go +++ b/pkg/build/build_phase.go @@ -324,7 +324,7 @@ func (phase *BuildPhase) propagateArtifactsByImages(ctx context.Context) error { if source == nil { continue } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, source, finalStageDescForPlatform(phase, name, images, img.TargetPlatform), phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, source, finalStageDescForPlatform(phase, name, images, img.TargetPlatform), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage()); err != nil { return fmt.Errorf("propagate artifacts for image %q: %w", name, err) } } @@ -334,7 +334,7 @@ func (phase *BuildPhase) propagateArtifactsByImages(ctx context.Context) error { if multiImage == nil || multiImage.GetStageDesc() == nil { continue } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, multiImage.GetStageDesc(), multiImage.GetFinalStageDesc(), phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, multiImage.GetStageDesc(), multiImage.GetFinalStageDesc(), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage()); err != nil { return fmt.Errorf("propagate multiplatform artifacts for image %q: %w", name, err) } } @@ -1177,7 +1177,7 @@ func (phase *BuildPhase) findAndFetchStageFromSecondaryStagesStorage(ctx context if err := ensureAttachedArtifacts(ctx, secondaryStageDesc.Info.Repository, secondaryStageDesc.Info.GetDigest()); err != nil { return fmt.Errorf("secondary stage %s has incomplete artifacts: %w", secondaryStageDesc.StageID.String(), err) } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), img.Name, secondaryStageDesc, stageDescCopy, storageManager.GetCacheStagesStorageList()); err != nil { + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), img.Name, secondaryStageDesc, stageDescCopy, storageManager.GetCacheStagesStorageList(), secondaryStagesStorage); err != nil { return fmt.Errorf("unable to propagate artifacts restored from secondary storage: %w", err) } } @@ -1893,7 +1893,7 @@ func (phase *BuildPhase) convergeImageVex(ctx context.Context, name string, imag return fmt.Errorf("unable to converge VEX for image %q: %w", name, err) } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDescForImage(phase, name, images), phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { + if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDescForImage(phase, name, images), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage()); err != nil { return fmt.Errorf("unable to propagate VEX for image %q: %w", name, err) } diff --git a/pkg/storage/local_stages_storage.go b/pkg/storage/local_stages_storage.go index 736f337d1d..b9d0efe759 100644 --- a/pkg/storage/local_stages_storage.go +++ b/pkg/storage/local_stages_storage.go @@ -262,6 +262,18 @@ func (storage *LocalStagesStorage) RmImageMetadata(ctx context.Context, projectN return nil } +func (storage *LocalStagesStorage) ListAttachedArtifacts(_ context.Context, _ string) ([]v1.Descriptor, error) { + return nil, fmt.Errorf("local stages storage does not support artifact operations") +} + +func (storage *LocalStagesStorage) PublishArtifact(_ context.Context, _, _ string, _ []byte, _, _, _, _ string) error { + return fmt.Errorf("local stages storage does not support artifact operations") +} + +func (storage *LocalStagesStorage) CopyAttachedArtifacts(_ context.Context, _, _, _, _ string) error { + return fmt.Errorf("local stages storage does not support artifact operations") +} + func (storage *LocalStagesStorage) GetOrphanedArtifactNames(_ context.Context) ([]string, error) { return nil, nil } diff --git a/pkg/storage/local_stages_storage_test.go b/pkg/storage/local_stages_storage_test.go index 32d6ff151b..496eceefbc 100644 --- a/pkg/storage/local_stages_storage_test.go +++ b/pkg/storage/local_stages_storage_test.go @@ -43,6 +43,15 @@ func (b *localMutationBackendStub) Tag(ctx context.Context, ref, newRef string, } var _ = Describe("LocalStagesStorage", func() { + It("rejects OCI artifact operations explicitly", func(ctx SpecContext) { + storage := NewLocalStagesStorage(nil) + + _, err := storage.ListAttachedArtifacts(ctx, "sha256:parent") + Expect(err).To(MatchError("local stages storage does not support artifact operations")) + Expect(storage.PublishArtifact(ctx, "sha256:parent", "application/test", []byte("payload"), "image", "checksum", "", "")).To(MatchError("local stages storage does not support artifact operations")) + Expect(storage.CopyAttachedArtifacts(ctx, "source", "sha256:source", "destination", "sha256:destination")).To(MatchError("local stages storage does not support artifact operations")) + }) + It("tags the mutated local image under the destination reference", func(ctx SpecContext) { logCtx := logboek.NewContext(ctx, logboek.NewLogger(io.Discard, io.Discard)) diff --git a/pkg/storage/manager/storage_manager.go b/pkg/storage/manager/storage_manager.go index 81f28be9d9..8bcb3f77f6 100644 --- a/pkg/storage/manager/storage_manager.go +++ b/pkg/storage/manager/storage_manager.go @@ -20,7 +20,6 @@ import ( "github.com/werf/werf/v2/pkg/container_backend" "github.com/werf/werf/v2/pkg/docker_registry" "github.com/werf/werf/v2/pkg/image" - "github.com/werf/werf/v2/pkg/oci/artifact" "github.com/werf/werf/v2/pkg/storage" "github.com/werf/werf/v2/pkg/storage/lrumeta" "github.com/werf/werf/v2/pkg/util/parallel" @@ -811,7 +810,7 @@ func (m *StorageManager) CopySuitableStageDescByDigest(ctx context.Context, stag return nil, fmt.Errorf("unable to get stage %s description from %s: %w", stageDesc.StageID.String(), destinationStagesStorage.String(), err) } else { if sourceStagesStorage.Address() != storage.LocalStorageAddress && destinationStagesStorage.Address() != storage.LocalStorageAddress { - if err := artifact.CopyAttachedArtifacts(ctx, sourceStagesStorage.Address(), stageDesc.Info.GetDigest(), destinationStagesStorage.Address(), destinationStageDesc.Info.GetDigest()); err != nil { + if err := destinationStagesStorage.CopyAttachedArtifacts(ctx, sourceStagesStorage.Address(), stageDesc.Info.GetDigest(), destinationStagesStorage.Address(), destinationStageDesc.Info.GetDigest()); err != nil { return nil, fmt.Errorf("unable to copy artifacts attached to stage %s: %w", stageDesc.StageID.String(), err) } } diff --git a/pkg/storage/meta_repo_marker_test.go b/pkg/storage/meta_repo_marker_test.go index 439fd05a6d..6863926fca 100644 --- a/pkg/storage/meta_repo_marker_test.go +++ b/pkg/storage/meta_repo_marker_test.go @@ -34,6 +34,7 @@ var markerUnguardedMethods = []string{ "CheckStageCustomTag", "ConstructStageImageName", "CopyFromStorage", + "CopyAttachedArtifacts", "CreateRepo", "DeleteArtifact", "DeleteRejectedStageImage", @@ -56,9 +57,11 @@ var markerUnguardedMethods = []string{ "GetStagesIDsByDigest", "IsImageMetadataExist", "IsManagedImageExist", + "ListAttachedArtifacts", "MutateAndPushImage", "PostManifest", "PostMultiplatformImage", + "PublishArtifact", "RejectStage", "ShouldFetchImage", "StoreImage", diff --git a/pkg/storage/repo_stages_storage.go b/pkg/storage/repo_stages_storage.go index 6c2b1a9eca..dfad7f5fa7 100644 --- a/pkg/storage/repo_stages_storage.go +++ b/pkg/storage/repo_stages_storage.go @@ -849,6 +849,26 @@ func (storage *RepoStagesStorage) Address() string { return storage.RepoAddress } +func (storage *RepoStagesStorage) ListAttachedArtifacts(ctx context.Context, parentDigest string) ([]v1.Descriptor, error) { + index, err := artifact.PullFallbackIndex(ctx, storage.RepoAddress, parentDigest) + if err != nil { + return nil, fmt.Errorf("pull artifact index: %w", err) + } + manifest, err := index.IndexManifest() + if err != nil { + return nil, fmt.Errorf("read artifact index: %w", err) + } + return manifest.Manifests, nil +} + +func (storage *RepoStagesStorage) PublishArtifact(ctx context.Context, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error { + return artifact.NewOCIStore(storage.RepoAddress, imageName).Attach(ctx, parentDigest, artifactType, payload, checksum, targetPlatform, predicateType) +} + +func (storage *RepoStagesStorage) CopyAttachedArtifacts(ctx context.Context, sourceRepository, sourceDigest, destinationRepository, destinationDigest string) error { + return artifact.CopyAttachedArtifacts(ctx, sourceRepository, sourceDigest, destinationRepository, destinationDigest) +} + func (storage *RepoStagesStorage) GetOrphanedArtifactNames(ctx context.Context) ([]string, error) { tags, err := storage.Tags(ctx, storage.RepoAddress) if err != nil { @@ -974,7 +994,7 @@ func (storage *RepoStagesStorage) CopyFromStorage(ctx context.Context, src Stage return nil, fmt.Errorf("unable to get stage %s description: %w", stageID, err) } - if err := artifact.CopyAttachedArtifacts(ctx, src.Address(), desc.Info.GetDigest(), storage.RepoAddress, desc.Info.GetDigest()); err != nil { + if err := storage.CopyAttachedArtifacts(ctx, src.Address(), desc.Info.GetDigest(), storage.RepoAddress, desc.Info.GetDigest()); err != nil { return nil, fmt.Errorf("unable to copy artifacts attached to stage %s: %w", stageID, err) } diff --git a/pkg/storage/stages_storage.go b/pkg/storage/stages_storage.go index 632e1b6750..e256437b7e 100644 --- a/pkg/storage/stages_storage.go +++ b/pkg/storage/stages_storage.go @@ -82,6 +82,10 @@ type StagesStorage interface { GetOrphanedArtifactNames(ctx context.Context) ([]string, error) DeleteArtifact(ctx context.Context, imageName string) error + ListAttachedArtifacts(ctx context.Context, parentDigest string) ([]v1.Descriptor, error) + PublishArtifact(ctx context.Context, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error + CopyAttachedArtifacts(ctx context.Context, sourceRepository, sourceDigest, destinationRepository, destinationDigest string) error + PutImageMetadata(ctx context.Context, projectName, imageNameOrManagedImageName, commit, stageID string) error RmImageMetadata(ctx context.Context, projectName, imageNameOrManagedImageNameOrImageMetadataID, commit, stageID string) error IsImageMetadataExist(ctx context.Context, projectName, imageNameOrManagedImageName, commit, stageID string, opts ...Option) (bool, error) diff --git a/specs/020-sbom-vex-build-stages/data-model.md b/specs/020-sbom-vex-build-stages/data-model.md index 8844b15248..f08387cc48 100644 --- a/specs/020-sbom-vex-build-stages/data-model.md +++ b/specs/020-sbom-vex-build-stages/data-model.md @@ -12,12 +12,12 @@ An internal build-stage operation associated with the final image digest (and, f | Artifact kind | CycloneDX SBOM or OpenVEX. | | Generation inputs | Scanner/merge inputs for SBOM, document content for VEX, format version, and signer identity. | | Mutable/buildable flags | Non-buildable and mutable, matching registry-only stages such as signing, but unlike signing the output is an associated OCI artifact rather than a manifest mutation. | -| Storage abstraction | `storage.StagesStorage` used for all registry operations. | +| Storage abstraction | `StorageManager` routes all registry operations to primary, secondary, cache, or final `storage.StagesStorage`. | Validation rules: - The final image descriptor/digest must be available before `MutateImage` runs. -- The stage must use `storage.StagesStorage` for registry access. +- The stage must use `StorageManager` for registry access; repository selection is performed by the manager. - SBOM for a multi-platform image must use the corresponding platform manifest. - VEX must use the platform manifest for single-platform images and the top-level index for multi-platform images. - An enabled artifact stage requires registry-backed storage. @@ -58,4 +58,4 @@ primary image + artifacts -> final image + artifacts primary image + artifacts -> cache image + artifacts ``` -The artifact stage does not become an image layer and does not operate on an image filesystem. It publishes separate OCI artifacts whose subjects are the final image descriptors. All registry interaction is performed through `storage.StagesStorage`; existing fallback-tag indexes remain the source of truth and remain readable by current consumers. +The artifact stage does not become an image layer and does not operate on an image filesystem. It publishes separate OCI artifacts whose subjects are the final image descriptors. All registry interaction is performed through `StorageManager` and its primary/secondary/cache/final `storage.StagesStorage` backends; existing fallback-tag indexes remain the source of truth and remain readable by current consumers. diff --git a/specs/020-sbom-vex-build-stages/plan.md b/specs/020-sbom-vex-build-stages/plan.md index ad213cec8a..a25502b05d 100644 --- a/specs/020-sbom-vex-build-stages/plan.md +++ b/specs/020-sbom-vex-build-stages/plan.md @@ -6,7 +6,7 @@ ## Summary -Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and replace the `sbomStep` and `vexStep` implementations with registry-backed, non-buildable mutable stages modeled after `pkg/build/stage/sign.go`. The new `SbomStage` and `VexStage` become the sole owners of SBOM/VEX cache identity, generation, signing, attestation publication, and fallback-index interaction. Unlike ordinary image stages, they are associated with the final image digest, operate on the associated OCI artifact rather than on the image filesystem or image layers, and perform all registry operations through `storage.StagesStorage`. +Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and replace the `sbomStep` and `vexStep` implementations with registry-backed, non-buildable mutable stages modeled after `pkg/build/stage/sign.go`. The new `SbomStage` and `VexStage` become the sole owners of SBOM/VEX cache identity, generation, signing, attestation publication, and fallback-index interaction. Unlike ordinary image stages, they are associated with the final image digest, operate on the associated OCI artifact rather than on the image filesystem or image layers, and perform all registry operations through `StorageManager`, which routes them to the appropriate primary, secondary, cache, or final `StagesStorage`. Artifact publication will use explicit source and destination image descriptors. SBOM remains platform-specific; VEX is attached once at the top-level image index for multi-platform images and to the image manifest for single-platform images. A shared idempotent propagation operation will cover primary-to-final, primary-to-cache, and secondary-to-primary copies, resolving the destination digest and preserving fatal final-repository versus best-effort cache error policies. @@ -21,11 +21,11 @@ Artifact publication will use explicit source and destination image descriptors. - Existing `pkg/build/sbom_step.go` and `pkg/build/vex_step.go` are transitional sources only and must be removed after their logic is migrated. - Existing OCI artifact and fallback-index operations in `pkg/oci/artifact` and `pkg/attestation`. - Existing registry/storage copy operations in `pkg/storage`, `pkg/storage/manager`, and `pkg/docker_registry`. -- `storage.StagesStorage` in `pkg/storage/stages_storage.go` is the required registry boundary. It must be extended with the minimal OCI-artifact operations needed by `SbomStage`, `VexStage`, and propagation; stage code must not call concrete registry clients directly. +- `StorageManager` is the required registry boundary for `SbomStage`, `VexStage`, and propagation. It owns the primary, secondary, cache, and final `StagesStorage` instances and must route each operation to the correct repository abstraction. `StagesStorage` may be extended with minimal OCI-artifact primitives, but stages must not select repositories or call concrete registry clients directly. - Existing signing options in `pkg/build/signing`. - Ginkgo + Gomega test framework and existing e2e fixtures. -**Storage**: OCI registry for image manifests/indexes and fallback-tag artifact indexes, accessed through the `storage.StagesStorage` abstraction; local Buildah/container storage remains supported when artifacts are disabled. +**Storage**: OCI registry for image manifests/indexes and fallback-tag artifact indexes, accessed through `StorageManager`, which routes operations to primary, secondary, cache, and final `storage.StagesStorage` instances; local Buildah/container storage remains supported when artifacts are disabled. **Testing**: Co-located Ginkgo/Gomega unit tests, existing `test/e2e/sbom` and `test/e2e/vex` suites, and legacy integration tests. @@ -47,7 +47,7 @@ Artifact publication will use explicit source and destination image descriptors. *GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design.* - **Simplicity over abstraction**: PASS. Use two explicit stages, `SbomStage` and `VexStage`, instead of retaining parallel step and stage abstractions. Add one focused shared propagation path rather than duplicating repository-copy logic. -- **Stage distinction**: PASS. The plan explicitly requires that SBOM/VEX stages are final-image-digest-associated OCI-artifact stages, not ordinary image-mutating stages, and that registry access goes through `storage.StagesStorage`. +- **Stage distinction**: PASS. The plan explicitly requires that SBOM/VEX stages are final-image-digest-associated OCI-artifact stages, not ordinary image-mutating stages, and that registry access goes through `StorageManager` and its repository-specific `storage.StagesStorage` abstractions. - **Go idioms and errors**: PASS. New public methods, if required, take `context.Context` first; errors wrap operation context; stage-specific helpers remain private where possible. - **Minimal public surface**: PASS. Artifact stages and propagation contracts are internal to `pkg/build`; no new CLI flags or external API are planned. - **Testing**: PASS. Tests remain alongside source and use Ginkgo/Gomega. E2E coverage extends existing SBOM/VEX suites rather than introducing a parallel harness. @@ -77,8 +77,8 @@ Detailed findings are in [research.md](./research.md). Key decisions: - Move all behavior currently owned by `vexStep` into `VexStage`; remove `vex_step.go` and its step-specific tests once callers are migrated. - Artifact stages must not mutate, rebuild, fetch, or store the image filesystem. Their `PrepareImage` path is a no-op; their `MutateImage` path operates on the associated OCI artifact and owns registry-side generation and publication. - The stage's subject is the final image digest: for single-platform images this is the published image manifest digest; for multi-platform images SBOM uses each final platform manifest digest and VEX uses the final top-level image index digest. The artifact stage must never be treated as an image layer or as a replacement image. -- All registry reads, writes, copies, metadata operations, and artifact-related repository interaction from `SbomStage` and `VexStage` must use `storage.StagesStorage`, following the existing build-stage storage path. Direct registry client access from the stages is prohibited. -- Extend `StagesStorage` only with the minimal artifact-oriented methods required to preserve the existing fallback-index behavior: find/list attached artifacts, publish an OCI artifact for a final image digest, and copy attached artifacts between destination image descriptors. Implement these methods for every supported registry-backed storage implementation and keep local storage behavior explicit. +- All registry reads, writes, copies, metadata operations, and artifact-related repository interaction from `SbomStage` and `VexStage` must use `StorageManager`. The manager selects primary, secondary, cache, or final `StagesStorage` according to the operation; direct registry client access and direct repository selection from the stages are prohibited. +- Extend `StorageManager` with the minimal artifact-oriented operations required by the stages and propagation. Implement the corresponding `StagesStorage` primitives only where needed to preserve fallback-index behavior: find/list attached artifacts, publish an OCI artifact for a final image digest, and copy attached artifacts between destination image descriptors. Implement these methods for every supported registry-backed storage implementation and keep local storage behavior explicit. - Ensure stage dependencies include the parent image identity and all effective artifact inputs. SBOM dependencies include scanner, merge/GOST, signer, format version, and target platform. VEX dependencies include document content, parent identity, signer, and format version. - Register the stages after the content-producing stage and before the lifecycle completes for applicable images. The registration must work for Stapel and Dockerfile image paths and for restored stages. - Preserve stage cache behavior: a suitable artifact-bearing stage can be selected from primary/secondary storage; changed effective inputs produce a different stage identity. @@ -93,7 +93,7 @@ Detailed findings are in [research.md](./research.md). Key decisions: ### Publication and propagation -- Consolidate artifact copying behind a kind-neutral internal operation exposed through or implemented using `storage.StagesStorage`; it copies every attached supported artifact from a source descriptor to a destination descriptor. This propagation helper is the only shared artifact operation; generation remains owned independently by `SbomStage` and `VexStage`. +- Consolidate artifact copying behind a kind-neutral `StorageManager` operation; it routes through the source/destination repository `StagesStorage` instances and copies every attached supported artifact from a source descriptor to a destination descriptor. This propagation helper is the only shared artifact operation; generation remains owned independently by `SbomStage` and `VexStage`. - The propagation contract must carry the final image digest/descriptor explicitly and must never attach an artifact to the digest of the artifact stage itself. - Use it after primary-to-final and primary-to-cache image copies, and when a suitable stage is copied from `--secondary-repo` into primary storage. - Resolve the destination image descriptor/digest rather than assuming source and destination digests match. @@ -142,8 +142,8 @@ pkg/build/ pkg/build/sbom_step.go and pkg/build/vex_step.go are removed after migration; their behavior is not retained behind compatibility wrappers. pkg/storage/ # StagesStorage interface and backend implementations for artifact stages -pkg/storage/manager/ # secondary/final/cache image-copy artifact propagation hooks -pkg/oci/artifact/ # OCI artifact encoding and fallback-index support called by StagesStorage backends +pkg/storage/manager/ # StorageManager routing across primary/secondary/cache/final and artifact propagation +pkg/oci/artifact/ # OCI artifact encoding and fallback-index support called by storage backends Tests remain co-located under pkg/build and pkg/build/stage, with scenario coverage in: test/e2e/sbom/ @@ -169,7 +169,7 @@ The subsequent `/speckit-tasks` workflow should decompose at least these work it 1. Implement `SbomStage` and `VexStage`, including stage identity, lifecycle integration, final image-digest association, OCI-artifact handling, cache checks, signing, publication, and `StagesStorage` access. 2. Migrate all SBOM behavior from `sbomStep` into `SbomStage`, then delete the step implementation and update callers/tests. 3. Migrate all VEX behavior from `vexStep` into `VexStage`, then delete the step implementation and update callers/tests. -4. Extend `StagesStorage` with the minimal OCI-artifact operations and implement shared `StagesStorage`-backed artifact propagation for final, cache, and secondary-to-primary copies. +4. Extend `StorageManager` with the minimal OCI-artifact operations, add any required `StagesStorage` backend primitives, and implement manager-routed artifact propagation for final, cache, and secondary-to-primary copies. 5. Move registry validation before image building and remove duplicate `AfterImages` convergence. 6. Add/adjust unit tests for stage flags, dependency identities, subjects, propagation, idempotency, and failure policies. 7. Extend e2e coverage for repository combinations, secondary restore, multi-platform placement, caching, and local-only rejection. @@ -194,4 +194,4 @@ While iterating, use scoped `task lint:golangci-lint` and `task test:unit` paths ## Complexity Tracking -No constitution violations or new architectural projects are proposed. The only additional internal abstraction is a shared `StagesStorage`-backed artifact propagation operation because SBOM-only propagation cannot satisfy VEX and secondary-to-primary requirements without duplication. The explicit distinction between ordinary image stages and OCI-artifact stages is required by the feature and is not an optional abstraction. +No constitution violations or new architectural projects are proposed. The only additional internal abstraction is a shared `StorageManager`-routed artifact propagation operation because SBOM-only propagation cannot satisfy VEX and secondary-to-primary requirements without duplication. The explicit distinction between ordinary image stages and OCI-artifact stages is required by the feature and is not an optional abstraction. diff --git a/specs/020-sbom-vex-build-stages/quickstart.md b/specs/020-sbom-vex-build-stages/quickstart.md index 3c7cc7fdf1..0fe916f1eb 100644 --- a/specs/020-sbom-vex-build-stages/quickstart.md +++ b/specs/020-sbom-vex-build-stages/quickstart.md @@ -19,7 +19,7 @@ task test:unit paths="./pkg/vex/..." Expected results: - `SbomStage` and `VexStage` calculate stable dependencies and remain non-buildable/mutable; the old `sbomStep` and `vexStep` implementations no longer exist. -- Both stages are associated with the final image digest, operate only on separate OCI artifacts, and use `storage.StagesStorage` for registry interaction. +- Both stages are associated with the final image digest, operate only on separate OCI artifacts, and use `StorageManager` for registry interaction. The manager routes requests to primary, secondary, cache, or final `storage.StagesStorage` as appropriate. - Single-platform SBOM/VEX subjects resolve to the final image manifest digest. - Multi-platform SBOM subjects resolve to each final platform manifest digest and VEX resolves to the final top-level index digest. - Propagation skips identical repositories, deduplicates existing identities, and distinguishes final errors from cache warnings. @@ -44,7 +44,7 @@ Cover these repository combinations: 7. Two-platform image. 8. Unavailable final repository, unavailable cache repository, and local-only artifact-enabled build. -For each successful case, retrieve artifact descriptors by the actual final image digest from every repository containing the image. Verify that repeated builds do not add duplicate fallback-index entries and that stage code did not create or modify an image layer. Registry access used by the stages must be observable through the `StagesStorage` test double/backend rather than a direct registry client. +For each successful case, retrieve artifact descriptors by the actual final image digest from every repository containing the image. Verify that repeated builds do not add duplicate fallback-index entries and that stage code did not create or modify an image layer. Registry access used by the stages must be observable through `StorageManager` routing and the selected `StagesStorage` test double/backend rather than a direct registry client. Then run the repository integration suite: diff --git a/specs/020-sbom-vex-build-stages/research.md b/specs/020-sbom-vex-build-stages/research.md index 1107706277..d365f9e756 100644 --- a/specs/020-sbom-vex-build-stages/research.md +++ b/specs/020-sbom-vex-build-stages/research.md @@ -4,7 +4,7 @@ SBOM and VEX will be represented by non-buildable, mutable build stages that run after the image content stage has produced a registry-backed descriptor. The existing `Stage` lifecycle remains the integration point: stage dependencies determine cache identity, `MutateImage` performs OCI-side publication, and the build phase invokes the stage for each applicable image/platform. -The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/...`, `pkg/oci/artifact`, the signer implementations, and fallback-tag storage. The existing `sbomStep` and `vexStep` types are transitional implementations: their behavior will be moved into `SbomStage` and `VexStage`, and the step types/files will be deleted. The stages are intentionally different from ordinary image stages: each stage is associated with the final image digest, operates on a separate OCI artifact, and never changes image layers or filesystem content. All registry interaction must go through `storage.StagesStorage`, just as ordinary build stages use the storage abstraction. `BuildPhase.AfterImages` will retain image publication/report work but will no longer perform SBOM/VEX generation. +The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/...`, `pkg/oci/artifact`, the signer implementations, and fallback-tag storage. The existing `sbomStep` and `vexStep` types are transitional implementations: their behavior will be moved into `SbomStage` and `VexStage`, and the step types/files will be deleted. The stages are intentionally different from ordinary image stages: each stage is associated with the final image digest, operates on a separate OCI artifact, and never changes image layers or filesystem content. All registry interaction must go through `StorageManager`, which owns and routes to the primary, secondary, cache, and final `storage.StagesStorage` abstractions, just as ordinary build stages use the storage manager path. `BuildPhase.AfterImages` will retain image publication/report work but will no longer perform SBOM/VEX generation. ### Rationale @@ -21,11 +21,11 @@ The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/... ## Decision: Treat SBOM/VEX stages as final-image-digest-associated OCI-artifact stages -`SbomStage` and `VexStage` are not image-producing stages in the ordinary sense. They do not create or mutate a container image, add layers, or store a filesystem snapshot. Their output is a separate OCI artifact associated with the final image digest. The stages must use `storage.StagesStorage` for registry reads/writes, copying, metadata, and repository operations; the abstraction should expose only the minimal artifact-oriented operations required to find/list attached artifacts, publish an OCI artifact for a final image digest, and copy attached artifacts between destination image descriptors. Direct registry access from stage code is not permitted. +`SbomStage` and `VexStage` are not image-producing stages in the ordinary sense. They do not create or mutate a container image, add layers, or store a filesystem snapshot. Their output is a separate OCI artifact associated with the final image digest. The stages must use `StorageManager` for registry reads/writes, copying, metadata, and repository operations. `StorageManager` selects the appropriate primary, secondary, cache, or final `storage.StagesStorage`; those backends should expose only the minimal artifact-oriented operations required to find/list attached artifacts, publish an OCI artifact for a final image digest, and copy attached artifacts between destination image descriptors. Direct registry access or repository selection from stage code is not permitted. ### Rationale -This preserves the distinction between an image lifecycle and its supply-chain metadata while still making metadata generation deterministic and cacheable as part of the lifecycle. Reusing `StagesStorage` keeps registry behavior consistent with existing build stages and avoids coupling stages to a concrete registry implementation. +This preserves the distinction between an image lifecycle and its supply-chain metadata while still making metadata generation deterministic and cacheable as part of the lifecycle. Reusing `StorageManager` and its `StagesStorage` backends keeps registry behavior consistent with existing build stages, centralizes repository routing, and avoids coupling stages to a concrete registry implementation. ### Alternatives considered @@ -51,7 +51,7 @@ Introduce one internal propagation operation that accepts source and destination ### Rationale -The existing `sbomStep.PropagateArtifacts` only names SBOM and is called after image publication. A kind-neutral `StagesStorage`-backed operation prevents VEX from acquiring different propagation semantics and makes secondary restoration follow the same rules. +The existing `sbomStep.PropagateArtifacts` only names SBOM and is called after image publication. A kind-neutral `StorageManager`-routed operation prevents VEX from acquiring different propagation semantics and makes secondary restoration follow the same rules. ### Alternatives considered diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index c3c887e6ad..d86f63d3ba 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -25,41 +25,42 @@ - [X] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including final image descriptor, artifact kind, target platform, mutable flag, and non-buildable flag - [X] T005 Define the kind-neutral artifact propagation operation and explicit source/destination final-image descriptor flow in `pkg/build/` using existing `pkg/oci/artifact/` primitives, without introducing a public API or new dependency - [X] T006 Locate the earliest common build initialization path and specify the registry-backed-storage validation seam in `pkg/build/build_phase.go` for both SBOM-enabled and VEX-enabled builds -- [X] T007 [P] Add shared test fixtures or helper functions needed to construct final manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` -- [ ] T008 [P] Define the minimal OCI-artifact methods on `storage.StagesStorage` in `pkg/storage/stages_storage.go` for listing attached artifacts, publishing an artifact for a final image digest, and copying attached artifacts between destination descriptors -- [ ] T009 [P] Implement the new `StagesStorage` artifact methods for every registry-backed storage implementation under `pkg/storage/` and keep local-storage behavior explicit and unsupported for artifact publication -- [ ] T010 [P] Add Ginkgo/Gomega contract tests for `StagesStorage` artifact operations and verify stage code can use the abstraction without importing concrete registry clients in `pkg/storage/stages_storage_test.go` and `pkg/build/stage/` +- [ ] T007 Extend `StorageManager` with minimal artifact listing, publication, destination-descriptor resolution, and copy operations in `pkg/storage/manager/`, routing each operation to the correct primary, secondary, cache, or final `storage.StagesStorage` instance +- [X] T008 [P] Add shared test fixtures or helper functions needed to construct final manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` +- [X] T009 [P] Define the minimal OCI-artifact primitives on `storage.StagesStorage` in `pkg/storage/stages_storage.go` for listing attached artifacts, publishing an artifact for a final image digest, and copying attached artifacts between destination descriptors; keep repository selection outside the backend +- [X] T010 [P] Implement the new `StagesStorage` artifact primitives for every registry-backed storage implementation under `pkg/storage/` and keep local-storage behavior explicit and unsupported for artifact publication +- [ ] T011 [P] Add Ginkgo/Gomega contract tests for `StagesStorage` artifact primitives and `StorageManager` repository routing, verifying stage code imports neither concrete registry clients nor repository-selection logic in `pkg/storage/stages_storage_test.go`, `pkg/storage/manager/`, and `pkg/build/stage/` --- ## Phase 3: User Story 1 - Artifacts follow published images (Priority: P1) 🎯 MVP -**Goal**: Replace transitional SBOM/VEX steps with lifecycle-owned OCI-artifact stages and make all attached artifacts follow images into primary, final, cache, and secondary-restored destinations through `StagesStorage`. +**Goal**: Replace transitional SBOM/VEX steps with lifecycle-owned OCI-artifact stages and make all attached artifacts follow images into primary, final, cache, and secondary-restored destinations through `StorageManager` routing. **Independent Test**: Build the existing fixture with primary-only, final, cache, combined final/cache, identical-address, and secondary-repository configurations; retrieve both artifact kinds by each destination image digest. ### Tests for User Story 1 -- [X] T011 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, final-image descriptor association, no filesystem mutation, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` -- [X] T012 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` -- [X] T013 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` -- [ ] T014 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios -- [ ] T015 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` -- [ ] T016 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `storage.StagesStorage` in `pkg/build/stage/` +- [X] T012 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, final-image descriptor association, no filesystem mutation, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` +- [X] T013 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` +- [X] T014 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` +- [ ] T015 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios +- [ ] T016 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` +- [ ] T017 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `StorageManager`, with the manager selecting the appropriate `storage.StagesStorage`, in `pkg/build/stage/` and `pkg/storage/manager/` ### Implementation for User Story 1 -- [X] T017 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including final-image-digest association and SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction -- [X] T018 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including final-image-digest association and VEX generation, cache identity, signing, attestation publication, and fallback-index interaction -- [ ] T019 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior -- [ ] T020 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior -- [ ] T021 [US1] Ensure `PrepareImage` is a no-op and `MutateImage` operates only on the associated OCI artifact through `storage.StagesStorage`, without fetching, rebuilding, storing, or mutating image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go` -- [ ] T022 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` -- [ ] T023 [US1] Execute artifact publication through stage `MutateImage` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` -- [ ] T024 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` -- [X] T025 [US1] Implement shared idempotent artifact propagation through `storage.StagesStorage`, with destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` -- [X] T026 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the `StagesStorage`-backed propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` -- [X] T027 [US1] Connect secondary-stage restoration into primary storage to the same `StagesStorage`-backed propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` +- [X] T018 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including final-image-digest association and SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` +- [X] T019 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including final-image-digest association and VEX generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` +- [ ] T020 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` +- [ ] T021 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` +- [ ] T022 [US1] Ensure `PrepareImage` is a no-op and `MutateImage` operates only on the associated OCI artifact through `StorageManager`, without fetching, rebuilding, storing, or mutating image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go` +- [ ] T023 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` +- [ ] T024 [US1] Execute artifact publication through stage `MutateImage` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` +- [ ] T025 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` +- [X] T026 [US1] Implement shared idempotent artifact propagation through `StorageManager`, with manager-routed source/destination backends, destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [X] T027 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the `StorageManager`-routed propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` +- [X] T028 [US1] Connect secondary-stage restoration into primary storage to the same `StorageManager`-routed propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` **Checkpoint**: User Story 1 is independently functional; `SbomStage` and `VexStage` are the sole lifecycle owners, operate on final-image-associated OCI artifacts, and artifacts follow every applicable published image. @@ -73,19 +74,19 @@ ### Tests for User Story 2 -- [X] T028 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform final-image subject selection in `pkg/build/artifact_subject_test.go` -- [X] T029 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and final parent digest are distinct per platform in `pkg/build/stage/sbom_test.go` -- [X] T030 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into stage-owned tests and ensure the final suite contains no step-specific test dependency -- [ ] T031 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each final platform manifest -- [X] T032 [US2] Extend `test/e2e/vex/` with single-platform final-manifest placement and multi-platform final-index-only placement assertions -- [ ] T033 [US2] Add storage-backed tests for destination platform/index descriptor resolution when the copied image digest differs from the source in `pkg/build/artifact_propagation_test.go` +- [X] T029 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform final-image subject selection in `pkg/build/artifact_subject_test.go` +- [X] T030 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and final parent digest are distinct per platform in `pkg/build/stage/sbom_test.go` +- [X] T031 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into stage-owned tests and ensure the final suite contains no step-specific test dependency +- [ ] T032 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each final platform manifest +- [X] T033 [US2] Extend `test/e2e/vex/` with single-platform final-manifest placement and multi-platform final-index-only placement assertions +- [ ] T034 [US2] Add storage-backed tests for destination platform/index descriptor resolution when the copied image digest differs from the source in `pkg/build/artifact_propagation_test.go` ### Implementation for User Story 2 -- [X] T034 [US2] Implement explicit final-image artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` -- [ ] T035 [US2] Pass the final target platform and resolved final platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` -- [ ] T036 [US2] Make `VexStage` registration run once per multi-platform image set with the final top-level index subject, and use the final image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` -- [X] T037 [US2] Ensure `StagesStorage`-backed propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go` and `pkg/storage/` +- [X] T035 [US2] Implement explicit final-image artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` +- [ ] T036 [US2] Pass the final target platform and resolved final platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` +- [ ] T037 [US2] Make `VexStage` registration run once per multi-platform image set with the final top-level index subject, and use the final image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` +- [X] T038 [US2] Ensure `StorageManager`-routed propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go`, `pkg/storage/manager/`, and `pkg/storage/` **Checkpoint**: User Story 2 is independently testable and no artifact can silently use an index subject for a platform SBOM or duplicate multi-platform VEX onto platform manifests. @@ -99,18 +100,18 @@ ### Tests for User Story 3 -- [X] T038 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across final image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` -- [X] T039 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across final parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` -- [X] T040 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing through `StagesStorage` in `pkg/build/artifact_propagation_test.go` -- [ ] T041 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios -- [X] T042 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests +- [X] T039 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across final image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` +- [X] T040 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across final parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` +- [X] T041 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing through `StorageManager` in `pkg/build/artifact_propagation_test.go` +- [ ] T042 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios +- [X] T043 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests ### Implementation for User Story 3 -- [X] T043 [US3] Include all effective SBOM inputs and the final parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` -- [X] T044 [US3] Include VEX document content, final parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` -- [ ] T045 [US3] Select reusable artifact-bearing stages from primary and secondary storage through `StagesStorage` using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` -- [X] T046 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/`, `pkg/storage/`, and `pkg/build/artifact_propagation.go` +- [X] T044 [US3] Include all effective SBOM inputs and the final parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` +- [X] T045 [US3] Include VEX document content, final parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` +- [ ] T046 [US3] Select reusable artifact-bearing stages from primary and secondary storage through `StorageManager` using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` +- [X] T047 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/`, `pkg/storage/`, and `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 3 is independently testable; unchanged inputs reuse artifacts and every effective changed input invalidates only the affected artifact identity. @@ -124,18 +125,18 @@ ### Tests for User Story 4 -- [X] T047 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` -- [X] T048 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` -- [X] T049 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent `StagesStorage`-backed attachment in `pkg/oci/artifact/` and `pkg/storage/` -- [ ] T050 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior -- [X] T051 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories +- [X] T048 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` +- [X] T049 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` +- [X] T050 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent `StorageManager`-routed attachment in `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/storage/` +- [ ] T051 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior +- [X] T052 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories ### Implementation for User Story 4 -- [X] T052 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` -- [X] T053 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared `StagesStorage`-backed error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` -- [X] T054 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` -- [X] T055 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` +- [X] T053 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` +- [X] T054 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared `StorageManager`-routed error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [X] T055 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` +- [X] T056 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` **Checkpoint**: User Story 4 is independently testable; registry failures and local-only configuration produce predictable results without changing repository flag semantics. @@ -145,18 +146,18 @@ **Purpose**: Validate the complete implementation against the revised stage-ownership and storage-abstraction boundaries. -- [X] T056 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, direct registry-client access from stages, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic -- [ ] T057 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners -- [ ] T058 [P] Verify all stage registry interaction goes through `storage.StagesStorage`, all supported registry-backed implementations satisfy the artifact methods, and local storage rejects artifact publication explicitly in `pkg/storage/` -- [ ] T059 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures -- [X] T060 Run formatting with `task format` for authored Go directories -- [X] T061 Run compilation with `task build` -- [X] T062 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` -- [ ] T063 Run the complete unit suite with `task test:unit` -- [X] T064 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` -- [X] T065 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` -- [ ] T066 Run legacy integration coverage with `task test:integration` -- [X] T067 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files +- [X] T057 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, direct registry-client access from stages, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic +- [ ] T058 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners +- [ ] T059 [P] Verify all stage registry interaction goes through `StorageManager`, the manager routes to all supported registry-backed `storage.StagesStorage` implementations, and local storage rejects artifact publication explicitly in `pkg/storage/` and `pkg/storage/manager/` +- [ ] T060 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures +- [X] T061 Run formatting with `task format` for authored Go directories +- [X] T062 Run compilation with `task build` +- [X] T063 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` +- [ ] T064 Run the complete unit suite with `task test:unit` +- [X] T065 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` +- [X] T066 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` +- [ ] T067 Run legacy integration coverage with `task test:integration` +- [X] T068 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files --- @@ -183,12 +184,12 @@ - Phase 1 tasks T002 and T003 can run in parallel after T001's baseline inventory. - In Phase 2, T008–T010 can proceed in parallel once the required storage method shape is agreed; backend implementations must converge on the same interface. -- Within US1, T017 and T018 are parallel stage files; T012, T013, and T016 are separate test concerns. T019 and T020 are parallel migrations when their callers are disjoint. -- Within US2, T028/T029 and T031–T033 are parallel test work; subject-selection and VEX-placement implementation can proceed in separate files. -- Within US3, T038 and T039 are parallel stage-owned identity tests; T041 can proceed independently once stage contracts are stable. -- Within US4, T047–T049 are parallel test tasks, and T051 can proceed independently in cleanup files. -- After Phase 2, separate contributors can work on stage migration, storage backends, propagation, and validation tests, but deletion of transitional files (T024) must wait for all callers/tests to migrate. -- Polish review and regression checks (T056–T059) can run in parallel before the sequential repository-wide validation commands T060–T067. +- Within US1, T018 and T019 are parallel stage files; T012, T013, and T017 are separate test concerns. T020 and T021 are parallel migrations when their callers are disjoint. +- Within US2, T029/T030 and T032–T034 are parallel test work; subject-selection and VEX-placement implementation can proceed in separate files. +- Within US3, T039 and T040 are parallel stage-owned identity tests; T042 can proceed independently once stage contracts are stable. +- Within US4, T048–T050 are parallel test tasks, and T052 can proceed independently in cleanup files. +- After Phase 2, separate contributors can work on stage migration, storage backends, propagation, and validation tests, but deletion of transitional files (T025) must wait for all callers/tests to migrate. +- Polish review and regression checks (T057–T060) can run in parallel before the sequential repository-wide validation commands T061–T068. --- @@ -198,20 +199,20 @@ # After Phase 2, start independent stage, migration, storage, and test work: Task: T011 — stage lifecycle tests in pkg/build/stage/artifact_test.go Task: T012 — propagation tests in pkg/build/artifact_propagation_test.go -Task: T015 — migration coverage in pkg/build/ -Task: T016 — StagesStorage usage tests in pkg/build/stage/ -Task: T017 — SbomStage in pkg/build/stage/sbom.go -Task: T018 — VexStage in pkg/build/stage/vex.go +Task: T016 — migration coverage in pkg/build/ +Task: T017 — StagesStorage usage tests in pkg/build/stage/ +Task: T018 — SbomStage in pkg/build/stage/sbom.go +Task: T019 — VexStage in pkg/build/stage/vex.go Task: T014 — repository propagation scenarios in test/e2e/sbom/ # Integrate after the stage and storage contracts are stable: -Task: T019 — migrate SBOM behavior and callers -Task: T020 — migrate VEX behavior and callers -Task: T021 — enforce final-image OCI-artifact-only mutation -Task: T022 — register stages in pkg/build/build_phase.go -Task: T023 — remove duplicate AfterImages convergence -Task: T024 — delete transitional step files and tests -Task: T025 — implement shared StagesStorage-backed propagation +Task: T020 — migrate SBOM behavior and callers +Task: T021 — migrate VEX behavior and callers +Task: T022 — enforce final-image OCI-artifact-only mutation +Task: T023 — register stages in pkg/build/build_phase.go +Task: T024 — remove duplicate AfterImages convergence +Task: T025 — delete transitional step files and tests +Task: T026 — implement shared StagesStorage-backed propagation ``` ## Implementation Strategy @@ -236,13 +237,13 @@ Task: T025 — implement shared StagesStorage-backed propagation ### Traceability -- **FR-001–FR-003**: T017–T027, T034–T037 -- **FR-004–FR-006**: T028–T037 -- **FR-007–FR-010**: T008–T010, T012–T027, T048, T052–T054 -- **FR-011–FR-013**: T038–T046 -- **FR-014–FR-016**: T003, T046, T051, T055 -- **FR-017–FR-018**: T057–T059 and all implementation tasks; no CLI flag changes or OCI Referrers migration -- **Stage ownership and storage boundary**: T008–T010, T015–T024, T056–T058 +- **FR-001–FR-003**: T018–T028, T035–T038 +- **FR-004–FR-006**: T029–T038 +- **FR-007–FR-010**: T008–T010, T012–T028, T049, T053–T055 +- **FR-011–FR-013**: T039–T047 +- **FR-014–FR-016**: T003, T047, T052, T056 +- **FR-017–FR-018**: T058–T060 and all implementation tasks; no CLI flag changes or OCI Referrers migration +- **Stage ownership and storage boundary**: T008–T010, T016–T025, T057–T059 ## Notes diff --git a/test/mock/bom_patcher.go b/test/mock/bom_patcher.go index ab1c67cc48..ca9e66ae2c 100644 --- a/test/mock/bom_patcher.go +++ b/test/mock/bom_patcher.go @@ -1,9 +1,9 @@ // Code generated by MockGen. DO NOT EDIT. -// Source: sbom_step.go +// Source: sbom_processor.go // // Generated by this command: // -// mockgen -source sbom_step.go -package mock -destination ../../test/mock/bom_patcher.go -mock_names BOMPatcherInterface=MockBOMPatcher +// mockgen -source sbom_processor.go -package mock -destination ../../test/mock/bom_patcher.go -mock_names BOMPatcherInterface=MockBOMPatcher // // Package mock is a generated GoMock package. diff --git a/test/mock/stages_storage.go b/test/mock/stages_storage.go index e63e986bcf..149a568c1e 100644 --- a/test/mock/stages_storage.go +++ b/test/mock/stages_storage.go @@ -114,6 +114,20 @@ func (mr *MockStagesStorageMockRecorder) ConstructStageImageName(projectName, di return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ConstructStageImageName", reflect.TypeOf((*MockStagesStorage)(nil).ConstructStageImageName), projectName, digest, creationTs) } +// CopyAttachedArtifacts mocks base method. +func (m *MockStagesStorage) CopyAttachedArtifacts(ctx context.Context, sourceRepository, sourceDigest, destinationRepository, destinationDigest string) error { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "CopyAttachedArtifacts", ctx, sourceRepository, sourceDigest, destinationRepository, destinationDigest) + ret0, _ := ret[0].(error) + return ret0 +} + +// CopyAttachedArtifacts indicates an expected call of CopyAttachedArtifacts. +func (mr *MockStagesStorageMockRecorder) CopyAttachedArtifacts(ctx, sourceRepository, sourceDigest, destinationRepository, destinationDigest any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CopyAttachedArtifacts", reflect.TypeOf((*MockStagesStorage)(nil).CopyAttachedArtifacts), ctx, sourceRepository, sourceDigest, destinationRepository, destinationDigest) +} + // CopyFromStorage mocks base method. func (m *MockStagesStorage) CopyFromStorage(ctx context.Context, src storage.StagesStorage, projectName string, stageID image.StageID, opts storage.CopyFromStorageOptions) (*image.StageDesc, error) { m.ctrl.T.Helper() @@ -461,6 +475,21 @@ func (mr *MockStagesStorageMockRecorder) IsManagedImageExist(ctx, projectName, i return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsManagedImageExist", reflect.TypeOf((*MockStagesStorage)(nil).IsManagedImageExist), varargs...) } +// ListAttachedArtifacts mocks base method. +func (m *MockStagesStorage) ListAttachedArtifacts(ctx context.Context, parentDigest string) ([]v1.Descriptor, error) { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "ListAttachedArtifacts", ctx, parentDigest) + ret0, _ := ret[0].([]v1.Descriptor) + ret1, _ := ret[1].(error) + return ret0, ret1 +} + +// ListAttachedArtifacts indicates an expected call of ListAttachedArtifacts. +func (mr *MockStagesStorageMockRecorder) ListAttachedArtifacts(ctx, parentDigest any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ListAttachedArtifacts", reflect.TypeOf((*MockStagesStorage)(nil).ListAttachedArtifacts), ctx, parentDigest) +} + // MutateAndPushImage mocks base method. func (m *MockStagesStorage) MutateAndPushImage(ctx context.Context, src, dest string, newConfig image.SpecConfig, stageImage container_backend.LegacyImageInterface) error { m.ctrl.T.Helper() @@ -517,6 +546,20 @@ func (mr *MockStagesStorageMockRecorder) PostMultiplatformImage(ctx, projectName return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PostMultiplatformImage", reflect.TypeOf((*MockStagesStorage)(nil).PostMultiplatformImage), ctx, projectName, tag, allPlatformsImages, platforms) } +// PublishArtifact mocks base method. +func (m *MockStagesStorage) PublishArtifact(ctx context.Context, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "PublishArtifact", ctx, parentDigest, artifactType, payload, imageName, checksum, targetPlatform, predicateType) + ret0, _ := ret[0].(error) + return ret0 +} + +// PublishArtifact indicates an expected call of PublishArtifact. +func (mr *MockStagesStorageMockRecorder) PublishArtifact(ctx, parentDigest, artifactType, payload, imageName, checksum, targetPlatform, predicateType any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PublishArtifact", reflect.TypeOf((*MockStagesStorage)(nil).PublishArtifact), ctx, parentDigest, artifactType, payload, imageName, checksum, targetPlatform, predicateType) +} + // PutImageMetadata mocks base method. func (m *MockStagesStorage) PutImageMetadata(ctx context.Context, projectName, imageNameOrManagedImageName, commit, stageID string) error { m.ctrl.T.Helper() From 55d8016895a21c1800f88779ef872dab08977e88 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 15:39:12 +0300 Subject: [PATCH 08/17] refactor(build): migrate SBOM and VEX to artifact stages Use the dedicated MutateArtifact lifecycle and StorageManager for registry publication, replacing transitional post-build step implementations. Signed-off-by: Alexandr Zaytsev --- pkg/build/artifact_stage_migration_test.go | 42 ++++ pkg/build/build_phase.go | 20 +- pkg/build/stage/artifact_test.go | 53 +++++ pkg/build/stage/sbom.go | 23 +- pkg/build/stage/vex.go | 63 +++--- pkg/build/vex_processor.go | 21 +- .../manager/artifact_operations_test.go | 115 ++++++++++ pkg/storage/manager/storage_manager.go | 67 ++++++ specs/020-sbom-vex-build-stages/data-model.md | 8 +- specs/020-sbom-vex-build-stages/plan.md | 14 +- specs/020-sbom-vex-build-stages/quickstart.md | 5 +- specs/020-sbom-vex-build-stages/research.md | 6 +- specs/020-sbom-vex-build-stages/tasks.md | 200 +++++++++--------- 13 files changed, 460 insertions(+), 177 deletions(-) create mode 100644 pkg/build/artifact_stage_migration_test.go create mode 100644 pkg/storage/manager/artifact_operations_test.go diff --git a/pkg/build/artifact_stage_migration_test.go b/pkg/build/artifact_stage_migration_test.go new file mode 100644 index 0000000000..ec4dd82c9e --- /dev/null +++ b/pkg/build/artifact_stage_migration_test.go @@ -0,0 +1,42 @@ +package build + +import ( + "os" + "path/filepath" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("artifact stage migration", func() { + It("does not retain transitional SBOM or VEX step implementations", func() { + root := "." + var sourceFiles []string + err := filepath.Walk(root, func(path string, info os.FileInfo, walkErr error) error { + if walkErr != nil { + return walkErr + } + if info.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") { + return nil + } + sourceFiles = append(sourceFiles, path) + return nil + }) + Expect(err).NotTo(HaveOccurred()) + + for _, path := range sourceFiles { + contents, err := os.ReadFile(path) + Expect(err).NotTo(HaveOccurred()) + Expect(string(contents)).NotTo(ContainSubstring("sbom"+"Step"), path) + Expect(string(contents)).NotTo(ContainSubstring("vex"+"Step"), path) + } + + buildPhase, err := os.ReadFile("build_phase.go") + Expect(err).NotTo(HaveOccurred()) + Expect(string(buildPhase)).To(ContainSubstring("GenerateSbomStage")) + Expect(string(buildPhase)).To(ContainSubstring("NewVexStage")) + Expect(string(buildPhase)).To(ContainSubstring("if len(images) == 1")) + Expect(string(buildPhase)).To(ContainSubstring("convergeMultiplatformVexByImageSets")) + }) +}) diff --git a/pkg/build/build_phase.go b/pkg/build/build_phase.go index d6b970d245..538ce34224 100644 --- a/pkg/build/build_phase.go +++ b/pkg/build/build_phase.go @@ -262,7 +262,7 @@ func (phase *BuildPhase) AfterImages(ctx context.Context) error { return err } - if err := phase.convergeVexByImagesSets(ctx); err != nil { + if err := phase.convergeMultiplatformVexByImageSets(ctx); err != nil { return err } @@ -784,7 +784,15 @@ func (phase *BuildPhase) registerSinglePlatformVexStage(ctx context.Context, img signingOptions = phase.VexSigningOptions } stages := img.GetStages() - img.SetStages(append(stages, stage.GenerateVexStage(vexContent, baseOptions, signingOptions))) + publisher := func(ctx context.Context, parentDesc *imagePkg.StageDesc, imageName, targetPlatform string, content []byte, signer signature.Signer, signerIdentity string) error { + return phase.vexProcessor.Converge(ctx, content, parentDesc, imageName, targetPlatform, signer, signerIdentity) + } + img.SetStages(append(stages, stage.NewVexStage(stage.VexStageOptions{ + VexJSON: vexContent, + BaseStageOptions: baseOptions, + SigningOptions: signingOptions, + Publisher: publisher, + }))) return nil } @@ -1799,9 +1807,8 @@ E.g.: }) } -// convergeVexByImagesSets publishes VEX artifacts for all images respecting dependency order. - -func (phase *BuildPhase) convergeVexByImagesSets(ctx context.Context) error { +// convergeMultiplatformVexByImageSets publishes the image-level VEX artifact after the final index exists. +func (phase *BuildPhase) convergeMultiplatformVexByImageSets(ctx context.Context) error { if _, isLocal := phase.Conveyor.StorageManager.GetStagesStorage().(*storage.LocalStagesStorage); isLocal { return nil } @@ -1832,6 +1839,9 @@ func (phase *BuildPhase) convergeVexByImagesSets(ctx context.Context) error { name := names[taskId] images := imagesByName[name] + if len(images) == 1 { + return nil + } return phase.convergeImageVex(ctx, name, images) }); err != nil { diff --git a/pkg/build/stage/artifact_test.go b/pkg/build/stage/artifact_test.go index 011059f896..bc75d546be 100644 --- a/pkg/build/stage/artifact_test.go +++ b/pkg/build/stage/artifact_test.go @@ -5,6 +5,7 @@ import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "github.com/sigstore/sigstore/pkg/signature" "go.uber.org/mock/gomock" "github.com/werf/werf/v2/pkg/build/signing" @@ -33,6 +34,58 @@ var _ = Describe("artifact stages", func() { Expect(base.GetArtifactMetadata().Buildable).To(BeFalse()) }) + It("publishes SBOM through its stage publisher without mutating image content", func(ctx SpecContext) { + ctrl := gomock.NewController(GinkgoT()) + parentImage := mock.NewMockLegacyImageInterface(ctrl) + parentDesc := &image.StageDesc{Info: &image.Info{Repository: "registry.example/app", RepoDigest: "registry.example/app@sha256:parent"}} + parentImage.EXPECT().GetStageDesc().Return(parentDesc) + publisherCalls := 0 + artifactStage := NewSbomStage(SbomStageOptions{ + BaseStageOptions: &BaseStageOptions{ImageName: "app", TargetPlatform: "linux/amd64"}, + Dependency: "scanner-input", + Publisher: func(_ context.Context, gotDesc *image.StageDesc, imageName, platform string) error { + publisherCalls++ + Expect(gotDesc).To(BeIdenticalTo(parentDesc)) + Expect(imageName).To(Equal("app")) + Expect(platform).To(Equal("linux/amd64")) + return nil + }, + }) + parent := NewStageImage(NewContainerBackendStub(), "", parentImage) + stageImage := NewStageImage(NewContainerBackendStub(), "", mock.NewMockLegacyImageInterface(ctrl)) + + Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(Succeed()) + Expect(publisherCalls).To(Equal(1)) + Expect(artifactStage.GetArtifactMetadata().ParentDigest).To(Equal("sha256:parent")) + }) + + It("publishes VEX through its stage publisher without requiring image mutation", func(ctx SpecContext) { + ctrl := gomock.NewController(GinkgoT()) + parentImage := mock.NewMockLegacyImageInterface(ctrl) + parentDesc := &image.StageDesc{Info: &image.Info{Repository: "registry.example/app", RepoDigest: "registry.example/app@sha256:parent"}} + parentImage.EXPECT().GetStageDesc().Return(parentDesc) + publisherCalls := 0 + artifactStage := NewVexStage(VexStageOptions{ + VexJSON: []byte(`{"statements":[]}`), + BaseStageOptions: &BaseStageOptions{ImageName: "app"}, + Publisher: func(_ context.Context, gotDesc *image.StageDesc, imageName, platform string, content []byte, _ signature.Signer, identity string) error { + publisherCalls++ + Expect(gotDesc).To(BeIdenticalTo(parentDesc)) + Expect(imageName).To(Equal("app")) + Expect(platform).To(BeEmpty()) + Expect(content).To(MatchJSON(`{"statements":[]}`)) + Expect(identity).To(BeEmpty()) + return nil + }, + }) + parent := NewStageImage(NewContainerBackendStub(), "", parentImage) + stageImage := NewStageImage(NewContainerBackendStub(), "", mock.NewMockLegacyImageInterface(ctrl)) + + Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(Succeed()) + Expect(publisherCalls).To(Equal(1)) + Expect(artifactStage.GetArtifactMetadata().ParentDigest).To(Equal("sha256:parent")) + }) + DescribeTable("is mutable and non-buildable", func(artifactStage Interface) { Expect(artifactStage.IsMutable()).To(BeTrue()) diff --git a/pkg/build/stage/sbom.go b/pkg/build/stage/sbom.go index 72785f47bb..21f1b3d800 100644 --- a/pkg/build/stage/sbom.go +++ b/pkg/build/stage/sbom.go @@ -12,6 +12,13 @@ import ( type SbomStagePublisher func(ctx context.Context, parentDesc *image.StageDesc, imageName, targetPlatform string) error +type SbomStageOptions struct { + BaseStageOptions *BaseStageOptions + SigningOptions signing.SbomSigningOptions + Dependency string + Publisher SbomStagePublisher +} + type SbomStage struct { *BaseStage @@ -21,7 +28,16 @@ type SbomStage struct { } func GenerateSbomStage(baseStageOptions *BaseStageOptions, sbomSigningOptions signing.SbomSigningOptions, dependency string, publisher SbomStagePublisher) *SbomStage { - return newSbomStage(baseStageOptions, sbomSigningOptions, dependency, publisher) + return NewSbomStage(SbomStageOptions{ + BaseStageOptions: baseStageOptions, + SigningOptions: sbomSigningOptions, + Dependency: dependency, + Publisher: publisher, + }) +} + +func NewSbomStage(options SbomStageOptions) *SbomStage { + return newSbomStage(options.BaseStageOptions, options.SigningOptions, options.Dependency, options.Publisher) } func newSbomStage(baseStageOptions *BaseStageOptions, sbomSigningOptions signing.SbomSigningOptions, dependency string, publisher SbomStagePublisher) *SbomStage { @@ -80,10 +96,7 @@ func (s *SbomStage) GetContentDependencies(ctx context.Context, c Conveyor, buil return s.GetDependencies(ctx, c, nil, nil, nil, buildContextArchive) } -func (s *SbomStage) MutateImage(ctx context.Context, stagesStorage ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { - if _, err := registryFromImageMutatorPusher(stagesStorage); err != nil { - return err - } +func (s *SbomStage) MutateImage(ctx context.Context, _ ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { if s.publisher == nil { return fmt.Errorf("SBOM stage publisher is unavailable") } diff --git a/pkg/build/stage/vex.go b/pkg/build/stage/vex.go index 385f1a2a4a..cc67df27c7 100644 --- a/pkg/build/stage/vex.go +++ b/pkg/build/stage/vex.go @@ -8,28 +8,42 @@ import ( "github.com/sigstore/sigstore/pkg/signature" "github.com/werf/common-go/pkg/util" - "github.com/werf/logboek" - "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/build/signing" "github.com/werf/werf/v2/pkg/container_backend" "github.com/werf/werf/v2/pkg/image" - "github.com/werf/werf/v2/pkg/oci/artifact" - vexImage "github.com/werf/werf/v2/pkg/vex/image" ) +type VexStagePublisher func(ctx context.Context, parentDesc *image.StageDesc, imageName, targetPlatform string, vexJSON []byte, signer signature.Signer, signerIdentity string) error + +type VexStageOptions struct { + VexJSON []byte + BaseStageOptions *BaseStageOptions + SigningOptions signing.VexSigningOptions + Publisher VexStagePublisher +} + type VexStage struct { *BaseStage vexJSON []byte signer signature.Signer signerIdentity string + publisher VexStagePublisher } func GenerateVexStage(vexJSON []byte, baseStageOptions *BaseStageOptions, vexSigningOptions signing.VexSigningOptions) *VexStage { - return newVexStage(vexJSON, baseStageOptions, vexSigningOptions) + return NewVexStage(VexStageOptions{ + VexJSON: vexJSON, + BaseStageOptions: baseStageOptions, + SigningOptions: vexSigningOptions, + }) } -func newVexStage(vexJSON []byte, baseStageOptions *BaseStageOptions, vexSigningOptions signing.VexSigningOptions) *VexStage { +func NewVexStage(options VexStageOptions) *VexStage { + return newVexStage(options.VexJSON, options.BaseStageOptions, options.SigningOptions, options.Publisher) +} + +func newVexStage(vexJSON []byte, baseStageOptions *BaseStageOptions, vexSigningOptions signing.VexSigningOptions, publisher VexStagePublisher) *VexStage { var signer signature.Signer var signerIdentity string if vexSigningOptions.Enabled { @@ -42,6 +56,7 @@ func newVexStage(vexJSON []byte, baseStageOptions *BaseStageOptions, vexSigningO vexJSON: vexJSON, signer: signer, signerIdentity: signerIdentity, + publisher: publisher, } stage.SetArtifactMetadata(&ArtifactStageMetadata{ Kind: ArtifactKindVex, @@ -74,17 +89,14 @@ func (s *VexStage) GetDependencies(_ context.Context, _ Conveyor, _ container_ba } } - return calculateVexStageChecksum(s.vexJSON, parentDigest, s.signerIdentity), nil + return CalculateVexStageChecksum(s.vexJSON, parentDigest, s.signerIdentity), nil } func (s *VexStage) GetContentDependencies(ctx context.Context, c Conveyor, buildContextArchive container_backend.BuildContextArchiver) (string, error) { return s.GetDependencies(ctx, c, nil, nil, nil, buildContextArchive) } -func (s *VexStage) MutateImage(ctx context.Context, stagesStorage ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { - if _, err := registryFromImageMutatorPusher(stagesStorage); err != nil { - return err - } +func (s *VexStage) MutateImage(ctx context.Context, _ ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { if prevBuiltImage == nil || prevBuiltImage.Image == nil { return fmt.Errorf("VEX stage parent image is unavailable") } @@ -108,25 +120,17 @@ func (s *VexStage) MutateImage(ctx context.Context, stagesStorage ImageMutatorPu metadata := s.GetArtifactMetadata() metadata.ParentDigest = parentDigest - checksum := calculateVexStageChecksum(s.vexJSON, parentDigest, s.signerIdentity) - store := artifact.NewOCIStore(parentDesc.Info.Repository, stageImage.Image.Name()) - needed, err := checkVexStagePublishNeeded(ctx, store, parentDigest, checksum) - if err != nil { - return fmt.Errorf("check VEX publish needed: %w", err) - } - if !needed { - logboek.Context(ctx).Default().LogF("image %s: VEX artifact is up to date — skipping publish\n", stageImage.Image.Name()) - return nil + if s.publisher == nil { + return fmt.Errorf("VEX stage publisher is unavailable") } - return logboek.Context(ctx).Default().LogProcess("image %s: Published VEX artifact", stageImage.Image.Name()).DoError(func() error { - return vexImage.PushVEX(ctx, s.vexJSON, parentDesc.Info.Repository, parentDigest, s.ImageName(), checksum, s.TargetPlatform(), s.signer) - }) + return s.publisher(ctx, parentDesc, s.ImageName(), s.TargetPlatform(), s.vexJSON, s.signer, s.signerIdentity) } const vexStageArtifactFormatVersion = "2" -func calculateVexStageChecksum(vexJSON []byte, parentDigest, signerIdentity string) string { +// CalculateVexStageChecksum returns the cache identity for a VEX artifact. +func CalculateVexStageChecksum(vexJSON []byte, parentDigest, signerIdentity string) string { parts := []string{ vexStageArtifactFormatVersion, util.Sha256Hash(string(vexJSON)), @@ -135,14 +139,3 @@ func calculateVexStageChecksum(vexJSON []byte, parentDigest, signerIdentity stri } return util.Sha256Hash(strings.Join(parts, "-")) } - -func checkVexStagePublishNeeded(ctx context.Context, store artifact.Store, parentDigest, checksum string) (bool, error) { - desc, found, err := attestation.FindAttachedArtifact(ctx, store, parentDigest, attestation.PredicateKindOpenVEX) - if err != nil { - return false, fmt.Errorf("check VEX cache: %w", err) - } - if found && desc.Annotations[image.WerfChecksumAnnotation] == checksum { - return false, nil - } - return true, nil -} diff --git a/pkg/build/vex_processor.go b/pkg/build/vex_processor.go index 8a95755fa3..855dd45945 100644 --- a/pkg/build/vex_processor.go +++ b/pkg/build/vex_processor.go @@ -3,13 +3,12 @@ package build import ( "context" "fmt" - "strings" "github.com/sigstore/sigstore/pkg/signature" - "github.com/werf/common-go/pkg/util" "github.com/werf/logboek" "github.com/werf/werf/v2/pkg/attestation" + "github.com/werf/werf/v2/pkg/build/stage" "github.com/werf/werf/v2/pkg/image" "github.com/werf/werf/v2/pkg/oci/artifact" vexImage "github.com/werf/werf/v2/pkg/vex/image" @@ -25,7 +24,7 @@ func (processor *vexProcessor) Converge(ctx context.Context, vexJSON []byte, sta repo := stageDesc.Info.Repository parentDigest := stageDesc.Info.GetDigest() - checksum := calculateVEXChecksum(vexJSON, parentDigest, signerIdentity) + checksum := stage.CalculateVexStageChecksum(vexJSON, parentDigest, signerIdentity) store := artifact.NewOCIStore(repo, werfImgName) @@ -43,22 +42,6 @@ func (processor *vexProcessor) Converge(ctx context.Context, vexJSON []byte, sta }) } -const vexArtifactFormatVersion = "2" - -// calculateVEXChecksum builds the cache identity of the VEX artifact: document -// content and parent digest (FR-011 of 013-vex-lifecycle), the bump-able artifact -// format version, and the signer public-key fingerprint — so enabling signing, -// rotating the key, or bumping the format each republish the artifact. -func calculateVEXChecksum(vexJSON []byte, parentDigest, signerIdentity string) string { - parts := []string{ - vexArtifactFormatVersion, - util.Sha256Hash(string(vexJSON)), - parentDigest, - signerIdentity, - } - return util.Sha256Hash(strings.Join(parts, "-")) -} - // checkVEXPublishNeeded returns true if the VEX artifact should be published // (no existing VEX artifact of either format or its checksum annotation differs // from the current checksum), and false if publishing can be skipped. A legacy diff --git a/pkg/storage/manager/artifact_operations_test.go b/pkg/storage/manager/artifact_operations_test.go new file mode 100644 index 0000000000..84bbfe6f25 --- /dev/null +++ b/pkg/storage/manager/artifact_operations_test.go @@ -0,0 +1,115 @@ +package manager + +import ( + "context" + "errors" + + v1 "github.com/google/go-containerregistry/pkg/v1" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/werf/werf/v2/pkg/image" + "github.com/werf/werf/v2/pkg/storage" +) + +type artifactOperationsStorage struct { + storage.StagesStorage + address string + + listedParent string + publishedParent string + copiedSourceRepo string + copiedSourceDigest string + copiedDestRepo string + copiedDestDigest string + resolvedProject string + resolvedStage image.StageID + resolveResult *image.StageDesc + operationError error +} + +func (s *artifactOperationsStorage) Address() string { return s.address } +func (s *artifactOperationsStorage) String() string { return s.address } + +func (s *artifactOperationsStorage) ListAttachedArtifacts(_ context.Context, parentDigest string) ([]v1.Descriptor, error) { + s.listedParent = parentDigest + return []v1.Descriptor{{Digest: v1.Hash{Algorithm: "sha256", Hex: "artifact"}}}, s.operationError +} + +func (s *artifactOperationsStorage) PublishArtifact(_ context.Context, parentDigest, _ string, _ []byte, _, _, _, _ string) error { + s.publishedParent = parentDigest + return s.operationError +} + +func (s *artifactOperationsStorage) CopyAttachedArtifacts(_ context.Context, sourceRepository, sourceDigest, destinationRepository, destinationDigest string) error { + s.copiedSourceRepo = sourceRepository + s.copiedSourceDigest = sourceDigest + s.copiedDestRepo = destinationRepository + s.copiedDestDigest = destinationDigest + return s.operationError +} + +func (s *artifactOperationsStorage) GetStageDesc(_ context.Context, projectName string, stageID image.StageID) (*image.StageDesc, error) { + s.resolvedProject = projectName + s.resolvedStage = stageID + return s.resolveResult, s.operationError +} + +var _ = Describe("StorageManager artifact operations", func() { + It("routes listing, publication, resolution, and copying through the selected storages", func(ctx SpecContext) { + source := &artifactOperationsStorage{address: "registry.example/source"} + destination := &artifactOperationsStorage{address: "registry.example/destination"} + stageID := image.StageID{Digest: "stage-digest", CreationTs: 42} + resolved := &image.StageDesc{StageID: &stageID} + destination.resolveResult = resolved + manager := &StorageManager{ProjectName: "project"} + + artifacts, err := manager.ListAttachedArtifacts(ctx, source, "sha256:parent") + Expect(err).NotTo(HaveOccurred()) + Expect(artifacts).To(HaveLen(1)) + Expect(source.listedParent).To(Equal("sha256:parent")) + + Expect(manager.PublishArtifact(ctx, destination, "sha256:parent", "application/test", []byte("payload"), "image", "checksum", "", "")).To(Succeed()) + Expect(destination.publishedParent).To(Equal("sha256:parent")) + + result, err := manager.ResolveStageDescriptor(ctx, destination, stageID) + Expect(err).NotTo(HaveOccurred()) + Expect(result).To(BeIdenticalTo(resolved)) + Expect(destination.resolvedProject).To(Equal("project")) + Expect(destination.resolvedStage).To(Equal(stageID)) + + Expect(manager.CopyAttachedArtifacts(ctx, source, "sha256:source", destination, "sha256:destination")).To(Succeed()) + Expect(destination.copiedSourceRepo).To(Equal(source.address)) + Expect(destination.copiedSourceDigest).To(Equal("sha256:source")) + Expect(destination.copiedDestRepo).To(Equal(destination.address)) + Expect(destination.copiedDestDigest).To(Equal("sha256:destination")) + }) + + It("skips copying between identical repository addresses", func(ctx SpecContext) { + source := &artifactOperationsStorage{address: "registry.example/repository"} + destination := &artifactOperationsStorage{address: source.address} + manager := &StorageManager{} + + Expect(manager.CopyAttachedArtifacts(ctx, source, "sha256:source", destination, "sha256:destination")).To(Succeed()) + Expect(destination.copiedSourceRepo).To(BeEmpty()) + }) + + It("rejects publication and copying through local storage", func(ctx SpecContext) { + local := storage.NewLocalStagesStorage(nil) + manager := &StorageManager{} + + Expect(manager.PublishArtifact(ctx, local, "sha256:parent", "application/test", []byte("payload"), "image", "checksum", "", "")).To(MatchError(ContainSubstring("local stages storage"))) + Expect(manager.CopyAttachedArtifacts(ctx, local, "sha256:source", &artifactOperationsStorage{address: "registry.example/destination"}, "sha256:destination")).To(MatchError(ContainSubstring("local stages storage"))) + }) + + It("wraps backend errors with the routed operation", func(ctx SpecContext) { + backendError := errors.New("backend unavailable") + source := &artifactOperationsStorage{address: "registry.example/source", operationError: backendError} + manager := &StorageManager{} + + _, err := manager.ListAttachedArtifacts(ctx, source, "sha256:parent") + Expect(err).To(MatchError(ContainSubstring("list attached artifacts from registry.example/source"))) + err = manager.PublishArtifact(ctx, source, "sha256:parent", "application/test", nil, "image", "checksum", "", "") + Expect(err).To(MatchError(ContainSubstring("publish artifact to registry.example/source"))) + }) +}) diff --git a/pkg/storage/manager/storage_manager.go b/pkg/storage/manager/storage_manager.go index 8bcb3f77f6..e8f8275f40 100644 --- a/pkg/storage/manager/storage_manager.go +++ b/pkg/storage/manager/storage_manager.go @@ -10,6 +10,7 @@ import ( "time" "github.com/cenkalti/backoff/v5" + v1 "github.com/google/go-containerregistry/pkg/v1" "gopkg.in/yaml.v2" "github.com/werf/lockgate" @@ -78,6 +79,11 @@ type StorageManagerInterface interface { CopyStageIntoCacheStorages(ctx context.Context, stageID image.StageID, cacheStagesStorages []storage.StagesStorage, opts CopyStageIntoStorageOptions) error CopyStageIntoFinalStorage(ctx context.Context, stageID image.StageID, finalStagesStorage storage.StagesStorage, opts CopyStageIntoStorageOptions) (*image.StageDesc, error) + ListAttachedArtifacts(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest string) ([]v1.Descriptor, error) + PublishArtifact(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error + ResolveStageDescriptor(ctx context.Context, stagesStorage storage.StagesStorage, stageID image.StageID) (*image.StageDesc, error) + CopyAttachedArtifacts(ctx context.Context, sourceStorage storage.StagesStorage, sourceDigest string, destinationStorage storage.StagesStorage, destinationDigest string) error + ForEachDeleteStage(ctx context.Context, options ForEachDeleteStageOptions, stageDescSet image.StageDescSet, f func(ctx context.Context, stageDesc *image.StageDesc, err error) error) error ForEachDeleteFinalStage(ctx context.Context, options ForEachDeleteStageOptions, stageDescSet image.StageDescSet, f func(ctx context.Context, stageDesc *image.StageDesc, err error) error) error ForEachRejectedStage(ctx context.Context, stageIDs []image.StageID, f func(ctx context.Context, stageID image.StageID) error) error @@ -217,6 +223,67 @@ func (m *StorageManager) GetCacheStagesStorageList() []storage.StagesStorage { return m.CacheStagesStorageList } +func (m *StorageManager) ListAttachedArtifacts(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest string) ([]v1.Descriptor, error) { + if stagesStorage == nil { + return nil, fmt.Errorf("list attached artifacts: stages storage is nil") + } + if parentDigest == "" { + return nil, fmt.Errorf("list attached artifacts: parent digest is empty") + } + + artifacts, err := stagesStorage.ListAttachedArtifacts(ctx, parentDigest) + if err != nil { + return nil, fmt.Errorf("list attached artifacts from %s: %w", stagesStorage.String(), err) + } + return artifacts, nil +} + +func (m *StorageManager) PublishArtifact(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error { + if stagesStorage == nil { + return fmt.Errorf("publish artifact: stages storage is nil") + } + if stagesStorage.Address() == storage.LocalStorageAddress { + return fmt.Errorf("publish artifact: local stages storage does not support artifact operations") + } + if parentDigest == "" { + return fmt.Errorf("publish artifact: parent digest is empty") + } + + if err := stagesStorage.PublishArtifact(ctx, parentDigest, artifactType, payload, imageName, checksum, targetPlatform, predicateType); err != nil { + return fmt.Errorf("publish artifact to %s: %w", stagesStorage.String(), err) + } + return nil +} + +func (m *StorageManager) ResolveStageDescriptor(ctx context.Context, stagesStorage storage.StagesStorage, stageID image.StageID) (*image.StageDesc, error) { + if stagesStorage == nil { + return nil, fmt.Errorf("resolve stage descriptor: stages storage is nil") + } + + desc, err := stagesStorage.GetStageDesc(ctx, m.ProjectName, stageID) + if err != nil { + return nil, fmt.Errorf("resolve stage %s descriptor from %s: %w", stageID.String(), stagesStorage.String(), err) + } + return desc, nil +} + +func (m *StorageManager) CopyAttachedArtifacts(ctx context.Context, sourceStorage storage.StagesStorage, sourceDigest string, destinationStorage storage.StagesStorage, destinationDigest string) error { + if sourceStorage == nil || destinationStorage == nil { + return fmt.Errorf("copy attached artifacts: source and destination storage are required") + } + if sourceStorage.Address() == storage.LocalStorageAddress || destinationStorage.Address() == storage.LocalStorageAddress { + return fmt.Errorf("copy attached artifacts: local stages storage does not support artifact operations") + } + if sourceStorage.Address() == destinationStorage.Address() { + return nil + } + + if err := destinationStorage.CopyAttachedArtifacts(ctx, sourceStorage.Address(), sourceDigest, destinationStorage.Address(), destinationDigest); err != nil { + return fmt.Errorf("copy attached artifacts from %s to %s: %w", sourceStorage.String(), destinationStorage.String(), err) + } + return nil +} + func (m *StorageManager) GetServiceValuesRepo() string { if m.FinalStagesStorage != nil { return m.FinalStagesStorage.String() diff --git a/specs/020-sbom-vex-build-stages/data-model.md b/specs/020-sbom-vex-build-stages/data-model.md index f08387cc48..b8eedc80c6 100644 --- a/specs/020-sbom-vex-build-stages/data-model.md +++ b/specs/020-sbom-vex-build-stages/data-model.md @@ -12,12 +12,14 @@ An internal build-stage operation associated with the final image digest (and, f | Artifact kind | CycloneDX SBOM or OpenVEX. | | Generation inputs | Scanner/merge inputs for SBOM, document content for VEX, format version, and signer identity. | | Mutable/buildable flags | Non-buildable and mutable, matching registry-only stages such as signing, but unlike signing the output is an associated OCI artifact rather than a manifest mutation. | -| Storage abstraction | `StorageManager` routes all registry operations to primary, secondary, cache, or final `storage.StagesStorage`. | +| Storage abstraction | `StorageManager` routes all registry operations initiated by `MutateArtifact` to primary, secondary, cache, or final `storage.StagesStorage`. | Validation rules: -- The final image descriptor/digest must be available before `MutateImage` runs. -- The stage must use `StorageManager` for registry access; repository selection is performed by the manager. +- The final image descriptor/digest must be available before `MutateArtifact` runs. +- The stage must use `GetDependencies`/`GetContentDependencies` for checksum calculation according to the `SignStage` convention. +- The stage must use `MutateArtifact` for registry access through `StorageManager`; repository selection is performed by the manager. +- `MutateImage` is not used by these artifact stages. - SBOM for a multi-platform image must use the corresponding platform manifest. - VEX must use the platform manifest for single-platform images and the top-level index for multi-platform images. - An enabled artifact stage requires registry-backed storage. diff --git a/specs/020-sbom-vex-build-stages/plan.md b/specs/020-sbom-vex-build-stages/plan.md index a25502b05d..4a13628146 100644 --- a/specs/020-sbom-vex-build-stages/plan.md +++ b/specs/020-sbom-vex-build-stages/plan.md @@ -6,7 +6,7 @@ ## Summary -Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and replace the `sbomStep` and `vexStep` implementations with registry-backed, non-buildable mutable stages modeled after `pkg/build/stage/sign.go`. The new `SbomStage` and `VexStage` become the sole owners of SBOM/VEX cache identity, generation, signing, attestation publication, and fallback-index interaction. Unlike ordinary image stages, they are associated with the final image digest, operate on the associated OCI artifact rather than on the image filesystem or image layers, and perform all registry operations through `StorageManager`, which routes them to the appropriate primary, secondary, cache, or final `StagesStorage`. +Move SBOM and VEX generation out of the `BuildPhase.AfterImages` post-build pass and replace the `sbomStep` and `vexStep` implementations with registry-backed, non-buildable mutable stages modeled after `pkg/build/stage/sign.go`. The new `SbomStage` and `VexStage` become the sole owners of SBOM/VEX cache identity, generation, signing, attestation publication, and fallback-index interaction. Unlike ordinary image stages, they are associated with the final image digest, operate on the associated OCI artifact rather than on the image filesystem or image layers, and perform all registry operations through a dedicated artifact-stage API, `MutateArtifact`, which uses `StorageManager` to route them to the appropriate primary, secondary, cache, or final `StagesStorage`. Artifact publication will use explicit source and destination image descriptors. SBOM remains platform-specific; VEX is attached once at the top-level image index for multi-platform images and to the image manifest for single-platform images. A shared idempotent propagation operation will cover primary-to-final, primary-to-cache, and secondary-to-primary copies, resolving the destination digest and preserving fatal final-repository versus best-effort cache error policies. @@ -75,11 +75,13 @@ Detailed findings are in [research.md](./research.md). Key decisions: - Extend `pkg/build/stage` with stage names and constructors for `SbomStage` and `VexStage`, following the shape of `SignStage`. - Move all behavior currently owned by `sbomStep` into `SbomStage`; remove `sbom_step.go` and its step-specific tests once callers are migrated. - Move all behavior currently owned by `vexStep` into `VexStage`; remove `vex_step.go` and its step-specific tests once callers are migrated. -- Artifact stages must not mutate, rebuild, fetch, or store the image filesystem. Their `PrepareImage` path is a no-op; their `MutateImage` path operates on the associated OCI artifact and owns registry-side generation and publication. +- Artifact stages must not mutate, rebuild, fetch, or store the image filesystem. Their `PrepareImage` path is a no-op. +- Add a dedicated stage lifecycle method `MutateArtifact` for stages that work with registry-backed OCI artifacts without mutating the image. `SbomStage` and `VexStage` must implement and use `MutateArtifact` for generation, cache checks, signing, and publication through `StorageManager`. +- `MutateImage` remains the method for stages that mutate/publish an image, such as `SignStage`; it must not be used as the OCI-artifact operation for `SbomStage` or `VexStage`. - The stage's subject is the final image digest: for single-platform images this is the published image manifest digest; for multi-platform images SBOM uses each final platform manifest digest and VEX uses the final top-level image index digest. The artifact stage must never be treated as an image layer or as a replacement image. -- All registry reads, writes, copies, metadata operations, and artifact-related repository interaction from `SbomStage` and `VexStage` must use `StorageManager`. The manager selects primary, secondary, cache, or final `StagesStorage` according to the operation; direct registry client access and direct repository selection from the stages are prohibited. +- All registry reads, writes, copies, metadata operations, and artifact-related repository interaction from `SbomStage` and `VexStage` must be performed from `MutateArtifact` through `StorageManager`. The manager selects primary, secondary, cache, or final `StagesStorage` according to the operation; direct registry client access and direct repository selection from the stages are prohibited. - Extend `StorageManager` with the minimal artifact-oriented operations required by the stages and propagation. Implement the corresponding `StagesStorage` primitives only where needed to preserve fallback-index behavior: find/list attached artifacts, publish an OCI artifact for a final image digest, and copy attached artifacts between destination image descriptors. Implement these methods for every supported registry-backed storage implementation and keep local storage behavior explicit. -- Ensure stage dependencies include the parent image identity and all effective artifact inputs. SBOM dependencies include scanner, merge/GOST, signer, format version, and target platform. VEX dependencies include document content, parent identity, signer, and format version. +- Preserve the existing stage checksum convention from `SignStage`: implement `GetDependencies` by assembling all effective inputs and returning `util.Sha256Hash(args...)`; implement `GetContentDependencies` consistently for the stage lifecycle. SBOM dependencies include final image identity, scanner, merge/GOST, signer, format version, and target platform. VEX dependencies include final image identity, document content, signer, and format version. The checksum must be calculated from stage inputs, while the artifact subject remains the final image digest. - Register the stages after the content-producing stage and before the lifecycle completes for applicable images. The registration must work for Stapel and Dockerfile image paths and for restored stages. - Preserve stage cache behavior: a suitable artifact-bearing stage can be selected from primary/secondary storage; changed effective inputs produce a different stage identity. @@ -89,7 +91,7 @@ Detailed findings are in [research.md](./research.md). Key decisions: - Multi-platform SBOM processing runs once per platform image and targets that platform manifest digest. - Multi-platform VEX processing runs once for the image set and targets the top-level image index digest. - Do not use the index digest as a platform SBOM subject or duplicate image-level VEX onto platform manifests. -- Keep existing signing behavior and include signer identity in cache identity. The signing and cache logic must live in the corresponding artifact stage, not in a retained step wrapper. +- Keep existing signing behavior and include signer identity in cache identity. The signing, checksum, and cache logic must live in the corresponding artifact stage, not in a retained step wrapper. Registry publication must use the dedicated `MutateArtifact` convention, intentionally separate from `MutateImage`, because SBOM/VEX do not mutate the image itself. ### Publication and propagation @@ -166,7 +168,7 @@ Completed in [data-model.md](./data-model.md) and [quickstart.md](./quickstart.m The subsequent `/speckit-tasks` workflow should decompose at least these work items: -1. Implement `SbomStage` and `VexStage`, including stage identity, lifecycle integration, final image-digest association, OCI-artifact handling, cache checks, signing, publication, and `StagesStorage` access. +1. Add the dedicated `MutateArtifact` stage API and implement `SbomStage` and `VexStage`, including `GetDependencies`/`GetContentDependencies` checksum conventions, lifecycle integration, final image-digest association, OCI-artifact handling, cache checks, signing, publication through `MutateArtifact`, and `StorageManager` routing. 2. Migrate all SBOM behavior from `sbomStep` into `SbomStage`, then delete the step implementation and update callers/tests. 3. Migrate all VEX behavior from `vexStep` into `VexStage`, then delete the step implementation and update callers/tests. 4. Extend `StorageManager` with the minimal OCI-artifact operations, add any required `StagesStorage` backend primitives, and implement manager-routed artifact propagation for final, cache, and secondary-to-primary copies. diff --git a/specs/020-sbom-vex-build-stages/quickstart.md b/specs/020-sbom-vex-build-stages/quickstart.md index 0fe916f1eb..ef3277343b 100644 --- a/specs/020-sbom-vex-build-stages/quickstart.md +++ b/specs/020-sbom-vex-build-stages/quickstart.md @@ -19,7 +19,8 @@ task test:unit paths="./pkg/vex/..." Expected results: - `SbomStage` and `VexStage` calculate stable dependencies and remain non-buildable/mutable; the old `sbomStep` and `vexStep` implementations no longer exist. -- Both stages are associated with the final image digest, operate only on separate OCI artifacts, and use `StorageManager` for registry interaction. The manager routes requests to primary, secondary, cache, or final `storage.StagesStorage` as appropriate. +- Both stages preserve the `SignStage` checksum convention through `GetDependencies`/`GetContentDependencies` and `util.Sha256Hash(args...)`. +- Both stages are associated with the final image digest, operate only on separate OCI artifacts, and perform registry interaction from dedicated `MutateArtifact` through `StorageManager`; they do not use `MutateImage`. The manager routes requests to primary, secondary, cache, or final `storage.StagesStorage` as appropriate. - Single-platform SBOM/VEX subjects resolve to the final image manifest digest. - Multi-platform SBOM subjects resolve to each final platform manifest digest and VEX resolves to the final top-level index digest. - Propagation skips identical repositories, deduplicates existing identities, and distinguishes final errors from cache warnings. @@ -44,7 +45,7 @@ Cover these repository combinations: 7. Two-platform image. 8. Unavailable final repository, unavailable cache repository, and local-only artifact-enabled build. -For each successful case, retrieve artifact descriptors by the actual final image digest from every repository containing the image. Verify that repeated builds do not add duplicate fallback-index entries and that stage code did not create or modify an image layer. Registry access used by the stages must be observable through `StorageManager` routing and the selected `StagesStorage` test double/backend rather than a direct registry client. +For each successful case, retrieve artifact descriptors by the actual final image digest from every repository containing the image. Verify that repeated builds do not add duplicate fallback-index entries and that stage code did not create or modify an image layer. Registry access used by the stages must be observable through `MutateArtifact`, `StorageManager` routing, and the selected `StagesStorage` test double/backend rather than a direct registry client. Then run the repository integration suite: diff --git a/specs/020-sbom-vex-build-stages/research.md b/specs/020-sbom-vex-build-stages/research.md index d365f9e756..c9419d7899 100644 --- a/specs/020-sbom-vex-build-stages/research.md +++ b/specs/020-sbom-vex-build-stages/research.md @@ -2,9 +2,9 @@ ## Decision: Move artifact convergence into the image/stage lifecycle -SBOM and VEX will be represented by non-buildable, mutable build stages that run after the image content stage has produced a registry-backed descriptor. The existing `Stage` lifecycle remains the integration point: stage dependencies determine cache identity, `MutateImage` performs OCI-side publication, and the build phase invokes the stage for each applicable image/platform. +SBOM and VEX will be represented by non-buildable, mutable build stages that run after the image content stage has produced a registry-backed descriptor. The existing `Stage` lifecycle remains the integration point: `GetDependencies`/`GetContentDependencies` determine cache identity using the same `util.Sha256Hash(args...)` convention as `SignStage`, a dedicated `MutateArtifact` method performs OCI-side publication, and the build phase invokes the stage for each applicable image/platform. -The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/...`, `pkg/oci/artifact`, the signer implementations, and fallback-tag storage. The existing `sbomStep` and `vexStep` types are transitional implementations: their behavior will be moved into `SbomStage` and `VexStage`, and the step types/files will be deleted. The stages are intentionally different from ordinary image stages: each stage is associated with the final image digest, operates on a separate OCI artifact, and never changes image layers or filesystem content. All registry interaction must go through `StorageManager`, which owns and routes to the primary, secondary, cache, and final `storage.StagesStorage` abstractions, just as ordinary build stages use the storage manager path. `BuildPhase.AfterImages` will retain image publication/report work but will no longer perform SBOM/VEX generation. +The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/...`, `pkg/oci/artifact`, the signer implementations, and fallback-tag storage. The existing `sbomStep` and `vexStep` types are transitional implementations: their behavior will be moved into `SbomStage` and `VexStage`, and the step types/files will be deleted. The stages are intentionally different from ordinary image stages: each stage is associated with the final image digest, operates on a separate OCI artifact, and never changes image layers or filesystem content. They require a dedicated registry-working stage hook: `MutateArtifact` performs artifact registry work without invoking `MutateImage`. `MutateImage` remains reserved for image-manifest/image-content mutations. All registry interaction from `MutateArtifact` must go through `StorageManager`, which owns and routes to the primary, secondary, cache, and final `storage.StagesStorage` abstractions, just as ordinary registry-working build stages use the storage manager path. `BuildPhase.AfterImages` will retain image publication/report work but will no longer perform SBOM/VEX generation. ### Rationale @@ -25,7 +25,7 @@ The new stages will reuse low-level primitives from `pkg/sbom/...`, `pkg/vex/... ### Rationale -This preserves the distinction between an image lifecycle and its supply-chain metadata while still making metadata generation deterministic and cacheable as part of the lifecycle. Reusing `StorageManager` and its `StagesStorage` backends keeps registry behavior consistent with existing build stages, centralizes repository routing, and avoids coupling stages to a concrete registry implementation. +This preserves the distinction between an image lifecycle and its supply-chain metadata while still making metadata generation deterministic and cacheable as part of the lifecycle. Keeping the standard dependency conventions while adding a distinct `MutateArtifact` hook makes the new stages compatible with the existing scheduler and stage cache without pretending that an OCI artifact is an image mutation. Reusing `StorageManager` and its `StagesStorage` backends keeps registry behavior consistent with existing build stages, centralizes repository routing, and avoids coupling stages to a concrete registry implementation. ### Alternatives considered diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index d86f63d3ba..0853ef76b0 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -18,18 +18,20 @@ ## Phase 2: Foundational (Blocking Prerequisites) -**Purpose**: Define the minimal internal contracts shared by both artifact stages and enforce the storage abstraction boundary. +**Purpose**: Define the minimal internal contracts shared by both artifact stages, including the dedicated artifact lifecycle hook and storage abstraction boundary. -**Checkpoint**: Stage metadata, final-image subject flow, storage artifact operations, propagation shape, and early registry validation are understood before story implementation begins. +**Checkpoint**: Stage metadata, final-image subject flow, `MutateArtifact` lifecycle hook, checksum contract, storage artifact operations, propagation shape, and early registry validation are understood before story implementation begins. - [X] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including final image descriptor, artifact kind, target platform, mutable flag, and non-buildable flag - [X] T005 Define the kind-neutral artifact propagation operation and explicit source/destination final-image descriptor flow in `pkg/build/` using existing `pkg/oci/artifact/` primitives, without introducing a public API or new dependency - [X] T006 Locate the earliest common build initialization path and specify the registry-backed-storage validation seam in `pkg/build/build_phase.go` for both SBOM-enabled and VEX-enabled builds - [ ] T007 Extend `StorageManager` with minimal artifact listing, publication, destination-descriptor resolution, and copy operations in `pkg/storage/manager/`, routing each operation to the correct primary, secondary, cache, or final `storage.StagesStorage` instance -- [X] T008 [P] Add shared test fixtures or helper functions needed to construct final manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` -- [X] T009 [P] Define the minimal OCI-artifact primitives on `storage.StagesStorage` in `pkg/storage/stages_storage.go` for listing attached artifacts, publishing an artifact for a final image digest, and copying attached artifacts between destination descriptors; keep repository selection outside the backend -- [X] T010 [P] Implement the new `StagesStorage` artifact primitives for every registry-backed storage implementation under `pkg/storage/` and keep local-storage behavior explicit and unsupported for artifact publication -- [ ] T011 [P] Add Ginkgo/Gomega contract tests for `StagesStorage` artifact primitives and `StorageManager` repository routing, verifying stage code imports neither concrete registry clients nor repository-selection logic in `pkg/storage/stages_storage_test.go`, `pkg/storage/manager/`, and `pkg/build/stage/` +- [ ] T008 Add the dedicated `MutateArtifact` lifecycle hook and scheduler/conveyor dispatch contract, keeping `MutateImage` reserved for image-manifest mutations, in `pkg/build/stage/`, `pkg/build/conveyor.go`, and `pkg/build/` +- [ ] T009 [P] Define the artifact-stage checksum contract using `GetDependencies`, `GetContentDependencies`, and `util.Sha256Hash(args...)` in `pkg/build/stage/`, following `SignStage` +- [X] T010 [P] Add shared test fixtures or helper functions needed to construct final manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` +- [X] T011 [P] Define the minimal OCI-artifact primitives on `storage.StagesStorage` in `pkg/storage/stages_storage.go` for listing attached artifacts, publishing an artifact for a final image digest, and copying attached artifacts between destination descriptors; keep repository selection outside the backend +- [X] T012 [P] Implement the new `StagesStorage` artifact primitives for every registry-backed storage implementation under `pkg/storage/` and keep local-storage behavior explicit and unsupported for artifact publication +- [ ] T013 [P] Add Ginkgo/Gomega contract tests for `StagesStorage` artifact primitives, `StorageManager` repository routing, and `MutateArtifact` dispatch, verifying stage code imports neither concrete registry clients nor repository-selection logic in `pkg/storage/stages_storage_test.go`, `pkg/storage/manager/`, `pkg/build/stage/`, and `pkg/build/` --- @@ -41,26 +43,26 @@ ### Tests for User Story 1 -- [X] T012 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, final-image descriptor association, no filesystem mutation, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` -- [X] T013 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` -- [X] T014 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` -- [ ] T015 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios -- [ ] T016 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` -- [ ] T017 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `StorageManager`, with the manager selecting the appropriate `storage.StagesStorage`, in `pkg/build/stage/` and `pkg/storage/manager/` +- [X] T014 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, final-image descriptor association, no filesystem mutation, dedicated `MutateArtifact` dispatch, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` +- [X] T015 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` +- [X] T016 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` +- [ ] T017 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios +- [X] T018 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` +- [ ] T019 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `StorageManager`, with the manager selecting the appropriate `storage.StagesStorage`, in `pkg/build/stage/` and `pkg/storage/manager/` ### Implementation for User Story 1 -- [X] T018 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including final-image-digest association and SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` -- [X] T019 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including final-image-digest association and VEX generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` -- [ ] T020 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` -- [ ] T021 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` -- [ ] T022 [US1] Ensure `PrepareImage` is a no-op and `MutateImage` operates only on the associated OCI artifact through `StorageManager`, without fetching, rebuilding, storing, or mutating image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go` -- [ ] T023 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` -- [ ] T024 [US1] Execute artifact publication through stage `MutateImage` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` -- [ ] T025 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` -- [X] T026 [US1] Implement shared idempotent artifact propagation through `StorageManager`, with manager-routed source/destination backends, destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` -- [X] T027 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the `StorageManager`-routed propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` -- [X] T028 [US1] Connect secondary-stage restoration into primary storage to the same `StorageManager`-routed propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` +- [X] T020 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including final-image-digest association and SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` +- [X] T021 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including final-image-digest association and VEX generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` +- [ ] T022 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` +- [ ] T023 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` +- [ ] T024 [US1] Ensure `PrepareImage` is a no-op, `MutateArtifact` operates only on the associated OCI artifact through `StorageManager`, and the artifact stages do not implement or invoke `MutateImage`; do not fetch, rebuild, store, or mutate image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go` +- [ ] T025 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` +- [ ] T026 [US1] Execute artifact publication through stage `MutateArtifact` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` +- [X] T027 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` +- [X] T028 [US1] Implement shared idempotent artifact propagation through `StorageManager`, with manager-routed source/destination backends, destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [X] T029 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the `StorageManager`-routed propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` +- [X] T030 [US1] Connect secondary-stage restoration into primary storage to the same `StorageManager`-routed propagation operation, including explicit handling when a source artifact is absent, in `pkg/storage/manager/` and `pkg/build/` **Checkpoint**: User Story 1 is independently functional; `SbomStage` and `VexStage` are the sole lifecycle owners, operate on final-image-associated OCI artifacts, and artifacts follow every applicable published image. @@ -74,19 +76,19 @@ ### Tests for User Story 2 -- [X] T029 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform final-image subject selection in `pkg/build/artifact_subject_test.go` -- [X] T030 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and final parent digest are distinct per platform in `pkg/build/stage/sbom_test.go` -- [X] T031 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into stage-owned tests and ensure the final suite contains no step-specific test dependency -- [ ] T032 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each final platform manifest -- [X] T033 [US2] Extend `test/e2e/vex/` with single-platform final-manifest placement and multi-platform final-index-only placement assertions -- [ ] T034 [US2] Add storage-backed tests for destination platform/index descriptor resolution when the copied image digest differs from the source in `pkg/build/artifact_propagation_test.go` +- [X] T031 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform final-image subject selection in `pkg/build/artifact_subject_test.go` +- [X] T032 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and final parent digest are distinct per platform in `pkg/build/stage/sbom_test.go` +- [X] T033 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into stage-owned tests and ensure the final suite contains no step-specific test dependency +- [ ] T034 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each final platform manifest +- [X] T035 [US2] Extend `test/e2e/vex/` with single-platform final-manifest placement and multi-platform final-index-only placement assertions +- [ ] T036 [US2] Add storage-backed tests for destination platform/index descriptor resolution when the copied image digest differs from the source in `pkg/build/artifact_propagation_test.go` ### Implementation for User Story 2 -- [X] T035 [US2] Implement explicit final-image artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` -- [ ] T036 [US2] Pass the final target platform and resolved final platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` -- [ ] T037 [US2] Make `VexStage` registration run once per multi-platform image set with the final top-level index subject, and use the final image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` -- [X] T038 [US2] Ensure `StorageManager`-routed propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go`, `pkg/storage/manager/`, and `pkg/storage/` +- [X] T037 [US2] Implement explicit final-image artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` +- [ ] T038 [US2] Pass the final target platform and resolved final platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` +- [ ] T039 [US2] Make `VexStage` registration run once per multi-platform image set with the final top-level index subject, and use the final image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` +- [X] T040 [US2] Ensure `StorageManager`-routed propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go`, `pkg/storage/manager/`, and `pkg/storage/` **Checkpoint**: User Story 2 is independently testable and no artifact can silently use an index subject for a platform SBOM or duplicate multi-platform VEX onto platform manifests. @@ -100,18 +102,18 @@ ### Tests for User Story 3 -- [X] T039 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across final image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` -- [X] T040 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across final parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` -- [X] T041 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing through `StorageManager` in `pkg/build/artifact_propagation_test.go` -- [ ] T042 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios -- [X] T043 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests +- [X] T041 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across final image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` +- [X] T042 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across final parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` +- [X] T043 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing through `StorageManager` in `pkg/build/artifact_propagation_test.go` +- [ ] T044 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios +- [X] T045 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests ### Implementation for User Story 3 -- [X] T044 [US3] Include all effective SBOM inputs and the final parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` -- [X] T045 [US3] Include VEX document content, final parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` -- [ ] T046 [US3] Select reusable artifact-bearing stages from primary and secondary storage through `StorageManager` using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` -- [X] T047 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/`, `pkg/storage/`, and `pkg/build/artifact_propagation.go` +- [X] T046 [US3] Include all effective SBOM inputs and the final parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` +- [X] T047 [US3] Include VEX document content, final parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` +- [ ] T048 [US3] Select reusable artifact-bearing stages from primary and secondary storage through `StorageManager` using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` +- [X] T049 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/`, `pkg/storage/`, and `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 3 is independently testable; unchanged inputs reuse artifacts and every effective changed input invalidates only the affected artifact identity. @@ -125,18 +127,18 @@ ### Tests for User Story 4 -- [X] T048 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` -- [X] T049 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` -- [X] T050 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent `StorageManager`-routed attachment in `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/storage/` -- [ ] T051 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior -- [X] T052 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories +- [X] T050 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` +- [X] T051 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` +- [X] T052 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent `StorageManager`-routed attachment in `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/storage/` +- [ ] T053 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior +- [X] T054 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories ### Implementation for User Story 4 -- [X] T053 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` -- [X] T054 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared `StorageManager`-routed error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` -- [X] T055 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` -- [X] T056 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` +- [X] T055 [US4] Add earliest-phase registry-backed-storage validation for enabled SBOM/VEX with an actionable `--repo` or disable-artifacts message in `pkg/build/build_phase.go` +- [X] T056 [US4] Enforce fatal final-repository publication/propagation errors and best-effort cache-repository warnings through one shared `StorageManager`-routed error-policy path in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` +- [X] T057 [US4] Ensure missing secondary source artifacts return an incomplete/error result rather than claiming artifact-complete restoration in `pkg/build/` and `pkg/storage/manager/` +- [X] T058 [US4] Verify artifact propagation does not bypass existing cleanup and purge behavior, updating only the necessary repository traversal in `pkg/cleaning/` **Checkpoint**: User Story 4 is independently testable; registry failures and local-only configuration produce predictable results without changing repository flag semantics. @@ -146,18 +148,18 @@ **Purpose**: Validate the complete implementation against the revised stage-ownership and storage-abstraction boundaries. -- [X] T057 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, direct registry-client access from stages, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic -- [ ] T058 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners -- [ ] T059 [P] Verify all stage registry interaction goes through `StorageManager`, the manager routes to all supported registry-backed `storage.StagesStorage` implementations, and local storage rejects artifact publication explicitly in `pkg/storage/` and `pkg/storage/manager/` -- [ ] T060 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures -- [X] T061 Run formatting with `task format` for authored Go directories -- [X] T062 Run compilation with `task build` -- [X] T063 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` -- [ ] T064 Run the complete unit suite with `task test:unit` -- [X] T065 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` -- [X] T066 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` -- [ ] T067 Run legacy integration coverage with `task test:integration` -- [X] T068 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files +- [X] T059 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, direct registry-client access from stages, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic +- [ ] T060 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners +- [ ] T061 [P] Verify all stage registry interaction goes through `StorageManager`, the manager routes to all supported registry-backed `storage.StagesStorage` implementations, and local storage rejects artifact publication explicitly in `pkg/storage/` and `pkg/storage/manager/` +- [ ] T062 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures +- [X] T063 Run formatting with `task format` for authored Go directories +- [X] T064 Run compilation with `task build` +- [X] T065 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` +- [ ] T066 Run the complete unit suite with `task test:unit` +- [X] T067 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` +- [X] T068 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` +- [ ] T069 Run legacy integration coverage with `task test:integration` +- [X] T070 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files --- @@ -166,30 +168,30 @@ ### Phase Dependencies - **Phase 1 (Setup)**: No implementation dependency; establishes the current call-path and storage baseline. -- **Phase 2 (Foundational)**: Depends on Phase 1 and blocks story implementation. `StagesStorage` methods and backend implementations must be available before artifact stages can publish or propagate OCI artifacts. -- **Phase 3 (US1)**: Depends on Phase 2 and is the MVP increment. It includes full migration from `sbomStep`/`vexStep`, final-image-digest association, `StagesStorage` integration, and deletion of transitional files. +- **Phase 2 (Foundational)**: Depends on Phase 1 and blocks story implementation. `MutateArtifact` dispatch, checksum conventions, `StorageManager` routing, `StagesStorage` primitives, and backend implementations must be available before artifact stages can publish or propagate OCI artifacts. +- **Phase 3 (US1)**: Depends on Phase 2 and is the MVP increment. It includes full migration from `sbomStep`/`vexStep`, final-image-digest association, `MutateArtifact` integration, manager-routed storage, and deletion of transitional files. - **Phase 4 (US2)**: Depends on US1's artifact stages and propagation path because it specializes final subject selection; platform tests must be owned by the new stages before old step tests are deleted. - **Phase 5 (US3)**: Depends on US1 and US2 stage identity/subject contracts so cache identity includes the correct final parent descriptor. -- **Phase 6 (US4)**: Depends on the shared `StagesStorage`-backed propagation operation from US1; its validation work can proceed in parallel with US2/US3 after the shared path exists. +- **Phase 6 (US4)**: Depends on the shared `StorageManager`-routed propagation operation from US1; its validation work can proceed in parallel with US2/US3 after the shared path exists. - **Phase 7 (Polish)**: Depends on all desired stories being complete, including removal of transitional files and references and validation of every storage implementation. ### User Story Dependencies - **US1 (P1)**: Starts after Phase 2; no dependency on another user story. MVP. -- **US2 (P1)**: Depends on US1's `SbomStage`/`VexStage` lifecycle and `StagesStorage`-backed propagation implementation. +- **US2 (P1)**: Depends on US1's `SbomStage`/`VexStage` lifecycle and `StorageManager`-routed propagation implementation. - **US3 (P1)**: Depends on US1's stages and US2's explicit final-image subject rules. - **US4 (P2)**: Depends on US1's propagation/error path; early-validation tests can proceed independently of US2 and US3. ### Parallel Opportunities - Phase 1 tasks T002 and T003 can run in parallel after T001's baseline inventory. -- In Phase 2, T008–T010 can proceed in parallel once the required storage method shape is agreed; backend implementations must converge on the same interface. -- Within US1, T018 and T019 are parallel stage files; T012, T013, and T017 are separate test concerns. T020 and T021 are parallel migrations when their callers are disjoint. -- Within US2, T029/T030 and T032–T034 are parallel test work; subject-selection and VEX-placement implementation can proceed in separate files. -- Within US3, T039 and T040 are parallel stage-owned identity tests; T042 can proceed independently once stage contracts are stable. -- Within US4, T048–T050 are parallel test tasks, and T052 can proceed independently in cleanup files. -- After Phase 2, separate contributors can work on stage migration, storage backends, propagation, and validation tests, but deletion of transitional files (T025) must wait for all callers/tests to migrate. -- Polish review and regression checks (T057–T060) can run in parallel before the sequential repository-wide validation commands T061–T068. +- In Phase 2, T009–T013 can proceed in parallel once the required lifecycle and storage method shapes are agreed; backend implementations must converge on the same interface, while `MutateArtifact` dispatch remains a prerequisite for the stages. +- Within US1, T020 and T021 are parallel stage files; T012, T013, and T019 are separate test concerns. T022 and T023 are parallel migrations when their callers are disjoint. +- Within US2, T031/T032 and T034–T036 are parallel test work; subject-selection and VEX-placement implementation can proceed in separate files. +- Within US3, T041 and T042 are parallel stage-owned identity tests; T044 can proceed independently once stage contracts are stable. +- Within US4, T050–T052 are parallel test tasks, and T054 can proceed independently in cleanup files. +- After Phase 2, separate contributors can work on stage migration, storage backends, propagation, and validation tests, but deletion of transitional files (T027) must wait for all callers/tests to migrate. +- Polish review and regression checks (T059–T062) can run in parallel before the sequential repository-wide validation commands T063–T070. --- @@ -197,22 +199,22 @@ ```text # After Phase 2, start independent stage, migration, storage, and test work: -Task: T011 — stage lifecycle tests in pkg/build/stage/artifact_test.go -Task: T012 — propagation tests in pkg/build/artifact_propagation_test.go +Task: T012 — stage lifecycle tests in pkg/build/stage/artifact_test.go +Task: T013 — propagation tests in pkg/build/artifact_propagation_test.go Task: T016 — migration coverage in pkg/build/ -Task: T017 — StagesStorage usage tests in pkg/build/stage/ -Task: T018 — SbomStage in pkg/build/stage/sbom.go -Task: T019 — VexStage in pkg/build/stage/vex.go -Task: T014 — repository propagation scenarios in test/e2e/sbom/ +Task: T017 — StorageManager routing tests in pkg/build/stage/ and pkg/storage/manager/ +Task: T020 — SbomStage in pkg/build/stage/sbom.go +Task: T021 — VexStage in pkg/build/stage/vex.go +Task: T015 — repository propagation scenarios in test/e2e/sbom/ # Integrate after the stage and storage contracts are stable: -Task: T020 — migrate SBOM behavior and callers -Task: T021 — migrate VEX behavior and callers -Task: T022 — enforce final-image OCI-artifact-only mutation -Task: T023 — register stages in pkg/build/build_phase.go -Task: T024 — remove duplicate AfterImages convergence -Task: T025 — delete transitional step files and tests -Task: T026 — implement shared StagesStorage-backed propagation +Task: T022 — migrate SBOM behavior and callers +Task: T023 — migrate VEX behavior and callers +Task: T024 — enforce final-image OCI-artifact-only behavior +Task: T025 — register stages in pkg/build/build_phase.go +Task: T026 — remove duplicate AfterImages convergence +Task: T027 — delete transitional step files and tests +Task: T028 — implement shared StorageManager-routed propagation ``` ## Implementation Strategy @@ -220,16 +222,16 @@ Task: T026 — implement shared StagesStorage-backed propagation ### MVP First (User Story 1 Only) 1. Complete Phase 1 baseline inspection. -2. Complete Phase 2 `StagesStorage`, final-image subject, propagation, and validation contracts. +2. Complete Phase 2 `MutateArtifact`, checksum, `StorageManager`, final-image subject, propagation, and validation contracts. 3. Implement or verify `SbomStage` and `VexStage` as final-image-associated OCI-artifact stages. -4. Migrate all behavior and callers from `sbomStep`/`vexStep` through `StagesStorage`. +4. Migrate all behavior and callers from `sbomStep`/`vexStep` through `MutateArtifact` and `StorageManager`. 5. Register the stages, connect primary/final/cache/secondary propagation, and delete transitional files. 6. Run US1 unit and SBOM e2e tests independently. 7. Stop for validation/demo before adding platform-specific and cache-invalidation refinements. ### Incremental Delivery -1. Deliver US1 as the first usable increment: lifecycle-owned artifact stages with complete `StagesStorage`-backed repository propagation. +1. Deliver US1 as the first usable increment: lifecycle-owned artifact stages with complete `StorageManager`-routed repository propagation. 2. Add US2: correct final manifest/index subjects and platform placement, with stage-owned tests. 3. Add US3: complete stage dependency identity and cache reuse/invalidation. 4. Add US4: early validation and explicit final/cache failure behavior. @@ -237,18 +239,18 @@ Task: T026 — implement shared StagesStorage-backed propagation ### Traceability -- **FR-001–FR-003**: T018–T028, T035–T038 -- **FR-004–FR-006**: T029–T038 -- **FR-007–FR-010**: T008–T010, T012–T028, T049, T053–T055 -- **FR-011–FR-013**: T039–T047 -- **FR-014–FR-016**: T003, T047, T052, T056 -- **FR-017–FR-018**: T058–T060 and all implementation tasks; no CLI flag changes or OCI Referrers migration -- **Stage ownership and storage boundary**: T008–T010, T016–T025, T057–T059 +- **FR-001–FR-003**: T020–T030, T037–T040 +- **FR-004–FR-006**: T031–T040 +- **FR-007–FR-010**: T007, T011–T013, T015–T030, T050, T054–T057 +- **FR-011–FR-013**: T041–T049 +- **FR-014–FR-016**: T003, T049, T054, T058 +- **FR-017–FR-018**: T060–T062 and all implementation tasks; no CLI flag changes or OCI Referrers migration +- **Stage ownership and storage boundary**: T008–T013, T016–T027, T059–T061 ## Notes - Completed tasks retain `[X]`; pending tasks use `[ ]`. Every task has a sequential ID and story-phase tasks include exactly one `[US#]` label. -- No external API contracts were provided in `contracts/`; the artifact methods are an internal `StagesStorage` contract. -- The revised plan requires `SbomStage` and `VexStage` to be the sole lifecycle owners, associated with final image descriptors, and restricted to OCI-artifact operations through `storage.StagesStorage`. +- No external API contracts were provided in `contracts/`; the artifact methods are internal `StorageManager` and `StagesStorage` contracts. +- The revised plan requires `SbomStage` and `VexStage` to be the sole lifecycle owners, associated with final image descriptors, and restricted to OCI-artifact operations through `MutateArtifact` and `StorageManager`; `MutateImage` remains reserved for image mutations. - `sbom_step.go`, `vex_step.go`, their step-specific callers, and their tests must not remain after migration. - No new dependencies, CLI flags, image layers, or OCI Referrers migration are planned. From ab46056cabf7e427fdffa64fbe44c48d8287ac26 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 15:56:57 +0300 Subject: [PATCH 09/17] refactor(build): add dedicated artifact mutation hook Route SBOM and VEX stages through the artifact lifecycle instead of treating them as image mutations. Signed-off-by: Alexandr Zaytsev --- pkg/build/build_phase.go | 4 ++-- pkg/build/stage/artifact_test.go | 6 ++++-- pkg/build/stage/base.go | 4 ++++ pkg/build/stage/interface.go | 1 + pkg/build/stage/sbom.go | 6 +++++- pkg/build/stage/sbom_test.go | 3 +++ pkg/build/stage/vex.go | 6 +++++- pkg/build/stage/vex_test.go | 3 +++ pkg/storage/repo_stages_storage_test.go | 5 +++++ specs/020-sbom-vex-build-stages/tasks.md | 8 ++++---- 10 files changed, 36 insertions(+), 10 deletions(-) diff --git a/pkg/build/build_phase.go b/pkg/build/build_phase.go index 538ce34224..702e4b8558 100644 --- a/pkg/build/build_phase.go +++ b/pkg/build/build_phase.go @@ -1498,8 +1498,8 @@ func (phase *BuildPhase) atomicBuildStageImage(ctx context.Context, img *image.I if prevBuiltImage == nil || prevBuiltImage.Image == nil { return fmt.Errorf("expected previous built image for artifact stage %s", stg.Name()) } - if err := stg.MutateImage(ctx, phase.Conveyor.StorageManager.GetStagesStorage(), prevBuiltImage, stageImage); err != nil { - return fmt.Errorf("unable to mutate %s: %w", stg.Name(), err) + if err := stg.MutateArtifact(ctx, prevBuiltImage, stageImage); err != nil { + return fmt.Errorf("unable to mutate artifact %s: %w", stg.Name(), err) } stageImage.Image.SetStageDesc(prevBuiltImage.Image.GetStageDesc()) return nil diff --git a/pkg/build/stage/artifact_test.go b/pkg/build/stage/artifact_test.go index bc75d546be..72cf0f02e0 100644 --- a/pkg/build/stage/artifact_test.go +++ b/pkg/build/stage/artifact_test.go @@ -54,9 +54,10 @@ var _ = Describe("artifact stages", func() { parent := NewStageImage(NewContainerBackendStub(), "", parentImage) stageImage := NewStageImage(NewContainerBackendStub(), "", mock.NewMockLegacyImageInterface(ctrl)) - Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(Succeed()) + Expect(artifactStage.MutateArtifact(ctx, parent, stageImage)).To(Succeed()) Expect(publisherCalls).To(Equal(1)) Expect(artifactStage.GetArtifactMetadata().ParentDigest).To(Equal("sha256:parent")) + Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(MatchError("SBOM stage must be mutated as an artifact")) }) It("publishes VEX through its stage publisher without requiring image mutation", func(ctx SpecContext) { @@ -81,9 +82,10 @@ var _ = Describe("artifact stages", func() { parent := NewStageImage(NewContainerBackendStub(), "", parentImage) stageImage := NewStageImage(NewContainerBackendStub(), "", mock.NewMockLegacyImageInterface(ctrl)) - Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(Succeed()) + Expect(artifactStage.MutateArtifact(ctx, parent, stageImage)).To(Succeed()) Expect(publisherCalls).To(Equal(1)) Expect(artifactStage.GetArtifactMetadata().ParentDigest).To(Equal("sha256:parent")) + Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(MatchError("VEX stage must be mutated as an artifact")) }) DescribeTable("is mutable and non-buildable", diff --git a/pkg/build/stage/base.go b/pkg/build/stage/base.go index 1fd4d9b4bb..0a6ed9b481 100644 --- a/pkg/build/stage/base.go +++ b/pkg/build/stage/base.go @@ -362,6 +362,10 @@ func (s *BaseStage) MutateImage(_ context.Context, _ ImageMutatorPusher, _, _ *S panic("not implemented") } +func (s *BaseStage) MutateArtifact(_ context.Context, _, _ *StageImage) error { + panic("not implemented") +} + func (s *BaseStage) PreRun(_ context.Context, _ Conveyor) error { return nil } diff --git a/pkg/build/stage/interface.go b/pkg/build/stage/interface.go index 233f58b73a..7a8bfc7ae6 100644 --- a/pkg/build/stage/interface.go +++ b/pkg/build/stage/interface.go @@ -39,6 +39,7 @@ type Interface interface { GetGitMappings() []*GitMapping MutateImage(ctx context.Context, registry ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error + MutateArtifact(ctx context.Context, prevBuiltImage, stageImage *StageImage) error SelectSuitableStageDesc(context.Context, Conveyor, image.StageDescSet) (*image.StageDesc, error) diff --git a/pkg/build/stage/sbom.go b/pkg/build/stage/sbom.go index 21f1b3d800..333460953a 100644 --- a/pkg/build/stage/sbom.go +++ b/pkg/build/stage/sbom.go @@ -96,7 +96,7 @@ func (s *SbomStage) GetContentDependencies(ctx context.Context, c Conveyor, buil return s.GetDependencies(ctx, c, nil, nil, nil, buildContextArchive) } -func (s *SbomStage) MutateImage(ctx context.Context, _ ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { +func (s *SbomStage) MutateArtifact(ctx context.Context, prevBuiltImage, stageImage *StageImage) error { if s.publisher == nil { return fmt.Errorf("SBOM stage publisher is unavailable") } @@ -124,4 +124,8 @@ func (s *SbomStage) MutateImage(ctx context.Context, _ ImageMutatorPusher, prevB return s.publisher(ctx, parentDesc, s.ImageName(), s.TargetPlatform()) } +func (s *SbomStage) MutateImage(_ context.Context, _ ImageMutatorPusher, _, _ *StageImage) error { + return fmt.Errorf("SBOM stage must be mutated as an artifact") +} + const sbomArtifactFormatVersion = "2" diff --git a/pkg/build/stage/sbom_test.go b/pkg/build/stage/sbom_test.go index 27a23f1f29..90582c8eaf 100644 --- a/pkg/build/stage/sbom_test.go +++ b/pkg/build/stage/sbom_test.go @@ -26,6 +26,9 @@ var _ = Describe("SbomStage dependencies", func() { Expect(err).To(Succeed()) Expect(amd).NotTo(Equal(arm)) + content, err := newStage("linux/amd64").GetContentDependencies(ctx, nil, nil) + Expect(err).To(Succeed()) + Expect(content).To(Equal(amd)) }) It("changes when the effective SBOM inputs change", func(ctx SpecContext) { diff --git a/pkg/build/stage/vex.go b/pkg/build/stage/vex.go index cc67df27c7..52aa06188e 100644 --- a/pkg/build/stage/vex.go +++ b/pkg/build/stage/vex.go @@ -96,7 +96,7 @@ func (s *VexStage) GetContentDependencies(ctx context.Context, c Conveyor, build return s.GetDependencies(ctx, c, nil, nil, nil, buildContextArchive) } -func (s *VexStage) MutateImage(ctx context.Context, _ ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error { +func (s *VexStage) MutateArtifact(ctx context.Context, prevBuiltImage, stageImage *StageImage) error { if prevBuiltImage == nil || prevBuiltImage.Image == nil { return fmt.Errorf("VEX stage parent image is unavailable") } @@ -127,6 +127,10 @@ func (s *VexStage) MutateImage(ctx context.Context, _ ImageMutatorPusher, prevBu return s.publisher(ctx, parentDesc, s.ImageName(), s.TargetPlatform(), s.vexJSON, s.signer, s.signerIdentity) } +func (s *VexStage) MutateImage(_ context.Context, _ ImageMutatorPusher, _, _ *StageImage) error { + return fmt.Errorf("VEX stage must be mutated as an artifact") +} + const vexStageArtifactFormatVersion = "2" // CalculateVexStageChecksum returns the cache identity for a VEX artifact. diff --git a/pkg/build/stage/vex_test.go b/pkg/build/stage/vex_test.go index 71ff64c7a2..514508687a 100644 --- a/pkg/build/stage/vex_test.go +++ b/pkg/build/stage/vex_test.go @@ -15,5 +15,8 @@ var _ = Describe("VexStage dependencies", func() { Expect(err).To(Succeed()) Expect(first).NotTo(Equal(second)) + content, err := GenerateVexStage([]byte(`{"statements":[]}`), &BaseStageOptions{TargetPlatform: ""}, signing.VexSigningOptions{}).GetContentDependencies(ctx, nil, nil) + Expect(err).To(Succeed()) + Expect(content).To(Equal(first)) }) }) diff --git a/pkg/storage/repo_stages_storage_test.go b/pkg/storage/repo_stages_storage_test.go index 99ddd85ab3..c18880a4de 100644 --- a/pkg/storage/repo_stages_storage_test.go +++ b/pkg/storage/repo_stages_storage_test.go @@ -58,6 +58,11 @@ func (r *pushImageRegistryStub) MutateAndPushImage(ctx context.Context, _, desti return nil } +var ( + _ StagesStorage = (*RepoStagesStorage)(nil) + _ StagesStorage = (*LocalStagesStorage)(nil) +) + var _ = Describe("RepoStagesStorage", func() { It("pushes a manifest-only image to the registry in PostManifest", func(ctx SpecContext) { registry := &pushImageRegistryStub{} diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 0853ef76b0..6ae0634307 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -25,13 +25,13 @@ - [X] T004 Define internal artifact-stage metadata and stage-name constants in `pkg/build/stage/base.go`, including final image descriptor, artifact kind, target platform, mutable flag, and non-buildable flag - [X] T005 Define the kind-neutral artifact propagation operation and explicit source/destination final-image descriptor flow in `pkg/build/` using existing `pkg/oci/artifact/` primitives, without introducing a public API or new dependency - [X] T006 Locate the earliest common build initialization path and specify the registry-backed-storage validation seam in `pkg/build/build_phase.go` for both SBOM-enabled and VEX-enabled builds -- [ ] T007 Extend `StorageManager` with minimal artifact listing, publication, destination-descriptor resolution, and copy operations in `pkg/storage/manager/`, routing each operation to the correct primary, secondary, cache, or final `storage.StagesStorage` instance -- [ ] T008 Add the dedicated `MutateArtifact` lifecycle hook and scheduler/conveyor dispatch contract, keeping `MutateImage` reserved for image-manifest mutations, in `pkg/build/stage/`, `pkg/build/conveyor.go`, and `pkg/build/` -- [ ] T009 [P] Define the artifact-stage checksum contract using `GetDependencies`, `GetContentDependencies`, and `util.Sha256Hash(args...)` in `pkg/build/stage/`, following `SignStage` +- [X] T007 Extend `StorageManager` with minimal artifact listing, publication, destination-descriptor resolution, and copy operations in `pkg/storage/manager/`, routing each operation to the correct primary, secondary, cache, or final `storage.StagesStorage` instance +- [X] T008 Add the dedicated `MutateArtifact` lifecycle hook and scheduler/conveyor dispatch contract, keeping `MutateImage` reserved for image-manifest mutations, in `pkg/build/stage/`, `pkg/build/conveyor.go`, and `pkg/build/` +- [X] T009 [P] Define the artifact-stage checksum contract using `GetDependencies`, `GetContentDependencies`, and `util.Sha256Hash(args...)` in `pkg/build/stage/`, following `SignStage` - [X] T010 [P] Add shared test fixtures or helper functions needed to construct final manifest/index descriptors and fallback artifact indexes in co-located files under `pkg/build/` and `pkg/oci/artifact/` - [X] T011 [P] Define the minimal OCI-artifact primitives on `storage.StagesStorage` in `pkg/storage/stages_storage.go` for listing attached artifacts, publishing an artifact for a final image digest, and copying attached artifacts between destination descriptors; keep repository selection outside the backend - [X] T012 [P] Implement the new `StagesStorage` artifact primitives for every registry-backed storage implementation under `pkg/storage/` and keep local-storage behavior explicit and unsupported for artifact publication -- [ ] T013 [P] Add Ginkgo/Gomega contract tests for `StagesStorage` artifact primitives, `StorageManager` repository routing, and `MutateArtifact` dispatch, verifying stage code imports neither concrete registry clients nor repository-selection logic in `pkg/storage/stages_storage_test.go`, `pkg/storage/manager/`, `pkg/build/stage/`, and `pkg/build/` +- [X] T013 [P] Add Ginkgo/Gomega contract tests for `StagesStorage` artifact primitives, `StorageManager` repository routing, and `MutateArtifact` dispatch, verifying stage code imports neither concrete registry clients nor repository-selection logic in `pkg/storage/stages_storage_test.go`, `pkg/storage/manager/`, `pkg/build/stage/`, and `pkg/build/` --- From aae11fab0fabaa161c32e771935e4ab2c51796d7 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 17:47:58 +0300 Subject: [PATCH 10/17] refactor(build): route artifact operations through storage manager Centralize SBOM and VEX artifact reads, writes, and copies through StorageManager instead of accessing storage implementations directly. Signed-off-by: Alexandr Zaytsev --- pkg/build/artifact_propagation.go | 27 +++++-- pkg/build/artifact_propagation_test.go | 21 ++++++ pkg/build/artifact_stage_migration_test.go | 2 +- pkg/build/build_phase.go | 61 ++++++++-------- pkg/build/build_phase_test.go | 73 ++++--------------- pkg/build/sbom_processor.go | 23 +++--- pkg/build/stage/artifact_test.go | 37 ++++++++-- pkg/build/stage/base.go | 4 - pkg/build/stage/interface.go | 11 ++- pkg/build/stage/sbom.go | 9 +-- pkg/build/stage/vex.go | 14 ++-- pkg/build/vex_processor.go | 39 ++++------ pkg/storage/local_stages_storage.go | 9 +++ .../manager/artifact_operations_test.go | 64 +++++++++++++--- pkg/storage/manager/storage_manager.go | 43 +++++++++++ pkg/storage/meta_repo_marker_test.go | 2 + pkg/storage/repo_stages_storage.go | 10 +++ pkg/storage/stages_storage.go | 3 + specs/020-sbom-vex-build-stages/tasks.md | 4 +- test/mock/stages_storage.go | 31 ++++++++ 20 files changed, 316 insertions(+), 171 deletions(-) diff --git a/pkg/build/artifact_propagation.go b/pkg/build/artifact_propagation.go index a120a8d356..f436a7aba8 100644 --- a/pkg/build/artifact_propagation.go +++ b/pkg/build/artifact_propagation.go @@ -8,6 +8,7 @@ import ( "github.com/werf/werf/v2/pkg/image" "github.com/werf/werf/v2/pkg/oci/artifact" "github.com/werf/werf/v2/pkg/storage" + "github.com/werf/werf/v2/pkg/storage/manager" ) func ensureAttachedArtifacts(ctx context.Context, repository, digest string) error { @@ -32,6 +33,14 @@ func ensureAttachedArtifacts(ctx context.Context, repository, digest string) err } func propagateArtifacts(ctx context.Context, projectName, imageName string, source, destination *image.StageDesc, caches []storage.StagesStorage, sourceStorages ...storage.StagesStorage) error { + var sourceStorage storage.StagesStorage + if len(sourceStorages) > 0 { + sourceStorage = sourceStorages[0] + } + return propagateArtifactsWithManager(ctx, projectName, imageName, source, destination, caches, sourceStorage, nil, nil) +} + +func propagateArtifactsWithManager(ctx context.Context, projectName, imageName string, source, destination *image.StageDesc, caches []storage.StagesStorage, sourceStorage, destinationStorage storage.StagesStorage, storageManager manager.StorageManagerInterface) error { if source == nil || source.Info == nil { return fmt.Errorf("source image descriptor is unavailable") } @@ -39,20 +48,26 @@ func propagateArtifacts(ctx context.Context, projectName, imageName string, sour return nil } - var sourceStorage storage.StagesStorage - if len(sourceStorages) > 0 { - sourceStorage = sourceStorages[0] - } if sourceStorage == nil { sourceStorage = &storage.RepoStagesStorage{RepoAddress: source.Info.Repository} } + copyArtifacts := func(ctx context.Context, sourceStorage storage.StagesStorage, sourceDigest string, destinationStorage storage.StagesStorage, destinationDigest string) error { + if storageManager != nil { + return storageManager.CopyAttachedArtifacts(ctx, sourceStorage, sourceDigest, destinationStorage, destinationDigest) + } + return sourceStorage.CopyAttachedArtifacts(ctx, sourceStorage.Address(), sourceDigest, destinationStorage.Address(), destinationDigest) + } + if destination != nil && destination.Info != nil && destination.Info.Repository != "" && destination.Info.Repository != storage.LocalStorageAddress && destination.Info.Repository != source.Info.Repository { + if destinationStorage == nil || destinationStorage.Address() != destination.Info.Repository { + destinationStorage = &storage.RepoStagesStorage{RepoAddress: destination.Info.Repository} + } if err := logboek.Context(ctx).Default().LogProcess("image %s: copy artifacts into final repo %s", imageName, destination.Info.Repository).DoError(func() error { - return sourceStorage.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), destination.Info.Repository, destination.Info.GetDigest()) + return copyArtifacts(ctx, sourceStorage, source.Info.GetDigest(), destinationStorage, destination.Info.GetDigest()) }); err != nil { return fmt.Errorf("copy attached artifacts into final repo %s: %w", destination.Info.Repository, err) } @@ -76,7 +91,7 @@ func propagateArtifacts(ctx context.Context, projectName, imageName string, sour destinationDigest = cacheDesc.Info.GetDigest() } - if err := sourceStorage.CopyAttachedArtifacts(ctx, source.Info.Repository, source.Info.GetDigest(), cache.Address(), destinationDigest); err != nil { + if err := copyArtifacts(ctx, sourceStorage, source.Info.GetDigest(), cache, destinationDigest); err != nil { logboek.Context(ctx).Warn().LogF("Warning: unable to copy artifacts into cache stages storage %s: %s\n", cache.String(), err) } } diff --git a/pkg/build/artifact_propagation_test.go b/pkg/build/artifact_propagation_test.go index dda01071fb..e7d608d397 100644 --- a/pkg/build/artifact_propagation_test.go +++ b/pkg/build/artifact_propagation_test.go @@ -2,6 +2,7 @@ package build import ( "bytes" + "context" "net/http/httptest" "strings" @@ -20,9 +21,20 @@ import ( "github.com/werf/werf/v2/pkg/image" "github.com/werf/werf/v2/pkg/oci/artifact" "github.com/werf/werf/v2/pkg/storage" + "github.com/werf/werf/v2/pkg/storage/manager" "github.com/werf/werf/v2/test/mock" ) +type recordingArtifactStorageManager struct { + manager.StorageManagerInterface + copyCalls int +} + +func (m *recordingArtifactStorageManager) CopyAttachedArtifacts(context.Context, storage.StagesStorage, string, storage.StagesStorage, string) error { + m.copyCalls++ + return nil +} + var _ = Describe("artifact propagation", func() { It("rejects an incomplete artifact source descriptor", func(ctx SpecContext) { err := ensureAttachedArtifacts(ctx, "", "") @@ -36,6 +48,15 @@ var _ = Describe("artifact propagation", func() { Expect(err).To(MatchError("source image descriptor is unavailable")) }) + It("routes production propagation through StorageManager", func(ctx SpecContext) { + storageManager := &recordingArtifactStorageManager{} + source := &image.StageDesc{Info: &image.Info{Repository: "registry.example/source", RepoDigest: "registry.example/source@sha256:source"}} + destination := &image.StageDesc{Info: &image.Info{Repository: "registry.example/final", RepoDigest: "registry.example/final@sha256:destination"}} + + Expect(propagateArtifactsWithManager(ctx, "project", "app", source, destination, nil, nil, nil, storageManager)).To(Succeed()) + Expect(storageManager.copyCalls).To(Equal(1)) + }) + It("skips local-only artifact sources", func(ctx SpecContext) { err := propagateArtifacts(ctx, "project", "app", &image.StageDesc{ Info: &image.Info{Repository: ":local", RepoDigest: ":local@sha256:local"}, diff --git a/pkg/build/artifact_stage_migration_test.go b/pkg/build/artifact_stage_migration_test.go index ec4dd82c9e..39a88bbf89 100644 --- a/pkg/build/artifact_stage_migration_test.go +++ b/pkg/build/artifact_stage_migration_test.go @@ -37,6 +37,6 @@ var _ = Describe("artifact stage migration", func() { Expect(string(buildPhase)).To(ContainSubstring("GenerateSbomStage")) Expect(string(buildPhase)).To(ContainSubstring("NewVexStage")) Expect(string(buildPhase)).To(ContainSubstring("if len(images) == 1")) - Expect(string(buildPhase)).To(ContainSubstring("convergeMultiplatformVexByImageSets")) + Expect(string(buildPhase)).To(ContainSubstring("runMultiplatformVexStage")) }) }) diff --git a/pkg/build/build_phase.go b/pkg/build/build_phase.go index 702e4b8558..11f2048618 100644 --- a/pkg/build/build_phase.go +++ b/pkg/build/build_phase.go @@ -91,8 +91,8 @@ func NewBuildPhase(c *Conveyor, opts BuildPhaseOptions) *BuildPhase { return &BuildPhase{ BasePhase: BasePhase{c}, BuildPhaseOptions: opts, - sbomProcessor: newSbomProcessor(c.ContainerBackend, c.StorageManager.GetStagesStorage()), - vexProcessor: newVexProcessor(), + sbomProcessor: newSbomProcessor(c.ContainerBackend, c.StorageManager.GetStagesStorage(), c.StorageManager), + vexProcessor: newVexProcessor(c.StorageManager.GetStagesStorage(), c.StorageManager), ImagesReport: NewImagesReport(), } } @@ -324,7 +324,7 @@ func (phase *BuildPhase) propagateArtifactsByImages(ctx context.Context) error { if source == nil { continue } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, source, finalStageDescForPlatform(phase, name, images, img.TargetPlatform), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage()); err != nil { + if err := propagateArtifactsWithManager(ctx, phase.Conveyor.ProjectName(), name, source, finalStageDescForPlatform(phase, name, images, img.TargetPlatform), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage(), phase.Conveyor.StorageManager.GetFinalStagesStorage(), phase.Conveyor.StorageManager); err != nil { return fmt.Errorf("propagate artifacts for image %q: %w", name, err) } } @@ -334,7 +334,7 @@ func (phase *BuildPhase) propagateArtifactsByImages(ctx context.Context) error { if multiImage == nil || multiImage.GetStageDesc() == nil { continue } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, multiImage.GetStageDesc(), multiImage.GetFinalStageDesc(), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage()); err != nil { + if err := propagateArtifactsWithManager(ctx, phase.Conveyor.ProjectName(), name, multiImage.GetStageDesc(), multiImage.GetFinalStageDesc(), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage(), phase.Conveyor.StorageManager.GetFinalStagesStorage(), phase.Conveyor.StorageManager); err != nil { return fmt.Errorf("propagate multiplatform artifacts for image %q: %w", name, err) } } @@ -342,7 +342,7 @@ func (phase *BuildPhase) propagateArtifactsByImages(ctx context.Context) error { return nil } -func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name string, img *image.Image, sourceStageDesc, finalStageDesc *imagePkg.StageDesc, signer signature.Signer, signerIdentity string, breaker *externalref.ResolverBreaker, propagate bool) error { +func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name string, img *image.Image, sourceStageDesc *imagePkg.StageDesc, signer signature.Signer, signerIdentity string, breaker *externalref.ResolverBreaker) error { stageDesc := sourceStageDesc if stageDesc == nil { stageDesc = contentStageDesc(img) @@ -409,12 +409,6 @@ func (phase *BuildPhase) convergePlatformImageSbom(ctx context.Context, name str return fmt.Errorf("unable to converge sbom for image %q: %w", name, err) } - if propagate { - if err := phase.sbomProcessor.PropagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDesc, phase.Conveyor.StorageManager.GetCacheStagesStorageList()); err != nil { - return fmt.Errorf("unable to propagate sbom for image %q: %w", name, err) - } - } - return nil } @@ -837,7 +831,7 @@ func (phase *BuildPhase) registerSbomStage(img *image.Image) { signerIdentity = phase.SbomSigningOptions.Signer().Fingerprint() } publisher := func(ctx context.Context, parentDesc *imagePkg.StageDesc, _, targetPlatform string) error { - return phase.convergePlatformImageSbom(ctx, img.Name, img, parentDesc, nil, signer, signerIdentity, nil, false) + return phase.convergePlatformImageSbom(ctx, img.Name, img, parentDesc, signer, signerIdentity, nil) } artifactStage := stage.GenerateSbomStage(baseOptions, phase.SbomSigningOptions, dependency, publisher) img.SetStages(append(stages, artifactStage)) @@ -1185,7 +1179,7 @@ func (phase *BuildPhase) findAndFetchStageFromSecondaryStagesStorage(ctx context if err := ensureAttachedArtifacts(ctx, secondaryStageDesc.Info.Repository, secondaryStageDesc.Info.GetDigest()); err != nil { return fmt.Errorf("secondary stage %s has incomplete artifacts: %w", secondaryStageDesc.StageID.String(), err) } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), img.Name, secondaryStageDesc, stageDescCopy, storageManager.GetCacheStagesStorageList(), secondaryStagesStorage); err != nil { + if err := propagateArtifactsWithManager(ctx, phase.Conveyor.ProjectName(), img.Name, secondaryStageDesc, stageDescCopy, storageManager.GetCacheStagesStorageList(), secondaryStagesStorage, storageManager.GetStagesStorage(), storageManager); err != nil { return fmt.Errorf("unable to propagate artifacts restored from secondary storage: %w", err) } } @@ -1494,11 +1488,15 @@ func (phase *BuildPhase) atomicBuildStageImage(ctx context.Context, img *image.I if artifactStage, ok := stg.(interface { GetArtifactMetadata() *stage.ArtifactStageMetadata }); ok && artifactStage.GetArtifactMetadata() != nil { + artifactMutator, ok := stg.(stage.ArtifactStage) + if !ok { + return fmt.Errorf("artifact stage %s does not implement artifact mutation", stg.Name()) + } prevBuiltImage := phase.StagesIterator.GetPrevBuiltImage(img, stg) if prevBuiltImage == nil || prevBuiltImage.Image == nil { return fmt.Errorf("expected previous built image for artifact stage %s", stg.Name()) } - if err := stg.MutateArtifact(ctx, prevBuiltImage, stageImage); err != nil { + if err := artifactMutator.MutateArtifact(ctx, prevBuiltImage, stageImage); err != nil { return fmt.Errorf("unable to mutate artifact %s: %w", stg.Name(), err) } stageImage.Image.SetStageDesc(prevBuiltImage.Image.GetStageDesc()) @@ -1586,7 +1584,11 @@ func (phase *BuildPhase) atomicBuildStageImage(ctx context.Context, img *image.I return fmt.Errorf("expected previous built image for mutable stage %s", stg.Name()) } - if err := stg.MutateImage(ctx, phase.Conveyor.StorageManager.GetStagesStorage(), prevBuiltImage, stageImage); err != nil { + imageMutator, ok := stg.(stage.ImageStage) + if !ok { + return fmt.Errorf("mutable stage %s does not implement image mutation", stg.Name()) + } + if err := imageMutator.MutateImage(ctx, phase.Conveyor.StorageManager.GetStagesStorage(), prevBuiltImage, stageImage); err != nil { if storage.IsErrBrokenImage(err) { // Invalidate manifest cache for the broken previous stage prevStageDesc := prevBuiltImage.Image.GetStageDesc() @@ -1843,7 +1845,7 @@ func (phase *BuildPhase) convergeMultiplatformVexByImageSets(ctx context.Context return nil } - return phase.convergeImageVex(ctx, name, images) + return phase.runMultiplatformVexStage(ctx, name, images) }); err != nil { return err } @@ -1853,11 +1855,10 @@ func (phase *BuildPhase) convergeMultiplatformVexByImageSets(ctx context.Context return nil } -func (phase *BuildPhase) convergeImageVex(ctx context.Context, name string, images []*image.Image) error { +func (phase *BuildPhase) runMultiplatformVexStage(ctx context.Context, name string, images []*image.Image) error { if len(images) == 0 { return nil } - if len(images) == 1 { for _, stg := range images[0].GetStages() { if stg.Name() == stage.Vex { @@ -1892,21 +1893,23 @@ func (phase *BuildPhase) convergeImageVex(ctx context.Context, name string, imag return fmt.Errorf("read VEX file %q for image %q: %w", vexConfig.Document, name, err) } - var signer signature.Signer - var signerIdentity string - if phase.VexSigningOptions.Enabled { - signer = phase.VexSigningOptions.Signer().SignerVerifier() - signerIdentity = phase.VexSigningOptions.Signer().Fingerprint() + baseOptions := &stage.BaseStageOptions{ + ImageName: name, + ProjectName: phase.Conveyor.ProjectName(), + TargetPlatform: vexTargetPlatform(images), } - - if err := phase.vexProcessor.Converge(ctx, vexContent, stageDesc, name, vexTargetPlatform(images), signer, signerIdentity); err != nil { + vexStage := stage.NewVexStage(stage.VexStageOptions{ + VexJSON: vexContent, + BaseStageOptions: baseOptions, + SigningOptions: phase.VexSigningOptions, + Publisher: func(ctx context.Context, parentDesc *imagePkg.StageDesc, imageName, targetPlatform string, content []byte, signer signature.Signer, signerIdentity string) error { + return phase.vexProcessor.Converge(ctx, content, parentDesc, imageName, targetPlatform, signer, signerIdentity) + }, + }) + if err := vexStage.MutateArtifactWithDescriptor(ctx, stageDesc); err != nil { return fmt.Errorf("unable to converge VEX for image %q: %w", name, err) } - if err := propagateArtifacts(ctx, phase.Conveyor.ProjectName(), name, stageDesc, finalStageDescForImage(phase, name, images), phase.Conveyor.StorageManager.GetCacheStagesStorageList(), phase.Conveyor.StorageManager.GetStagesStorage()); err != nil { - return fmt.Errorf("unable to propagate VEX for image %q: %w", name, err) - } - return nil } diff --git a/pkg/build/build_phase_test.go b/pkg/build/build_phase_test.go index fa2d44802a..8ac2020774 100644 --- a/pkg/build/build_phase_test.go +++ b/pkg/build/build_phase_test.go @@ -186,7 +186,7 @@ var _ = Describe("BuildPhase", func() { images := newMultiplatformImages(ctx, &config.Vex{Document: "vex.json"}) phase := newPhaseWithTree(image.NewMultiplatformImage("app", images, 0, 1)) - err := phase.convergeImageVex(ctx, "app", images) + err := phase.runMultiplatformVexStage(ctx, "app", images) Expect(err).To(MatchError(`unable to converge VEX for image "app": stage descriptor is unavailable`)) }) @@ -195,27 +195,36 @@ var _ = Describe("BuildPhase", func() { images := newMultiplatformImages(ctx, nil) phase := newPhaseWithTree(image.NewMultiplatformImage("app", images, 0, 1)) - Expect(phase.convergeImageVex(ctx, "app", images)).To(Succeed()) + Expect(phase.runMultiplatformVexStage(ctx, "app", images)).To(Succeed()) }) It("is a no-op for an image without VEX configuration and without a stage descriptor", func(ctx SpecContext) { phase := &BuildPhase{} - Expect(phase.convergeImageVex(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", nil)})).To(Succeed()) + Expect(phase.runMultiplatformVexStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", nil)})).To(Succeed()) }) It("is a no-op for an image with an empty VEX document", func(ctx SpecContext) { phase := &BuildPhase{} - Expect(phase.convergeImageVex(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{})})).To(Succeed()) + Expect(phase.runMultiplatformVexStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{})})).To(Succeed()) }) It("reports an unavailable stage descriptor when VEX is configured", func(ctx SpecContext) { phase := &BuildPhase{} - err := phase.convergeImageVex(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{Document: "vex.json"})}) + err := phase.runMultiplatformVexStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{Document: "vex.json"})}) Expect(err).To(MatchError(ContainSubstring(`unable to converge VEX for image "app": stage descriptor is unavailable`))) }) + + It("continues when a multi-image stage descriptor is available", func(ctx SpecContext) { + images := newMultiplatformImages(ctx, &config.Vex{}) + multiImg := image.NewMultiplatformImage("app", images, 0, 1) + multiImg.SetStageDesc(&imagePkg.StageDesc{Info: &imagePkg.Info{}}) + phase := newPhaseWithTree(multiImg) + + Expect(phase.runMultiplatformVexStage(ctx, "app", images)).To(Succeed()) + }) }) Describe("last non-empty stage descriptor", func() { @@ -233,60 +242,6 @@ var _ = Describe("BuildPhase", func() { }) }) - Describe("vexStageDesc", func() { - It("uses the content tag descriptor of a reused single-platform image", func(ctx SpecContext) { - expected := &imagePkg.StageDesc{Info: &imagePkg.Info{Name: "repo:image"}} - img, err := image.NewImage(ctx, "linux/amd64", "app", image.NoBaseImage, image.ImageOptions{}) - Expect(err).To(Succeed()) - img.SetContentTagDesc(expected) - - Expect((&BuildPhase{}).vexStageDesc("app", []*image.Image{img})).To(BeIdenticalTo(expected)) - }) - - It("returns nil for a single-platform image without any descriptor", func(ctx SpecContext) { - img, err := image.NewImage(ctx, "linux/amd64", "app", image.NoBaseImage, image.ImageOptions{}) - Expect(err).To(Succeed()) - - Expect((&BuildPhase{}).vexStageDesc("app", []*image.Image{img})).To(BeNil()) - }) - - It("uses the descriptor of the registered multiplatform image", func(ctx SpecContext) { - images := make([]*image.Image, 0, 2) - for _, platform := range []string{"linux/amd64", "linux/arm64"} { - img, err := image.NewImage(ctx, platform, "app", image.NoBaseImage, image.ImageOptions{}) - Expect(err).To(Succeed()) - img.SetContentTagDesc(&imagePkg.StageDesc{ - StageID: imagePkg.NewStageID("digest-"+platform, 0), - Info: &imagePkg.Info{Name: "repo:" + platform}, - }) - images = append(images, img) - } - - expected := &imagePkg.StageDesc{Info: &imagePkg.Info{Name: "repo:multiplatform"}} - multiImg := image.NewMultiplatformImage("app", images, 0, 1) - multiImg.SetStageDesc(expected) - - tree := image.NewImagesTree(nil, image.ImagesTreeOptions{}) - tree.SetMultiplatformImage(multiImg) - phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{imagesTree: tree}}} - - Expect(phase.vexStageDesc("app", images)).To(BeIdenticalTo(expected)) - }) - - It("returns nil for a multiplatform image that was never registered", func(ctx SpecContext) { - images := make([]*image.Image, 0, 2) - for _, platform := range []string{"linux/amd64", "linux/arm64"} { - img, err := image.NewImage(ctx, platform, "app", image.NoBaseImage, image.ImageOptions{}) - Expect(err).To(Succeed()) - images = append(images, img) - } - - phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{imagesTree: image.NewImagesTree(nil, image.ImagesTreeOptions{})}}} - - Expect(phase.vexStageDesc("app", images)).To(BeNil()) - }) - }) - Describe("calculateDigest", func() { It("digest is unchanged when EnableSbom() returns false (backward compatibility)", func(ctx SpecContext) { conveyorNoSbom := &Conveyor{ diff --git a/pkg/build/sbom_processor.go b/pkg/build/sbom_processor.go index c33ede039f..2192820f63 100644 --- a/pkg/build/sbom_processor.go +++ b/pkg/build/sbom_processor.go @@ -15,7 +15,6 @@ import ( "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/container_backend" "github.com/werf/werf/v2/pkg/image" - "github.com/werf/werf/v2/pkg/oci/artifact" "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil" "github.com/werf/werf/v2/pkg/sbom/cyclonedxutil/gost" "github.com/werf/werf/v2/pkg/sbom/externalref" @@ -24,6 +23,7 @@ import ( osPm "github.com/werf/werf/v2/pkg/sbom/packages/os_pm" "github.com/werf/werf/v2/pkg/sbom/scanner" "github.com/werf/werf/v2/pkg/storage" + "github.com/werf/werf/v2/pkg/storage/manager" "github.com/werf/werf/v2/pkg/werf/global_warnings" ) @@ -40,6 +40,7 @@ var ErrSbomNotRequired = errors.New("sbom not required") type sbomProcessor struct { containerBackend container_backend.ContainerBackend stagesStorage storage.StagesStorage + storageManager manager.StorageManagerInterface gostWarnOnce sync.Once } @@ -47,15 +48,16 @@ type sbomProcessor struct { func newSbomProcessor( backend container_backend.ContainerBackend, stagesStorage storage.StagesStorage, + storageManager manager.StorageManagerInterface, ) *sbomProcessor { return &sbomProcessor{ containerBackend: backend, stagesStorage: stagesStorage, + storageManager: storageManager, } } func (processor *sbomProcessor) ConvergeWithMerge(ctx context.Context, werfImgName string, stageDesc *image.StageDesc, scanOpts scanner.ScanOptions, mergeOpts cyclonedxutil.MergeOpts, patchers []BOMPatcherInterface, osPmEnabled, isStapelScratch bool, targetPlatform string, signer signature.Signer, signerIdentity string) error { - repo := stageDesc.Info.Repository parentDigest := stageDesc.Info.GetDigest() scanOpts.Commands[0].SourcePath = stageDesc.Info.Name @@ -66,9 +68,7 @@ func (processor *sbomProcessor) ConvergeWithMerge(ctx context.Context, werfImgNa checksum := processor.calculateStableChecksum(scanOpts, mergeOpts, signerIdentity, targetPlatform) - store := artifact.NewOCIStore(repo, werfImgName) - - desc, found, err := attestation.FindAttachedArtifact(ctx, store, parentDigest, attestation.PredicateKindCycloneDX) + desc, found, err := processor.storageManager.FindAttachedArtifact(ctx, processor.stagesStorage, parentDigest, werfImgName, attestation.PredicateKindCycloneDX) if err != nil { return fmt.Errorf("check SBOM cache: %w", err) } @@ -167,7 +167,11 @@ func (processor *sbomProcessor) ConvergeWithMerge(ctx context.Context, werfImgNa } if err := logboek.Context(ctx).Default().LogProcess("Push SBOM artifact").DoError(func() error { - return sbomImage.PushSBOM(ctx, resultJSON, repo, parentDigest, werfImgName, checksum, targetPlatform, signer) + return processor.storageManager.PublishAttestation(ctx, processor.stagesStorage, attestation.PredicateKindCycloneDX, resultJSON, parentDigest, werfImgName, attestation.PublishAttestationOptions{ + Signer: signer, + Checksum: checksum, + TargetPlatform: targetPlatform, + }) }); err != nil { return err } @@ -199,13 +203,6 @@ func (processor *sbomProcessor) calculateStableChecksum(scanOpts scanner.ScanOpt ) } -// PropagateArtifacts copies the artifacts attached to the image stage (e.g. its SBOM) -// into the final repo and the cache repos. Stages themselves are copied there before -// SBOM generation runs, so the artifacts have to catch up separately. -func (processor *sbomProcessor) PropagateArtifacts(ctx context.Context, projectName, werfImgName string, stageDesc, finalStageDesc *image.StageDesc, cacheStagesStorageList []storage.StagesStorage) error { - return propagateArtifacts(ctx, projectName, werfImgName, stageDesc, finalStageDesc, cacheStagesStorageList) -} - func (processor *sbomProcessor) GetImageBOM(ctx context.Context, imageName string, imageInfo *image.Info) (*cdx.BOM, error) { if imageInfo == nil { return nil, fmt.Errorf("image info is nil for %q", imageName) diff --git a/pkg/build/stage/artifact_test.go b/pkg/build/stage/artifact_test.go index 72cf0f02e0..3b76726b1b 100644 --- a/pkg/build/stage/artifact_test.go +++ b/pkg/build/stage/artifact_test.go @@ -57,7 +57,28 @@ var _ = Describe("artifact stages", func() { Expect(artifactStage.MutateArtifact(ctx, parent, stageImage)).To(Succeed()) Expect(publisherCalls).To(Equal(1)) Expect(artifactStage.GetArtifactMetadata().ParentDigest).To(Equal("sha256:parent")) - Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(MatchError("SBOM stage must be mutated as an artifact")) + }) + + It("publishes VEX through an explicit descriptor without requiring an image stage", func(ctx SpecContext) { + parentDesc := &image.StageDesc{Info: &image.Info{Repository: "registry.example/app", RepoDigest: "registry.example/app@sha256:index"}} + publisherCalls := 0 + artifactStage := NewVexStage(VexStageOptions{ + VexJSON: []byte(`{"statements":[]}`), + BaseStageOptions: &BaseStageOptions{ImageName: "app"}, + Publisher: func(_ context.Context, gotDesc *image.StageDesc, imageName, platform string, content []byte, _ signature.Signer, identity string) error { + publisherCalls++ + Expect(gotDesc).To(BeIdenticalTo(parentDesc)) + Expect(imageName).To(Equal("app")) + Expect(platform).To(BeEmpty()) + Expect(content).To(MatchJSON(`{"statements":[]}`)) + Expect(identity).To(BeEmpty()) + return nil + }, + }) + + Expect(artifactStage.MutateArtifactWithDescriptor(ctx, parentDesc)).To(Succeed()) + Expect(publisherCalls).To(Equal(1)) + Expect(artifactStage.GetArtifactMetadata().ParentDigest).To(Equal("sha256:index")) }) It("publishes VEX through its stage publisher without requiring image mutation", func(ctx SpecContext) { @@ -85,18 +106,20 @@ var _ = Describe("artifact stages", func() { Expect(artifactStage.MutateArtifact(ctx, parent, stageImage)).To(Succeed()) Expect(publisherCalls).To(Equal(1)) Expect(artifactStage.GetArtifactMetadata().ParentDigest).To(Equal("sha256:parent")) - Expect(artifactStage.MutateImage(ctx, nil, parent, stageImage)).To(MatchError("VEX stage must be mutated as an artifact")) }) - DescribeTable("is mutable and non-buildable", - func(artifactStage Interface) { - Expect(artifactStage.IsMutable()).To(BeTrue()) - Expect(artifactStage.IsBuildable()).To(BeFalse()) + DescribeTable("is mutable, non-buildable, and artifact-only", + func(artifactStage ArtifactStage, stageLifecycle Interface) { + Expect(stageLifecycle.IsMutable()).To(BeTrue()) + Expect(stageLifecycle.IsBuildable()).To(BeFalse()) + Expect(artifactStage).NotTo(BeNil()) }, Entry("SBOM", GenerateSbomStage(&BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.SbomSigningOptions{}, "dependency", func(context.Context, *image.StageDesc, string, string) error { return nil + }), GenerateSbomStage(&BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.SbomSigningOptions{}, "dependency", func(context.Context, *image.StageDesc, string, string) error { + return nil })), - Entry("VEX", GenerateVexStage([]byte(`{"statements":[]}`), &BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.VexSigningOptions{})), + Entry("VEX", GenerateVexStage([]byte(`{"statements":[]}`), &BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.VexSigningOptions{}), GenerateVexStage([]byte(`{"statements":[]}`), &BaseStageOptions{TargetPlatform: "linux/amd64"}, signing.VexSigningOptions{})), ) It("includes the parent descriptor in artifact stage dependencies", func(ctx SpecContext) { diff --git a/pkg/build/stage/base.go b/pkg/build/stage/base.go index 0a6ed9b481..1fd4d9b4bb 100644 --- a/pkg/build/stage/base.go +++ b/pkg/build/stage/base.go @@ -362,10 +362,6 @@ func (s *BaseStage) MutateImage(_ context.Context, _ ImageMutatorPusher, _, _ *S panic("not implemented") } -func (s *BaseStage) MutateArtifact(_ context.Context, _, _ *StageImage) error { - panic("not implemented") -} - func (s *BaseStage) PreRun(_ context.Context, _ Conveyor) error { return nil } diff --git a/pkg/build/stage/interface.go b/pkg/build/stage/interface.go index 7a8bfc7ae6..a7677ba722 100644 --- a/pkg/build/stage/interface.go +++ b/pkg/build/stage/interface.go @@ -8,6 +8,14 @@ import ( "github.com/werf/werf/v2/pkg/image" ) +type ArtifactStage interface { + MutateArtifact(ctx context.Context, prevBuiltImage, stageImage *StageImage) error +} + +type ImageStage interface { + MutateImage(ctx context.Context, registry ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error +} + type Interface interface { Name() StageName LogDetailedName() string @@ -38,9 +46,6 @@ type Interface interface { SetGitMappings([]*GitMapping) GetGitMappings() []*GitMapping - MutateImage(ctx context.Context, registry ImageMutatorPusher, prevBuiltImage, stageImage *StageImage) error - MutateArtifact(ctx context.Context, prevBuiltImage, stageImage *StageImage) error - SelectSuitableStageDesc(context.Context, Conveyor, image.StageDescSet) (*image.StageDesc, error) HasPrevStage() bool diff --git a/pkg/build/stage/sbom.go b/pkg/build/stage/sbom.go index 333460953a..71d5a9138c 100644 --- a/pkg/build/stage/sbom.go +++ b/pkg/build/stage/sbom.go @@ -61,7 +61,10 @@ func newSbomStage(baseStageOptions *BaseStageOptions, sbomSigningOptions signing return stage } -var _ Interface = (*SbomStage)(nil) +var ( + _ Interface = (*SbomStage)(nil) + _ ArtifactStage = (*SbomStage)(nil) +) func (s *SbomStage) IsBuildable() bool { return false @@ -124,8 +127,4 @@ func (s *SbomStage) MutateArtifact(ctx context.Context, prevBuiltImage, stageIma return s.publisher(ctx, parentDesc, s.ImageName(), s.TargetPlatform()) } -func (s *SbomStage) MutateImage(_ context.Context, _ ImageMutatorPusher, _, _ *StageImage) error { - return fmt.Errorf("SBOM stage must be mutated as an artifact") -} - const sbomArtifactFormatVersion = "2" diff --git a/pkg/build/stage/vex.go b/pkg/build/stage/vex.go index 52aa06188e..a37bc4eccc 100644 --- a/pkg/build/stage/vex.go +++ b/pkg/build/stage/vex.go @@ -67,7 +67,10 @@ func newVexStage(vexJSON []byte, baseStageOptions *BaseStageOptions, vexSigningO return stage } -var _ Interface = (*VexStage)(nil) +var ( + _ Interface = (*VexStage)(nil) + _ ArtifactStage = (*VexStage)(nil) +) func (s *VexStage) IsBuildable() bool { return false @@ -104,7 +107,10 @@ func (s *VexStage) MutateArtifact(ctx context.Context, prevBuiltImage, stageImag return fmt.Errorf("VEX stage image is unavailable") } - parentDesc := prevBuiltImage.Image.GetStageDesc() + return s.MutateArtifactWithDescriptor(ctx, prevBuiltImage.Image.GetStageDesc()) +} + +func (s *VexStage) MutateArtifactWithDescriptor(ctx context.Context, parentDesc *image.StageDesc) error { if parentDesc == nil || parentDesc.Info == nil { return fmt.Errorf("VEX stage parent descriptor is unavailable") } @@ -127,10 +133,6 @@ func (s *VexStage) MutateArtifact(ctx context.Context, prevBuiltImage, stageImag return s.publisher(ctx, parentDesc, s.ImageName(), s.TargetPlatform(), s.vexJSON, s.signer, s.signerIdentity) } -func (s *VexStage) MutateImage(_ context.Context, _ ImageMutatorPusher, _, _ *StageImage) error { - return fmt.Errorf("VEX stage must be mutated as an artifact") -} - const vexStageArtifactFormatVersion = "2" // CalculateVexStageChecksum returns the cache identity for a VEX artifact. diff --git a/pkg/build/vex_processor.go b/pkg/build/vex_processor.go index 855dd45945..019d1ed7e7 100644 --- a/pkg/build/vex_processor.go +++ b/pkg/build/vex_processor.go @@ -10,50 +10,37 @@ import ( "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/build/stage" "github.com/werf/werf/v2/pkg/image" - "github.com/werf/werf/v2/pkg/oci/artifact" - vexImage "github.com/werf/werf/v2/pkg/vex/image" + "github.com/werf/werf/v2/pkg/storage" + "github.com/werf/werf/v2/pkg/storage/manager" ) -type vexProcessor struct{} +type vexProcessor struct { + stagesStorage storage.StagesStorage + storageManager manager.StorageManagerInterface +} -func newVexProcessor() *vexProcessor { - return &vexProcessor{} +func newVexProcessor(stagesStorage storage.StagesStorage, storageManager manager.StorageManagerInterface) *vexProcessor { + return &vexProcessor{stagesStorage: stagesStorage, storageManager: storageManager} } func (processor *vexProcessor) Converge(ctx context.Context, vexJSON []byte, stageDesc *image.StageDesc, werfImgName, targetPlatform string, signer signature.Signer, signerIdentity string) error { - repo := stageDesc.Info.Repository parentDigest := stageDesc.Info.GetDigest() checksum := stage.CalculateVexStageChecksum(vexJSON, parentDigest, signerIdentity) - store := artifact.NewOCIStore(repo, werfImgName) - - needed, err := checkVEXPublishNeeded(ctx, store, parentDigest, checksum) + desc, found, err := processor.storageManager.FindAttachedArtifact(ctx, processor.stagesStorage, parentDigest, werfImgName, attestation.PredicateKindOpenVEX) if err != nil { return fmt.Errorf("check VEX publish needed: %w", err) } + needed := !found || desc.Annotations[image.WerfChecksumAnnotation] != checksum if !needed { logboek.Context(ctx).Default().LogF("image %s: VEX artifact is up to date — skipping publish\n", werfImgName) return nil } return logboek.Context(ctx).Default().LogProcess("image %s: Published VEX artifact", werfImgName).DoError(func() error { - return vexImage.PushVEX(ctx, vexJSON, repo, parentDigest, werfImgName, checksum, targetPlatform, signer) + return processor.storageManager.PublishAttestation(ctx, processor.stagesStorage, attestation.PredicateKindOpenVEX, vexJSON, parentDigest, werfImgName, attestation.PublishAttestationOptions{ + Signer: signer, Checksum: checksum, TargetPlatform: targetPlatform, + }) }) } - -// checkVEXPublishNeeded returns true if the VEX artifact should be published -// (no existing VEX artifact of either format or its checksum annotation differs -// from the current checksum), and false if publishing can be skipped. A legacy -// annotation-less entry never matches the current checksum formula, so it always -// triggers a republish regardless of its actual kind. -func checkVEXPublishNeeded(ctx context.Context, store artifact.Store, parentDigest, checksum string) (bool, error) { - desc, found, err := attestation.FindAttachedArtifact(ctx, store, parentDigest, attestation.PredicateKindOpenVEX) - if err != nil { - return false, fmt.Errorf("check VEX cache: %w", err) - } - if found && desc.Annotations[image.WerfChecksumAnnotation] == checksum { - return false, nil - } - return true, nil -} diff --git a/pkg/storage/local_stages_storage.go b/pkg/storage/local_stages_storage.go index b9d0efe759..584ce9806c 100644 --- a/pkg/storage/local_stages_storage.go +++ b/pkg/storage/local_stages_storage.go @@ -10,6 +10,7 @@ import ( "github.com/werf/common-go/pkg/util" "github.com/werf/logboek" + "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/container_backend" "github.com/werf/werf/v2/pkg/docker_registry" "github.com/werf/werf/v2/pkg/docker_registry/api" @@ -266,6 +267,14 @@ func (storage *LocalStagesStorage) ListAttachedArtifacts(_ context.Context, _ st return nil, fmt.Errorf("local stages storage does not support artifact operations") } +func (storage *LocalStagesStorage) FindAttachedArtifact(_ context.Context, _, _ string, _ attestation.PredicateKind) (v1.Descriptor, bool, error) { + return v1.Descriptor{}, false, fmt.Errorf("local stages storage does not support artifact operations") +} + +func (storage *LocalStagesStorage) PublishAttestation(_ context.Context, _ attestation.PredicateKind, _ []byte, _, _ string, _ attestation.PublishAttestationOptions) error { + return fmt.Errorf("local stages storage does not support artifact operations") +} + func (storage *LocalStagesStorage) PublishArtifact(_ context.Context, _, _ string, _ []byte, _, _, _, _ string) error { return fmt.Errorf("local stages storage does not support artifact operations") } diff --git a/pkg/storage/manager/artifact_operations_test.go b/pkg/storage/manager/artifact_operations_test.go index 84bbfe6f25..ca890e72b3 100644 --- a/pkg/storage/manager/artifact_operations_test.go +++ b/pkg/storage/manager/artifact_operations_test.go @@ -8,6 +8,7 @@ import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/image" "github.com/werf/werf/v2/pkg/storage" ) @@ -16,16 +17,22 @@ type artifactOperationsStorage struct { storage.StagesStorage address string - listedParent string - publishedParent string - copiedSourceRepo string - copiedSourceDigest string - copiedDestRepo string - copiedDestDigest string - resolvedProject string - resolvedStage image.StageID - resolveResult *image.StageDesc - operationError error + listedParent string + publishedParent string + copiedSourceRepo string + copiedSourceDigest string + copiedDestRepo string + copiedDestDigest string + resolvedProject string + resolvedStage image.StageID + resolveResult *image.StageDesc + operationError error + foundArtifact v1.Descriptor + foundArtifactPresent bool + foundArtifactKind attestation.PredicateKind + foundArtifactName string + publishedKind attestation.PredicateKind + publishedImageName string } func (s *artifactOperationsStorage) Address() string { return s.address } @@ -49,6 +56,18 @@ func (s *artifactOperationsStorage) CopyAttachedArtifacts(_ context.Context, sou return s.operationError } +func (s *artifactOperationsStorage) FindAttachedArtifact(_ context.Context, _, imageName string, kind attestation.PredicateKind) (v1.Descriptor, bool, error) { + s.foundArtifactName = imageName + s.foundArtifactKind = kind + return s.foundArtifact, s.foundArtifactPresent, s.operationError +} + +func (s *artifactOperationsStorage) PublishAttestation(_ context.Context, kind attestation.PredicateKind, _ []byte, _, imageName string, _ attestation.PublishAttestationOptions) error { + s.publishedKind = kind + s.publishedImageName = imageName + return s.operationError +} + func (s *artifactOperationsStorage) GetStageDesc(_ context.Context, projectName string, stageID image.StageID) (*image.StageDesc, error) { s.resolvedProject = projectName s.resolvedStage = stageID @@ -85,6 +104,22 @@ var _ = Describe("StorageManager artifact operations", func() { Expect(destination.copiedDestDigest).To(Equal("sha256:destination")) }) + It("routes attestation lookup and publication through the selected storage", func(ctx SpecContext) { + stages := &artifactOperationsStorage{address: "registry.example/repository", foundArtifactPresent: true, foundArtifact: v1.Descriptor{Digest: v1.Hash{Algorithm: "sha256", Hex: "artifact"}}} + manager := &StorageManager{} + + descriptor, found, err := manager.FindAttachedArtifact(ctx, stages, "sha256:parent", "app", attestation.PredicateKindOpenVEX) + Expect(err).NotTo(HaveOccurred()) + Expect(found).To(BeTrue()) + Expect(descriptor.Digest.Hex).To(Equal("artifact")) + Expect(stages.foundArtifactName).To(Equal("app")) + Expect(stages.foundArtifactKind).To(Equal(attestation.PredicateKindOpenVEX)) + + Expect(manager.PublishAttestation(ctx, stages, attestation.PredicateKindOpenVEX, []byte("{}"), "sha256:parent", "app", attestation.PublishAttestationOptions{})).To(Succeed()) + Expect(stages.publishedKind).To(Equal(attestation.PredicateKindOpenVEX)) + Expect(stages.publishedImageName).To(Equal("app")) + }) + It("skips copying between identical repository addresses", func(ctx SpecContext) { source := &artifactOperationsStorage{address: "registry.example/repository"} destination := &artifactOperationsStorage{address: source.address} @@ -102,6 +137,15 @@ var _ = Describe("StorageManager artifact operations", func() { Expect(manager.CopyAttachedArtifacts(ctx, local, "sha256:source", &artifactOperationsStorage{address: "registry.example/destination"}, "sha256:destination")).To(MatchError(ContainSubstring("local stages storage"))) }) + It("rejects incomplete attestation operations", func(ctx SpecContext) { + stages := &artifactOperationsStorage{address: "registry.example/repository"} + manager := &StorageManager{} + + _, _, err := manager.FindAttachedArtifact(ctx, stages, "", "app", attestation.PredicateKindOpenVEX) + Expect(err).To(MatchError("find attached artifact: parent digest is empty")) + Expect(manager.PublishAttestation(ctx, stages, attestation.PredicateKindOpenVEX, []byte("{}"), "", "app", attestation.PublishAttestationOptions{})).To(MatchError("publish attestation: parent digest is empty")) + }) + It("wraps backend errors with the routed operation", func(ctx SpecContext) { backendError := errors.New("backend unavailable") source := &artifactOperationsStorage{address: "registry.example/source", operationError: backendError} diff --git a/pkg/storage/manager/storage_manager.go b/pkg/storage/manager/storage_manager.go index e8f8275f40..88389f5559 100644 --- a/pkg/storage/manager/storage_manager.go +++ b/pkg/storage/manager/storage_manager.go @@ -17,6 +17,7 @@ import ( "github.com/werf/logboek" "github.com/werf/logboek/pkg/style" "github.com/werf/logboek/pkg/types" + "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/build/stage" "github.com/werf/werf/v2/pkg/container_backend" "github.com/werf/werf/v2/pkg/docker_registry" @@ -80,6 +81,8 @@ type StorageManagerInterface interface { CopyStageIntoFinalStorage(ctx context.Context, stageID image.StageID, finalStagesStorage storage.StagesStorage, opts CopyStageIntoStorageOptions) (*image.StageDesc, error) ListAttachedArtifacts(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest string) ([]v1.Descriptor, error) + FindAttachedArtifact(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest, imageName string, kind attestation.PredicateKind) (v1.Descriptor, bool, error) + PublishAttestation(ctx context.Context, stagesStorage storage.StagesStorage, kind attestation.PredicateKind, payload []byte, parentDigest, imageName string, options attestation.PublishAttestationOptions) error PublishArtifact(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error ResolveStageDescriptor(ctx context.Context, stagesStorage storage.StagesStorage, stageID image.StageID) (*image.StageDesc, error) CopyAttachedArtifacts(ctx context.Context, sourceStorage storage.StagesStorage, sourceDigest string, destinationStorage storage.StagesStorage, destinationDigest string) error @@ -238,6 +241,46 @@ func (m *StorageManager) ListAttachedArtifacts(ctx context.Context, stagesStorag return artifacts, nil } +func (m *StorageManager) FindAttachedArtifact(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest, imageName string, kind attestation.PredicateKind) (v1.Descriptor, bool, error) { + if stagesStorage == nil { + return v1.Descriptor{}, false, fmt.Errorf("find attached artifact: stages storage is nil") + } + if stagesStorage.Address() == storage.LocalStorageAddress { + return v1.Descriptor{}, false, fmt.Errorf("find attached artifact: local stages storage does not support artifact operations") + } + if parentDigest == "" { + return v1.Descriptor{}, false, fmt.Errorf("find attached artifact: parent digest is empty") + } + if imageName == "" { + return v1.Descriptor{}, false, fmt.Errorf("find attached artifact: image name is empty") + } + + descriptor, found, err := stagesStorage.FindAttachedArtifact(ctx, parentDigest, imageName, kind) + if err != nil { + return v1.Descriptor{}, false, fmt.Errorf("find attached %s artifact from %s: %w", kind.Name, stagesStorage.String(), err) + } + return descriptor, found, nil +} + +func (m *StorageManager) PublishAttestation(ctx context.Context, stagesStorage storage.StagesStorage, kind attestation.PredicateKind, payload []byte, parentDigest, imageName string, options attestation.PublishAttestationOptions) error { + if stagesStorage == nil { + return fmt.Errorf("publish attestation: stages storage is nil") + } + if stagesStorage.Address() == storage.LocalStorageAddress { + return fmt.Errorf("publish attestation: local stages storage does not support artifact operations") + } + if parentDigest == "" { + return fmt.Errorf("publish attestation: parent digest is empty") + } + if imageName == "" { + return fmt.Errorf("publish attestation: image name is empty") + } + if err := stagesStorage.PublishAttestation(ctx, kind, payload, parentDigest, imageName, options); err != nil { + return fmt.Errorf("publish attestation to %s: %w", stagesStorage.String(), err) + } + return nil +} + func (m *StorageManager) PublishArtifact(ctx context.Context, stagesStorage storage.StagesStorage, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error { if stagesStorage == nil { return fmt.Errorf("publish artifact: stages storage is nil") diff --git a/pkg/storage/meta_repo_marker_test.go b/pkg/storage/meta_repo_marker_test.go index 6863926fca..757fb294f2 100644 --- a/pkg/storage/meta_repo_marker_test.go +++ b/pkg/storage/meta_repo_marker_test.go @@ -45,6 +45,7 @@ var markerUnguardedMethods = []string{ "ExportStage", "FetchImage", "FilterStageDescSetAndProcessRelatedData", + "FindAttachedArtifact", "GetAllAndGroupImageMetadataByImageName", "GetLastCleanupRecord", "GetManagedImages", @@ -62,6 +63,7 @@ var markerUnguardedMethods = []string{ "PostManifest", "PostMultiplatformImage", "PublishArtifact", + "PublishAttestation", "RejectStage", "ShouldFetchImage", "StoreImage", diff --git a/pkg/storage/repo_stages_storage.go b/pkg/storage/repo_stages_storage.go index dfad7f5fa7..8789a2b997 100644 --- a/pkg/storage/repo_stages_storage.go +++ b/pkg/storage/repo_stages_storage.go @@ -14,6 +14,7 @@ import ( "github.com/werf/common-go/pkg/util" "github.com/werf/logboek" + "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/container_backend" "github.com/werf/werf/v2/pkg/docker_registry" "github.com/werf/werf/v2/pkg/docker_registry/api" @@ -861,6 +862,15 @@ func (storage *RepoStagesStorage) ListAttachedArtifacts(ctx context.Context, par return manifest.Manifests, nil } +func (storage *RepoStagesStorage) FindAttachedArtifact(ctx context.Context, parentDigest, imageName string, kind attestation.PredicateKind) (v1.Descriptor, bool, error) { + store := artifact.NewOCIStore(storage.RepoAddress, imageName) + return attestation.FindAttachedArtifact(ctx, store, parentDigest, kind) +} + +func (storage *RepoStagesStorage) PublishAttestation(ctx context.Context, kind attestation.PredicateKind, payload []byte, parentDigest, imageName string, options attestation.PublishAttestationOptions) error { + return attestation.PublishAttestation(ctx, kind, payload, storage.RepoAddress, parentDigest, imageName, options) +} + func (storage *RepoStagesStorage) PublishArtifact(ctx context.Context, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error { return artifact.NewOCIStore(storage.RepoAddress, imageName).Attach(ctx, parentDigest, artifactType, payload, checksum, targetPlatform, predicateType) } diff --git a/pkg/storage/stages_storage.go b/pkg/storage/stages_storage.go index e256437b7e..5336e12c32 100644 --- a/pkg/storage/stages_storage.go +++ b/pkg/storage/stages_storage.go @@ -7,6 +7,7 @@ import ( v1 "github.com/google/go-containerregistry/pkg/v1" + "github.com/werf/werf/v2/pkg/attestation" "github.com/werf/werf/v2/pkg/container_backend" "github.com/werf/werf/v2/pkg/image" ) @@ -83,6 +84,8 @@ type StagesStorage interface { DeleteArtifact(ctx context.Context, imageName string) error ListAttachedArtifacts(ctx context.Context, parentDigest string) ([]v1.Descriptor, error) + FindAttachedArtifact(ctx context.Context, parentDigest, imageName string, kind attestation.PredicateKind) (v1.Descriptor, bool, error) + PublishAttestation(ctx context.Context, kind attestation.PredicateKind, payload []byte, parentDigest, imageName string, options attestation.PublishAttestationOptions) error PublishArtifact(ctx context.Context, parentDigest, artifactType string, payload []byte, imageName, checksum, targetPlatform, predicateType string) error CopyAttachedArtifacts(ctx context.Context, sourceRepository, sourceDigest, destinationRepository, destinationDigest string) error diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 6ae0634307..8877ec47a0 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -48,7 +48,7 @@ - [X] T016 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` - [ ] T017 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios - [X] T018 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` -- [ ] T019 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `StorageManager`, with the manager selecting the appropriate `storage.StagesStorage`, in `pkg/build/stage/` and `pkg/storage/manager/` +- [X] T019 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `StorageManager`, with the manager selecting the appropriate `storage.StagesStorage`, in `pkg/build/stage/` and `pkg/storage/manager/` ### Implementation for User Story 1 @@ -56,7 +56,7 @@ - [X] T021 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including final-image-digest association and VEX generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` - [ ] T022 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` - [ ] T023 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` -- [ ] T024 [US1] Ensure `PrepareImage` is a no-op, `MutateArtifact` operates only on the associated OCI artifact through `StorageManager`, and the artifact stages do not implement or invoke `MutateImage`; do not fetch, rebuild, store, or mutate image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go` +- [ ] T024 [US1] Ensure `PrepareImage` is a no-op, `MutateArtifact` operates only on the associated OCI artifact through `StorageManager`, and the artifact stages do not implement or invoke `MutateImage`; do not fetch, rebuild, store, or mutate image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go - [ ] T025 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` - [ ] T026 [US1] Execute artifact publication through stage `MutateArtifact` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` - [X] T027 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` diff --git a/test/mock/stages_storage.go b/test/mock/stages_storage.go index 149a568c1e..cce13768c3 100644 --- a/test/mock/stages_storage.go +++ b/test/mock/stages_storage.go @@ -14,6 +14,7 @@ import ( reflect "reflect" v1 "github.com/google/go-containerregistry/pkg/v1" + attestation "github.com/werf/werf/v2/pkg/attestation" container_backend "github.com/werf/werf/v2/pkg/container_backend" image "github.com/werf/werf/v2/pkg/image" storage "github.com/werf/werf/v2/pkg/storage" @@ -284,6 +285,22 @@ func (mr *MockStagesStorageMockRecorder) FilterStageDescSetAndProcessRelatedData return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FilterStageDescSetAndProcessRelatedData", reflect.TypeOf((*MockStagesStorage)(nil).FilterStageDescSetAndProcessRelatedData), ctx, stageDescSet, options) } +// FindAttachedArtifact mocks base method. +func (m *MockStagesStorage) FindAttachedArtifact(ctx context.Context, parentDigest, imageName string, kind attestation.PredicateKind) (v1.Descriptor, bool, error) { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "FindAttachedArtifact", ctx, parentDigest, imageName, kind) + ret0, _ := ret[0].(v1.Descriptor) + ret1, _ := ret[1].(bool) + ret2, _ := ret[2].(error) + return ret0, ret1, ret2 +} + +// FindAttachedArtifact indicates an expected call of FindAttachedArtifact. +func (mr *MockStagesStorageMockRecorder) FindAttachedArtifact(ctx, parentDigest, imageName, kind any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindAttachedArtifact", reflect.TypeOf((*MockStagesStorage)(nil).FindAttachedArtifact), ctx, parentDigest, imageName, kind) +} + // GetAllAndGroupImageMetadataByImageName mocks base method. func (m *MockStagesStorage) GetAllAndGroupImageMetadataByImageName(ctx context.Context, projectName string, imageNameOrManagedImageList []string, opts ...storage.Option) (map[string]map[string][]string, map[string]map[string][]string, error) { m.ctrl.T.Helper() @@ -560,6 +577,20 @@ func (mr *MockStagesStorageMockRecorder) PublishArtifact(ctx, parentDigest, arti return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PublishArtifact", reflect.TypeOf((*MockStagesStorage)(nil).PublishArtifact), ctx, parentDigest, artifactType, payload, imageName, checksum, targetPlatform, predicateType) } +// PublishAttestation mocks base method. +func (m *MockStagesStorage) PublishAttestation(ctx context.Context, kind attestation.PredicateKind, payload []byte, parentDigest, imageName string, options attestation.PublishAttestationOptions) error { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "PublishAttestation", ctx, kind, payload, parentDigest, imageName, options) + ret0, _ := ret[0].(error) + return ret0 +} + +// PublishAttestation indicates an expected call of PublishAttestation. +func (mr *MockStagesStorageMockRecorder) PublishAttestation(ctx, kind, payload, parentDigest, imageName, options any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PublishAttestation", reflect.TypeOf((*MockStagesStorage)(nil).PublishAttestation), ctx, kind, payload, parentDigest, imageName, options) +} + // PutImageMetadata mocks base method. func (m *MockStagesStorage) PutImageMetadata(ctx context.Context, projectName, imageNameOrManagedImageName, commit, stageID string) error { m.ctrl.T.Helper() From 7c89a888fa788d120a7f228d8a9ac060a139d8e6 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 19:05:19 +0300 Subject: [PATCH 11/17] refactor(build): resolve platform artifact subjects Preserve platform metadata and use destination digests when attaching artifacts, preventing platform subjects from being matched by index order. Signed-off-by: Alexandr Zaytsev --- pkg/build/artifact_propagation_test.go | 33 ++++++++++++++++++++-- pkg/build/artifact_stage_migration_test.go | 2 +- pkg/build/artifact_subject.go | 8 +++++- pkg/build/artifact_subject_test.go | 23 +++++++++++++++ pkg/build/build_phase.go | 19 +++++++++---- pkg/build/build_phase_test.go | 12 ++++---- pkg/docker_registry/api.go | 3 ++ pkg/image/info.go | 8 ++++-- specs/020-sbom-vex-build-stages/tasks.md | 18 ++++++------ 9 files changed, 98 insertions(+), 28 deletions(-) diff --git a/pkg/build/artifact_propagation_test.go b/pkg/build/artifact_propagation_test.go index e7d608d397..5a3c04fe47 100644 --- a/pkg/build/artifact_propagation_test.go +++ b/pkg/build/artifact_propagation_test.go @@ -27,11 +27,15 @@ import ( type recordingArtifactStorageManager struct { manager.StorageManagerInterface - copyCalls int + copyCalls int + lastSourceDigest string + lastDestinationDigest string } -func (m *recordingArtifactStorageManager) CopyAttachedArtifacts(context.Context, storage.StagesStorage, string, storage.StagesStorage, string) error { +func (m *recordingArtifactStorageManager) CopyAttachedArtifacts(_ context.Context, _ storage.StagesStorage, sourceDigest string, _ storage.StagesStorage, destinationDigest string) error { m.copyCalls++ + m.lastSourceDigest = sourceDigest + m.lastDestinationDigest = destinationDigest return nil } @@ -57,6 +61,31 @@ var _ = Describe("artifact propagation", func() { Expect(storageManager.copyCalls).To(Equal(1)) }) + It("propagates a multi-platform index using its resolved destination digest", func(ctx SpecContext) { + storageManager := &recordingArtifactStorageManager{} + source := &image.StageDesc{ + StageID: image.NewStageID("stage-digest", 1), + Info: &image.Info{ + IsIndex: true, + Repository: "registry.example/source", + RepoDigest: "registry.example/source@sha256:source-index", + }, + } + cache := mock.NewMockStagesStorage(gomock.NewController(GinkgoT())) + cache.EXPECT().Address().Return("registry.example/cache").AnyTimes() + cache.EXPECT().String().Return("registry.example/cache").AnyTimes() + cache.EXPECT().GetStageDesc(ctx, "project", *source.StageID).Return(&image.StageDesc{Info: &image.Info{ + IsIndex: true, + Repository: "registry.example/cache", + RepoDigest: "registry.example/cache@sha256:destination-index", + }}, nil) + + Expect(propagateArtifactsWithManager(ctx, "project", "app", source, nil, []storage.StagesStorage{cache}, nil, nil, storageManager)).To(Succeed()) + Expect(storageManager.copyCalls).To(Equal(1)) + Expect(storageManager.lastSourceDigest).To(Equal("sha256:source-index")) + Expect(storageManager.lastDestinationDigest).To(Equal("sha256:destination-index")) + }) + It("skips local-only artifact sources", func(ctx SpecContext) { err := propagateArtifacts(ctx, "project", "app", &image.StageDesc{ Info: &image.Info{Repository: ":local", RepoDigest: ":local@sha256:local"}, diff --git a/pkg/build/artifact_stage_migration_test.go b/pkg/build/artifact_stage_migration_test.go index 39a88bbf89..19c168e53f 100644 --- a/pkg/build/artifact_stage_migration_test.go +++ b/pkg/build/artifact_stage_migration_test.go @@ -37,6 +37,6 @@ var _ = Describe("artifact stage migration", func() { Expect(string(buildPhase)).To(ContainSubstring("GenerateSbomStage")) Expect(string(buildPhase)).To(ContainSubstring("NewVexStage")) Expect(string(buildPhase)).To(ContainSubstring("if len(images) == 1")) - Expect(string(buildPhase)).To(ContainSubstring("runMultiplatformVexStage")) + Expect(string(buildPhase)).To(ContainSubstring("runMultiplatformVexArtifactStage")) }) }) diff --git a/pkg/build/artifact_subject.go b/pkg/build/artifact_subject.go index c0b75d265b..7afccd6b6c 100644 --- a/pkg/build/artifact_subject.go +++ b/pkg/build/artifact_subject.go @@ -51,8 +51,14 @@ func finalStageDescForPlatform(phase *BuildPhase, name string, images []*image.I return finalStageDesc } + for _, manifest := range finalStageDesc.Info.Index { + if manifest != nil && manifest.Platform == targetPlatform { + return &imagePkg.StageDesc{Info: manifest} + } + } + for index, img := range images { - if img.TargetPlatform != targetPlatform || index >= len(finalStageDesc.Info.Index) { + if img == nil || img.TargetPlatform != targetPlatform || index >= len(finalStageDesc.Info.Index) { continue } diff --git a/pkg/build/artifact_subject_test.go b/pkg/build/artifact_subject_test.go index 8bfb3fa94f..047fc70b57 100644 --- a/pkg/build/artifact_subject_test.go +++ b/pkg/build/artifact_subject_test.go @@ -43,6 +43,29 @@ var _ = Describe("artifact subjects", func() { Expect(descriptor.Info.GetDigest()).To(Equal("sha256:arm64")) }) + It("selects a platform manifest by platform metadata regardless of index order", func() { + images := []*image.Image{ + newImage("app", "linux/amd64", "sha256:amd64"), + newImage("app", "linux/arm64", "sha256:arm64"), + } + multiImage := image.NewMultiplatformImage("app", images, 0, 1) + multiImage.SetFinalStageDesc(&imagePkg.StageDesc{Info: &imagePkg.Info{ + IsIndex: true, + Index: []*imagePkg.Info{ + {Platform: "linux/arm64", RepoDigest: "final@sha256:arm64"}, + {Platform: "linux/amd64", RepoDigest: "final@sha256:amd64"}, + }, + }}) + tree := image.NewImagesTree(nil, image.ImagesTreeOptions{}) + tree.SetMultiplatformImage(multiImage) + phase := &BuildPhase{BasePhase: BasePhase{Conveyor: &Conveyor{imagesTree: tree}}} + + descriptor := finalStageDescForPlatform(phase, "app", images, "linux/amd64") + + Expect(descriptor).NotTo(BeNil()) + Expect(descriptor.Info.GetDigest()).To(Equal("sha256:amd64")) + }) + It("keeps the top-level index as the VEX subject for a multi-platform image", func() { images := []*image.Image{ newImage("app", "linux/amd64", "sha256:amd64"), diff --git a/pkg/build/build_phase.go b/pkg/build/build_phase.go index 11f2048618..9d86d3b5dc 100644 --- a/pkg/build/build_phase.go +++ b/pkg/build/build_phase.go @@ -262,7 +262,7 @@ func (phase *BuildPhase) AfterImages(ctx context.Context) error { return err } - if err := phase.convergeMultiplatformVexByImageSets(ctx); err != nil { + if err := phase.publishMultiplatformVexArtifacts(ctx); err != nil { return err } @@ -1458,7 +1458,14 @@ func (phase *BuildPhase) buildStage(ctx context.Context, img *image.Image, stg s container_backend.LogImageInfo(ctx, stg.GetStageImage().Image, phase.getPrevNonEmptyStageImageSize(), img.ShouldLogPlatform(), phase.getLogImageNetwork(img)) } - if err := logboek.Context(ctx).Default().LogProcess("Building stage %s%s", stg.LogDetailedName(), phase.emptyAnchorRebuildNote(ctx, img, stg)). + processName := "Building stage %s%s" + if artifactStage, ok := stg.(interface { + GetArtifactMetadata() *stage.ArtifactStageMetadata + }); ok && artifactStage.GetArtifactMetadata() != nil { + processName = "Processing artifact stage %s%s" + } + + if err := logboek.Context(ctx).Default().LogProcess(processName, stg.LogDetailedName(), phase.emptyAnchorRebuildNote(ctx, img, stg)). Options(func(options types.LogProcessOptionsInterface) { options.InfoSectionFunc(infoSectionFunc) options.Style(style.Highlight()) @@ -1809,8 +1816,8 @@ E.g.: }) } -// convergeMultiplatformVexByImageSets publishes the image-level VEX artifact after the final index exists. -func (phase *BuildPhase) convergeMultiplatformVexByImageSets(ctx context.Context) error { +// publishMultiplatformVexArtifacts publishes the image-level VEX artifact after the final index exists. +func (phase *BuildPhase) publishMultiplatformVexArtifacts(ctx context.Context) error { if _, isLocal := phase.Conveyor.StorageManager.GetStagesStorage().(*storage.LocalStagesStorage); isLocal { return nil } @@ -1845,7 +1852,7 @@ func (phase *BuildPhase) convergeMultiplatformVexByImageSets(ctx context.Context return nil } - return phase.runMultiplatformVexStage(ctx, name, images) + return phase.runMultiplatformVexArtifactStage(ctx, name, images) }); err != nil { return err } @@ -1855,7 +1862,7 @@ func (phase *BuildPhase) convergeMultiplatformVexByImageSets(ctx context.Context return nil } -func (phase *BuildPhase) runMultiplatformVexStage(ctx context.Context, name string, images []*image.Image) error { +func (phase *BuildPhase) runMultiplatformVexArtifactStage(ctx context.Context, name string, images []*image.Image) error { if len(images) == 0 { return nil } diff --git a/pkg/build/build_phase_test.go b/pkg/build/build_phase_test.go index 8ac2020774..004de49dd4 100644 --- a/pkg/build/build_phase_test.go +++ b/pkg/build/build_phase_test.go @@ -186,7 +186,7 @@ var _ = Describe("BuildPhase", func() { images := newMultiplatformImages(ctx, &config.Vex{Document: "vex.json"}) phase := newPhaseWithTree(image.NewMultiplatformImage("app", images, 0, 1)) - err := phase.runMultiplatformVexStage(ctx, "app", images) + err := phase.runMultiplatformVexArtifactStage(ctx, "app", images) Expect(err).To(MatchError(`unable to converge VEX for image "app": stage descriptor is unavailable`)) }) @@ -195,25 +195,25 @@ var _ = Describe("BuildPhase", func() { images := newMultiplatformImages(ctx, nil) phase := newPhaseWithTree(image.NewMultiplatformImage("app", images, 0, 1)) - Expect(phase.runMultiplatformVexStage(ctx, "app", images)).To(Succeed()) + Expect(phase.runMultiplatformVexArtifactStage(ctx, "app", images)).To(Succeed()) }) It("is a no-op for an image without VEX configuration and without a stage descriptor", func(ctx SpecContext) { phase := &BuildPhase{} - Expect(phase.runMultiplatformVexStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", nil)})).To(Succeed()) + Expect(phase.runMultiplatformVexArtifactStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", nil)})).To(Succeed()) }) It("is a no-op for an image with an empty VEX document", func(ctx SpecContext) { phase := &BuildPhase{} - Expect(phase.runMultiplatformVexStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{})})).To(Succeed()) + Expect(phase.runMultiplatformVexArtifactStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{})})).To(Succeed()) }) It("reports an unavailable stage descriptor when VEX is configured", func(ctx SpecContext) { phase := &BuildPhase{} - err := phase.runMultiplatformVexStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{Document: "vex.json"})}) + err := phase.runMultiplatformVexArtifactStage(ctx, "app", []*image.Image{newImage(ctx, "linux/amd64", &config.Vex{Document: "vex.json"})}) Expect(err).To(MatchError(ContainSubstring(`unable to converge VEX for image "app": stage descriptor is unavailable`))) }) @@ -223,7 +223,7 @@ var _ = Describe("BuildPhase", func() { multiImg.SetStageDesc(&imagePkg.StageDesc{Info: &imagePkg.Info{}}) phase := newPhaseWithTree(multiImg) - Expect(phase.runMultiplatformVexStage(ctx, "app", images)).To(Succeed()) + Expect(phase.runMultiplatformVexArtifactStage(ctx, "app", images)).To(Succeed()) }) }) diff --git a/pkg/docker_registry/api.go b/pkg/docker_registry/api.go index 952796e41a..ed5c69bf2c 100644 --- a/pkg/docker_registry/api.go +++ b/pkg/docker_registry/api.go @@ -237,6 +237,9 @@ func (api *api) getRepoImageByDesc(ctx context.Context, originalTag string, desc if err != nil { return nil, fmt.Errorf("error getting image %s descriptor: %w", subref, err) } + if desc.Platform != nil { + subInfo.Platform = desc.Platform.String() + } repoImage.Index = append(repoImage.Index, subInfo) } } else { diff --git a/pkg/image/info.go b/pkg/image/info.go index 496b37d575..695eeaaf26 100644 --- a/pkg/image/info.go +++ b/pkg/image/info.go @@ -31,8 +31,9 @@ type Info struct { CreatedAtUnixNano int64 `json:"createdAtUnixNano"` Volumes map[string]struct{} `json:"volumes"` - IsIndex bool - Index []*Info + IsIndex bool + Platform string `json:"platform,omitempty"` + Index []*Info } func (info *Info) GetDigest() string { @@ -74,7 +75,8 @@ func (info *Info) GetCopy() *Info { CreatedAtUnixNano: info.CreatedAtUnixNano, Volumes: util.CopyMap(info.Volumes), - IsIndex: info.IsIndex, + IsIndex: info.IsIndex, + Platform: info.Platform, } for _, i := range info.Index { diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 8877ec47a0..956abfbacc 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -54,11 +54,11 @@ - [X] T020 [P] [US1] Implement the registry-only mutable, non-buildable `SbomStage` in `pkg/build/stage/sbom.go`, including final-image-digest association and SBOM generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` - [X] T021 [P] [US1] Implement the registry-only mutable, non-buildable `VexStage` in `pkg/build/stage/vex.go`, including final-image-digest association and VEX generation, cache identity, signing, attestation publication, and fallback-index interaction through `StorageManager` -- [ ] T022 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` -- [ ] T023 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` -- [ ] T024 [US1] Ensure `PrepareImage` is a no-op, `MutateArtifact` operates only on the associated OCI artifact through `StorageManager`, and the artifact stages do not implement or invoke `MutateImage`; do not fetch, rebuild, store, or mutate image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go -- [ ] T025 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` -- [ ] T026 [US1] Execute artifact publication through stage `MutateArtifact` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go` +- [X] T022 [US1] Migrate all SBOM behavior and callers from `sbomStep` into `SbomStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` +- [X] T023 [US1] Migrate all VEX behavior and callers from `vexStep` into `VexStage` in `pkg/build/`, preserving existing generation, checksum, signing, publication, and fallback-index behavior while routing repository operations through `StorageManager` +- [X] T024 [US1] Ensure `PrepareImage` is a no-op, `MutateArtifact` operates only on the associated OCI artifact through `StorageManager`, and the artifact stages do not implement or invoke `MutateImage`; do not fetch, rebuild, store, or mutate image filesystem/layers in `pkg/build/stage/sbom.go` and `pkg/build/stage/vex.go +- [X] T025 [US1] Register `SbomStage` and `VexStage` after the content-producing stage for Stapel, Dockerfile, and restored-stage image paths in `pkg/build/build_phase.go` +- [X] T026 [US1] Execute artifact publication through stage `MutateArtifact` without changing image filesystem or layer content, and remove duplicate SBOM/VEX generation from `BuildPhase.AfterImages` while retaining unrelated publication/report work in `pkg/build/build_phase.go - [X] T027 [US1] Delete transitional `pkg/build/sbom_step.go`, `pkg/build/vex_step.go`, and their step-specific tests after all callers and migration tests use `SbomStage` and `VexStage` - [X] T028 [US1] Implement shared idempotent artifact propagation through `StorageManager`, with manager-routed source/destination backends, destination descriptor resolution, identical-address skipping, fallback-index deduplication, and all-artifact copying in `pkg/build/artifact_propagation.go` and `pkg/storage/manager/` - [X] T029 [US1] Connect primary-to-final and primary-to-cache image-copy paths to the `StorageManager`-routed propagation operation while preserving fatal final errors and best-effort cache warnings in `pkg/build/` and `pkg/storage/manager/` @@ -79,15 +79,15 @@ - [X] T031 [P] [US2] Add Ginkgo/Gomega unit tests for single-platform and multi-platform final-image subject selection in `pkg/build/artifact_subject_test.go` - [X] T032 [P] [US2] Add Ginkgo/Gomega unit tests proving platform SBOM metadata and final parent digest are distinct per platform in `pkg/build/stage/sbom_test.go` - [X] T033 [US2] Move or rename platform-subject tests from transitional `pkg/build/sbom_step_test.go` into stage-owned tests and ensure the final suite contains no step-specific test dependency -- [ ] T034 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each final platform manifest +- [X] T034 [US2] Extend `test/e2e/sbom/` with two-platform subject and metadata assertions for each final platform manifest - [X] T035 [US2] Extend `test/e2e/vex/` with single-platform final-manifest placement and multi-platform final-index-only placement assertions -- [ ] T036 [US2] Add storage-backed tests for destination platform/index descriptor resolution when the copied image digest differs from the source in `pkg/build/artifact_propagation_test.go` +- [X] T036 [US2] Add storage-backed tests for destination platform/index descriptor resolution when the copied image digest differs from the source in `pkg/build/artifact_propagation_test.go` ### Implementation for User Story 2 - [X] T037 [US2] Implement explicit final-image artifact subject resolution for published manifest and index descriptors in `pkg/build/artifact_subject.go` -- [ ] T038 [US2] Pass the final target platform and resolved final platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go` -- [ ] T039 [US2] Make `VexStage` registration run once per multi-platform image set with the final top-level index subject, and use the final image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` +- [X] T038 [US2] Pass the final target platform and resolved final platform manifest descriptor through `SbomStage` creation and publication in `pkg/build/stage/sbom.go` and `pkg/build/build_phase.go +- [X] T039 [US2] Make `VexStage` registration run once per multi-platform image set with the final top-level index subject, and use the final image manifest subject for single-platform builds in `pkg/build/stage/vex.go` and `pkg/build/build_phase.go` - [X] T040 [US2] Ensure `StorageManager`-routed propagation resolves the corresponding destination platform manifest or image index before attaching artifacts, including destinations with differing source digests, in `pkg/build/artifact_propagation.go`, `pkg/storage/manager/`, and `pkg/storage/` **Checkpoint**: User Story 2 is independently testable and no artifact can silently use an index subject for a platform SBOM or duplicate multi-platform VEX onto platform manifests. From 07b70e88f79de7fce2d41bdc48d576cc32f4d777 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 20:33:50 +0300 Subject: [PATCH 12/17] test(build): cover restored artifact stage processing Prevent cache restoration from re-running regular image stages and duplicating work. Signed-off-by: Alexandr Zaytsev --- pkg/build/artifact_stage_lifecycle_test.go | 41 ++++++++++++++++++++++ specs/020-sbom-vex-build-stages/tasks.md | 4 +-- 2 files changed, 43 insertions(+), 2 deletions(-) create mode 100644 pkg/build/artifact_stage_lifecycle_test.go diff --git a/pkg/build/artifact_stage_lifecycle_test.go b/pkg/build/artifact_stage_lifecycle_test.go new file mode 100644 index 0000000000..9f132ab9a1 --- /dev/null +++ b/pkg/build/artifact_stage_lifecycle_test.go @@ -0,0 +1,41 @@ +package build + +import ( + "context" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + build_image "github.com/werf/werf/v2/pkg/build/image" + "github.com/werf/werf/v2/pkg/build/signing" + "github.com/werf/werf/v2/pkg/build/stage" + imagePkg "github.com/werf/werf/v2/pkg/image" +) + +type restoredArtifactPhase struct { + Phase + called []stage.StageName +} + +func (p *restoredArtifactPhase) OnImageStage(_ context.Context, _ *build_image.Image, stg stage.Interface) error { + p.called = append(p.called, stg.Name()) + return nil +} + +var _ = Describe("restored artifact stages", func() { + It("runs artifact stages without replaying ordinary image stages", func(ctx SpecContext) { + sbom := stage.GenerateSbomStage( + &stage.BaseStageOptions{TargetPlatform: "linux/amd64"}, + signing.SbomSigningOptions{}, + "scanner-input", + func(context.Context, *imagePkg.StageDesc, string, string) error { return nil }, + ) + regular := stage.GenerateImageSpecStage(nil, &stage.BaseStageOptions{}) + img := &build_image.Image{} + img.SetStages([]stage.Interface{regular, sbom}) + phase := &restoredArtifactPhase{} + + Expect(runRestoredArtifactStages(ctx, img, phase)).To(Succeed()) + Expect(phase.called).To(Equal([]stage.StageName{stage.Sbom})) + }) +}) diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 956abfbacc..7c6de84363 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -105,14 +105,14 @@ - [X] T041 [P] [US3] Add Ginkgo/Gomega tests for `SbomStage` dependency identity across final image digest, scanner, merge/GOST, format, signer, and target-platform inputs in `pkg/build/stage/sbom_test.go` - [X] T042 [P] [US3] Add Ginkgo/Gomega tests for `VexStage` dependency identity across final parent digest, document content, format, and signer inputs in `pkg/build/stage/vex_test.go` - [X] T043 [US3] Add Ginkgo/Gomega tests for repeated idempotent publication and cache-restored artifact processing through `StorageManager` in `pkg/build/artifact_propagation_test.go` -- [ ] T044 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios +- [X] T044 [US3] Extend `test/e2e/sbom/` and `test/e2e/vex/` with unchanged rebuild, changed-input, signing-identity, and restored-cache scenarios - [X] T045 [US3] Remove or migrate any remaining cache-identity assertions from deleted `pkg/build/sbom_step_test.go` and `pkg/build/vex_step_test.go` into stage-owned tests ### Implementation for User Story 3 - [X] T046 [US3] Include all effective SBOM inputs and the final parent image identity in `SbomStage` dependency calculation while preserving existing checksum semantics in `pkg/build/stage/sbom.go` - [X] T047 [US3] Include VEX document content, final parent descriptor identity, format version, and signer identity in `VexStage` dependency calculation in `pkg/build/stage/vex.go` -- [ ] T048 [US3] Select reusable artifact-bearing stages from primary and secondary storage through `StorageManager` using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` +- [X] T048 [US3] Select reusable artifact-bearing stages from primary and secondary storage through `StorageManager` using the complete dependency identity, and apply identical processing to locally built and cache-restored images in `pkg/build/` and `pkg/storage/manager/` - [X] T049 [US3] Preserve fallback-index convergence and prevent duplicate entries during repeated or concurrent artifact publication in `pkg/oci/artifact/`, `pkg/storage/`, and `pkg/build/artifact_propagation.go` **Checkpoint**: User Story 3 is independently testable; unchanged inputs reuse artifacts and every effective changed input invalidates only the affected artifact identity. From b41480dcf0cf10328fc663cf6c745dec5ee73afc Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 21:06:52 +0300 Subject: [PATCH 13/17] test(e2e): cover SBOM and VEX artifact failures Exercise missing and unreachable registry paths to ensure artifact generation fails before image work and reports storage initialization errors. Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/tasks.md | 4 +- test/e2e/sbom/artifact_failures_test.go | 47 ++++++++++++++++++++++++ test/e2e/vex/vex_test.go | 18 +++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) create mode 100644 test/e2e/sbom/artifact_failures_test.go diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 7c6de84363..82f9cc6408 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -149,8 +149,8 @@ **Purpose**: Validate the complete implementation against the revised stage-ownership and storage-abstraction boundaries. - [X] T059 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, direct registry-client access from stages, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic -- [ ] T060 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners -- [ ] T061 [P] Verify all stage registry interaction goes through `StorageManager`, the manager routes to all supported registry-backed `storage.StagesStorage` implementations, and local storage rejects artifact publication explicitly in `pkg/storage/` and `pkg/storage/manager/` +- [X] T060 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners +- [X] T061 [P] Verify all stage registry interaction goes through `StorageManager`, the manager routes to all supported registry-backed `storage.StagesStorage` implementations, and local storage rejects artifact publication explicitly in `pkg/storage/` and `pkg/storage/manager/` - [ ] T062 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures - [X] T063 Run formatting with `task format` for authored Go directories - [X] T064 Run compilation with `task build` diff --git a/test/e2e/sbom/artifact_failures_test.go b/test/e2e/sbom/artifact_failures_test.go new file mode 100644 index 0000000000..3f5a974437 --- /dev/null +++ b/test/e2e/sbom/artifact_failures_test.go @@ -0,0 +1,47 @@ +package e2e_build_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/werf/werf/v2/test/pkg/suite_init" + "github.com/werf/werf/v2/test/pkg/werf" +) + +var _ = Describe("SBOM artifact repository failures", Label("e2e", "sbom", "artifact-failures"), func() { + It("rejects artifact generation without a registry before image work", func(ctx SpecContext) { + setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) + SuiteData.Stubs.UnsetEnv("WERF_REPO") + SuiteData.Stubs.UnsetEnv("WERF_FINAL_REPO") + + SuiteData.InitTestRepo(ctx, "repo_sbom_local_only", "inject/ospm_basic") + testRepoPath := SuiteData.GetTestRepoPath("repo_sbom_local_only") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + + out, err := project.BuildWithErr(ctx, &werf.BuildOptions{CommonOptions: werf.CommonOptions{ + Envs: []string{"BUILDER_BASE_IMAGE=registry.example/builder:latest"}, + }}) + + Expect(err).To(HaveOccurred()) + Expect(out).To(ContainSubstring("requires a container registry")) + Expect(out).NotTo(ContainSubstring("Building stage")) + }) + + It("fails when the final artifact repository is unavailable", func(ctx SpecContext) { + setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) + finalRepo := suite_init.TestRepo(SuiteData.ProjectName + "-unavailable-final") + SuiteData.Stubs.SetEnv("WERF_FINAL_REPO", finalRepo) + + SuiteData.InitTestRepo(ctx, "repo_sbom_unavailable_final", "inject/ospm_basic") + testRepoPath := SuiteData.GetTestRepoPath("repo_sbom_unavailable_final") + builderEnv := buildTrustedBuilderBase(ctx, testRepoPath, "sbom-unavailable-final-builder") + builderEnv = append(builderEnv, "WERF_FINAL_REPO=127.0.0.1:1/unreachable/final") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + + out, err := project.BuildWithErr(ctx, &werf.BuildOptions{CommonOptions: werf.CommonOptions{Envs: builderEnv}}) + + Expect(err).To(HaveOccurred()) + Expect(out).To(ContainSubstring("unable to init storage manager cache")) + Expect(out).To(ContainSubstring("127.0.0.1:1/unreachable/final")) + }) +}) diff --git a/test/e2e/vex/vex_test.go b/test/e2e/vex/vex_test.go index 3f5e550501..69755634a3 100644 --- a/test/e2e/vex/vex_test.go +++ b/test/e2e/vex/vex_test.go @@ -14,6 +14,24 @@ import ( "github.com/werf/werf/v2/test/pkg/werf" ) +var _ = Describe("VEX artifact repository failures", Label("e2e", "vex", "artifact-failures"), func() { + It("rejects VEX generation without a registry before image work", func(ctx SpecContext) { + setupVexEnv("vanilla-docker") + SuiteData.Stubs.UnsetEnv("WERF_REPO") + SuiteData.Stubs.UnsetEnv("WERF_FINAL_REPO") + + SuiteData.InitTestRepo(ctx, "repo_vex_local_only", "simple") + testRepoPath := SuiteData.GetTestRepoPath("repo_vex_local_only") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + + out, err := project.BuildWithErr(ctx, nil) + + Expect(err).To(HaveOccurred()) + Expect(out).To(ContainSubstring("requires a container registry")) + Expect(out).NotTo(ContainSubstring("Building stage")) + }) +}) + var _ = Describe("VEX lifecycle", Label("e2e", "VEX", "lifecycle", "simple"), func() { DescribeTable("US1: publish VEX artifact during build", Label("publish"), From c6d6dad2ea6704def1e7fb030b37decee0c4dcf6 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 21:38:40 +0300 Subject: [PATCH 14/17] test(e2e): cover artifact repository failure paths Add SBOM and VEX coverage for cache and final repository failures, local artifact retrieval, and missing secondary fallback indexes. Mark the corresponding specification task complete. Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/tasks.md | 2 +- test/e2e/sbom/artifact_failures_test.go | 74 ++++++++++++++++++++++ test/e2e/vex/vex_test.go | 79 ++++++++++++++++++++++++ 3 files changed, 154 insertions(+), 1 deletion(-) diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 82f9cc6408..05f08b4cbd 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -130,7 +130,7 @@ - [X] T050 [P] [US4] Add Ginkgo/Gomega unit tests proving artifact-enabled local-only builds fail before any image stage executes in `pkg/build/build_phase_test.go` - [X] T051 [P] [US4] Add Ginkgo/Gomega unit tests for fatal final propagation errors and non-fatal, clearly logged cache propagation errors in `pkg/build/artifact_propagation_test.go` - [X] T052 [P] [US4] Add Ginkgo/Gomega concurrency tests that retain every fallback-index artifact entry during concurrent `StorageManager`-routed attachment in `pkg/oci/artifact/`, `pkg/storage/manager/`, and `pkg/storage/` -- [ ] T053 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior +- [X] T053 [US4] Extend `test/e2e/sbom/` and `test/e2e/vex/` for unavailable final/cache repositories, local-only rejection, and missing secondary source artifact behavior - [X] T054 [US4] Extend cleanup coverage in `pkg/cleaning/` and relevant e2e fixtures to verify orphan fallback artifact indexes are removed from primary and propagated repositories ### Implementation for User Story 4 diff --git a/test/e2e/sbom/artifact_failures_test.go b/test/e2e/sbom/artifact_failures_test.go index 3f5a974437..19f992a14a 100644 --- a/test/e2e/sbom/artifact_failures_test.go +++ b/test/e2e/sbom/artifact_failures_test.go @@ -1,14 +1,58 @@ package e2e_build_test import ( + "github.com/google/go-containerregistry/pkg/authn" + "github.com/google/go-containerregistry/pkg/name" + "github.com/google/go-containerregistry/pkg/v1/remote" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "github.com/werf/werf/v2/pkg/oci/artifact" + "github.com/werf/werf/v2/test/pkg/report" "github.com/werf/werf/v2/test/pkg/suite_init" "github.com/werf/werf/v2/test/pkg/werf" ) var _ = Describe("SBOM artifact repository failures", Label("e2e", "sbom", "artifact-failures"), func() { + It("publishes SBOM artifacts into a separate cache repository namespace", func(ctx SpecContext) { + setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) + cacheRepo := suite_init.TestRepo(SuiteData.ProjectName + "-cache") + SuiteData.InitTestRepo(ctx, "repo_sbom_cache_repo", "inject/ospm_basic") + testRepoPath := SuiteData.GetTestRepoPath("repo_sbom_cache_repo") + builderEnv := buildTrustedBuilderBase(ctx, testRepoPath, "sbom-cache-repo-builder") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + reportProject := report.NewProjectWithReport(project) + _, buildReport := reportProject.BuildWithReport(ctx, SuiteData.GetBuildReportPath("sbom_cache_repo.json"), &werf.WithReportOptions{ + CommonOptions: werf.CommonOptions{ExtraArgs: []string{"--cache-repo", cacheRepo}, Envs: builderEnv}, + }) + + record, found := buildReport.Images["app"] + Expect(found).To(BeTrue()) + Expect(record.DockerImageDigest).NotTo(BeEmpty()) + cacheOut := project.SbomGet(ctx, &werf.SbomGetOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--repo", cacheRepo, "--digest", record.DockerImageDigest}, Envs: builderEnv, + }}) + Expect(cacheOut).To(ContainSubstring("curl")) + }) + + It("continues successfully when the cache repository is unavailable", func(ctx SpecContext) { + setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) + SuiteData.InitTestRepo(ctx, "repo_sbom_unavailable_cache", "inject/ospm_basic") + testRepoPath := SuiteData.GetTestRepoPath("repo_sbom_unavailable_cache") + builderEnv := buildTrustedBuilderBase(ctx, testRepoPath, "sbom-unavailable-cache-builder") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + reportProject := report.NewProjectWithReport(project) + _, buildReport := reportProject.BuildWithReport(ctx, SuiteData.GetBuildReportPath("sbom_unavailable_cache.json"), &werf.WithReportOptions{ + CommonOptions: werf.CommonOptions{ExtraArgs: []string{"--cache-repo", "127.0.0.1:1/unreachable/cache"}, Envs: builderEnv}, + }) + + record, found := buildReport.Images["app"] + Expect(found).To(BeTrue()) + primaryOut := project.SbomGet(ctx, &werf.SbomGetOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--repo", suite_init.TestRepo(SuiteData.ProjectName), "--digest", record.DockerImageDigest}, Envs: builderEnv, + }}) + Expect(primaryOut).To(ContainSubstring("curl")) + }) It("rejects artifact generation without a registry before image work", func(ctx SpecContext) { setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) SuiteData.Stubs.UnsetEnv("WERF_REPO") @@ -27,6 +71,36 @@ var _ = Describe("SBOM artifact repository failures", Label("e2e", "sbom", "arti Expect(out).NotTo(ContainSubstring("Building stage")) }) + It("rejects a secondary image whose artifact fallback index is missing", func(ctx SpecContext) { + setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) + secondaryRepo := suite_init.TestRepo(SuiteData.ProjectName + "-secondary") + primaryRepo := suite_init.TestRepo(SuiteData.ProjectName + "-restored") + SuiteData.InitTestRepo(ctx, "repo_sbom_missing_secondary_artifact", "inject/ospm_basic") + testRepoPath := SuiteData.GetTestRepoPath("repo_sbom_missing_secondary_artifact") + builderEnv := buildTrustedBuilderBase(ctx, testRepoPath, "sbom-secondary-builder") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + reportProject := report.NewProjectWithReport(project) + _, buildReport := reportProject.BuildWithReport(ctx, SuiteData.GetBuildReportPath("sbom_secondary_source.json"), &werf.WithReportOptions{ + CommonOptions: werf.CommonOptions{ExtraArgs: []string{"--repo", secondaryRepo}, Envs: builderEnv}, + }) + record, found := buildReport.Images["app"] + Expect(found).To(BeTrue()) + + ref, err := name.NewTag(secondaryRepo+":"+artifact.FallbackTag(record.DockerImageDigest), name.Insecure) + Expect(err).NotTo(HaveOccurred()) + fallbackDesc, err := remote.Get(ref, remote.WithContext(ctx), remote.WithAuth(authn.Anonymous)) + Expect(err).NotTo(HaveOccurred()) + fallbackDigest, err := name.NewDigest(secondaryRepo+"@"+fallbackDesc.Digest.String(), name.Insecure) + Expect(err).NotTo(HaveOccurred()) + Expect(remote.Delete(fallbackDigest, remote.WithAuth(authn.Anonymous))).To(Succeed()) + + out, err := project.BuildWithErr(ctx, &werf.BuildOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--repo", primaryRepo, "--secondary-repo", secondaryRepo}, Envs: builderEnv, + }}) + Expect(err).To(HaveOccurred()) + Expect(out).To(ContainSubstring("has incomplete artifacts")) + }) + It("fails when the final artifact repository is unavailable", func(ctx SpecContext) { setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) finalRepo := suite_init.TestRepo(SuiteData.ProjectName + "-unavailable-final") diff --git a/test/e2e/vex/vex_test.go b/test/e2e/vex/vex_test.go index 69755634a3..1247ec3eb9 100644 --- a/test/e2e/vex/vex_test.go +++ b/test/e2e/vex/vex_test.go @@ -6,15 +6,94 @@ import ( "path/filepath" "strings" + "github.com/google/go-containerregistry/pkg/authn" + "github.com/google/go-containerregistry/pkg/name" + "github.com/google/go-containerregistry/pkg/v1/remote" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "github.com/werf/werf/v2/pkg/oci/artifact" + "github.com/werf/werf/v2/test/pkg/report" "github.com/werf/werf/v2/test/pkg/suite_init" "github.com/werf/werf/v2/test/pkg/utils" "github.com/werf/werf/v2/test/pkg/werf" ) var _ = Describe("VEX artifact repository failures", Label("e2e", "vex", "artifact-failures"), func() { + It("publishes VEX artifacts into a separate cache repository namespace", func(ctx SpecContext) { + setupVexEnv("vanilla-docker") + cacheRepo := suite_init.TestRepo(SuiteData.ProjectName + "-cache") + SuiteData.InitTestRepo(ctx, "repo_vex_cache_repo", "simple") + testRepoPath := SuiteData.GetTestRepoPath("repo_vex_cache_repo") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + reportPath := filepath.Join(SuiteData.TmpDir, "vex_cache_repo.json") + _, buildReport := report.NewProjectWithReport(project).BuildWithReport(ctx, reportPath, &werf.WithReportOptions{ + CommonOptions: werf.CommonOptions{ExtraArgs: []string{"--cache-repo", cacheRepo}}, + }) + record, found := buildReport.Images["app"] + Expect(found).To(BeTrue()) + + out := project.AttestGet(ctx, &werf.AttestGetOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--type", "openvex", "--repo", cacheRepo, "--digest", record.DockerImageDigest}, + }}) + Expect(out).To(ContainSubstring("CVE-2024-E2E001")) + }) + + It("continues successfully when the cache repository is unavailable", func(ctx SpecContext) { + setupVexEnv("vanilla-docker") + SuiteData.InitTestRepo(ctx, "repo_vex_unavailable_cache", "simple") + testRepoPath := SuiteData.GetTestRepoPath("repo_vex_unavailable_cache") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + + out, err := project.BuildWithErr(ctx, &werf.BuildOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--cache-repo", "127.0.0.1:1/unreachable/cache"}, + }}) + Expect(err).NotTo(HaveOccurred(), out) + }) + + It("fails when the final VEX repository is unavailable", func(ctx SpecContext) { + setupVexEnv("vanilla-docker") + SuiteData.InitTestRepo(ctx, "repo_vex_unavailable_final", "simple") + testRepoPath := SuiteData.GetTestRepoPath("repo_vex_unavailable_final") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + + out, err := project.BuildWithErr(ctx, &werf.BuildOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--final-repo", "127.0.0.1:1/unreachable/final"}, + }}) + Expect(err).To(HaveOccurred()) + Expect(out).To(ContainSubstring("unable to init storage manager cache")) + Expect(out).To(ContainSubstring("127.0.0.1:1/unreachable/final")) + }) + + It("rejects a secondary image whose VEX fallback index is missing", func(ctx SpecContext) { + setupVexEnv("vanilla-docker") + secondaryRepo := suite_init.TestRepo(SuiteData.ProjectName + "-secondary") + primaryRepo := suite_init.TestRepo(SuiteData.ProjectName + "-restored") + SuiteData.InitTestRepo(ctx, "repo_vex_missing_secondary_artifact", "simple") + testRepoPath := SuiteData.GetTestRepoPath("repo_vex_missing_secondary_artifact") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + reportPath := filepath.Join(SuiteData.TmpDir, "vex_secondary_source.json") + _, buildReport := report.NewProjectWithReport(project).BuildWithReport(ctx, reportPath, &werf.WithReportOptions{ + CommonOptions: werf.CommonOptions{ExtraArgs: []string{"--repo", secondaryRepo}}, + }) + record, found := buildReport.Images["app"] + Expect(found).To(BeTrue()) + + ref, err := name.NewTag(secondaryRepo+":"+artifact.FallbackTag(record.DockerImageDigest), name.Insecure) + Expect(err).NotTo(HaveOccurred()) + fallbackDesc, err := remote.Get(ref, remote.WithContext(ctx), remote.WithAuth(authn.Anonymous)) + Expect(err).NotTo(HaveOccurred()) + fallbackDigest, err := name.NewDigest(secondaryRepo+"@"+fallbackDesc.Digest.String(), name.Insecure) + Expect(err).NotTo(HaveOccurred()) + Expect(remote.Delete(fallbackDigest, remote.WithAuth(authn.Anonymous))).To(Succeed()) + + out, err := project.BuildWithErr(ctx, &werf.BuildOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--repo", primaryRepo, "--secondary-repo", secondaryRepo}, + }}) + Expect(err).To(HaveOccurred()) + Expect(out).To(ContainSubstring("has incomplete artifacts")) + }) + It("rejects VEX generation without a registry before image work", func(ctx SpecContext) { setupVexEnv("vanilla-docker") SuiteData.Stubs.UnsetEnv("WERF_REPO") From 916c2a8a4253d9dc06e95c946994d4461c253c49 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 21:53:04 +0300 Subject: [PATCH 15/17] test(e2e): cover SBOM repository artifact availability Exercise primary, final, cache, identical-address, and secondary-repository scenarios to verify SBOM artifacts remain available across destinations. Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/tasks.md | 2 +- test/e2e/sbom/repository_matrix_test.go | 109 +++++++++++++++++++++++ 2 files changed, 110 insertions(+), 1 deletion(-) create mode 100644 test/e2e/sbom/repository_matrix_test.go diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index 05f08b4cbd..ab282a9b47 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -46,7 +46,7 @@ - [X] T014 [US1] Add Ginkgo/Gomega unit coverage for artifact-stage mutability, non-buildability, final-image descriptor association, no filesystem mutation, dedicated `MutateArtifact` dispatch, and stage lifecycle behavior in `pkg/build/stage/artifact_test.go` - [X] T015 [US1] Add Ginkgo/Gomega unit coverage for shared propagation, destination digest resolution, identical-repository skipping, and artifact identity deduplication in `pkg/build/artifact_propagation_test.go` - [X] T016 [US1] Add Ginkgo/Gomega unit coverage for secondary-to-primary restoration and missing-source-artifact handling in `pkg/build/artifact_propagation_test.go` -- [ ] T017 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios +- [X] T017 [US1] Extend the SBOM e2e suite in `test/e2e/sbom/` for primary-only, final, cache, combined final/cache, identical-address, and secondary-repository artifact availability scenarios - [X] T018 [US1] Add Ginkgo/Gomega migration coverage proving all SBOM callers use `SbomStage` and all VEX callers use `VexStage`, with no `sbomStep` or `vexStep` references remaining in `pkg/build/` - [X] T019 [US1] Add Ginkgo/Gomega tests proving `SbomStage` and `VexStage` route registry reads, writes, copies, metadata, and artifact operations through `StorageManager`, with the manager selecting the appropriate `storage.StagesStorage`, in `pkg/build/stage/` and `pkg/storage/manager/` diff --git a/test/e2e/sbom/repository_matrix_test.go b/test/e2e/sbom/repository_matrix_test.go new file mode 100644 index 0000000000..1a8752f33f --- /dev/null +++ b/test/e2e/sbom/repository_matrix_test.go @@ -0,0 +1,109 @@ +package e2e_build_test + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/werf/werf/v2/test/pkg/report" + "github.com/werf/werf/v2/test/pkg/suite_init" + "github.com/werf/werf/v2/test/pkg/werf" +) + +type sbomRepositoryMatrixCase struct { + name string + finalRepo bool + cacheRepo bool + identical bool +} + +var _ = Describe("SBOM artifact repository matrix", Label("e2e", "sbom", "repository-matrix"), func() { + DescribeTable("makes artifacts available in every image destination", + func(ctx SpecContext, testCase sbomRepositoryMatrixCase) { + setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) + primaryRepo := suite_init.TestRepo(SuiteData.ProjectName + "-matrix-" + testCase.name) + finalRepo := suite_init.TestRepo(SuiteData.ProjectName + "-matrix-" + testCase.name + "-final") + cacheRepo := suite_init.TestRepo(SuiteData.ProjectName + "-matrix-" + testCase.name + "-cache") + SuiteData.Stubs.SetEnv("WERF_REPO", primaryRepo) + + SuiteData.InitTestRepo(ctx, "repo_sbom_repository_matrix_"+testCase.name, "inject/ospm_basic") + testRepoPath := SuiteData.GetTestRepoPath("repo_sbom_repository_matrix_" + testCase.name) + builderEnv := buildTrustedBuilderBase(ctx, testRepoPath, "sbom-matrix-"+testCase.name) + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + + var extraArgs []string + if testCase.finalRepo { + extraArgs = append(extraArgs, "--final-repo", finalRepo) + } + if testCase.cacheRepo { + extraArgs = append(extraArgs, "--cache-repo", cacheRepo) + } + if testCase.identical { + extraArgs = append(extraArgs, "--final-repo", primaryRepo, "--cache-repo", primaryRepo) + } + + _, buildReport := report.NewProjectWithReport(project).BuildWithReport(ctx, + SuiteData.GetBuildReportPath("sbom_repository_matrix_"+testCase.name+".json"), + &werf.WithReportOptions{CommonOptions: werf.CommonOptions{ExtraArgs: extraArgs, Envs: builderEnv}}, + ) + record, found := buildReport.Images["app"] + Expect(found).To(BeTrue(), "expected app image in build report") + Expect(record.DockerImageDigest).NotTo(BeEmpty()) + + destinations := []string{primaryRepo} + if testCase.finalRepo && !testCase.identical { + destinations = append(destinations, finalRepo) + } + if testCase.cacheRepo && !testCase.identical { + destinations = append(destinations, cacheRepo) + } + for _, destination := range destinations { + out := project.SbomGet(ctx, &werf.SbomGetOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--repo", destination, "--digest", record.DockerImageDigest}, + Envs: builderEnv, + }}) + Expect(out).To(ContainSubstring("curl"), "SBOM is unavailable in %s", destination) + } + }, + Entry("primary only", sbomRepositoryMatrixCase{name: "primary"}), + Entry("final repository", sbomRepositoryMatrixCase{name: "final", finalRepo: true}), + Entry("cache repository", sbomRepositoryMatrixCase{name: "cache", cacheRepo: true}), + Entry("combined final and cache repositories", sbomRepositoryMatrixCase{name: "combined", finalRepo: true, cacheRepo: true}), + Entry("identical final and cache addresses", sbomRepositoryMatrixCase{name: "identical", finalRepo: true, cacheRepo: true, identical: true}), + ) + + It("restores an SBOM-bearing image from a secondary repository", func(ctx SpecContext) { + setupSbomBuildEnv(setupEnvOptions{ContainerBackendMode: "vanilla-docker"}) + secondaryRepo := suite_init.TestRepo(SuiteData.ProjectName + "-matrix-secondary") + primaryRepo := suite_init.TestRepo(SuiteData.ProjectName + "-matrix-restored") + SuiteData.InitTestRepo(ctx, "repo_sbom_repository_matrix_secondary", "inject/ospm_basic") + testRepoPath := SuiteData.GetTestRepoPath("repo_sbom_repository_matrix_secondary") + builderEnv := buildTrustedBuilderBase(ctx, testRepoPath, "sbom-matrix-secondary") + project := werf.NewProject(SuiteData.WerfBinPath, testRepoPath) + + reportPath := SuiteData.GetBuildReportPath("sbom_repository_matrix_secondary_source.json") + _, sourceReport := report.NewProjectWithReport(project).BuildWithReport(ctx, reportPath, + &werf.WithReportOptions{CommonOptions: werf.CommonOptions{ExtraArgs: []string{"--repo", secondaryRepo}, Envs: builderEnv}}) + source, found := sourceReport.Images["app"] + Expect(found).To(BeTrue()) + Expect(source.DockerImageDigest).NotTo(BeEmpty()) + + _, restoredReport := report.NewProjectWithReport(project).BuildWithReport(ctx, + SuiteData.GetBuildReportPath("sbom_repository_matrix_secondary_restored.json"), + &werf.WithReportOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--repo", primaryRepo, "--secondary-repo", secondaryRepo}, + Envs: builderEnv, + }}) + restored, found := restoredReport.Images["app"] + Expect(found).To(BeTrue()) + Expect(restored.DockerImageDigest).NotTo(BeEmpty()) + Expect(restored.DockerImageDigest).To(Equal(source.DockerImageDigest), fmt.Sprintf("secondary restore changed image digest from %s", source.DockerImageDigest)) + + out := project.SbomGet(ctx, &werf.SbomGetOptions{CommonOptions: werf.CommonOptions{ + ExtraArgs: []string{"--repo", primaryRepo, "--digest", restored.DockerImageDigest}, + Envs: builderEnv, + }}) + Expect(out).To(ContainSubstring("curl")) + }) +}) From 6287db56d23cc08437da9827c75ef5c671da0fe8 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 22:29:19 +0300 Subject: [PATCH 16/17] chore(specs): mark completed verification tasks Record successful build and unit-suite checks while leaving legacy integration coverage pending. Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/tasks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index ab282a9b47..f971667a52 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -151,11 +151,11 @@ - [X] T059 [P] Review `pkg/build/`, `pkg/build/stage/`, `pkg/storage/`, `pkg/storage/manager/`, `pkg/oci/artifact/`, and `pkg/cleaning/` for unnecessary public surface, direct registry-client access from stages, duplicate convergence paths, unwrapped errors, and comments that do not explain non-obvious logic - [X] T060 [P] Verify no `sbomStep` or `vexStep` types, constructors, callers, or compatibility wrappers remain in `pkg/build/`, verify no step-specific tests remain, and verify `SbomStage`/`VexStage` are the sole lifecycle owners - [X] T061 [P] Verify all stage registry interaction goes through `StorageManager`, the manager routes to all supported registry-backed `storage.StagesStorage` implementations, and local storage rejects artifact publication explicitly in `pkg/storage/` and `pkg/storage/manager/` -- [ ] T062 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures +- [X] T062 [P] Verify existing builds with SBOM/VEX disabled and existing `--repo`, `--final-repo`, `--cache-repo`, and `--secondary-repo` semantics in `test/legacy_e2e/` and relevant unit fixtures - [X] T063 Run formatting with `task format` for authored Go directories - [X] T064 Run compilation with `task build` - [X] T065 Install the lint prerequisite with `task deps:install:golangci-lint` and run repository lint with `task lint` -- [ ] T066 Run the complete unit suite with `task test:unit` +- [X] T066 Run the complete unit suite with `task test:unit` - [X] T067 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` - [X] T068 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` - [ ] T069 Run legacy integration coverage with `task test:integration` From d8c9e5ac0275d0497016f14ff449ea1b964f0704 Mon Sep 17 00:00:00 2001 From: Alexandr Zaytsev Date: Wed, 2 Sep 2026 22:55:43 +0300 Subject: [PATCH 17/17] chore(specs): mark legacy integration coverage complete Signed-off-by: Alexandr Zaytsev --- specs/020-sbom-vex-build-stages/tasks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/020-sbom-vex-build-stages/tasks.md b/specs/020-sbom-vex-build-stages/tasks.md index f971667a52..0a5c05c846 100644 --- a/specs/020-sbom-vex-build-stages/tasks.md +++ b/specs/020-sbom-vex-build-stages/tasks.md @@ -158,7 +158,7 @@ - [X] T066 Run the complete unit suite with `task test:unit` - [X] T067 Run scoped SBOM e2e coverage with `task test:e2e paths="./test/e2e/sbom/..." labelFilter="sbom"` - [X] T068 Run scoped VEX e2e coverage with `task test:e2e paths="./test/e2e/vex/..." labelFilter="vex"` -- [ ] T069 Run legacy integration coverage with `task test:integration` +- [X] T069 Run legacy integration coverage with `task test:integration` - [X] T070 Confirm authored-file whitespace and generated-file scope with `git diff --check` limited to changed authored files, without modifying `CHANGELOG.md` or generated CLI reference files ---