Skip to content

fix(sbom): stop sbom validate failing on checker warnings - #331

Draft
reyreavman wants to merge 3 commits into
mainfrom
fix/sbom/validate-warnings-non-fatal
Draft

reyreavman wants to merge 3 commits into
mainfrom
fix/sbom/validate-warnings-non-fatal

Conversation

@reyreavman

@reyreavman reyreavman commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

werf sbom validate no longer fails on sbom-checker warnings: they are printed and counted, but only errors set a non-zero exit code. With the current pinned checker image this is invisible (it emits no purl warnings); with the current upstream ISPRAS checker a production OSS SBOM yields dozens of WARNING: pkg:golang/github.com/Azure/... не подходит под спецификацию purl lines, and today every one of them fails the run.

What

  • BREAKING: the default flips from strict to lenient — a pipeline that relied on sbom validate exiting 1 on WARNING lines (with the pinned image those come only from --check-vcs) now passes; the way back is --fail-on-warnings.
  • A file whose checker output has WARNING lines and no ERROR lines is reported as (i/N) file.json... OK (M warning(s)) and the run exits 0.
  • New flag --fail-on-warnings (default false): with it, such a file is reported FAILED, its warnings are listed in the error, and the run exits 1.
  • A file with at least one ERROR line is FAILED and exits 1 regardless of the flag; its warnings are still printed under it, as before.
  • WARNING lines are emitted on stderr (the warning log stream); the (i/N) file... OK|FAILED header and ERROR lines stay on stdout, so sbom validate > out.txt no longer captures the warnings — use 2>&1.
  • The summary line becomes Result: N passed, M failed; X error(s), Y warning(s); previously it stopped at M failed.
  • The error returned for a failed file (validation failed for <file>: followed by the lines) is unchanged.
  • Usage docs (EN and RU) and the CLI reference describe the split and the flag.

Why

parseResult appended warnings to the same list as errors, so any warning marked the file FAILED. The checker's own --errors 0 already distinguishes the two classes; werf collapsed them. Left alone, bumping the checker image (tracked separately) immediately turns CI red for both oss and container formats on findings that are not blockers, while a team that does want hard acceptance had no way to opt in either — hence the flag rather than a silent behavior change alone.

Shipped as fix rather than a major bump: the strict default was never documented as a contract, the affected population is pipelines gating on --check-vcs warnings, and the opt-back is a single flag.

A --strictness=errors|all enum was the alternative; a boolean covers the only two levels the checker emits and leaves room to add the enum later if a third class appears.

Treat WARNING lines from sbom-checker as non-fatal: the file is reported
as "OK (N warning(s))", warnings go to the warning stream, and the exit
code stays 0. Only ERROR lines fail the run unless the new
--fail-on-warnings flag is passed. The summary now splits the counts:
"N passed, M failed; X error(s), Y warning(s)".

Previously parseResult put warnings into the same list as errors, so a
single warning marked the file FAILED and exited 1. The pinned checker
image does not validate purl yet, so nothing surfaced; the current
upstream does, and production OSS SBOMs produce dozens of purl-case
warnings that would turn both formats red as soon as the checker image
is bumped.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
State in the usage pages (EN and RU) that sbom validate reports errors
and warnings separately, that only errors fail the run, and that
--fail-on-warnings turns warnings into failures.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
@reyreavman

Copy link
Copy Markdown
Collaborator Author

Verification

  • Hand-run on macOS with Docker Desktop against test/e2e/sbom/_fixtures/validate: oss_multiple_vcs_urls.json alone → OK (1 warning(s)), exit 0; same file with --fail-on-warningsFAILED, exit 1; mixed run with missing_bom_format.jsonResult: 2 passed, 1 failed; 2 error(s), 1 warning(s), exit 1.
  • task test:e2e paths="./test/e2e/sbom" labelFilter="validate" parallel=1 on macOS/Docker: 31/31 passed.
  • Mutation: failed reverted to errs > 0 || warnings > 0 → unit fileResult.report / warnings alone pass by default failed.
  • Mutation: failOnWarnings ignored in fileResult.report → unit warnings alone fail with failOnWarnings failed.
  • Mutation: --fail-on-warnings not wired into checker.RunOptions → e2e should fail validation / warnings with --fail-on-warnings failed.
  • Not run: the card's acceptance check against the new upstream checker image (34 purl warnings on a real merged OSS SBOM) — that image is not pinned in this branch; covered by the fixture-based warning case instead.

Review focus

  • fileResult.report in pkg/sbom/checker/checker.go: the single condition deciding pass/fail.
  • Warning lines now go to logboek ... Warn(); confirm this stream is captured the way CI consumers expect (e2e asserts on combined output).

Follow-up

  • Merge before the checker image bump, otherwise the bump turns sbom validate red on purl-case warnings.

Replace the nested append with slices.Concat and make the warnings-only
e2e entry assert the "OK (1 warning(s))" line rather than just "WARNING",
so the per-file summary format is pinned.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
@reyreavman

Copy link
Copy Markdown
Collaborator Author

Verification

  • Hand-run on macOS with Docker Desktop against test/e2e/sbom/_fixtures/validate: oss_multiple_vcs_urls.json alone → OK (1 warning(s)), exit 0; same file with --fail-on-warningsFAILED, exit 1; mixed run with missing_bom_format.jsonResult: 2 passed, 1 failed; 2 error(s), 1 warning(s), exit 1.
  • Stream split: 2>/dev/null drops the WARNING lines and keeps the OK (1 warning(s)) header; 2>&1 keeps them in order.
  • task test:e2e paths="./test/e2e/sbom" labelFilter="validate" parallel=1 on macOS/Docker: 31/31 passed.
  • Mutation: failed reverted to errs > 0 || warnings > 0 → unit fileResult.report / warnings alone pass by default failed.
  • Mutation: failOnWarnings ignored in fileResult.report → unit warnings alone fail with failOnWarnings failed.
  • Mutation: --fail-on-warnings not wired into checker.RunOptions → e2e should fail validation / warnings with --fail-on-warnings failed.
  • Mutation: per-file line printed as plain OK without the warning count → e2e should pass validation / warnings only pass by default failed.
  • Not run: the card's acceptance check against the new upstream checker image (34 purl warnings on a real merged OSS SBOM) — that image is not pinned in this branch; covered by the fixture-based warning case instead.

Review focus

  • fileResult.report in pkg/sbom/checker/checker.go: the single condition deciding pass/fail.
  • qlty reports "1 blocking issue" on the PR while the status is green and golangci-lint is clean locally; the dashboard needs a login — someone with access should read what it flags before merge.

Follow-up

  • Merge before the checker image bump, otherwise the bump turns sbom validate red on purl-case warnings.
  • Decide whether the strict→lenient default flip warrants a BREAKING CHANGE: footer (major bump) or ships as fix — the description currently argues for fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant