Skip to content

test(conformance): report multipart regression details - #1551

Merged
overtrue merged 1 commit into
mainfrom
p1/1547-multipart-diagnostics
Oct 29, 2026
Merged

overtrue merged 1 commit into
mainfrom
p1/1547-multipart-diagnostics

Conversation

@overtrue

Copy link
Copy Markdown
Collaborator

Summary

Include each case's existing failing diagnostics in the multipart-family regression assertion. When details are absent, identify the missing diagnostic or the skip reason instead of reporting only a case ID. The baseline regression selection, case set, fixture load, assertions and deadlines are unchanged.

Related to #1547. This test-only diagnostic does not reproduce or fix the historical c-mpu-0015 failure. It is gateway-specific; no corresponding s3s change is identified.

Verification

On main 06af3805, with one Cargo build worker and incremental compilation disabled:

  • cargo fmt --all --check: exit 0.
  • cargo clippy --workspace --all-targets -- -D warnings: exit 0.
  • cargo test --workspace: exit 0.
  • cargo xtask verify --crate rustfs-gateway-conformance: exit 0 in 42.55 seconds wall time, with 24.67 seconds preparation and a 17.39-second checked loop. This exceeds the 30-second command target; see verify: conformance preparation exceeds command feedback budget #1504. An earlier base's complete invocation took 46.09 seconds.
  • The restored multipart-family module passed all 6 tests. The diagnostic control first failed against an empty implementation. Independent mutations of omitted failure, included warning, case ID, absent failure detail, skip reason and missing skip reason each made its assertion fail. An injected synthetic regression also made the existing family assertion print both the case ID and the injected diagnostic. This control is not the historical failure.

Role Verdicts

  • simplicity-adversary: attacked baseline selection, failure-only filtering, missing observations and skip reporting — no break found

Breaking Change

  • BREAKING — this PR touches a protected file or changes a public contract.

Checklist

  • The four-command gate passed (fmt, clippy, test, cargo xtask verify)
  • New or changed public API has rustdoc
  • Negative test cases outnumber positive ones
  • Every new assertion was mutated — the implementation was broken on purpose and the assertion
    went red. The PR description names which ones
  • No Protected File touched; if one was, the PR description contains BREAKING and a migration path
  • No unsafe introduced
  • No inventory / linkme introduced
  • No #[non_exhaustive] added to a DTO
  • No s3s code copied; behavioural evidence is URL + self-written summary
  • No credentials, signatures, or expected-signature values reachable from logs or error bodies
  • No plan document, analysis report, or working note committed
  • Commits follow Conventional Commits; PR title ≤72 characters
  • A Handoff comment has been appended to the issue

@overtrue

Copy link
Copy Markdown
Collaborator Author

Role review

  • simplicity-adversary: attacked the unchanged baseline regression selector, preservation of each failing diagnostic, exclusion of advisory warnings, explicit missing details and skipped-reason handling. The one local formatter gives the assertion independently testable output while leaving the report and its selection unchanged. No break found. The historical failure has no saved diagnostic, so this only improves future failures.

@overtrue

Copy link
Copy Markdown
Collaborator Author

CI evidence

Run 40962462759 on head d5a9c36c02784eefa29da846bbee26eb134b755f passed all 33 jobs. Test completed 381 seconds after run creation. This diagnostic does not reproduce or explain the historical c-mpu-0015 failure. The 42.55-second local verifier remains disclosed above.

@overtrue overtrue left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed d5a9c36 against main 06af380. No actionable finding in this test-only delta.

The reference report, baseline regression selection, family census, skip ratchet, recovered-case verdicts, fixture load and deadlines remain unchanged. The mapping preserves one entry per selected regression and adds only existing deny diagnostics, a missing-diagnostic marker or the skip reason. The Diagnostic formatter and CaseOutcome failure filter were inspected at this head.

Independent verification compiled the diagnostic function and exact diagnostic test with actual Diagnostic and CaseOutcome source in a standalone Rust harness; only the unrelated schema Violation shape was supplied locally. The control passed. Six separately compiled faults for omitted failure, included warning, wrong case ID, missing failure detail, omitted skip reason and missing skip reason marker each failed the assertion, and the restored control passed. git diff --check passed and the review worktree is clean. This check does not independently execute the live multipart family or the full local four-command gate.

Hosted CI run 40962462759 completed with all 33 jobs successful on this exact head. The original c-mpu-0015 failure remains unexplained in #1547; synthetic diagnostic controls and later green runs do not establish its cause or repair. Whole-command verifier latency remains tracked separately in #1504.

Role Verdicts

  • simplicity-adversary: attacked baseline selection preservation, regression cardinality, deny-only filtering, missing observations and skipped-case reporting — no break found

@overtrue
overtrue merged commit be6f360 into main Oct 29, 2026
33 checks passed
@overtrue
overtrue deleted the p1/1547-multipart-diagnostics branch October 29, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant