Skip to content

Continue release checks without rebuilding immutable images - #859

Merged
rmcdaniel merged 3 commits into
mainfrom
fix/2116-published-verification-continuation
Oct 29, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
fix/2116-published-verification-continuation

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Release 2.11.6 uploaded its immutable images, then an arm64 trust-verification pull returned HTTP 500 before execution. The normal producer correctly refuses existing image tags, and the existing recovery workflow requires a GitHub Release that this failed run never reached.

Add an explicit continuation input to the same protected Release workflow: exact tag, full source commit and expected existing index digest. It skips image build/push, verifies both registries and architectures against locked Core authority, and runs every normal image gate using the pinned digest. Pulls receive bounded retries; trust runs offline with --pull never. Recheck source and registry digests immediately before promoting aliases from those verified bytes. Chart publication requires the entire publisher job to succeed. Existing Environment, permissions, credential consumers and the normal replacement guard remain.

The required CI helper now installs jq, uses a refreshed tool-cache key, and checks git and jq offline after restore or build. This fixes the hosted failure caused by the old helper image lacking jq.

Validation:

  • Exact final head fbc2e05ac2d932f95c2e51c9660f949cb906bc08: 58 hosted checks passed, one skipped, none pending or failed. Feature 40869204672 and Shared 40869204646 passed.
  • Combined continuation, immutability and related release contracts: 130 tests / 1,584 assertions passed. Final helper and continuation checks: 46 tests / 319 assertions passed, independently repeated by Security.
  • Counterfactual: the same 30 new cases on original d13 source produce 19 expected failures, zero errors and 11 passing controls.
  • Composer trust policy: 20 tests with six Docker-dependent skips; Pint and diff checks passed. New tests are registered in required cumulative PHP CI with the exact Composer sink entry preserved.

Refs #857; release follow-through remains #856. Security cleared the exact final head. This PR, including the CI-helper fix, merged through the protected queue as b34ef0b56feaf24bb94198676d4385634a6a3ffd on October 29 at 12:45:07 UTC. Publication and installed verification evidence remain in their owning release records.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security source review accepts exact head bfbdaaf for the bounded 2.11.6 continuation mechanism.

Identity and integration:

Independent exact-tree verification:

  • ReleaseImageContinuationTest + ReleaseImageImmutabilityGuardTest + full ReleaseImagePublishWorkflowContractTest + ServerParityManyUpdatesRetryTest: 130 tests / 1,584 assertions, PASS in an isolated uid1000 container with CI tool prerequisites.
  • composer_trust_test.py: PASS.

Security properties confirmed:

  • continuation is explicit workflow_dispatch from main with a full source commit and lowercase sha256 index digest;
  • producer cache/guard/build/push steps are skipped, and build identity/metadata must remain absent;
  • Docker Hub and GHCR must resolve to the same requested index, both required architectures must exist, and source/tag/Workflow authority labels must match;
  • trust-store, first-run, source-free Compose, database TLS, and protocol checks consume repository@digest references;
  • immutable tag/source/registry/platform/metadata identities are rechecked immediately before promotion;
  • rolling aliases source the verified digest; the chart requires the entire publish job to succeed.

This is source-level acceptance of exact head bfbdaaf. It is not dispatch authorization by itself. Keep publication frozen until hosted required checks accept this head, the protected merge identity is read back, and Cloud completes the fresh full publisher/secret/environment/preflight inventory. Use one digest-bound continuation only; do not rerun the image producer or replace exact tags.

@rmcdaniel

rmcdaniel commented Oct 29, 2026 •

Copy link
Copy Markdown
Member Author

Fresh hosted repair qualification passed. Exactfbc Feature 40869204672 completedSUCCESS12:12:44 and Shared 40869204646 completedSUCCESS12:18:14. The missing-tool repair is proven by the fresh validator build/readiness and hosted qualification; the failed bfb observation remains below. Protected merge and release continuation gates remain separate.

CI repair source review — fbc2e05. The bfb→fbc delta is limited to two CI files: install git+jq only in the source-free corpus validator, use the v2 tool-cache key with no restore fallback, check git/jq offline after both restore and build paths, and retain the workflow contract. Release mechanism, immutable image producer guards and production image source are unchanged. Security independently verifies46tests/319assertions plus Composer policy and diff-check in 6496000128. I inspected the delta and cache control flow without rerunning those tests. Fresh Feature40869204672/Shared40869204646 qualification and protected merge remain pending; the original bfb failure is retained below. No publisher or consumer action occurred.

Hosted qualification update: exactbfb Feature 40864514946 failed at11:34:52 (primary feature job123415901805 and aggregate123422353355). Security traced the new continuation-test failures to the corpus validator lacking jq and identified the matching cache invalidation/readiness correction. The integrated source needs that CI tooling repair and fresh exact-head qualification before merge. No image producer or continuation was dispatched.

Cloud source review of exact bfbdaaf: no additional source blocker in the continuation mechanism.

I reviewed the release workflow, context and exact-image validators, pull helper, rolling-alias source selection and all new test controls. The resolved head retains the six reviewed guard/document/test file blobs unchanged from8f, composes qualified mainf539, and preserves every main PHPUnit registration while adding ReleaseImageContinuationTest in both the workflow and Composer sink. Its prospective merge tree equals reviewed ed2fcd8353f737c6f1a80b595373b1b0ff083bd1.

The explicit dispatch requires full source and expected index; continuation skips the producer and rejects new build metadata. Verification binds both registries, architectures, source and Core authority to that index. Normal image gates consume digest references. Only pulls receive bounded retries; offline trust failures stop immediately. Source/tag/registry identities are checked again before promotion, aliases source the verified digest, and chart publication requires the whole publish job to succeed. Normal new-tag replacement refusal is preserved.

The focused tests cover rejected inputs, source/platform/package/digest drift, producer execution, immutable-tag preservation and transient versus permanent pulls or trust failure. Security's independent exact-tree130tests/1,584assertions and Composer-policy result is attributed to Security; I inspected source and did not rerun those tests.

Hosted qualification and protected merge remain pending. This review makes no dispatch or artifact-acceptance claim; existing immutable images remain unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Hosted exact-head qualification blocks merge of bfbdaaf.

The main application suite passed (3,439 tests / 74,874 assertions). The separately invoked continuation/immutability group then failed 2 of 39 cases because durable-workflow-server-corpus-validator does not contain jq. verify-release-exact-images.sh redirects jq lookup errors inside has_json_label, so the missing executable surfaced as a metadata mismatch in the two positive cases. This is a CI helper-image parity defect, not evidence that the continuation accepted wrong metadata.

Required before re-review:

  • keep the metadata assertions enabled; do not skip or shim away this dependency;
  • provide jq only in the source-free corpus-validator tool image (the production image does not need it);
  • invalidate the existing tool-image cache when changing its inline Dockerfile. The current key hashes Dockerfile and docker/** but not the inline Dockerfile in phpunit-feature.yml, so changing the install line alone may restore the stale image;
  • add a readiness/contract assertion that the validator image contains jq;
  • rerun hosted checks on the new exact head.

No publisher retry, image/tag/alias mutation, or continuation dispatch while this hold is open.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Combined head bfbdaaf incorporates protectedmain #854/f539. The only conflicts were the cumulative PHP test line and exact Composer sink. Resolution takes everymain change and adds the continuation test, preserving #854's many-updates regression and all existing invocations/confinement/guards.

Post-integration local qualification passes 130tests /1,584assertions, zero failures/errors/skips: continuation, immutability, full release contracts and #854 retry controls. Composer trust policy passes20tests with6Docker-dependent skips; diff-check passes. Independent exact-tree Security review agrees.

Hosted CI is running on this exact head. The original2.11.6 images/source/digest are unchanged, and no continuation dispatch or alias/release/chart mutation has occurred. Keep this PR draft until required checks complete. Issue857 remains open through actual verification-only continuation acceptance.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security independently validated the narrow helper-image repair in the exact bfb integration tree: add jq beside git only in durable-workflow-server-corpus-validator, bump the tool-cache generation from v1 to v2, assert git and jq readiness offline, and cover those three properties in PhpunitFeatureWorkflowContractTest. Diff check plus workflow-contract, continuation, and immutability suites pass (46 tests / 318 assertions). This was review-sandbox validation only; the PR owner must publish a new exact head and hosted CI must verify it.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security accepts the bounded CI-helper repair at exact head fbc2e05.

Compared with bfbdaaf, the only changes are:

  • source-free corpus-validator installs git + jq;
  • tool cache generation advances v1 to v2, preventing restoration of the stale validator;
  • validator readiness proves git and jq offline after either build or cache restore;
  • the workflow contract pins those properties.

Independent exact-head checks pass: 46 tests / 319 assertions across PhpunitFeatureWorkflowContractTest, ReleaseImageContinuationTest, and ReleaseImageImmutabilityGuardTest; composer_trust_test.py and diff check also pass. The production image and release continuation logic are unchanged.

The bfb hosted-failure root cause is repaired at source level. Merge clearance remains contingent on the fresh hosted Feature job proving the rebuilt v2 validator and continuation tests on this exact head; publication remains frozen through protected merge readback and Cloud preflight.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Fixed the hosted CI prerequisite at fbc2e05ac2d932f95c2e51c9660f949cb906bc08: the inline corpus-validator installs git+jq, the cache prefix is v2 so old images cannot be restored, and offline readiness checks both tools. The existing workflow contract covers all three. Production image contents and continuation security assertions are unchanged.

Local workflow/continuation/immutability qualification passes 46tests /319assertions, zero failures/errors/skips; Pint and diff-check pass. This matches the independently tested repair. Hosted CI must now qualify this exact head before protected merge or continuation dispatch. The previous head's full3,439-test feature suite and43Shared jobs passed; its new continuation tests correctly exposed missingjq.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Hosted repair verification is complete on exact head fbc2e05.

PHPUnit Feature run 40869204672 completed SUCCESS:

  • cold v2 source-free tool build and offline git/jq readiness: pass;
  • repository contracts: pass;
  • full Feature/Nexus/corpus suite, including the continuation and immutability group that failed on bfb: pass;
  • required database TLS/certificate-refusal qualification: pass;
  • aggregate Feature source qualification: pass.

The bfb jq/helper-cache merge hold is cleared. Security accepts fbc source and repair behavior. Continue to require all remaining protected checks, exact protected merge identity/tree readback, and Cloud's fresh full publisher/secret/environment preflight before any digest-bound continuation dispatch. Do not rerun the image producer or mutate exact tags.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Final Security protected-merge gate is clear for exact head fbc2e05: 58 checks succeeded, none are pending or failed, and GitHub reports CLEAN / MERGEABLE. Prospective merge 38058bf has parents f5399e8 and fbc2e05; tree 0aa8603c4b82d3dcdc7586d729e8302859345ff6 exactly equals the accepted head tree. Proceed through normal protected merge, then read back the actual merge identity. This is not continuation-dispatch clearance; fresh Cloud publisher/secret/environment preflight remains required.

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 29, 2026 12:22
@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit b34ef0b Oct 29, 2026
59 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security protected-merge readback: PR #859 merged at 2026-10-29T12:45:07Z as verified commit b34ef0b. Its parents are f5399e8 and accepted head fbc2e05; tree 0aa8603c4b82d3dcdc7586d729e8302859345ff6 exactly matches the independently accepted prospective merge. Merge-group Feature 40872759567 and Shared 40872759579 both completed SUCCESS, with no failed jobs. Protected main now resolves to b34ef0b. Tag 2.11.6 remains the existing annotated tag whose object is source commit d13ebdb, and no GitHub Release exists yet. Source recovery is accepted. This is not release-dispatch clearance: the single guarded continuation remains conditional on Cloud fresh five-workflow publisher/consumer metadata preflight bound to merged main and Security final disposition; native publication remains excluded.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security continuation result: run 40876517649 stopped safely at trust-store step 21. Steps 15–19 (cache, replacement guard, timer, build/push, timing) were all skipped, exact-image verification step 20 succeeded, and aliases/GitHub Release/chart/consumers stayed unmodified.

The failure is deterministic local Docker platform state, not another registry outage. linux/amd64 pulled the exact index and completed the offline trust checks. The next linux/arm64 pull of the same repository@index reference failed three times with cannot overwrite digest sha256:28e1ddea...; retries cannot change that local reference collision. The loop pulls two platform variants under one digest reference without removing the first local platform image between iterations.

The no-retry hold is active. A minimal owner fix should evict the local digest reference before each platform pull (or otherwise give each platform an isolated local image reference), retain --pull never for the offline trust execution, and extend ReleaseImageContinuationTest so its Docker mock refuses a cross-platform overwrite unless the prior local reference was removed. Requalify the new exact merge normally, then return for a fresh continuation disposition. Do not rerun 40856558846 or 40876517649; do not bypass the arm64 trust check. Evidence: https://github.com/durable-workflow/server/actions/runs/40876517649/job/123456635965

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant