Repository navigation
M8+M9: Migration Verification Lab + Evidence-Backed Automated Migration PRs - #67
Merged
Merged
Conversation
…ed execution, evidence bundle, decision engine Introduces patchfrog/migration_verification/ (M8): a verification plan is derived from a migration's own MigrationStep/PatchLinkage evidence (never "run everything" by default), targeted tests are selected from M6's already-computed BlastRadius.related_tests (DIRECT/TRANSITIVE/FALLBACK/ UNKNOWN, never a filename guess), and every executable step runs through the existing VerificationSandbox (bwrap+prlimit) -- no new subprocess execution path. Baseline-vs-patched comparison reuses the same sandbox to distinguish "this migration actually fixed something" from "an unrelated test happens to pass". Contract verification reuses M7's own StepResult evidence rather than re-deriving it with string matching. A deterministic decision engine (mirroring fix_verification's evidence-combination discipline) produces VERIFIED/PARTIALLY_VERIFIED/UNVERIFIED/FAILED/ REGRESSION_DETECTED/HUMAN_REQUIRED -- never an LLM call anywhere in this package. Adds migration_verification_runs persistence (0036, one Alembic head), mirroring MigrationStore's idempotent fingerprint-keyed pattern so a different patch/requirement-set is never matched against stale evidence. Adds a bundled `demo_verified` fixture (a fully-automatic single-consumer variant of the existing M7 demo) that reaches a real VERIFIED outcome with genuine sandboxed baseline-fail/patched-pass evidence; the existing M7 `demo` fixture (which has one human-required step) now demonstrates the HUMAN_REQUIRED path unchanged. Neither changes M6/M7 behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the `migrations verify` subcommand alongside the existing plan/generate/demo commands: plans + generates a patch for each given repository, runs it through Migration Verification, and prints a human-readable evidence report or `--json`. `--persist` records the event/plan/patch (as `generate` already does) plus a migration_verification_runs row linked to the exact patch id -- a re-run with unchanged inputs is idempotent (verification_run_created: false, same run id). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…list/update GitHubClient previously had no way to create a branch, commit, or pull request at all -- only PR review/check-run/metadata reads and review submission. Adds get_ref/create_ref/update_ref, get_commit_tree_sha/ create_tree/create_commit (Git Data API, content given directly so no separate blob-creation round trip), and create_pull_request/ list_pull_requests/update_pull_request, following the same typed-domain/ error-translation discipline as the rest of the client. This is the real new surface M9's migration PR publisher needs; no local git clone or credential-bearing remote URL is ever used for it. Covered by respx-intercepted unit tests only -- no real GitHub request. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ation, durable publisher Introduces patchfrog/migration_pr/: MigrationPRPlan/MigrationPRLinkage/ MigrationPRStatus/MigrationPREligibility (M9.1), deterministic branch naming (M9.2, branch.py), publication eligibility policy that never silently lowers the bar (M9.3, eligibility.py), an evidence-backed Change Dossier renderer (M9.4, dossier.py), a second GitHub Check name/ external-id scheme reusing patchfrog.publishing.checks.CheckRunClient verbatim (M9.5, check.py), and the durable publish sequence -- idempotent by identity_key, stale-base protected, force-moves a PatchFrog-owned branch, never reopens a human-closed PR (M9.6/M9.7, publisher.py). M9.8's evidence/patch integrity check (integrity.py) is enforced inside the planner itself, before a plan is even built: a MigrationEvidenceBundle whose change/patch/repository-head fingerprints don't correspond exactly to what's about to be published raises MigrationPRIntegrityError rather than silently building a plan on mismatched evidence. FakeMigrationGitHubPublisher (M9.9) satisfies the same Protocol structurally, so tests never open a real GitHub PR -- covers dry-run, first-open, idempotent republish of identical/regenerated patches, stale-base rejection, every non-eligible verification outcome, a human-closed PR never being reopened, and check-run reconciliation being correctly per-commit-SHA (a regenerated patch's new commit legitimately gets its own check run; a retried identical commit reconciles the existing one). Persistence: migration_pull_requests (0037, chained after 0036_migration_verification), one row per identity_key, mirroring migration/store.py's advisory-lock/fingerprint-keyed-upsert pattern. GitHubClient.list_check_runs gained an optional check_name parameter (default unchanged) so MigrationCheckPublisher can query its own check name without colliding with review publishing's. A structural test (test_migration_verification_and_pr_module_boundaries.py) enforces that neither migration_verification nor migration_pr ever imports a provider abstraction or SDK -- mirrors migration/'s own guard for the deterministic M7 modules.
…l product demo, docs `patchfrog migrations publish --dry-run` (M9.10) builds the exact MigrationPRPlan a real publish would use and renders it -- proposed branch, commit/PR title, PR body, check title/summary, and whether policy allows publication -- without ever touching GitHub. Requires exactly one --repo (a migration PR targets one repository, never a fleet) and --repository OWNER/REPO; --allow-partially-verified maps to MigrationPRPolicy. Real publication is intentionally not reachable from this CLI in this milestone. `patchfrog migrations demo --verified` (M9.12) is the complete product demo: upstream change -> affected consumers -> blast radius -> migration plan -> generated patch -> bounded verification -> evidence bundle -> VERIFIED -> AUTO_OPEN-eligible dry-run PR dossier, against the bundled demo_verified fixture (real baseline-vs-patched evidence via its SDK stub). The default `demo` (no flag) now also carries the existing human-required M7.10 fixture through M8/M9, demonstrating the non-verified path in the same command: HUMAN_REQUIRED / PLAN_ONLY, never an automatic code PR. Both extensions are additive -- the pre-existing default demo's exact text/JSON assertions (M7.8) are all still satisfied unchanged. An opt-in, disabled-by-default real-GitHub E2E harness (M9.11) is added (tests/integration/test_migration_pr_real_github_e2e.py, gated on PATCHFROG_MIGRATION_PR_REAL_GITHUB_E2E=1) but deliberately left unimplemented and never run automatically -- it fails loudly if its own skip gate is ever bypassed, rather than silently doing nothing under a green check. docs/migration-verification.md and docs/migration-pr.md document M8 and M9 respectively, following docs/migration.md's own module-by-module convention; docs/migration.md's "deliberately out of scope" section now points forward to both instead of describing them as unimplemented.
Both were listed as "not started" -- update to reflect what M8 (Migration Verification Lab) and M9 (Evidence-Backed Automated Migration PRs) actually implement, pointing at their own docs instead of duplicating detail here.
Records the commit list, what was reused vs. genuinely new, the one broken fixture and one incorrect test expectation found and fixed in the interrupted prior session's uncommitted work, the M9.8 integrity check that was the one genuinely missing piece, exact validation-gate results (ruff/mypy repo-wide, every M4-M9 suite individually, Alembic, Docker, Celery, secret scan), and known limitations -- including that the repo-wide full pytest run could not be completed locally this session due to host memory pressure unrelated to this milestone's code, with CI treated as the authoritative full-suite confirmation for this PR.
… verification decision engine The human-steps check only ran when no mandatory requirement was unresolved, so a plan with an unresolved human step plus any unavailable automatic requirement downgraded to PARTIALLY_VERIFIED. Precedence is now REGRESSION_DETECTED > FAILED > HUMAN_REQUIRED > VERIFIED/PARTIALLY_VERIFIED > UNVERIFIED.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the two milestones after M6+M7 (upstream change → impact →
migration plan → generated patch): M8 — Migration Verification Lab
(executable/runtime evidence that a generated migration patch actually
restores compatibility) and M9 — Evidence-Backed Automated Migration
PRs (a policy-gated, idempotent, evidence-bound GitHub PR built from a
sufficiently verified migration). This is the milestone where PatchFrog
becomes a compatibility-migration product, not just a diff/plan/patch
generator — the full flow is now:
No LLM ever decides final verification truth or PR eligibility/content —
every decision is deterministic, explainable, and traceable to already-
computed evidence. Never a live provider call anywhere in this milestone.
What's new
M8 (
patchfrog/migration_verification/): deterministic verification-requirement generation, targeted test selection (DIRECT/TRANSITIVE/
FALLBACK/UNKNOWN), a bounded execution plan run through the existing
VerificationSandbox(bwrap + prlimit — no new sandbox), boundedbaseline-vs-patched comparison, contract-specific verification reusing
M6's own diff evidence, regression detection distinct from mere
insufficient evidence, a reproducible evidence bundle with explicit
evidence strength, and a deterministic outcome engine (
VERIFIED/PARTIALLY_VERIFIED/UNVERIFIED/FAILED/REGRESSION_DETECTED/HUMAN_REQUIRED).patchfrog migrations verify(with--persist).M9 (
patchfrog/migration_pr/): a publication-eligibility policy thatnever silently lowers the bar, deterministic branch naming, an evidence-
backed Change Dossier, a second GitHub Check name/external-id scheme
reusing the existing
CheckRunClientProtocol verbatim, and a durablepublish sequence that is idempotent by migration identity, stale-base
protected, and never reopens a human-closed PR. An explicit M9.8
integrity check (
verify_evidence_integrity) rejects building a plan atall if the evidence bundle doesn't correspond exactly to the patch/
change/repository state about to be published.
patchfrog migrations publish --dry-runprints the exact branch/commit/PR/check plan withoutever touching GitHub; real publication is deliberately not exposed from
this CLI in this milestone (an opt-in, disabled-by-default real-GitHub
E2E harness exists for the future but is left unimplemented and never
runs automatically).
Full product demo:
patchfrog migrations demo --verifiedruns thecomplete flow to
VERIFIED/AUTO_OPENwith real baseline-vs-patchedevidence and prints the full dry-run PR dossier; the existing
migrations demo(unchanged fixture) now also demonstrates the non-verified path,reaching
HUMAN_REQUIRED/PLAN_ONLY.Docs:
docs/migration-verification.md,docs/migration-pr.md;docs/migration.mdanddocs/roadmap.mdupdated to point forwardinstead of describing M8/M9 as future work.
Notable fixes to interrupted prior-session work
This branch had 3 commits and a large amount of uncommitted M9 work
already in progress from a prior session that hit a prompt limit. Auditing
and completing it found:
exist) — fixed to use the real
session_factoryfixture.per-commit-SHA, so a regenerated patch that force-moves the branch to a
new commit correctly gets a new check run rather than "reconciling"
one attached to a commit the branch no longer points at. Rewritten into
two separate, correctly-named tests.
actually implemented or enforced — added.
Validation
baseline; M5 (35), M6 (83), M7 (67 passed, 5 expected skips), M8 (49),
M9 (36 across 5 new suites), demo suite (9) — all passing
upgrade headsucceeds on real Postgresthrough all 37 revisions
api/worker/verifierall succeed; new modulesconfirmed importable inside the built
apiimagelocally this session — the harness OOM-killed it due to genuine host
memory pressure, unrelated to this code. Every suite this PR actually
touches was run to completion instead (see
validation/m8_m9_verification_migration_pr/latest-summary.mdfor thefull gate-by-gate table). CI is the authoritative full-suite check for
this PR.
Test plan
python -m patchfrog.cli migrations demo --verifiedoutput against the dossier example in the validation summary