M10+M11 (engine): organization compatibility campaigns and continuous upstream watchers

.github/workflows/ci.yml

@@ -37,43 +37,16 @@ jobs:
       APP_ENV: test
       DATABASE_URL: postgresql+asyncpg://patchfrog:patchfrog@localhost:5432/patchfrog
       REDIS_URL: redis://localhost:6379/0
-      # A GitHub App identity is required for Settings to construct, not
-      # for any real GitHub call CI makes -- every job below runs against
+      # A GitHub App identity is required for Settings to construct, not for
+      # any real GitHub call CI makes -- every job below runs against
       # fixtures/oracle providers only, never live credentials.
-      # GITHUB_PRIVATE_KEY below is a disposable RSA keypair generated
-      # solely for this workflow (`openssl genrsa 2048`) -- never
-      # registered with any real GitHub App, never used outside CI.
+      #
+      # Secret-looking values are deliberately NOT written here: a job-level
+      # `env:` block is printed, resolved, in the header of every step's log.
+      # The disposable App key and webhook secret are generated at run time
+      # by the "Generate disposable credentials" step below, written to a
+      # file / $GITHUB_ENV, and masked. See tests/unit/test_ci_workflow_hygiene.py.
       GITHUB_APP_ID: "0"
-      GITHUB_PRIVATE_KEY: |
-        -----BEGIN PRIVATE KEY-----
-        MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDc1Br1N+2s8fLE
-        2r1+/42VyQRbJ/XColFATS4htUGInCp3vrp2lJD7rfhlQLCCj6+oL8KFOI9wRjFD
-        HQUfUbSyOAiPVjwwLw4wg/fegw/rPIqk0sbtgLX6tNsiDvF+jeZ2ILIiA6O5h/of
-        tAPwHZFbM0F28CNC/FczaAN49ICnGg/5cbkCmWu6ZFRHYLBdKeQK2326zOjLZAAf
-        3U08BjvSwjxGuytSAlETUTLnn9NmwSiAAIzhqfsufIOJ21aS30lmmfUdehmUuDOM
-        Gx3o+C5DTwX0zqHB4Te3aLk1PBzKxVMFzPsxTJDKBzuAWE8RyuvKN3HRGPgluIM9
-        o2WsR8EZAgMBAAECggEAMWTUHtOCuzNnLPcMedKplb5f1g/rHbz21Z4t2mZToT3l
-        FeG4jXuPrmvsaqF8mceP79u289uB4gghTrS4B5hcBZ0Y9Q7B6bkswFdUYxYHwoNJ
-        M7LdwhQPIGPBvC2BgOA2k2ee0JLl1UMABIVceCH0+GRPmU3/ZSVST/JNDIoMGNQG
-        EjuHnM8Hjs4WP2r7ous2CeEcUdDrnhqrVfcFOzIKMEiffoWIaw12oBhRR6D+stP1
-        wB95YJ57326Z/yQyQUZw+iOOpkG0url3zyzJUu5jvpF9mAOVHt1i61N0OxktYNd4
-        b5OuXfOTbYoaLl+jn3mzW/pLQpkfgQa28NBqDDcUMQKBgQD49XsudnZQ0NoeuU8+
-        LTxQ/e6R0pBKjqS/4yv0nxrt5aEY8CKp7nrVGhubX1btRIZNS3xc+UrtWrdivSzM
-        9Y3LAIw67XOa1oYID0hg0C6q5jftpMpqb9C9ZKEIz9bP4V4xpNDM67k48qAujvm2
-        XGZ6AgRT2RSFtr4H+CzongSlZQKBgQDjEvQvsgogK/XfMDJmKA34Y/8q5mGmp4PU
-        8lf7KTwsBg2rgABqU5ZDoDuguBzrch1eE5NxECN1mQyDG2eX201eL3zerCalXXYT
-        p+nDuHHwQBiua+cjAKPOHny+IV4Y2ijij6nXK10Trb4POOdjuMZQU8aBnlE3LTlR
-        AADs6sGbpQKBgAzlkpsGkfx/xTzgNqWu7Df535j4QndKSmiSWR2T/G4kfiLvlawd
-        6YkOkC2NU9B4w0hPx8J9O49uKp/X0vFPKyhAAAOxd1KnGExklkhFoJVD1dPevImk
-        /zhAdUMimSstrpZ9xj9HWtIDZ57h0zfKM5jQb+70COcgX1nEVnc6cF0tAoGAHOEZ
-        O0OqdAMDRzpq/oETncah+kPIoZhmdNvJgsDwzSyL6obiBqPQ1ZL4Ym1oCC3APGnC
-        87s4t0kaxUXj+/VBNrxVZkCAvcFQNhnB6kl3/DU6rlw8zj/TlXBdLxUxUXnVf1ME
-        J8B1Gv6a0HbNPuqOFkNBWNTYwWq4Amw8seT9cJ0CgYEA5ovXV2T4++ws2Qt3cPO5
-        im8J2aCh1fYKE/1ZNnuO0xnUUu3VeH8Rz2iw4mFzELKc64v9BSw2xON7tWSDtY6z
-        0VKkLmokNeUbcCpEs/lLy6/bdP0/pVTBLesXBDsm6qnAw1Stnc/MTnXPsiaTYwke
-        VgAFWZ9YEAZPVROqM6i76BU=
-        -----END PRIVATE KEY-----
-      GITHUB_WEBHOOK_SECRET: ci-placeholder-not-a-real-secret
       # The database-specific suite must fail if the declared CI service is
       # unavailable or unmigrated. Local runs may skip these tests when no
       # compatible Postgres is present.
@@ -84,6 +57,24 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      - name: Generate disposable credentials (written to a file, masked, never printed)
+        run: |
+          set +x
+          umask 077
+          key="$RUNNER_TEMP/ci-github-app-key.pem"
+          openssl genrsa -out "$key" 2048 2>/dev/null
+          # Belt and braces: if any later step ever echoed this key, the runner redacts it.
+          while IFS= read -r line; do
+            [ "${#line}" -ge 8 ] && echo "::add-mask::$line"
+          done < "$key"
+          chmod 0644 "$key"  # a disposable test key; the container user must be able to read the mounted file
+          webhook_secret="$(openssl rand -hex 24)"
+          echo "::add-mask::$webhook_secret"
+          {
+            echo "GITHUB_PRIVATE_KEY_PATH=$key"
+            echo "GITHUB_WEBHOOK_SECRET=$webhook_secret"
+          } >> "$GITHUB_ENV"
+
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
@@ -142,8 +133,10 @@ jobs:
         run: |
           docker run --rm \
             -e DATABASE_URL="$DATABASE_URL" -e REDIS_URL="$REDIS_URL" \
-            -e GITHUB_APP_ID="$GITHUB_APP_ID" -e GITHUB_PRIVATE_KEY="$GITHUB_PRIVATE_KEY" \
-            -e GITHUB_WEBHOOK_SECRET="$GITHUB_WEBHOOK_SECRET" \
+            -e GITHUB_APP_ID="$GITHUB_APP_ID" \
+            -v "$GITHUB_PRIVATE_KEY_PATH:/run/ci-github-app-key.pem:ro" \
+            -e GITHUB_PRIVATE_KEY_PATH=/run/ci-github-app-key.pem \
+            -e GITHUB_WEBHOOK_SECRET \
             --network host \
             patchfrog-worker:ci python -c "
           from apps.worker.celery_app import celery_app

docs/campaigns.md

@@ -0,0 +1,117 @@
+# Compatibility campaigns (M10)
+
+One upstream API/SDK change usually reaches several repositories of one organization. A
+**compatibility campaign** is the single, durable, honest record of that fact: which repositories were
+in scope, how each was classified, how far each migration got, and what is still unresolved. It is
+built entirely from the M5 → M9 machinery (dependency registry, upstream change, blast radius,
+migration plan/patch, verification, PR publication); it adds no new analysis and no model call.
+
+```
+upstream change ──► every enrolled repository evaluated independently
+                        (fresh? readable? affected? migration? verified? PR?)
+                ──► one CompatibilityCampaign  (idempotent per workspace + change + engine version)
+```
+
+Package: `patchfrog/campaigns/`. Version constant: `CAMPAIGN_ENGINE_VERSION = 1`.
+
+## The organization graph (M10.1)
+
+`campaigns.graph.build_org_graph` derives, on demand, `workspace → repositories → dependency identities →
+per-repository instances (version, usage sites, contract)`. It is a **view**, never stored: a dependency
+identity is shared logically across repositories (an SDK consumer, a direct-REST consumer and a
+generated-client consumer all land under one provider identity) while every per-repository fact stays owned
+by that repository's own M5 rows.
+
+## Repository and campaign states (M10.3)
+
+Repository states: `NOT_AFFECTED`, `IMPACTED`, `MIGRATION_PLANNED`, `PATCH_GENERATED`, `VERIFYING`,
+`VERIFIED`, `PARTIALLY_VERIFIED`, `HUMAN_REQUIRED`, `FAILED`, `PR_OPENED`, `RESOLVED`, plus the
+uncertainty states that exist so doubt is representable instead of rounded away: `STALE`, `UNKNOWN`,
+`ACCESS_LOST`, `UNVERIFIED`.
+
+Campaign states: `DETECTED`, `ANALYZING`, `ACTION_REQUIRED`, `MIGRATING`, `PARTIALLY_RESOLVED`,
+`RESOLVED`, `BLOCKED`. Precedence (`campaigns/state.py`, first match wins):
+
+1. nothing evaluated → `DETECTED`; some unevaluated → `ANALYZING`
+2. every repository `NOT_AFFECTED`/`RESOLVED` → `RESOLVED` (the **only** road to it)
+3. at least one `RESOLVED` and anything unfinished → `PARTIALLY_RESOLVED`
+4. blocking repositories, all system-blocked (`FAILED`/`ACCESS_LOST`), nothing in flight → `BLOCKED`
+5. blocking repositories (`IMPACTED`, `HUMAN_REQUIRED`, `FAILED`, `PARTIALLY_VERIFIED`, `UNVERIFIED`,
+   `STALE`, `UNKNOWN`, `ACCESS_LOST`) → `ACTION_REQUIRED`
+6. otherwise → `MIGRATING`
+
+`RESOLVED` is reached only by **fresh re-evaluation** showing a previously affected repository is no
+longer affected — never by "a PR was opened" or "a patch exists".
+
+## Honesty rules
+
+* **Freshness (M10.7).** Each repository's evidence is `FRESH`, `STALE` (older than the window,
+  default 7 days, configurable), `UNKNOWN` (never discovered, or a clock/record fault) or `ACCESS_LOST`.
+  A repository whose evidence is not fresh is **not analyzed at all** and is never reported unaffected.
+* **Uncertainty is never "safe".** `OrgBlastRadius.safe` is true only if every enrolled repository is
+  positively known unaffected. Version-level-only evidence is `UNKNOWN`, not `AFFECTED`.
+* **Failure is isolated.** Any exception evaluating or publishing one repository becomes that
+  repository's `FAILED` record (credentials scrubbed) and cannot touch another.
+* **Evidence that could not be obtained is not weaker evidence.** A caller that tried and failed to
+  check a repository out passes `evidence_error`; the repository is `FAILED`, never classified from
+  registry data that, for an SDK without a built-in discovery adapter, would read as "not affected".
+* **Scope is explicit (M10.6).** Only repositories passed in are evaluated — there is no implicit
+  access. A repository no longer enrolled leaves the campaign; one enrolled but unreadable stays as
+  `ACCESS_LOST`.
+
+## Orchestration (M10.4) and publication
+
+`campaigns.orchestrate.run_campaign` evaluates every repository (M6 impact → M7 plan/patch → M8
+verification → M9 PR plan), then publishes **per repository** under the workspace policy
+(`campaigns/policy.py`: watch mode `off | detect_only | migrate | migrate_and_open_pr`, verification
+requirement `require_verified | allow_partially_verified`, publication `automatic | draft_only |
+manual_approval`). Safe defaults: detect and report, require verified, manual approval. A partially
+verified PR is always a draft. A human approval (`approved_repositories`) turns `AWAIT_APPROVAL` into a
+publication only after a **fresh** evaluation, and never lowers the bar. Each repository keeps its own base
+SHA, patch fingerprint, evidence bundle, PR and publication state. Nothing ever merges.
+
+## Internal contracts (M10.5)
+
+An organization's own contract (a published package, an OpenAPI spec, a generated client, an internal SDK)
+is an `InternalContract` with an **explicit** identity: package (ecosystem + name), declared API hosts, or an
+M5 dependency key. No relationship is inferred from repository names, organization, layout or similarity.
+Consumers are matched by their *own* M5 evidence through the same `match_dependency` external changes use;
+the producer repository is the origin, not a consumer. There is no second graph and no new relation table.
+
+## Idempotency (M10.9) and persistence
+
+`campaign_identity_key(workspace_key, change_fingerprint, engine_version)` is the campaign's identity;
+re-running with unchanged inputs yields an equal campaign (same `version`). Repository-level operations
+stay separately idempotent (M9's PR identity is repository + change + engine version). Tables
+`compatibility_campaigns` and `campaign_repository_records` (Alembic `0038`); `workspace_key` is an opaque
+string the caller chooses (a hosted workspace id, an installation id, an org login).
+
+## Dossier (M10.8)
+
+`campaigns.dossier.campaign_to_dict` / `render_campaign_markdown`: upstream change, blast-radius counts,
+per-repository state/freshness/impact/verification/PR, residual risk, human-required actions, a single
+next action per repository, and per-repository data freshness — derivable from persisted state alone.
+
+## CLI
+
+```bash
+patchfrog campaigns demo           # the bundled four-repository scenario, offline, deterministic
+patchfrog campaigns analyze --old OLD --new NEW --hints H \
+    --repo web=./web --repo legacy=./legacy --stale legacy --policy migrate
+```
+
+The demo (`patchfrog/campaigns/demo.py`, fixtures in `tests/fixtures/campaigns/acme`): a watcher baselines
+then detects `client.chat.create → client.responses.create`; one campaign covers four repositories — one
+auto-fixable and verified (one PR), one human-required, one not affected, one with stale discovery — and is
+`ACTION_REQUIRED`, not resolved; an identical re-run creates no second campaign or PR; after the PR is
+merged, the human fix is made and the stale repository is rediscovered, it reconciles to `RESOLVED`.
+
+## Known limitations
+
+* Verification needs a working `bwrap` sandbox; without it nothing is ever `VERIFIED`
+  (`docs/executable-verification.md`).
+* An SDK with no built-in discovery adapter is recorded in the registry by package declaration only
+  (no call sites). Consumers are therefore found by **live analysis of a checkout** (using the change's own
+  target adapters), not by the registry alone — see `docs/watchers.md`.
+* Transitive blast radius follows the in-repository code graph only; cross-repository transitivity is not
+  inferred.

docs/ci-health.md

@@ -66,3 +66,13 @@ Pydantic forward-reference warning during one governance-tool test, and a rare
 Python asyncio subprocess-transport finalizer warning after a deliberately
 killed analyzer process. Neither changes a test result or represents an ignored
 assertion; they remain visible so dependency/runtime upgrades can remove them.
+
+## CI never prints key material
+
+The workflow generates its disposable GitHub App key and webhook secret at run time (written to a file or
+`$GITHUB_ENV`, and masked) instead of writing them in the job-level `env:` block, which GitHub prints --
+resolved -- in every step header. `tests/unit/test_ci_workflow_hygiene.py` fails if a secret-looking literal
+reappears in any `env:` block, if key or encoded-blob text is embedded in a workflow, or if a workflow prints
+the environment or resolved Docker config. This guards the *configuration going forward*; it does not and
+cannot erase anything already present in historical logs.
+

docs/product-boundary.md

@@ -56,6 +56,20 @@ all:
 None of this is being built as part of this PR, and none of it is being
 implied to already exist.
 
+## Compatibility campaigns and watchers (M10/M11)
+
+The same governing rule applies to the compatibility pipeline:
+
+| Public engine (this repository) | Private PatchFrog Cloud |
+|---|---|
+| campaign domain, states, freshness, org blast radius, dossier, internal-contract identity | which workspaces opted in, their stored policy values |
+| watcher interface, source adapters, snapshot comparison, watch-registry aggregation, launch gate | when to poll: schedules, cadence, claims/leases, retry/backoff, per-host limits |
+| the publication *decision* and dossier (`decide_publication`, `build_pr_plan`) and its exact-head / stale-base / integrity / idempotency checks | the official GitHub App credentials and executing the authorized GitHub side effect |
+| lossless event serialization, tenant-scoped registry reads | cursor and delivery storage, per-workspace fan-out, usage counters, dashboard, notifications |
+
+The engine contains no hosted-service code and no credentials; Cloud contains no analysis, diffing, impact,
+migration, verification or eligibility logic.
+
 ## GitHub App boundary
 
 Two GitHub App identities exist in this model, and they are never shared:

docs/roadmap.md

@@ -231,6 +231,32 @@ GitHub publication is wired (`GitHubClientMigrationPublisher`) but not
 yet called from the production webhook/worker pipeline -- see
 `docs/migration-pr.md`'s "Known limitations".
 
+**M10 — Cross-repository / organization intelligence.** Implemented: the
+`CompatibilityCampaign` (one upstream change across a workspace's repositories,
+idempotent per workspace + change + engine version), a derived organization
+compatibility graph, an organization blast radius that never rounds uncertainty
+to "safe", deterministic repository and campaign states, freshness (`FRESH`,
+`STALE`, `UNKNOWN`, `ACCESS_LOST`), per-repository isolation, a bounded
+explicit-identity internal-contract model, a machine-readable and renderable
+dossier, and `patchfrog campaigns analyze|demo`. See `docs/campaigns.md`.
+
+**M11 — Continuous upstream watchers and hosted operation.** Implemented:
+the engine-side watcher interface and adapters (PyPI, npm, GitHub Releases,
+OpenAPI URLs, structured changelog feeds, manual snapshots), snapshot
+comparison with cursors and duplicate suppression, the watch registry, the
+campaign launch gate, an SSRF-safe fetcher, lossless event serialization, and
+the single-workspace ingest function (event → campaign → audit trail → policy-
+gated PR). Hosted scheduling, cursor storage, workspace policy, real GitHub
+publication with the official App identity, the dashboard and usage counters
+live in the private PatchFrog Cloud repository. See `docs/watchers.md` and
+`validation/m10_m11/latest-summary.md`.
+
+**PatchFrog v1 is complete for beta.** No further feature milestone is planned.
+What remains is operational: deploy when infrastructure budget allows, onboard
+five to ten real users, observe real provider/API changes, and measure detection
+precision, migration success rate and verification quality. Build only from
+what that feedback shows.
+
 ## Then — Advanced Verification
 
 **AA — Multi-Repository System Intelligence**