Skip to content

fix(orgs): contain organization merge and its operator bypasses (#6827) - #7561

Merged
bokelley merged 1 commit into
mainfrom
fix/organization-merge-containment
Sep 18, 2026
Merged

bokelley merged 1 commit into
mainfrom
fix/organization-merge-containment

Conversation

@bokelley

@bokelley bokelley commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Organization merge could delete local organization state before provider deletion
completed. This change makes merge execution temporarily unavailable at the admin
HTTP route, Addie tool, and shared service boundary. Read-only preview remains
available. It also removes the incident script's destructive mode and the
invariant's raw-delete remediation hint, with regression coverage for both.

This follow-on applies directly over landed #7560 on main, preserving main's
separate admin authorization correction and the containment document's SEO
metadata. The archived pure reconstruction preserves the original 16-path
added/deleted payload verbatim. Separate corrections add the required Addie
CODE_VERSION bump to 2026.09.60 and fix two documentation claims: identity
merge is already separately contained, and conditional preview wording applies
to the WorkOS section while existing Stripe/data-movement warnings remain.
The original commit rationale and co-author attribution are restored. There are
17 changed paths, with no migration or protocol release-surface changes.

Current qualified candidate:

  • Commit: d29c6ac1f8cbdc5b3340591068caa08c5733e59e
  • Sole parent: 82b429ace832d9355f49e71f8948ba07d7b69283
  • Tree: da6fc8fc7fb413d92926e1969b2dfdaa6dbcd208
  • Stable patch ID: 1dbfdf6fd075aa91923175bbbc4738997b915752
  • Binary diff SHA256: 967c45176eec7947b0590dd89868df31d555991c2795278f380c5549753810e5

Qualification repeated on that repaired tree:

  • Focused containment and interaction coverage: 29 files / 743 tests passed.
  • Complete unmodified commit hook: root 71 files / 1,063 tests; server 648 files
    / 10,801 tests passed, with 31 existing skips; both typechecks passed.
  • Unmodified pre-push, fresh local migrations/idempotence, migration policy,
    docs/nav, schema links, SEO, tool-surface checks, and applicable Semgrep passed.
  • The real incident script refuses --execute before configuration work.
    Deliberately injected TypeScript and SQL bypasses made the guard fail; both
    controls were removed and the pristine guard passed.
  • Fresh independent Claude Opus review at max effort: APPROVED, with no blocking findings, bound to the exact commit, sole parent, tree, and both patch identities above.

The first unpublished candidate was rejected by independent review for the lost
commit rationale/attribution and an inaccurate containment bullet. Its review is
preserved; both findings were repaired before complete qualification and fresh
independent review of the candidate above. The nearby preview-wording claim was
also narrowed without changing preview code.

The earlier checkout capacity failure and Semgrep temporary-directory startup
failure remain preserved in local evidence. Only the environment was repaired
for those failures; qualification was repeated and no failed result was waived.
The standard local migration runner intentionally excludes evaluator-only
migrations 584/585. Fresh public CI, Ready-triggered Ladon, and native exact-head
approval remain required before merge. No admin bypass or auto-merge is used.

The earlier description below is preserved as historical evidence. Its old
stack, commit identities, reviews, test counts, and blanket preview-wording
claim describe those earlier heads, not the corrected candidate above. Other
documented limitations still apply.

Original published description and composition history

Summary

Organization merge was the remaining organization-deletion writer. It is now contained on the same terms as #7560, and the operator bypasses that let someone walk around that containment are closed too.

Stacked on #7560 (organization deletion containment), which is stacked on #7538.

{ "error": "organization_merge_unavailable", "message": "Organization merge is temporarily unavailable." }

Why

Merge deleted the secondary organization as its final step. mergeOrganizations removed the local organization row inside its transaction, committed, and only then attempted workos.organizations.deleteOrganization, downgrading a provider failure to a warning while still reporting success. The Addie path attempted the provider delete twice — once inside mergeOrganizations and again at step 4 of the tool handler — with WorkOS membership creation interleaved between the local commit and that second attempt, so its unknown-outcome surface was strictly larger than the HTTP route's.

That is the same split provider/local state #7560 closes for the two DELETE endpoints.

What changed

Three layers, each refusing before any merge-specific effect:

Surface Containment Auth
POST /api/admin/cleanup/merge stable 503 before body validation and every merge-specific effect requireGlobalAdmin kept (it authenticates first, so this path does perform non-locking authentication reads — see the note below)
Addie merge_organizations (execute) refuses before its provider membership reads; no longer acquires a WorkOS client at all on that path admin-gated as before
mergeOrganizations service boundary throws OrganizationMergeUnavailableError before getPool(), pool.connect() and BEGIN defense in depth for any caller

Administrative gating is deliberately unchanged here, unlike the contained deletion routes — nothing in merge needs to run before authentication, so the refusal is never reachable anonymously.

To be precise about what "before any effect" means on each surface: the two contained deletion routes in #7560 return before authentication and therefore perform no database work at all. The contained merge route does not make that claim — requireGlobalAdmin runs first and performs non-locking authentication reads. What is guaranteed here is that the refusal precedes every merge-specific effect: the merge transaction, WorkOS membership writes, provider organization deletion, Stripe operations, audit rows, cache invalidation, and notifications.

The unsafe body is removed rather than left unreachable behind the guard: that sequence is the hazard, so re-enabling merge has to be a new reviewed implementation meeting the lifecycle contract, not the deletion of a guard. After this change no production call site of workos.organizations.deleteOrganization remains in server/src.

previewMerge and GET /api/admin/cleanup/preview-merge are untouched and remain read-only (SELECT/COUNT only). Preview wording no longer instructs anyone to execute.

Operator bypasses closed

A containment that only covers routes and services is one an operator can walk around. Two checked-in surfaces handed them the prohibited operation:

  • scripts/incidents/2026-05-cleanup-duplicate-prospect-stubs.ts deleted organization rows with direct SQL, and its own header reasoned from the absence of an admin delete endpoint that a script was the sanctioned route. Its destructive path is removed, not gated: no database client, no writes, and --execute hard-refuses above every other statement. Read-only analysis and reporting are preserved.
  • The unique-org-per-email-domain invariant emitted a ready-to-paste raw delete with the organization ID pre-filled, and pointed at a merge endpoint that does not exist. Its remediation_hint now routes to read-only preview-merge inspection plus escalation.

server/tests/unit/organization-delete-statement-guard.test.ts is the static guard: it scans the JS/TS/SQL files under server/src, server/scripts and scripts for a raw organization delete and fails on anything outside an allowlist recording why each remaining occurrence is not operator-reachable. SQL is covered because a .sql runbook is the likeliest shape for the next bypass. server/scripts/setup-sandbox.ts is documented and allowlisted rather than changed — it is sk_test-gated and ID-bounded.

Qualification

  • New unit suites for the service boundary, the route (including a test that pins the route refuses on its own rather than delegating and mapping the service throw), and both Addie branches — with a positive control proving the "no WorkOS client acquired" assertion is observable.
  • New mounted-HTTPServer + real-PostgreSQL suite: unauthenticated 401, non-admin rejection, CSRF precedence, platform admin refused across input variants, static admin key refused, preview still 200 and effect-free, direct service calls refused, Addie execution refused, third organization untouched.
  • The three pre-existing org-merge suites keep their fixtures and now prove the contained service performs no work on them; a pre-existing identities cleanup leak was fixed deterministically by id.
  • An existing assertion that pinned the raw delete this PR removes now pins the inverse contract.
  • Broadened affected-suite sweep on a fresh disposable database: 131 files / 2626 tests passed, zero fixture residue. Root Addie project 31/31.
  • Mandatory .husky/pre-commit passed: 71 files / 1063 tests, then the full server unit suite at 622 files / 9551 passed + 31 skipped, then typecheck.
  • Independent security and code review bound to the exact commit tree, with the static guard verified by negative control (it fails on an injected .ts bypass and an injected .sql runbook).

No migration, lifecycle journal, compensation, partial merge, or rollback fiction.

Known follow-up

The merge_organizations tool's declared description and usage_hints are intentionally unchanged: scripts/addie-tool-surface-budget.json pins a reviewed profile-contract hash over every declared tool profile, so a one-word edit invalidates it and re-blessing is its own review. The runtime refusal is authoritative meanwhile. Recorded in docs/contributing/organization-deletion-containment.md.

Published composition (restacked)

Replayed from d116704d48a9e0c91a4e780ed8e5c4775d02ef4b onto the restacked #7560 head, itself replayed onto the corrected #7538 head. Clean linear replay, no conflict, no manual resolution.

Patch identity, proven three ways:

  1. git patch-id --stable unchanged: 9c3ba179e9a257685ba7f8ee4b20319b90c0c7a5.
  2. Diff text byte-identical to the original once ^index lines are stripped — strictly stronger than the patch-id check, which normalizes whitespace.
  3. git diff <old tip>..<new tip> equals git diff <old #7538 head>..<new #7538 head>, so the only content delta across the whole restack is the new base commit itself.

Commit message byte-identical; zero file overlap with the new base commit; no conflict residue.

Interaction with the new base. The base commit touches server/src/routes/workos-webhooks.ts only. It adds no organization-row writer and no merge caller; its newly exported upsertMembership has no caller outside its own module. SET LOCAL lock_timeout = '2s' is scoped to the webhook transaction and reverts at commit or rollback, so it can only ever abort the webhook — never a contained refusal. The static guard's allowlist counts were re-verified on this exact tree (999 files scanned, 4 SQL files in the walk, zero migration-directory leakage, hit counts exactly matching the allowlist), so widening the guard to SQL still holds on the new base.

Qualification on this exact tree:

  • Focused webhook + deletion + merge + guard + invariants suites on a fresh disposable PostgreSQL database: 21 files / 309 tests passed.
  • Both typechecks passed.
  • Mandatory .husky/pre-commit on this tree, single run, exit 0: root unit 71 files / 1063 tests, full server unit 623 files / 9554 passed + 31 skipped, then typecheck. No timeout, no failed tests, no TypeScript errors.
  • Independent exact-tree review bound to this commit and tree: no blocking findings.

Non-blocking follow-up limitations

Two observations from independent review, recorded rather than fixed here so this stack stays scoped to the containment:

  1. Out-of-band provider deletion is not covered and cannot be. An operator refused by these routes can still delete an organization in the WorkOS console. The resulting organization.deleted webhook removes organization_domains rows while the local organizations row and its memberships survive, with no audit or journal record. The static guard cannot catch this — it matches a raw delete against the organizations table, and that path targets a different table. The operator-handling section's "no bypass" wording is about in-product paths, not out-of-band provider actions.
  2. The static guard's pattern is deliberately narrow. It matches DELETE FROM organizations and would not match a quoted, schema-qualified, or TRUNCATE form. None of those exists under the scan roots today, so this is forward-looking hardening rather than a present gap.

🤖 Generated with Claude Code

@bokelley
bokelley requested a review from a team September 16, 2026 18:24
@bokelley
bokelley marked this pull request as ready for review September 17, 2026 07:27

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — organization-merge containment follow-up (#7561).

Checked: the route (503, requireGlobalAdmin retained), the Addie merge_organizations tool (refuses before acquiring a WorkOS client), and the mergeOrganizations service boundary (Promise, throws before getPool/connect/BEGIN) all refuse cleanly. The two operator bypasses (incident script, invariant remediation hint) are closed and a static raw-delete guard is added. previewMerge remains read-only (SELECT/COUNT) and its deps survive the removed imports. No protocol surface (static/schemas/source/**), no migration, no changeset-eligible surface implicated. Thorough unit + mounted-Postgres integration coverage.

high_risk is true only from (modified) entries on admin-tools.ts and cleanup.ts; with no medium-or-higher findings, row 5 does not fire and the modifications are presumed safe. No deletions in sensitive paths, gated_paths false, no author team gate. No blocking or medium findings — falls through to row 9.

@bokelley
bokelley marked this pull request as draft September 17, 2026 08:04
@bokelley
bokelley force-pushed the fix/organization-merge-containment branch from d116704 to 98cf884 Compare September 17, 2026 08:32
@bokelley
bokelley force-pushed the fix/organization-deletion-containment branch from 97652f8 to 468adfa Compare September 17, 2026 08:32
@bokelley
bokelley marked this pull request as ready for review September 17, 2026 08:35
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Sep 17, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — extends #6827 deletion containment to the organization merge writer.

This PR closes the last remaining organization-deletion writer path: mergeOrganizations now throws OrganizationMergeUnavailableError before any DB connection or provider call, POST /cleanup/merge returns a stable 503 while retaining requireGlobalAdmin, the Addie tool refuses execution (acquiring the WorkOS client only in the read-only preview branch), and operator bypasses (incident script, invariant remediation hint) are closed and locked by a static guard test. Fail-closed and well-tested.

Checks:

  • No Critical/High/Medium findings from the reviewer (clean pass).
  • high_risk is true only via (modified) on server/src/addie/mcp/admin-tools.ts and server/src/routes/admin/cleanup.ts; with no medium-or-higher finding on those files, row 5 does not fire and the modifications are presumed safe. No deletions in high-risk paths (row 3 n/a).
  • No protocol wire surface touched (schemas/docs/dist untouched), so no changeset required and no spec-drift/immutability concerns apply.
  • gated_paths false; review_decision APPROVED; no no-auto-approve team match.
  • Prior decision was approve with no findings — net state unchanged/improved.

None of rows 1–8 fire; falls through to row 9 → approve.

@bokelley
bokelley force-pushed the fix/organization-deletion-containment branch 2 times, most recently from 2a97bee to 95f9e0f Compare September 18, 2026 01:05
Base automatically changed from fix/organization-deletion-containment to main September 18, 2026 01:19
@bokelley
bokelley dismissed aao-secretariat[bot]’s stale review September 18, 2026 01:19

The base branch was changed.

Organization merge deleted the secondary organization as its final step.
mergeOrganizations removed the local organization row inside its transaction,
committed, and only then attempted workos.organizations.deleteOrganization,
downgrading a provider failure to a warning while still reporting success. The
Addie path attempted the provider delete twice — once inside mergeOrganizations
and again at step 4 of the tool handler — with WorkOS membership creation
interleaved between the local commit and that second attempt. That is the split
provider/local state commit 97652f8 closed for the two DELETE endpoints, so
merge is now unavailable on the same terms and with the same stable shape.

Three layers, each refusing before any merge-specific effect:

- POST /api/admin/cleanup/merge returns a stable 503
  organization_merge_unavailable before body validation and before any
  merge-specific transaction, WorkOS membership write, provider organization
  deletion, Stripe operation, audit row, cache invalidation or notification.
  Administrative gating is deliberately UNCHANGED — requireGlobalAdmin still
  authenticates and authorizes, unlike the contained deletion routes, because
  nothing here needs to run before authentication.
- The Addie merge_organizations tool refuses execution before it reads provider
  memberships and no longer acquires a WorkOS client on that path; the client
  moved into the read-only preview branch.
- mergeOrganizations itself throws OrganizationMergeUnavailableError before
  getPool(), pool.connect() and BEGIN, so an overlooked caller, a new caller or
  a script bypassing both entry points cannot execute the sequence either.

The unsafe body is removed rather than left unreachable behind the guard: that
sequence is the hazard, so re-enabling merge must be a new reviewed
implementation meeting the lifecycle contract in
docs/contributing/organization-deletion-containment.md, not the deletion of a
guard. After this change no production call site of
workos.organizations.deleteOrganization remains in server/src.

A containment that only covers routes and services is one an operator can walk
around, and two checked-in surfaces handed them the prohibited operation:

- scripts/incidents/2026-05-cleanup-duplicate-prospect-stubs.ts deleted
  organization rows with direct SQL, reasoning from the absence of an admin
  delete endpoint that a script was the sanctioned route. Its destructive path
  is removed, not gated: no database client, no writes, and --execute
  hard-refuses above every other statement. Read-only analysis is preserved.
- The unique-org-per-email-domain invariant emitted a ready-to-paste raw delete
  with the organization ID pre-filled, and pointed at a merge endpoint that does
  not exist. Its remediation_hint now routes to read-only preview-merge
  inspection plus escalation and names both containment codes.

server/tests/unit/organization-delete-statement-guard.test.ts is the static
guard: it scans the JavaScript, TypeScript and SQL files under server/src,
server/scripts and scripts for a raw organization delete and fails on anything
outside an allowlist recording why each remaining occurrence is not
operator-reachable. SQL is covered because a .sql runbook is the likeliest shape
for the next bypass. server/scripts/setup-sandbox.ts is documented and
allowlisted rather than changed: it is sk_test-gated and ID-bounded.

previewMerge and GET /api/admin/cleanup/preview-merge are untouched and remain
read-only (SELECT/COUNT only). The WorkOS preview section describes provider
effects conditionally and the tool footer states that execution refuses;
the existing Stripe and data-movement warnings remain unchanged.

An existing assertion in server/tests/unit/integrity/invariants.test.ts pinned
the raw delete this change removes; it now pins the inverse contract. The three
org-merge suites keep their fixtures and prove the contained service performs no
work on them, and a pre-existing identities cleanup leak in the
primary-organization suite is fixed deterministically by id.

No migration, lifecycle journal, compensation, partial merge or rollback fiction
is introduced. The merge_organizations tool's declared description and usage
hints are intentionally left alone because the Addie tool-surface budget pins a
reviewed profile-contract hash across every declared tool profile; that
re-blessing is recorded as its own follow-up.

Reconstructed directly over landed #7560 at 82b429a.
Preserve main's authorization correction and SEO metadata. Bump Addie
CODE_VERSION to 2026.09.60 as required for this tool implementation change.
Correct the containment document to acknowledge the identity consolidation
refusal already on main, and narrow its conditional-wording claim to the
WorkOS preview section. Original rationale and co-author attribution are
retained; the original public commit and description remain archived.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bokelley
bokelley marked this pull request as draft September 18, 2026 02:57
@bokelley
bokelley force-pushed the fix/organization-merge-containment branch from 98cf884 to d29c6ac Compare September 18, 2026 02:57
@bokelley
bokelley marked this pull request as ready for review September 18, 2026 03:08

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — organization merge deletion containment follow-on to #7560.

This PR contains the remaining organization-deletion writer (the local-delete-then-best-effort-provider-delete split-state path) at the HTTP route, the Addie tool, and the shared service boundary. Operator bypasses (incident script, invariant remediation hint) are closed and a static raw-delete guard is added.

Checks performed:

  • Refusal precedes all I/O; provider-delete SDK call sites removed.
  • Admin gating and read-only preview preserved.
  • Change is correctly out of protocol/changeset scope (no static/schemas/source/**, no docs/reference/**, no dist artifacts touched) — no changeset required.
  • CODE_VERSION bump present.

No Critical/High/Medium findings. No inline comments.

high_risk is true, but all matching files are (modified) with no medium-or-higher finding, so the modification is presumed safe (no escalation on the flag alone). gated_paths is false. No no-auto-approve team match. Prior decision was approve; this remains clean. Falls through to row 9.

@bokelley
bokelley merged commit ca4d45d into main Sep 18, 2026
40 checks passed
@bokelley
bokelley deleted the fix/organization-merge-containment branch September 18, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant