feat: combine collapse and delete-marker correctness, plus per-package release automation (RPL-6000, RPL-6780)

[johnnyC9000]

What's on this branch

Three separable concerns, in commit order. The middle one is unrelated to the other two and is here because per-package publishing had to exist before either library change could reach a consumer.

	Commits
1. Same-batch create+delete discards the insert's data	09e7a72
2. Per-package npm release automation	4aef416, 0d3dc73, 0019b04, a05e0a6, 35a3901
3. Cross-group ordering hooks (merged from #252, #253)	60a5f53, 193ec6e

---

1. Same-batch create+delete discards the insert's data

Problem. A dimension/fact delete in this connector is a soft close, so it needs the target row to exist. When an entity is created and then deleted within one combine batch, the collapse in common/datawarehouse/combine.js discarded the insert's data (lastObj = data), leaving a bare tombstone that the loader wrote as a sparse row — data columns null, date/dimension FKs 1, never closed. Confirmed against a production dimension table, where a measurable population of such rows had accumulated.

Fix. Collapse same-key records with last-event-wins ordering, but never discard data. The decision moves into a pure module, common/datawarehouse/combine-records.js:

• insert/update → delete: the delete wins, its markers stamped onto the accumulated record → created-then-soft-closed, not a bare tombstone.
• delete → insert/update: the later write wins and reactivates the entity — loading data clears the deleted flag.
• two writes: deep-merge (unchanged). Lone/leading delete: unchanged bare tombstone.

Extracting it lets the collapse be unit-tested without importing combine.js, which pulls in leo-sdk/aws-sdk.

Same-batch sequence (arrival order)	Before	After
insert, delete	bare sparse tombstone (data lost)	populated + soft-closed
delete, insert	reactivated (write wins)	unchanged
insert, update	deep-merged	unchanged
lone delete	bare tombstone	unchanged

---

2. Per-package npm release automation

Adds .github/workflows/release.yaml and .github/scripts/plan-release.mjs, plus .nvmrc pinning Node 22.

• Push to master publishes clean conventional-commit semver to the latest dist-tag. workflow_dispatch on any branch publishes x.y.z-rc.<run_id> to the rc tag, which is how release candidates come off feature branches without touching latest.

• plan-release.mjs derives each package's bump from conventional commits, baselines against the newest stable version actually published to npm (taking the max with package.json rather than trusting it alone), and enforces a per-package major ceiling so a bump cannot stray into a reserved major line.

• Publishing uses npm Trusted Publishing with OIDC provenance. Per the workflow's own header, each package is configured on npmjs.com to accept publishes only from this repo running this exact workflow file — so the file must not be renamed.

• Manifests gain repository.directory (required for provenance in a monorepo), and the nine packages are normalized off their -rc and drifting versions onto clean minors:

  common 4.0.13-rc → 4.1.0 · postgres 4.0.24-rc → 4.1.0 · entity-table 3.0.22-rc → 3.1.0 · elasticsearch 2.0.5 → 2.1.0 · mongo 3.0.7 → 3.1.0 · mysql 3.0.3 → 3.1.0 · oracle 2.0.1 → 2.1.0 · redshift 3.0.3 → 3.1.0 · sqlserver 4.0.1 → 4.1.0

  0d3dc73 briefly pushed common to 5.1.0; 0019b04 and a05e0a6 pin it back to the v4 line, and 35a3901 adds the ceilings that prevent a repeat.

---

3. Cross-group ordering hooks (#252, #253)

Both were merged into this branch rather than into development, so they land together with the collapse fix they build on.

#252 — opt-in per-record arrival sequence (60a5f53). combine() already assigns every record a batch-global arrival counter before grouping by natural key, then discards it at line.substr(42). This carries it on the record as __leo_seq__ when a client opts in via client.emitCombineSequence, and restores the opts argument combine() accepted but dropped. Default off — with emitSequence false the emitted records are byte-identical to before, so the postgres/Redshift path is unaffected.

#253 — key a delete marker by the table's real natural key (193ec6e). checkforDelete wrote every marker under a column literally named id. For a table whose natural key is named something else, that left the natural-key column undefined on every marker, so all of them hashed into one combine group and every delete but the last was silently dropped — data loss, not just misordering. Marker construction moves into common/datawarehouse/delete-marker.js and resolves the table's real natural key. Composite and unresolvable keys keep the historical id shape rather than guess.

---

Tests

common/ had no combine coverage before this branch. Added:

• common/test/datawarehouse/combine.test.js — 7 cases across every collapse path
• common/test/datawarehouse/combine-sequence.test.js — 4 cases on sequence carry-over through the fold
• common/test/datawarehouse/delete-marker.test.js — 6 cases on marker keying, including distinct-group and non-id natural keys

All three are pure-function tests with no leo-sdk import, so they run without AWS credentials or a configured RStreams environment. npm test in common/ is green (17 tests). eslint reports 41 pre-existing style errors on these files, unchanged by this branch — the linter and the committed style have been out of sync here for a while.

Not covered: release.yaml and plan-release.mjs have no tests. The release path has been exercised by running it, not by a suite.

---

Scope and rollout

• Shared leo-connector-common; consumers pin an exact version, so nothing floats in automatically — a change reaches a connector only when that connector bumps its pin.
• First consumer of the collapse fix is a downstream Delta-table connector, which carries the companion importDimension change; the two close the defect together.
• Redshift/postgres unaffected until a consumer on that path bumps its pin. The same collapse defect is latent there behind a short read window; the mirror fix is to run the soft-close flush after the insert.
• feat(common): opt-in order-field fold in datawarehouse combine — 4.x line (PMT-4302) #254 (combineOrder) is stacked on this branch and targets it directly, so this branch must not be renamed or deleted while that PR is open.

---

Open review findings

Three Cursor Bugbot findings are unresolved as of 193ec6e, all in the release automation:

1. High — release.yaml:14. The push trigger still lists fix/rpl-5795-combine-soft-delete-ordering alongside master, so every push to this branch runs the release pipeline. Non-master refs publish as -rc.<run_id>, so this is rc-channel only, but the branch name would remain in the trigger on master after merge.
2. High — plan-release.mjs:207. git cat-file -e succeeds with empty stdout, so !gitOrNull(...) is always true and readEventBefore always returns null. resolveRange then falls back to HEAD~1..HEAD, which with --no-merges can miss a whole push.
3. Medium — plan-release.mjs:344-361. The ceiling check only rejects when the npm-baseline candidate exceeds the cap. If package.json is already above the ceiling, maxVersion keeps the higher value and the package stays in the publish matrix.

---

🤖 Generated with Claude Code

---

Note

Medium Risk
Warehouse load/combine behavior changes affect dimension/fact correctness when consumers upgrade leo-connector-common; new release automation can publish wrong versions or miss commits if misconfigured (open review notes on push triggers and commit-range logic).

Overview
Fixes datawarehouse batch collapse and delete-marker grouping in leo-connector-common, and adds per-package npm release automation for the monorepo.

Combine / soft-delete: Same-natural-key collapse moves to combine-records.js with last-event-wins rules that keep row data when insert/update is followed by delete (populated soft-close instead of a sparse tombstone), and drop delete intent when a write follows a delete. combine() also honors its opts again and can opt in to stamping __leo_seq__ via client.emitCombineSequence (default off, so existing emitted records stay unchanged).

Delete markers: delete-marker.js keys markers on the table’s real natural-key column (via load.js), fixing batches where non-id keys left the NK undefined and collapsed many deletes into one combine group.

Release: New .github/workflows/release.yaml and plan-release.mjs plan bumps from conventional commits per package folder, baseline against npm’s max stable version (with temporary major ceilings), commit version bumps/tags on master, and publish via OIDC trusted publishing (latest on master, x.y.z-rc.<run_id> on other branches). Adds .nvmrc (Node 22), repository.directory on manifests, and bumps all nine connector packages to clean minor versions.

Tests: New unit tests for combineRecords, delete-marker keying, and sequence carry-over in common/test/datawarehouse/.

^{Reviewed by Cursor Bugbot for commit 34819d4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[ch-snyk-sa]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[cursor]

Feature branch left in push trigger

High Severity

The push trigger still lists fix/rpl-5795-combine-soft-delete-ordering, so every push to that branch runs the full release pipeline and can publish rc packages to npm. The header comments say RCs are meant to come only from workflow_dispatch; this looks like temporary test scaffolding that would also remain on master after merge.

 

^{Reviewed by Cursor Bugbot for commit 0d3dc73. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a05e0a6. Configure here.}