Repository navigation
fix(compaction): keep memory clusters within one scope - #1020
Conversation
rwmjhb
left a comment
There was a problem hiding this comment.
Reviewed 95e7d2e through the orchestrator. Approved: partitioning candidates by exact scope before clustering, and retaining each group's concrete members, prevents cross-scope merging and deletion.
Independent follow-up verification: all 34 tests in compaction-scope-isolation.test.mjs and memory-compactor.test.mjs passed; npm run build passed; regenerated dist/src/memory-compactor.js matched the committed artifact.
Please register the new scope-isolation regression in package.json and scripts/ci-test-manifest.mjs: the normal automated entry points currently omit it. Broad regression verification remains incomplete: npm test timed out after 600 seconds in the real LanceDB update rollback suite, with no compaction assertion failure shown. This approval is not a claim that the full suite passed.
Problem and behavior
An all-scope compaction pass clusters rows by vector similarity before checking their scopes. Two identical memories belonging to different scopes can therefore become one merged row under the first member's scope, after which both original rows are deleted. The registered compaction caller passes
undefinedfor the scope filter, so this is a reachable maintenance path.Group candidates by scope before calling
buildClusters, and retain each plan's group-local members. Same-scope compaction, bounded cluster/delete concurrency, failure handling, and dry-run behavior are preserved. This does not migrate or repair previously compacted rows.Validation
c0e2c7a3a8b5780b6133ddfd9e2f426bf0ea1b6c. The four added regressions produce 1 pass / 3 failures without the fix: cross-scope separation and the two-cluster real/dry-run expectations fail, while ordinary same-scope compaction passes.node --test test/compaction-scope-isolation.test.mjs test/memory-compactor.test.mjs— 34 passed, 0 failed.npm run buildandgit diff --checkpass; generateddist/src/memory-compactor.jsis included.The regressions execute the real compaction runner with a synthetic in-memory store and embedder. No production database, Gateway, or real model was used, and no historical-data repair or hosted-CI success is claimed.
Related: #923 handles different legacy scope/reflection ownership read paths; its changes do not group compaction clusters by scope. This patch addresses the cluster/write/delete boundary only.