[TriDefender]

Fixes #1031

What was broken

Two write paths mishandled databases where several physical rows share one memory id (the reporter had 3,404 rows in 332 duplicate-ID groups, 331 of them same-text/metadata with different vectors):

1. backfillSearchTextRows (used by both backfill-search-text --apply and the search_text schema migration) fetched the whole duplicate-ID group per stale id and passed multiple source rows with the same key to mergeInsert("id").whenMatchedUpdateAll() — LanceDB rejects that with Ambiguous merge insert: multiple source rows match the same target row. Because backfill commits per batch, a failed run could leave earlier batches already committed.
2. applyManualRecallMetadataBatch (silent data corruption) deduped its read by id keeping the last physical row only, then merge-inserted one source row per candidate id with whenMatchedUpdateAll — LanceDB overwrote every same-id target row with that single source row, collapsing the duplicate group's distinct vectors to one while returning success (verified by the reporter in a disposable database).

Per the issue triage, the duplicate-ID provenance (importEntry append / upsert delete-suppression / reembed defaults) is intentionally not addressed here — this change only makes both write paths safe when duplicate ids already exist.

How it is fixed

backfillSearchTextRows (src/store.ts)

• Batches are cut from distinct stale ids, so one duplicate id can never be processed in two batches (which would break updated === stale accounting on clean runs).
• Fetched rows are regrouped by id per batch — this doubles as the recheck for duplicates introduced after the initial scan.
• Ids occurring exactly once keep the existing batched mergeInsert("id").whenMatchedUpdateAll() — semantics unchanged for the common case.
• Duplicate ids are rewritten per (text, metadata) subgroup via table.update({ where, values }) touching only search_text (null-aware IS NULL matchers, escapeSqlLiteral on every literal, a search_text IS NULL OR != expected guard). Vectors and all other fields are untouched by construction; updated counts actual rowsUpdated, and an undercount emits a console.warn instead of inflating the counter.
• The schema-migration caller and the dryRun contract are unchanged (dry-run stays fully read-only; counters still count stale rows).

applyManualRecallMetadataBatch (src/store.ts)

• The read now keeps all rows per id instead of last-row-wins.
• Per-candidate scope matching uses the normalized scope (row.scope ?? "global"); a candidate with no matching row gets the existing non-retryable scope-changed error.
• Candidates are partitioned per chunk: ids with exactly one matching row (and unique among the chunk's candidates) stay on the byte-identical single-row mergeInsert path including classifyUncertainMerge; everything else goes to a new duplicate update path.
• The duplicate path merges each row's own metadata snapshot: parseSmartMetadata derives valid_from/last_accessed_at from the row's own timestamp and memory_category/fact_key from the row's own category, so subgroups are keyed by (text, old metadata, timestamp, category) and each table.update carries null-aware id/text/metadata/timestamp/category matchers plus the same scope guard as the merge path. values contain only metadata and the re-derived search_text (when searchTextColumnReady) — vectors are never written, so they cannot collapse.
• rowsUpdated is accounted per subgroup; a thrown or mismatched statement classifies each row by re-read (succeeded / pending / unknown), mirroring classifyUncertainMerge's recovery semantics, and noteDataModification() fires once per chunk whenever data may have changed (keeps FTS index folding fresh).

Tests

Two new files following the repo's node --test + jiti conventions, synthetic fixtures, float32-exact vectors so preserved vectors deepEqual bit-for-bit:

• test/backfill-search-text-duplicate-ids.test.mjs (5 cases): dry-run read-only, duplicate/mixed groups keep per-row distinct vectors and per-row expected search_text, apostrophe/newline escaping round-trip, and the schema migration path on a pre-upgrade duplicate-ID table.
• test/manual-recall-duplicate-ids.test.mjs (7 cases): the first tests to drive the real applyManualRecallMetadataBatch against a real LanceDB table — per-row snapshot increments on duplicate groups, mixed-scope groups (only matching rows updated, sibling byte-identical), scope-mismatch errors, mergeInsert/update failure injection with recovery classification, two candidates with the same id under different scopes, and a regression case for per-row derived metadata (valid_from) when siblings share text+metadata but differ in timestamp (this one was red against an intermediate implementation and is green now).

Regression suites re-run green: fts-search-text-column 7/7, manual-recall-metadata-queue 4/4, migrate-legacy-schema 2/2, cli-subcommand-attachment 22/22, update-consistency-lancedb 6/6; tsc --noEmit and npm run build clean (dist/src/store.js rebuilt — dist/ is committed in this repo).

The issue reporter's own repro script, run against the rebuilt build, prints exactly the reporter's post-fix expectation (the unpatched build prints Ambiguous merge insert):

{"result":{"scanned":2,"stale":2,"updated":2},"rows":2,"distinctVectors":2,"stale":0}

Notes

• Known consequence, consistent with the approved recovery semantics: retrying a candidate after a partial duplicate-path failure re-applies accessCountDelta to rows that were already updated (the queue re-enqueues the original update on retryable errors). Worth an idempotent-delta follow-up.
• update({ where: "_rowid = …" }) is not usable on @lancedb/lancedb 0.26.2 (schema planner rejects _rowid in update statements), which is why rows are disambiguated by column matchers rather than rowids.