Skip to content

feat(reindex): metadata-only mode to backfill MongoDB missing_params (#2008) - #2213

Draft
mauripunzueta wants to merge 2 commits into
mainfrom
feat/2008-mongo-reindex-missing-params
Draft

mauripunzueta wants to merge 2 commits into
mainfrom
feat/2008-mongo-reindex-missing-params

Conversation

@mauripunzueta

Copy link
Copy Markdown
Contributor

Closes #2008.

What

A metadata-only $reindex mode that fills in missing_params on MongoDB resources written before #1984, without rebuilding the value index.

{"resourceType": "Parameters", "parameter": [
  {"name": "metadataOnly", "valueBoolean": true},
  {"name": "skipExistingMetadata", "valueBoolean": true}
]}
  • metadataOnly: same walk, same extraction (search_index_documents_checked) and same version guard (resource_version_filter, upsert: false) as a rebuild. Only the resources.missing_params update runs; nothing is deleted from or inserted into search_index / search_index_contained.
  • skipExistingMetadata: adds missing_params: {$exists: false} to each update's filter, so resources that already carry the field are left alone.
  • $reindex-status adds metadataOnly: true and metadataUpdated, the number of resources the updates matched. entriesCreated stays 0. Per-resource failures are reported as they are today.
  • Validation: 400 when combined with clearExisting, clearOnly or a bulk index rebuild, and 400 for skipExistingMetadata without metadataOnly. It does combine with POST /{type}/$reindex, with idStart/idEnd, and with named resources.
  • 501 on every backend except MongoDB, including MongoDB with search offloaded (mongodb-elasticsearch never reads missing_params). This is checked before a job id is issued.

How

  • ReindexTarget gains supports_search_metadata() (default false) and write_search_metadata_page() (default: refuse). In metadata mode the driver writes only through writers that support it.
  • MongoDB: a new write_metadata_page in reindex_pipeline.rs. It reuses extract_one (on the rayon pool for large pages), update_missing_metadata (now with a counted variant) and page_outcomes.
  • Docs: in docs/mongodb/search-indexes.md, upgrade step 3 now uses metadataOnly, and there is a new "Metadata-only reindex" section. The definition-maintenance procedure still says full rebuild, because its value-index rows were extracted under the old definitions too. The issue suggested this mode for that case as well, but that would leave stale value rows behind.

Acceptance criteria

  • After a metadata-only run on a pre-feat(mongodb): index missing search parameters on resources #1984 dataset, every live resource has missing_params, and search_index is untouched. mongodb_metadata_only_reindex_backfills_without_touching_the_index unsets the field across a tenant, runs the mode, and asserts that the stored arrays equal what the original writes stored. It also asserts that every search_index + search_index_contained row is byte-for-byte identical, including _id, so a rewrite would show, and that a second tenant is untouched.
  • :missing=true results are identical before and after. The same test checks birthdate:missing=true and =false both ways.
  • Documented in docs/mongodb/search-indexes.md.

Also covered: skipExistingMetadata keeps a sentinel array and respects type scope, offloaded MongoDB returns Unsupported, plus REST parse and validation unit tests.

Verified locally

  • cargo test -p helios-persistence --features mongodb --test mongodb_tests reindex: 81 passed, including the 3 new tests, against a real MongoDB testcontainer.
  • cargo test -p helios-persistence --features mongodb --lib reindex: 167 passed.
  • cargo test -p helios-rest --lib reindex: 20 passed.
  • clippy: no new warnings on the changed lines. The local toolchain flags collapsible_if on existing code.

Not done / follow-ups

  • skipExistingMetadata skips only the write. Every resource in scope is still read and extracted, because the reindex source has no "field absent" filter. Pushing missing_params: {$exists: false} into the Mongo walk (id phase and catch-up rounds) would make a mostly-backfilled tenant cheap to re-run. I left it out to keep the walk untouched.
  • Not run at scale. No measurement yet of the metadata-only throughput against a full rebuild on the Synthea corpus.

…2008)

A `metadataOnly` $reindex recomputes each resource's `missing_params`
with the rebuild's extraction and version guard and writes nothing to
search_index or search_index_contained. `skipExistingMetadata` leaves
resources that already carry the field alone.

- ReindexTarget gains supports_search_metadata/write_search_metadata_page;
  only MongoDB (search not offloaded) implements them, everything else is
  501 before a job starts.
- $reindex-status reports metadataOnly and metadataUpdated.
- Rejected with clearExisting, clearOnly or a bulk index rebuild; combines
  with type scope and idStart/idEnd.
- Upgrade step 3 in docs/mongodb/search-indexes.md now uses it.
@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.18972% with 40 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/persistence/src/search/reindex.rs 77.96% 26 Missing ⚠️
crates/persistence/src/backends/mongodb/storage.rs 38.46% 8 Missing ⚠️
...rsistence/src/backends/mongodb/reindex_pipeline.rs 91.17% 6 Missing ⚠️

📢 Thoughts on this report? Let us know!

…#2008)

- Driver unit tests: metadata goes only through writers that keep it,
  failures are recorded, metadataUpdated/metadataOnly in status, the
  default trait methods refuse, and the validation branches.
- MongoDB: a 40-resource page on a multi-thread runtime takes the
  prepare-pool extraction path; PageFailure::fan_out_metadata replaces the
  inline closure and has its own unit test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mongodb $reindex: metadata-only mode to backfill missing_params on existing resources

1 participant