Repository navigation
feat(reindex): metadata-only mode to backfill MongoDB missing_params (#2008) - #2213
Draft
mauripunzueta wants to merge 2 commits into
Draft
mauripunzueta wants to merge 2 commits into
mauripunzueta wants to merge 2 commits into
Conversation
…2008) A `metadataOnly` $reindex recomputes each resource's `missing_params` with the rebuild's extraction and version guard and writes nothing to search_index or search_index_contained. `skipExistingMetadata` leaves resources that already carry the field alone. - ReindexTarget gains supports_search_metadata/write_search_metadata_page; only MongoDB (search not offloaded) implements them, everything else is 501 before a job starts. - $reindex-status reports metadataOnly and metadataUpdated. - Rejected with clearExisting, clearOnly or a bulk index rebuild; combines with type scope and idStart/idEnd. - Upgrade step 3 in docs/mongodb/search-indexes.md now uses it.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…#2008) - Driver unit tests: metadata goes only through writers that keep it, failures are recorded, metadataUpdated/metadataOnly in status, the default trait methods refuse, and the validation branches. - MongoDB: a 40-resource page on a multi-thread runtime takes the prepare-pool extraction path; PageFailure::fan_out_metadata replaces the inline closure and has its own unit test.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2008.
What
A metadata-only
$reindexmode that fills inmissing_paramson MongoDB resources written before #1984, without rebuilding the value index.{"resourceType": "Parameters", "parameter": [ {"name": "metadataOnly", "valueBoolean": true}, {"name": "skipExistingMetadata", "valueBoolean": true} ]}metadataOnly: same walk, same extraction (search_index_documents_checked) and same version guard (resource_version_filter,upsert: false) as a rebuild. Only theresources.missing_paramsupdate runs; nothing is deleted from or inserted intosearch_index/search_index_contained.skipExistingMetadata: addsmissing_params: {$exists: false}to each update's filter, so resources that already carry the field are left alone.$reindex-statusaddsmetadataOnly: trueandmetadataUpdated, the number of resources the updates matched.entriesCreatedstays 0. Per-resource failures are reported as they are today.clearExisting,clearOnlyor a bulk index rebuild, and 400 forskipExistingMetadatawithoutmetadataOnly. It does combine withPOST /{type}/$reindex, withidStart/idEnd, and with named resources.mongodb-elasticsearchnever readsmissing_params). This is checked before a job id is issued.How
ReindexTargetgainssupports_search_metadata()(defaultfalse) andwrite_search_metadata_page()(default: refuse). In metadata mode the driver writes only through writers that support it.write_metadata_pageinreindex_pipeline.rs. It reusesextract_one(on the rayon pool for large pages),update_missing_metadata(now with a counted variant) andpage_outcomes.docs/mongodb/search-indexes.md, upgrade step 3 now usesmetadataOnly, and there is a new "Metadata-only reindex" section. The definition-maintenance procedure still says full rebuild, because its value-index rows were extracted under the old definitions too. The issue suggested this mode for that case as well, but that would leave stale value rows behind.Acceptance criteria
missing_params, andsearch_indexis untouched.mongodb_metadata_only_reindex_backfills_without_touching_the_indexunsets the field across a tenant, runs the mode, and asserts that the stored arrays equal what the original writes stored. It also asserts that everysearch_index+search_index_containedrow is byte-for-byte identical, including_id, so a rewrite would show, and that a second tenant is untouched.:missing=trueresults are identical before and after. The same test checksbirthdate:missing=trueand=falseboth ways.docs/mongodb/search-indexes.md.Also covered:
skipExistingMetadatakeeps a sentinel array and respects type scope, offloaded MongoDB returnsUnsupported, plus REST parse and validation unit tests.Verified locally
cargo test -p helios-persistence --features mongodb --test mongodb_tests reindex: 81 passed, including the 3 new tests, against a real MongoDB testcontainer.cargo test -p helios-persistence --features mongodb --lib reindex: 167 passed.cargo test -p helios-rest --lib reindex: 20 passed.collapsible_ifon existing code.Not done / follow-ups
skipExistingMetadataskips only the write. Every resource in scope is still read and extracted, because the reindex source has no "field absent" filter. Pushingmissing_params: {$exists: false}into the Mongo walk (id phase and catch-up rounds) would make a mostly-backfilled tenant cheap to re-run. I left it out to keep the walk untouched.