[mauripunzueta]

Refs #2006. This PR adds an opt-out for missing_params, removes one never-read entry from it, adds a cost benchmark, and reports 1M-resource measurements with a recommendation.

Changes

• HFS_MONGODB_MISSING_METADATA (MongoBackendConfig::missing_metadata, default true). When it is false:

  • writes record no missing_params, and each update or reindex removes the field;
  • idx_resources_missing_params is neither built nor required (an existing one is left in place);
  • :missing=true takes the value-index route, as it does before mongodb: build idx_resources_missing_params under HFS_MONGODB_INDEX_BUILD instead of blocking startup #2001's build finishes, and returns the same results.

  The switch hooks into search_index_documents_checked, which every writer (create, update, transaction, bulk ingest, deferred index, reindex) goes through.

• _lastUpdated and _id are no longer recorded. is_missing_resource_parameter never answers :missing on them from the array, because both are columns of every resource document. Even so, every resource without meta.lastUpdated in its body recorded _lastUpdated as absent, which covers all of Synthea. That was about 1 of the ~18 entries per resource, each one written and indexed but never read. Both places now share records_missing_metadata.

• missing_params_cost_bench (#[ignore]). It loads the same Synthea type mix into a fresh database with metadata on and then off, and reports:

  • throughput and p50/p99/p99.9 for create, update and $bulk-submit;
  • $collStats sizes;
  • WiredTiger cache counter deltas per phase;
  • the per-type absent-name profile.

• Tests.

  • The recorded set omits _id and _lastUpdated but keeps other absent meta parameters.
  • With the switch off, the field an enabled writer left is removed, the index is not built, and :missing=true and :missing=false return exactly what they return with metadata (Accurate and Estimate totals).
  • Local runs: all 17 missing_index tests pass; 129 missing/deferred/reindex integration tests pass; 489 Mongo lib tests pass.

Measurements

Setup

• Local MongoDB 8.0.13, single-node replica set, wiredTigerCacheSizeGB=2; release build; 32 concurrent writers.
• Load: 1M creates in Synthea's natural type mix (41% Observation), then 50k updates with unchanged content, then 200k $bulk-submit into a second tenant. That makes 1.2M resources.
• No sharded cluster was available.

Run 2 (clean, off then on)

	metadata off	metadata on
create ops/s (p50 / p99 / p99.9 ms)	503 (55 / 186 / 591)	585 (50 / 95 / 420)
update ops/s (p50 / p99 / p99.9 ms)	434 (67 / 211 / 651)	487 (57 / 275 / 793)
$bulk-submit res/s	782	782
resources data	2,904 MiB	3,347 MiB (+15.2%, 2,537 → 2,924 B/doc)
idx_resources_missing_params	—	1,803 MiB (run 1: 1,488 MiB)
all other resources indexes	287 MiB	288 MiB
WT bytes read into cache, create phase	71.7 GB	102.0 GB (+42%)
WT bytes written from cache, create phase	286 GB	318 GB (+11%)

Reading the numbers

• Write throughput and latency: no cost was measurable on one node. The on arm ran second and came out slightly faster, which is the corpus being warm in the OS page cache, not an effect of the metadata.

• Run 1 (on then off): the on arm overlapped other builds and benchmarks on the host and showed −7% create, −62% update and −50% bulk. Run 2 does not reproduce those gaps, so they were contention.

• The real costs are storage and cache:

  • the index is about 6× every other resources index combined, roughly 1.5–1.6 KiB per resource;
  • it adds 15% to resources data;
  • it raises cache read traffic by 42% under a cache smaller than the working set.

  On a single node this does not yet show up as write latency. A deployment whose working set already fills the cache is where it would.

• What the array holds: ~18.0 names per resource, ranging from 25.7 for Observation to 6 for CareTeam. 80% of entries are names absent from at least 99% of resources of their type. Examples: _security, _source, _tag and identifier on most types, and performer, specimen and method on Observation.

Recommendation

1. Keep metadata on by default. No write-throughput or latency cost was measurable.
2. Ship the opt-out (this PR) for deployments that never use :missing=true. It saves the index (about 1.5 GiB per million resources here) and 15% of resource data.
3. No static reduced parameter set beyond _id/_lastUpdated.
   • Which names are near-universally absent depends on the data, not the spec.
   • Those names are exactly the ones a :missing=true query would match broadly.
   • Excluding them would push those queries back to the value-index route, the cost feat(mongodb): index missing search parameters on resources #1984 removed.
   • The only always-safe exclusions are names search never reads from the array, and those are now excluded.
4. Not measured here: sharded clusters, and a cache much smaller than the index at 10M+ resources. If a deployment reports cache pressure, the opt-out is the lever.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

[smunini]

Review: tested with and without the change, plus a cost bench

Verdict: the opt-out is correct and costs nothing measurable on writes. Dropping _id/_lastUpdated is safe. I'd merge it once CI is green. Notes are below. I merged main into the branch twice (#2181, #2206, #2207). The conflicts were only tests appended at the end of tests/mongodb/missing_index.rs and tests/mongodb_tests.rs, and it is pushed as 3044e8301.

Correctness

suite	this PR (merged with main)	main
missing_index	24/24	20/20 (without the PR's tests)
reindex	82/82	78/78
deferred / bulk / transaction	all pass	—

• Round-trip check, run locally and not pushed. On, then off with the index left in place, then on again. Resources created while off carry no array. A stale array is removed when its resource is updated while off. :missing=true and :missing=false return exactly the right ids and totals (Accurate and Estimate) in all three phases.
  • The PR's own test drops the index before switching off, so "an existing index is left in place and must not be used" is not covered. A case like this would be worth adding.
• perf(mongodb): skip unchanged missing_params writes on $reindex (#2005) #2207 interaction. MissingUpdate::is_unchanged skips a resource only if it already has neither the array nor the stamp, so a reindex with metadata off still clears what an enabled writer left.
• Remaining failures are not from this PR. The full Mongo suite has 4 deferred_search_index failures, and they fail on main too on the same mongod (enableTestCommands=1). The Mongo lib tests have one failure per full run, a different test each time, and each passes on its own.

Cost bench

Setup: MongoDB 8.0 single-node replica set, wiredTigerCacheSizeGB=1. Load per arm: 300k Synthea creates, 30k updates, 100k $bulk-submit. Arms interleaved off,on / main / on,off / main to cancel the order effect seen in run 2 above.

	off	on (this PR)	main
create ops/s (two runs)	982 / 1225	1054 / 1182	946 / 1143
update ops/s	815 / 988	931 / 916	820 / 803
resources data	971 MiB	1126 MiB (+16%)	1134 MiB
idx_resources_missing_params	—	715–775 MiB	724–837 MiB
WT bytes read into cache, create phase	21–25 GB	33–54 GB	43–64 GB
absent names per resource	—	17.01	18.01

This agrees with the PR body. The write differences are run-to-run noise. The cost is storage and cache traffic. The _lastUpdated change removes exactly one name per resource.

Read side

The recommendation to keep metadata on covers only writes. For searches, the metadata route is currently slower whenever many resources match. This is already on main and not caused by this PR. Medians of 7 warm runs, page with an accurate total. Totals were identical on every row.

query	on	main	off
Observation specimen:missing=true (121,890)	10.2 s	9.4 s	0.56 s
Procedure reason-code:missing=true (30,430)	2.4 s	2.1 s	0.09 s
Observation value-quantity:missing=true (27,059)	2.4 s	1.9 s	0.41 s
Observation encounter:missing=true (0)	3 ms	2 ms	3.7 s

The cause is the $or/$exists:false filter: it scans all 3.0M Observation keys and then runs a $lookup per match. Details and a suggested fix are in #2233. The opt-out is still useful as shipped. It's worth one sentence in the README row: off also gives back the fast "nothing is missing" case, which then takes seconds per page.

Smaller points

1. Deleted resources still get metadata when off. In deferred_index.rs, the deleted-resource branch of the deferred rebuild still writes missing_params: [] and a fingerprint when missing_metadata is false. It's harmless because the scope is is_deleted: false, but it contradicts "writes record no missing_params".
2. Off saves storage only, not CPU. The extractor still computes the absent set; search_index_documents_checked drops it afterwards.
3. Old arrays keep _lastUpdated. Arrays written before this change keep the entry until the resource is rewritten or reindexed. That's harmless because nothing reads it; worth a line in the changelog.