/search paginates in application memory, so every page costs the server everything up to that page #309

Summary

/search fans out to two Atlas Search indexes as two separate db.aggregate() calls, then merges, deduplicates, sorts, and slices the combined result in Node (controllers/search.js:282). Each branch caps at $limit: limit + skip (controllers/search.js:211,216), so a request for page 40 makes Atlas produce and return 4000 documents per index and makes Node process up to 8000 of them in order to hand back 100.

/query pushes the same work down to Mongo — db.find(props).limit(limit).skip(skip) (controllers/crud.js:87) — and its latency is flat with depth. /search cannot, and its latency is not.

Nothing about $search requires this. A single Atlas Search pipeline sorts by score natively, returns each document once, and can end in $skip/$limit. The blocker is the two-index fan-out, not the search stage.

This is the performance half of the /search work. The correctness half is #307, which should land first and independently — a client can page /search honestly once that lands, just not cheaply.

Why this matters

Cost scales with depth, not page size. Dev, {"searchText":"line"}, limit=100:

skip=0     -> 100 documents in  379ms
skip=1000  -> 100 documents in  664ms
skip=2000  -> 100 documents in  781ms
skip=3000  -> 100 documents in 1172ms
skip=3700  -> 100 documents in 2021ms
skip=3800  ->  25 documents in 1954ms
skip=3900  ->   0 documents in 1784ms
skip=5000  ->   0 documents in 1802ms
skip=20000 ->   0 documents in 1815ms

/query at the same depths, {"__rerum.APIversion":{"$exists":true}}, limit=100:

skip=0     -> 100 documents in 340ms
skip=1000  -> 100 documents in 296ms
skip=5000  -> 100 documents in 323ms
skip=20000 -> 100 documents in 369ms
skip=99000 -> 100 documents in 445ms

The two seconds spent to return nothing at skip=5000 is the cost of producing and merging the entire matching set. Cost plateaus once the result set is exhausted, which confirms the bound is min(total matches, limit + skip) per branch. Production shows the same shape at smaller scale: line has 574 matches there, and the empty pages at skip=1000, 2000, and 20000 still cost roughly 850-900 ms each.

The ceiling is the skip maximum. At the effective maximum of 100000, a sufficiently broad term at maximum depth asks Node to hold on the order of 100000 full RERUM documents per branch in memory and sort them. Not identifiers — whole documents.

It is a shared-process cost. This work happens in the API process, not in Atlas. Under PM2 cluster mode a few concurrent deep searches contend for the same workers that serve every other endpoint.

Affected lines

Proposed change

Three options, best first. This issue should settle which one before any code is written.

1. One combined Atlas Search index (recommended)

Define a single index covering both vocabularies — the IIIF 3.0 paths (body.value, bodyValue, and the items / annotations embedded documents) and the IIIF 2.1 paths (resource.chars, resource.cnt:chars, and the resources / otherContent / sequences embedded documents).

/search becomes one $search with all the existing should clauses, followed by $skip and $limit:

• Ranking becomes native. Atlas returns $search results in descending score order across all clauses, and every document is scored once against one index rather than compared across two.
• Deduplication disappears. Atlas returns each document exactly once; there is no key to derive and nothing to drop. mergeSearchResults() can be deleted, and with it both bugs from /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307.
• Paging runs in the database, so cost stops scaling with depth.

Cost: an Atlas-side index change, and both index generations must exist during the transition.

2. $unionWith in a single pipeline

Keep both indexes, run the IIIF 2.1 branch as a $unionWith sub-pipeline against the same collection, then $group on _id, $sort by score, and $skip/$limit — all server-side. Grouping on _id in Mongo compares structurally, so embedded-object ids group correctly there.

This moves the work off the API process but not off the cluster: the $group is a blocking stage over the union, so Atlas still materializes the merged set. Smaller change, smaller payoff. Confirm $search is permitted as the first stage of a $unionWith sub-pipeline on our cluster tier before committing to this — it is not universally available.

3. Keep the shape, bound the damage

If neither restructuring is scheduled soon, enforce a /search-specific skip maximum far below the /query maximum, since the two endpoints pay very different prices for the same depth. This is a mitigation, and it makes the two endpoints inconsistent — exactly what the shared-contract work argues against — so it should be a deliberate, documented decision rather than a default.

Notes

• Breaking. Option 1 changes relevance scoring, and should: a single index scores a document once across all matched clauses. Today the two branches' scores are never compared against each other at all, so ordering will shift substantially. That is the point, not a regression. Membership should change only by the currently dropped documents reappearing.
• Depends on Atlas Search index definitions are not in the repository #308. Reviewing a new index definition against two that exist only in a browser tab is not a review.
• /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307 is closed. Its ranking bug is fixed: mergeSearchResults() sorts on __rerum.score as of bd91fc7, merged in Behavioral Fixes for limit and skip #310. Its dedup key was deliberately not fixed and moves here, because there is nothing for the dedup to do and option 1 removes it outright.
• rel="next" for /search belongs to this issue. /query paged responses carry no rel="next", so no client can tell a full page from the last page #302 delivers it for /query only. /search needs to over-fetch one record to know whether another page exists. Under the current shape that is limit + skip + 1 per branch; under option 1 it becomes the ordinary single-pipeline limit + 1. Reuse setNextPageLink(), which /query paged responses carry no rel="next", so no client can tell a full page from the last page #302 adds to controllers/utils.js, so both endpoints emit rel="next" the same way, appended after the JSON-LD context link. /query paged responses carry no rel="next", so no client can tell a full page from the last page #302 deliberately sends no first, prev, or last.
• 2026-09-17 ruling: objects whose _id is not a string need not appear in /query or /search results. They are old, bad data. That settles the dedup question — the documents the toString() key collapses do not have to be served, so this is no longer a membership defect. What is left is that the drop is accidental and partial: one such document per response survives the key, and pages under-fill (manuscript at limit=100 returns 50 against 68 raw matches). Exclude non-string _id in the pipeline instead, and the toString() key becomes harmless, because strings key uniquely. Measured 2026-09-17: prod holds 3 such documents, all ObjectId _id test junk with no @id or __rerum; dev holds over 100000.
• The /search half of that exclusion is tracked in Objects whose _id is not a string are served by /query and dropped by accident in /search #313, which also covers /query. If this issue's restructuring lands first, the condition belongs in its single pipeline instead of the two $match stages Objects whose _id is not a string are served by /query and dropped by accident in /search #313 describes.
• Decide what next does at the skip maximum. /query still sends it past the maximum, where following it is answered with the 400 (/query paged responses carry no rel="next", so no client can tell a full page from the last page #302), because omitting it tells the client the walk is complete when it is not. /search should match unless there is a reason not to. A search cursor, below, makes the question moot.
• Once this lands, /search can take a real cursor. Atlas Search's searchAfter / searchSequenceToken is the keyset equivalent for $search results — confirm availability on our cluster tier and server version before planning it. See Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303 for why a cursor before this lands would be dishonest.
• Whatever lands must keep the property already verified: sequential pages equal to a single larger request, in the same order.

Acceptance criteria

• An option is chosen and the reasoning is recorded on the issue before implementation
• /search pagination is applied in MongoDB rather than in application memory
• /search latency at depth is flat with respect to skip, comparable to /query, measured with the same probes used above
• Documents whose _id is not a string are excluded by the search pipeline rather than collapsed by the application-side dedup, so every page fills to limit while matches remain and a walk total is honest (Objects whose _id is not a string are served by /query and dropped by accident in /search #313; 2026-09-17 ruling, inherited from /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307)
• Results are ordered by descending relevance score, verified by asserting the score sequence never increases
• Sequential paged results still equal a single larger request, in the same order
• Relevance ordering change is measured on representative queries and accepted before cutover
• /search and /search/phrase both covered, and the unmounted search variants updated to match
• The ordering guarantee is stated in openapi/contracts/core-provider.openapi.yaml, not only in JSDoc
• /search and /search/phrase responses carry Link: …; rel="next" while more results exist, and omit it on the final page
• The JSON-LD context link is still present alongside rel="next"
• The over-fetched record is never returned to the client, at any limit including the maximum
• A client following only rel="next" walks a /search result set to completion and terminates, with no length-based stop condition
• The behavior of next at the skip maximum is decided and documented
• The unmounted searchFuzzily, searchWildly, and searchAlikes emit the same links, so the pattern is not carried forward without them when they are routed
• Regression tests cover rel="next" presence, its absence on the final page, and the untrimmed extra record never appearing
• public/API.html documents rel="next" for the search endpoints, and the Link response header is declared for /api/search and /api/search/phrase in openapi/contracts/core-provider.openapi.yaml
• The mergeSearchResults() JSDoc matches the code: no stale toString() keying, and the caveat that independently scored branches are not strictly comparable while that is still true (inherited from /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307)