/search paging is unfinished: cost scales with depth, no rel="next", and pages under-fill #306

Activity

1. thehabes

   added sub-issues on Sep 3, 2026

   • /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307
   • Atlas Search index definitions are not in the repository #308
   • /search paginates in application memory, so every page costs the server everything up to that page #309
2. thehabes commented on Sep 17, 2026

   

   thehabes

   on Sep 17, 2026

   MemberAuthor

   More actions

   Original investigation, verified 2026-09-02 and re-run 2026-09-03. This was the body of this issue until 2026-09-17, kept here because the evidence is still the source for the raw match counts. Superseded by the current body: the ranking bug it describes is fixed (#310), and the dedup fix it proposes is not wanted (see #313).

   Full report as written

   Detailed verification report. This is the full investigation this issue and its sub-issues were written from.

   Summary

   /search fans out to two Atlas Search indexes as two separate db.aggregate() calls, then merges, deduplicates, sorts, and slices the combined result in Node (controllers/search.js:282). Three problems follow from that design, two of them user-visible:

   1. Relevance ranking does not work at all. The merge sorts on a field that does not exist, so the comparator returns 0 for every pair and the sort is a no-op. Results come back as "every IIIF 3.0 hit, then every IIIF 2.1 hit," regardless of score. On production the single best-scoring match for line — scoring 84 — is ranked 6th, behind five matches scoring under 2.
   2. The deduplication step drops matching documents. The merge keys each document by _id?.$oid || _id?.toString(). Legacy documents whose _id is an embedded object (the v0 import from 165.134.105.29/annotationstore) all key as the string "[object Object]", so the first one wins and every other one is silently discarded. On the dev collection, 182 of the 4007 documents matching line are unreachable through /search on any page.
   3. Paging costs scale with depth. /query pushes pagination down to Mongo — db.find(props).limit(limit).skip(skip) (controllers/crud.js:87). /search cannot, so every page costs the server everything up to that page, on every request.

   Nothing about $search requires any of these. A single Atlas Search pipeline sorts by score natively, returns each document once, and can end in $skip/$limit. The blocker is the two-index fan-out, not the search stage — which is why one change fixes all three.

   The pagination contract problems shared by both endpoints — silent clamping, the repeating page past the skip maximum, the inert environment variables — are tracked separately in the paged-reads issue. This issue is about /search's own internals.

   Why this matters

   The ranking bug

   buildDualIndexQueries() writes each document's relevance score to __rerum.score (controllers/search.js:210,215):

   { $addFields: { "__rerum.score": { $meta: "searchScore" } } }

   mergeSearchResults() then sorts on a top-level score (controllers/search.js:45):

   return merged.sort((a, b) => (b.score || 0) - (a.score || 0))

   No document has a top-level score. Every comparison evaluates 0 - 0, the sort does nothing, and the merged array keeps its construction order — [...results1, ...results2], IIIF 3.0 first. The JSDoc on both search handlers promises results "sorted by relevance score (highest first)". They are not, and have not been.

   This is not subtle in practice. On production, searching line returns five IIIF 3.0 matches scoring between 1.69 and 2.25, and then at index 5 the score jumps to 83.9 — the IIIF 2.1 boundary and the best match in the entire set, ranked behind five documents that score fifty times lower. On the dev collection, where the IIIF 3.0 index has 205 hits for the same term, the best match (score 12.0) is ranked 206th; a client showing "top 10 results" never sees it, and a client paging at 100 per page finds it on page 3.

   Any deployment where one vocabulary dominates the corpus will see the other vocabulary's matches effectively buried, no matter how well they score.

   The dropped documents

   mergeSearchResults() (controllers/search.js:37) derives its deduplication key as:

   const id = result._id?.$oid || result._id?.toString()

   That is correct for an ObjectId and for a string. It is wrong for a plain object: ({}).toString() is "[object Object]". The dev collection holds 6427 documents whose _id is an embedded object — the serialized Java ObjectId shape ({date, inc, machine, time, …}) left behind by the v0 import. Every one of them that matches a search collapses onto the same key, the first is kept, and the rest are dropped before paging begins.

   Measured directly against the two branch pipelines on the dev collection at full depth ($limit 100100 per branch):

   Term	presi3 raw	presi2 raw	Cross-index overlap	Embedded-object _id	Merged	Dropped
   line	205	3802	0	183	3825	182
   text	2301	46	0	4	2344	3
   manuscript	0	68	0	19	50	18
   page	63	3	0	0	66	0

   Two consequences for a paging client:

   • Records are unreachable. No skip value ever returns the 182 dropped line documents. A client that walks /search to the empty page believes it has everything.
   • Pages under-fill before results are exhausted. manuscript with the default limit=100 and skip=0 returns 50 documents from 68 matches. A length < limit client concludes there are 50 results and stops. This is the hazard the per-branch $limit-before-dedup shape always had; it is not hypothetical.

   Cross-index overlap, by contrast, really is zero for every term tried — a document is written in either IIIF 3.0 or IIIF 2.1 shape, and the two indexes cover disjoint field paths. The deduplication is not doing the job it was written for; it is doing a different, harmful one.

   Production currently has no embedded-object _id documents ({"_id.inc":{"$exists":true}} returns nothing on store.rerum.io), so the drop is not observable there today. It is a defect in the code path regardless: any document whose _id is neither an ObjectId nor a string will be lost the same way, and the dev deployment is where clients are developed against.

   The paging cost

   Cost scales with depth, not page size. Each branch caps at $limit: limit + skip (controllers/search.js:211,216), so a request for page 40 makes Atlas produce and return 4000 documents per index, and makes Node sort and deduplicate up to 8000 of them, in order to hand back 100. /query at the same depth asks Mongo to skip and returns 100.

   The ceiling is set by the skip maximum. With the effective maximum at 100000 (see the paged-reads issue — the configured 10000 is not in effect on any deployment), a sufficiently broad search term at maximum depth asks Node to hold on the order of 100000 documents per branch in memory and sort them. Full RERUM documents, not identifiers. The two issues compound: the cap that is too high is also the cap that bounds this blast radius.

   It is a shared-process cost. This work happens in the API process, not in Atlas. Under PM2 cluster mode a few concurrent deep searches contend for the same workers that serve every other endpoint.

   Evidence

   Verified 2026-09-02. "Dev" means the local pm2 instance at localhost:3001, which reads the same collection as devstore.rerum.io (the rerum-test cluster, annotationStore.alpha); the same numbers were confirmed on devstore.rerum.io where noted. "Prod" means https://store.rerum.io. All probes are read-only.

   Independently re-run twice on 2026-09-03, read-only, against localhost:3001 and directly against the presi3AnnotationText / presi2AnnotationText indexes (bypassing the endpoint to isolate the merge step), with the score-ordering check also re-run against devstore.rerum.io and store.rerum.io. Every number reproduced: the line/text/manuscript/page raw and dropped counts, the score jump for line at index 205 (1.958 → 12.013, rank 206th of 500 returned) on both local and devstore, the identical boundary on /search/phrase, the manuscript under-fill (68 raw matches, 50 returned at limit=100), the empty pages for line at skip=100000/100100/150000, the 6427 embedded-object _id documents on dev against none on production, and the tiling and latency-scaling shapes. No document keyed on any embedded-object _id was found to collide with a distinct document's key by anything other than the "[object Object]" collision already described — the mechanism is exactly as diagnosed. One number moved: production's best line score now reads 83.928 rather than 83.925, which is ordinary drift as the corpus and index change and does not affect the rank.

   Ranking is broken, measured through the endpoint

   The results of a single /search?limit=500 request, read back via .__rerum.score. A correctly sorted list never increases.

   Deployment	Term	Returned	Score increases at	Values at the increase	Best score in set	Rank of best
   prod	line	500	index 5	1.690 → 83.928	83.928	6th
   prod	text	24	index 2	2.452 → 6.958	6.958	3rd
   prod	page	298	index 5	1.843 → 3.971	3.971	6th
   dev, devstore	line	500	index 205	1.958 → 12.013	12.013	206th

   In every case there is exactly one increase, and it sits exactly at the boundary between the two branches. /search/phrase shows the identical boundary (dev, line: index 205, 1.958 → 12.013). Confirmed at the source by running the pipelines directly: every returned document has __rerum.score populated and top-level score undefined, so (b.score || 0) - (a.score || 0) is 0 for every pair.

   Documents are dropped by the merge

   The table in "The dropped documents" above. Confirmed by grouping raw branch output by the merge's own key: for manuscript, 68 raw results yield 50 distinct keys, and the one colliding key — "[object Object]" — has 19 members, each a different document with a different _id, all @type: oa:Annotation from the v0 import.

   Confirmed a second way, end to end through the endpoint rather than against the pipelines. Walking /search for line at limit=500 until it returns an empty page reaches exactly 3825 records — the merged count, not the 4007 raw matches. The 182 dropped documents are not on a later page; they are not on any page:

   POST /v1/api/search?limit=500&skip=0 … until empty  -> 3825 records total
   raw branch matches (205 presi3 + 3802 presi2)       -> 4007
   

   The under-fill is visible in a single request with no walking at all:

   POST /v1/api/search?limit=100&skip=0 {"searchText":"manuscript"}
   -> 50 documents   (100 requested, 68 actually match)
   

   A length < limit client stops right there and reports 50 of 68.

   Latency scales with depth

   Dev, {"searchText":"line"}, limit=100:

   skip=0     -> 100 documents in  379ms
   skip=1000  -> 100 documents in  664ms
   skip=2000  -> 100 documents in  781ms
   skip=3000  -> 100 documents in 1172ms
   skip=3700  -> 100 documents in 2021ms
   skip=3800  ->  25 documents in 1954ms
   skip=3900  ->   0 documents in 1784ms
   skip=5000  ->   0 documents in 1802ms
   skip=20000 ->   0 documents in 1815ms
   

   /query at the same depths for contrast, {"__rerum.APIversion":{"$exists":true}}, limit=100:

   skip=0     -> 100 documents in 340ms
   skip=1000  -> 100 documents in 296ms
   skip=5000  -> 100 documents in 323ms
   skip=20000 -> 100 documents in 369ms
   skip=99000 -> 100 documents in 445ms
   

   The two seconds spent to return nothing at skip=5000 is the cost of producing and merging the entire matching set. Cost plateaus once the result set is exhausted, which confirms the bound is min(total matches, limit + skip) per branch — for a broad term on a large collection, that bound is the skip maximum. Prod shows the same shape at a smaller scale. line has 574 prod matches: skip=0 returns 100 in 621 ms, skip=500 returns the last 74 in 941 ms, and the empty pages at skip=1000, 2000, and 20000 still cost roughly 850–900 ms each.

   Paging is self-consistent, even though the ordering is wrong

   Five sequential limit=100 pages return exactly the same 500 records in exactly the same order as a single limit=500 request. The tail terminates: skip=3700 returns 100, skip=3800 returns 25, skip=3900 returns 0 — 3825 records reachable, which is exactly the merged count after the 182 drops.

   Worth being precise about why: each branch returns a deterministic prefix of its own result set, the caps grow with skip, and the no-op sort leaves the concatenation order stable. So pages tile correctly — membership and page boundaries are stable. They are tiling a list that is in the wrong order and that is missing documents, and no client can page its way to either the best match or the dropped ones.

   Past the skip maximum, /search returns an empty page rather than the repeated page described in the paged-reads issue, because the slice runs off the end of the merged set: line, the, and of all return 0 documents at skip=100000, 100100, and 150000 on dev. The repeated page would need a term with more than 100000 merged matches, which no term tried reaches.

   Affected lines

   File	Line	Current
   controllers/search.js	82	buildDualIndexQueries() builds two independent pipelines
   controllers/search.js	84, 153	Two separate indexes: presi3AnnotationText, presi2AnnotationText
   controllers/search.js	211, 216	$limit: limit + skip per branch — the depth cost
   controllers/search.js	210, 215	Score written to __rerum.score
   controllers/search.js	37	Dedup key _id?.$oid || _id?.toString() collapses embedded-object _id values to "[object Object]" — the drop bug
   controllers/search.js	45	Sorts on top-level score, which is never set — the ranking bug
   controllers/search.js	32	mergeSearchResults() deduplicates in Node, after the per-branch cap
   controllers/search.js	230, 238, 300, 386, 464, 557	JSDoc promising results "sorted by relevance score"
   controllers/search.js	282, 368	merged.slice(skip, skip + limit) — pagination in application memory
   controllers/search.js	642, 660	Same shape hand-rolled in searchAlikes()
   controllers/crud.js	87	/query for contrast: .limit(limit).skip(skip) in Mongo
   openapi/contracts/core-provider.openapi.yaml	241, 267	/api/search and /api/search/phrase declare no limit or skip parameter and no ordering guarantee

   Live endpoints are searchAsWords (:263) and searchAsPhrase (:346), mounted at /search and /search/phrase (routes/search.js:6,13). searchFuzzily (:420), searchWildly (:510), and searchAlikes (:614) have the same structure but no routes yet — worth fixing together so the pattern is not carried forward when they are mounted.

   Proposed change

   Three options, best first.

   1. One combined Atlas Search index (recommended)

   Define a single index covering both vocabularies — the IIIF 3.0 paths (body.value, bodyValue, and the items / annotations embedded documents) and the IIIF 2.1 paths (resource.chars, resource.cnt:chars, and the resources / otherContent / sequences embedded documents).

   /search then becomes one $search with all the existing should clauses, followed by $skip and $limit. This fixes all three problems at once:

   • Ranking becomes native. Atlas returns $search results in descending score order across all clauses, so there is no cross-branch merge to get wrong.
   • Dropped documents cannot happen. Atlas returns each matching document exactly once; there is nothing to deduplicate and no key to derive.
   • Paging runs in the database, so cost stops scaling with depth.

   mergeSearchResults() can be deleted, and with it both bugs.

   Cost: an Atlas-side index change, and both indexes must exist during the transition.

   2. $unionWith in a single pipeline

   Keep both indexes, run the IIIF 2.1 branch as a $unionWith sub-pipeline against the same collection, then $group on _id to deduplicate, $sort by score, and $skip/$limit — all server-side. Grouping on _id in Mongo compares the value structurally, so embedded-object ids group correctly there.

   This removes the work from the API process but not from the cluster: the $group is a blocking stage over the union, so Atlas still materializes the merged set. It is a smaller change than option 1 with a smaller payoff. Before committing to it, confirm that $search is permitted as the first stage of a $unionWith sub-pipeline on our cluster tier.

   3. Keep the current shape, bound the damage

   If neither restructuring is scheduled soon, the depth cost can at least be capped by enforcing a /search-specific maximum on skip that is far below the /query maximum, since the two endpoints pay very different prices for the same depth. This is a mitigation, not a fix, and it makes the two endpoints inconsistent — which is exactly what the paged-reads issue argues against — so it should be a deliberate, documented decision rather than a default.

   Notes

   • The index definitions are not in this repository. They exist only in the Atlas UI, which means the current search behavior is not reviewable or reproducible from source. Whichever option is chosen, the definitions should be checked in — that is arguably worth doing first, independently of this issue.
   • The checked-in OpenAPI contract (openapi/contracts/core-provider.openapi.yaml) describes /api/search and /api/search/phrase as taking a body and returning a GenericArray, with no limit or skip parameter and no statement about ordering. Whatever ordering guarantee comes out of this work should be written there, not only in the JSDoc — the paged-reads issue covers the parameters themselves.
   • Option 1 changes relevance scoring, and should. A single index scores a document once across all matched clauses and sorts natively. Today the two branches' scores are never compared at all, so ordering will change substantially — that is the point, not a regression. Membership should change only by the documents that are currently dropped reappearing. Worth a before-and-after comparison on representative queries so the shift is understood before cutover.
   • Two one-line mitigations are available immediately, independent of any restructuring, and are worth landing first regardless of which option is chosen for the paging work:
     • sort on __rerum.score instead of score in mergeSearchResults(), which makes cross-index ordering real without touching the index layout;
     • derive the dedup key from something that is unique for every _id shape — JSON.stringify(result._id), or the document's @id / id — instead of toString().
       Both change results for existing clients (order for everyone; membership on dev), so they are behavioral even though the diffs are trivial.
   • The rel="next" work proposed in the paged-reads issue needs /search to over-fetch one record per branch (limit + skip + 1) to know whether another page exists. That applies under any of the three options here, and under option 1 it becomes the ordinary single-pipeline limit + 1.
   • Whatever lands must keep the property verified above: sequential pages equal to a single larger request, in the same order.

   Acceptance criteria

   • /search results are ordered by descending relevance score across both vocabularies, verified by asserting the returned score sequence never increases
   • The JSDoc promise of relevance ordering is either true or removed
   • Every document matched by the search is reachable by paging: the number of records a client can walk to equals the number of raw matches, including documents whose _id is an embedded object
   • A regression test feeds mergeSearchResults() (or its replacement) documents with ObjectId, string, and embedded-object _id values and asserts none are dropped
   • /search pagination is applied in MongoDB rather than in application memory
   • /search latency at depth is flat with respect to skip, comparable to /query
   • Search index definitions are checked into the repository
   • Sequential paged results still equal a single larger request, in the same order
   • Relevance ordering change is measured and accepted before cutover
   • /search and /search/phrase both covered; the unmounted search variants updated to match
   • The ordering guarantee is stated in openapi/contracts/core-provider.openapi.yaml, not only in JSDoc
3. thehabes

   changed the title [-]`/search` misranks results, drops matching documents, and pages in application memory[/-] [+]`/search` paging is unfinished: cost scales with depth, no `rel="next"`, and pages under-fill[/+] on Sep 17, 2026
4. thehabes

   added a sub-issue on Sep 17, 2026

   • Objects whose _id is not a string are served by /query and dropped by accident in /search #313
5. thehabes

   closed this as completedon Sep 25, 2026