Objects whose _id is not a string are served by /query and dropped by accident in /search #313

Summary

Objects whose _id is not a string are old, bad data and do not need to appear in /query or /search results. Today both endpoints serve them, and /search removes some of them by accident rather than by intent.

Exclude them where the results are gathered:

• controllers/crud.js:90 — db.find(props) applies only the client's filter.
• controllers/search.js:209,214 — the two $search pipelines for /search and /search/phrase, plus :640,658 for the unmounted searchAlikes.

A {"_id": {"$type": "string"}} condition in those four places makes the exclusion deliberate, fills pages back to limit, and retires the dedup-key problem described in #307, because strings key uniquely under toString().

Why this matters

The data

Measured 2026-09-17, read-only, by paging /query to the first skip that returns nothing:

The three on production are test junk with no @id and no __rerum: {"born":"today"}, {"New":"Item","Who":"Dis?"}, {"a":"same","b":"same"}. They were inserted outside the API, since nothing in the write path produces a document in that shape. The dev population is the v0 import from 165.134.105.29/annotationstore, whose @id values are the serialized Java ObjectId interpolated into a URL and are not resolvable.

/query serves them

POST /v1/api/query?limit=3  {"_id":{"$not":{"$type":"string"}}}
-> 200, three documents, on dev and on production

They also appear in any broad query that happens to match them. A client cannot do anything with them: idNegotiation() strips _id, so a document with no @id arrives as bare content like {"born":"today"}, with no identity to follow or update.

/search removes them by accident, and under-fills its pages

mergeSearchResults() keys deduplication on result._id?.$oid || result._id?.toString() (controllers/search.js:37). Every embedded-object _id keys to "[object Object]", so one such document per response survives and the rest are discarded. The outcome is close to what we want, but the mechanism is wrong in two visible ways:

• One legacy document per response still leaks through, arbitrarily chosen by branch order.
• Pages under-fill while matches remain. manuscript at limit=100 returns 50 documents, re-verified 2026-09-17. /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307 measured 68 raw matches behind that. A client that stops on a short page reports 50 and believes it is done — the same trap as a clamped limit.

Filtering in the pipeline fixes both: the page is filled from documents that are actually servable.

It is not measurably more expensive

/query at limit=500&skip=100000 on dev, three runs each:

The type predicate brackets to the string range on the _id index, so the sort and the deep skip stay index-driven.

Proposed change

/query

Combine rather than merge, so a client's own _id condition survives:

// controllers/crud.js — not {...props, _id: …}, which would silently replace a client's own _id filter
let matches = await db.find({ $and: [props, { _id: { $type: "string" } }] })
    .sort({ _id: 1 }).limit(limit + 1).skip(skip).toArray()

The limit + 1 over-fetch from #302 stays correct, since hasNext is then computed over servable documents only.

/search

A $match stage after $search and before the $limit, in all four pipelines, so the limit is reached with servable documents:

{ $search: presi3Query },
{ $addFields: { "__rerum.score": { $meta: "searchScore" } } },
{ $match: { _id: { $type: "string" } } },
{ $limit: limit + skip }

Atlas Search cannot express a BSON type condition inside $search, so this belongs in a $match.

If #309's restructuring lands first, the condition goes in its single pipeline instead, once.

Alternative considered: clean the data instead

Deleting the three junk documents on production and the v0 import on dev would remove the need for any code change. Rejected as the primary fix: a direct insert outside the API can recreate the shape at any time, and a permanent guarantee in the read path is cheaper than a promise about the collection's contents. Removing production's three junk documents is still worth doing on its own, and no dev data should be deleted without an explicit decision — over 100000 documents is not a cleanup to run casually.

Notes

• Breaking on dev only. Production loses three unusable documents from query results. Dev result sets shrink substantially, which is the intent. Capture before-and-after counts on representative queries so the shift is understood rather than discovered by a client.
• Relates to /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307, now closed: this retires its dedup-key concern rather than fixing the key. JSON.stringify(result._id) is explicitly not wanted, because it would preserve documents that do not need serving.
• Relates to /search paginates in application memory, so every page costs the server everything up to that page #309: that issue carries the /search half, and its acceptance criteria were updated on 2026-09-17 to ask for this exclusion.
• Filed as a sub-issue of /search paging is unfinished: cost scales with depth, no rel="next", and pages under-fill #306, the /search paging umbrella, because that is where the exclusion is load-bearing. The /query half stands on its own and needs nothing from /search paging is unfinished: cost scales with depth, no rel="next", and pages under-fill #306 or /search paginates in application memory, so every page costs the server everything up to that page #309.
• Helps Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303. A cursor of the form {_id: {$gt: lastId}} assumes every _id is a string. With this filter the assumption is enforced by the read path rather than by the state of the collection.
• /since, /history and /id need no change. They take an _id from the URL, and a non-string _id has no addressable URL.
• public/API.html should state that objects whose _id is not a string are not served, so the omission is documented rather than surprising.

Acceptance criteria

• POST /v1/api/query with {"_id":{"$not":{"$type":"string"}}} returns 200 [] on both deployments
• A client's own _id condition still applies, verified with a filter that names an _id explicitly and one that uses an operator
• /search and /search/phrase return no document whose _id is not a string, and manuscript at limit=100 fills the page from servable documents instead of returning 50
• The rel="next" contract from /query paged responses carry no rel="next", so no client can tell a full page from the last page #302 still holds: hasNext is computed over servable documents, and a walk terminates without repeating or skipping a record
• Deep-skip /query latency is unchanged, measured with the probes in this issue
• The unmounted searchFuzzily, searchWildly, and searchAlikes carry the same exclusion, so it is not lost when they are routed
• Regression tests assert the filter reaches the database call for /query and appears before $limit in each search pipeline
• public/API.html documents that non-string _id objects are not served
• Before-and-after result counts on representative dev queries are posted here before cutover