/query paged responses carry no rel="next", so no client can tell a full page from the last page #302

Summary

Nothing in a /query response says whether more results exist. A client has to infer it from the number of records it received, and both available inferences are wrong:

• Stopping when results.length < limit silently drops records, because a clamped limit produces a short page that is not a final page.
• Stopping when results.length === 0 and advancing by results.length never reaches the end of a result set larger than the skip maximum. Before limit and skip silently guess at invalid input, and an over-maximum skip returns the same page forever #301 every request past the maximum returned the same page forever. Since limit and skip silently guess at invalid input, and an over-maximum skip returns the same page forever #301 the walk ends in a 400 instead, which is honest but still leaves the rest of the result set unread.

The fix is to stop making clients guess. /query emits Link: <…>; rel="next" (RFC 8288) while another page exists and omits it when the result set is exhausted. Clients follow next and stop when it is absent, which makes every length-based stop condition irrelevant in one move.

This issue is scoped to /query. /search paging is tracked under #306 and its sub-issues, and the /gog/*InManuscript endpoints are out of scope (see Notes).

Why this matters

Both client shapes fail, in opposite directions. From the parent draft (#299), before #301 landed, both run against the same query on the same deployment:

# stop on short page, limit=1000
req #1: skip=0 -> 500 docs
=> 500 records (client believes the walk is complete; the query matches more than 100000)

# stop on empty page, limit=100, entered at skip=99800
req #3: skip=100000 -> 100 docs, first=4c0ef019
req #4: skip=100100 -> 100 docs, first=4c0ef019
req #5: skip=100200 -> 100 docs, first=4c0ef019
...never terminates

With rel="next" the first client gets a next link on its 500-record page and keeps going; the second gets no next link at the end and stops. Neither has to know what the server's maximums are.

A client that wants to defend itself still has to do arithmetic. #301 added Pagination-Limit, Pagination-Skip, Pagination-Limit-Max, and Pagination-Skip-Max, so a client can now detect a clamped limit and the depth ceiling. It still has to combine record counts with those headers to decide when to stop, and every client has to get that logic right independently. The only Link emitted today is the JSON-LD context (utils.js:163).

The published example still guesses. pagedQuery at public/API.html:556 stops when results.length < appliedLimit and treats the skip ceiling 400 as the end of the walk. It is correct for the current contract, but it is exactly the counting logic a next link makes unnecessary, and it cannot read past the maximum.

Affected lines

Design

Know whether another page exists

/query over-fetches one record and trims it before serialization. The extra record is never returned; its existence is the signal.

let matches = await db.find(props).sort({ _id: 1 }).limit(limit + 1).skip(skip).toArray()
const hasNext = matches.length > limit
matches = matches.slice(0, limit).map(o => idNegotiation(o))

Emit the header

A new helper, setNextPageLink(req, res, { limit, skip, hasNext }) in controllers/utils.js, adds the next link. /query calls it after res.set(utils.configureLDHeadersFor(matches)).

Link: <http://store.rerum.io/v1/context.json>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"
Link: <https://store.rerum.io/v1/api/query?limit=3&skip=9>; rel="next"

That is the response to ?limit=3&skip=6. The rules:

• Append, do not replace. configureLDHeadersFor() is unchanged. The helper calls res.append("Link", …), so the next link arrives as a second Link field after the context link, which RFC 9110 treats the same as one comma-joined value.
• Absolute URLs on the process.env.RERUM_PREFIX origin, never the request host. The path is the route that was paged (req.baseUrl + req.path, trailing slash trimmed), so the legacy getByProperties.action alias links to /v1/api/query. If RERUM_PREFIX is unset, the links are path-only references, which RFC 8288 allows.
• Only paging parameters are carried: limit (the applied value, after clamping) and skip.
• next is present only while another page exists. Absence is the termination signal, so it is genuinely absent on the final page, including a final page that is exactly limit records long. It advances skip by the applied limit.
• next is still sent past the skip maximum. When the next page would start beyond MAX_QUERY_SKIP, the link is present anyway, and following it is answered with the limit and skip silently guess at invalid input, and an over-maximum skip returns the same page forever #301 400 that names the maximum. Leaving it out would tell a client the walk was complete when it was not, which is the silent truncation this issue exists to stop. A result set longer than the maximum therefore ends loudly. Unbounded depth is Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303, deferred.
• Only next. first, prev, and last are not sent. A client already has the URL it started from. prev would need sizing rules for a skip smaller than limit that no client has asked for, and would have no offset to go back from under a future cursor (Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303). last would need a total count.
• /query is a POST endpoint. The link carries no request body; the client re-sends the same body to it. public/API.html says so explicitly.
• Access-Control-Expose-Headers is already * (app.js:53), so browser clients can read the header without further CORS work.

Documentation

• public/API.html: the Custom Query text tells clients to follow rel="next" until there is none. pagedQuery is rewritten as a loop that re-sends the same request to each next link, stops when the header has none, and warns that its results are incomplete if a next link past the skip maximum is answered with the 400. A new "Following rel="next"" section under Pagination parameters describes the link and its behavior at the maximum, shows an example header, and notes that the same body goes to the link.
• openapi/contracts/core-provider.openapi.yaml: a NextPageLink header component describes the link, and the /api/query 200 response references it as Link.

Verification

Tests:

• routes/__tests__/query.test.js, rel="next" links on /query: next is present and advances skip while more records exist, and is absent on the final page. The JSON-LD context link survives, a final page carries no paging link at all, and the extra record is never served at the maximum limit. At the skip maximum next is still present, and following it is a 400 carrying Pagination-Skip-Max. A walk that follows only next returns every record once.
• __tests__/utils.test.js, setNextPageLink: an absolute link on the RERUM_PREFIX origin that advances skip by the limit, a path-only link without it, nothing added on the final page, and an append rather than a replace.

Read-only, against the local pm2 app reading the dev collection (rerum-test):

• ?limit=3&skip=6 over {"__rerum.history":{"$exists":true}} returned the context link plus next, as in the example above.
• A query matching one record, and a query matching exactly limit=2 records, each returned the context link only, with no next.
• Following only rel="next" at limit=500 over {"_id":{"$type":"string","$gte":"69e9"}} read 2433 records in 5 pages and stopped when next was absent. The walk was identical, document for document, to an independent skip walk at limit=250.
• Following only rel="next" at limit=500 from skip=99000 over {"__rerum.history":{"$exists":true}}, which matches 102001 records, read 1500 records and then got a 400 naming the maximum of 100000 from the skip=100500 link.

Notes

• Additive and non-breaking. Existing clients ignore the header and behave exactly as they do now. This can ship without client coordination.
• /query paginates with no sort, so page boundaries rest on MongoDB natural order #300 is closed, so /query pages over a deterministic _id order and a next link promises nothing the server cannot keep.
• Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303 is deferred as a feature to consider later. If it lands, next on /query should carry a cursor instead of a skip, which lifts the skip maximum for clients that follow the link, without any change on their side. A conforming client never builds a URL, so that swap is invisible to it.
• HEAD /query is not part of this issue. HEAD /query answers an empty page with 404 where POST /query answers 200 [] #304 removes it, because a POST endpoint has no HEAD counterpart.
• /search is out of scope. Its paging is tracked under /search paging is unfinished: cost scales with depth, no rel="next", and pages under-fill #306. /search paginates in application memory, so every page costs the server everything up to that page #309 now carries the rel="next" criteria for /search, and should reuse setNextPageLink(). Until /search merge sorts on a field that does not exist and drops documents whose _id is an embedded object #307 lands, the result pool a /search next link would walk is still missing documents.
• /gog/*InManuscript is out of scope. Those endpoints page over a window of candidate Annotations rather than over the entities they return, which is why limit and skip silently guess at invalid input, and an over-maximum skip returns the same page forever #301 left the Pagination-* headers off them. They need their own treatment.
• The pagination contract is undocumented, misdocumented, and absent from the OpenAPI contract #305 remains the final documentation sweep.
• /history and /since load an object's entire version chain into memory, unpaged #252 is still open and its third recommendation is cursor-based pagination, which this issue is the first half of. Add cursor-based continuation to /query so paging depth is unbounded and cost is flat #303 is the second.

Acceptance criteria

• /query responses carry Link: …; rel="next" while more results exist, and omit it on the final page
• The JSON-LD context link is still present alongside rel="next"
• The next link is absolute on the RERUM_PREFIX origin and carries only paging parameters
• rel="next" is still present when the next page would start past the skip maximum, and following it returns the 400 that names the maximum
• The over-fetched record is never returned to the client, at any limit including the maximum
• A client following only rel="next" walks a /query result set to completion and terminates, with no length-based stop condition
• Regression tests cover rel="next" presence, its absence on the final page, and the untrimmed extra record never appearing
• public/API.html publishes a pagedQuery example that follows rel="next", and says the same request body goes to the link
• The Link response header is declared for /api/query in openapi/contracts/core-provider.openapi.yaml