When an ingest response reports records_rejected > 0 inside a 2xx envelope, the runtime still records the batch as succeeded (reference-implementation/runtime/index.ts:4340–4364, run.batch_ingested with records_rejected and status: "succeeded") and the stream's cursor is staged and committed as normal. The rejected records are counted but not retained anywhere the runtime can replay from, and the cursor moves past them.
On an instance with five months of history, 715 run/stream pairs committed a cursor in a run that also reported rejected records:
| connectors |
pairs |
| the two largest remote connectors |
511 |
| the two local collectors |
177 |
| seven others |
27 |
A committed cursor therefore does not mean every emitted record was stored. The next run does not revisit the rejected items, because the cursor has advanced past them.
Proposal: treat a non-zero records_rejected as blocking for that stream's cursor, the same way a drift-skipped stream is not staged today, and persist the rejected records with their rejection reason so they can be replayed once the cause is fixed. Streams with zero rejections in the same run keep committing.
Not verified: how many of the 715 pairs correspond to distinct source items still absent from records, since a rejected record can be re-emitted and accepted in a later run. The count is of runs, not of lost items.
When an ingest response reports
records_rejected > 0inside a 2xx envelope, the runtime still records the batch as succeeded (reference-implementation/runtime/index.ts:4340–4364,run.batch_ingestedwithrecords_rejectedandstatus: "succeeded") and the stream's cursor is staged and committed as normal. The rejected records are counted but not retained anywhere the runtime can replay from, and the cursor moves past them.On an instance with five months of history, 715 run/stream pairs committed a cursor in a run that also reported rejected records:
A committed cursor therefore does not mean every emitted record was stored. The next run does not revisit the rejected items, because the cursor has advanced past them.
Proposal: treat a non-zero
records_rejectedas blocking for that stream's cursor, the same way a drift-skipped stream is not staged today, and persist the rejected records with their rejection reason so they can be replayed once the cause is fixed. Streams with zero rejections in the same run keep committing.Not verified: how many of the 715 pairs correspond to distinct source items still absent from
records, since a rejected record can be re-emitted and accepted in a later run. The count is of runs, not of lost items.