Skip to content

runtime: rejected records inside a 2xx ingest response do not block the stream cursor #86

Description

@tnunamak

When an ingest response reports records_rejected > 0 inside a 2xx envelope, the runtime still records the batch as succeeded (reference-implementation/runtime/index.ts:43404364, run.batch_ingested with records_rejected and status: "succeeded") and the stream's cursor is staged and committed as normal. The rejected records are counted but not retained anywhere the runtime can replay from, and the cursor moves past them.

On an instance with five months of history, 715 run/stream pairs committed a cursor in a run that also reported rejected records:

connectors pairs
the two largest remote connectors 511
the two local collectors 177
seven others 27

A committed cursor therefore does not mean every emitted record was stored. The next run does not revisit the rejected items, because the cursor has advanced past them.

Proposal: treat a non-zero records_rejected as blocking for that stream's cursor, the same way a drift-skipped stream is not staged today, and persist the rejected records with their rejection reason so they can be replayed once the cause is fixed. Streams with zero rejections in the same run keep committing.

Not verified: how many of the 715 pairs correspond to distinct source items still absent from records, since a rejected record can be re-emitted and accepted in a later run. The count is of runs, not of lost items.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions