Skip to content

runtime: per-stream cursor commits precede the terminal event, so a failed or abandoned run can leave cursors advanced #88

Description

@tnunamak

commitState PUTs each stream's cursor to /v1/state/<connectorId> one stream at a time (reference-implementation/runtime/index.ts:44354461) and emits run.state_advanced after each successful PUT, before the run's terminal event is written. If the process stops between two PUTs, or the terminal handling then fails, some streams are committed and the run ends as run.failed or run.abandoned.

On an instance with five months of history, 11 runs advanced at least one cursor and did not end with a succeeded completion:

runs terminal event when
2 run.failed (runtime_error) 2026-04-19, 04-20
9 run.abandoned (controller) 2026-08-21, 08-27

In the two failed runs, a local collector's sessions, skills and slash_commands cursors advanced and then the run failed; the sessions that run had emitted but not fully ingested are now behind the cursor.

Proposal: commit all staged cursors for a run in one request, or write the terminal event first and commit only after it records succeeded, so a run can never end failed or abandoned with a subset of its cursors advanced. A migration is not needed; the affected runs can be re-driven by resetting the four streams' cursors to their pre-run values, which the event log holds.

Not verified: whether the nine abandoned runs lost records, since abandonment came from the controller after the PUTs and the batches before them were accepted.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions