Skip to content

fix: Kafka subtask reported idle on slow split assignment - #452

Merged
ferenc-csaky merged 5 commits into
mainfrom
fix/issue-451
Nov 17, 2026
Merged

ferenc-csaky merged 5 commits into
mainfrom
fix/issue-451

Conversation

@mateczagany

Copy link
Copy Markdown
Collaborator

What does this change?

Closes #451.

With idle advancement on, a kafka-safe source subtask could be reported idle while it still had a backlog to read. A temporal join downstream then followed its other input to wall-clock time, and every row of that backlog arrived late. The cause was the reader's main output, which never sees a row: it went idle on its own timeout, before splits were assigned or before an empty split next to a backlog split had gone idle.

  • The main output is now idle only while no assigned partition has produced a row and each is read to the end of its log. Otherwise it calls markActive(), which also turns an idle subtask active when it gets a partition discovered later.
  • No subtask is idle before the enumerator has assigned the splits of its first discovery. Flink does not tell a reader that it gets no split, so InitialAssignmentSignalingEnumerator wraps the Kafka enumerator, counts the async calls it makes from start() and their handlers, and sends each reader an InitialAssignmentDoneEvent when none is pending. This applies after a restore too.

What to look at:

  • The wrapper depends on KafkaSourceEnumerator (connector 5.0.0) doing its first assignment in callAsync calls started from start(). If a later version assigned somewhere else, the event would go out early and [BUG] kafka-safe subtask is reported idle at a fresh start while it delivers rows, so a temporal join's clock runs ahead of the backlog #451 would be back for slow enumerators.
  • The strategy treats the first generator Flink creates from it as the main output's, as createMainOutput does in Flink 2.3.0.
  • After every start and restore, a subtask holds downstream watermarks until the enumerator's first discovery is done.
  • An idle subtask now asks the brokers once per broker-check TTL. On main it stopped once it was idle.

FLINK-40504, on Flink's release-2.3 branch, reports a subtask active from the first row of any split. That does not cover a backlog that has not produced a row yet.

Area

  • Runner CLI and Docker image (flink-sql-runner)
  • Connectors (connectors)
  • Formats (formats)
  • Function libraries (stdlib)
  • Environment utilities (env-utils)
  • Tests and samples (testing)
  • Build / CI / dependencies
  • Docs (README.md, stdlib-docs)

How did you test it?

  • The 130 unit tests in kafka-safe-connector pass on Flink 2.3.0, and with FLINK-40504's classes copied over it.
  • Seven tests in KafkaRecordTimestampWatermarkStrategyTest fail with the main output's rule from main. They cover a fresh start with a slow enumerator, a subtask without splits, a restored subtask, and a partition discovered later.
  • InitialAssignmentSignalingEnumeratorTest runs the wrapper around the real KafkaSourceEnumerator, without a broker: fresh start, late reader, two restores, periodic discovery and a failed discovery.
  • The 9 tests in KafkaSourceWatermarkIT pass. The new one runs two source subtasks for one partition, and times out when the reader drops the event.
  • Not run on a cluster: a subtask failover, and a restore with a changed parallelism.

Checklist

  • The PR title starts with a type such as feat:, fix: or docs:, optionally with a scope (e.g. fix(flexible-json):)
  • mvn clean install passes locally and the code is formatted with mvn spotless:apply (ran: kafka-safe-connector tests, KafkaSourceWatermarkIT and spotless:check)
  • New or changed behavior is covered by tests (given_when_then naming, AssertJ assertions) (unit tests keep the test… naming of the existing test classes)
  • New dependency and plugin versions are defined as properties in the root pom.xml (none added)
  • Commits are signed off and signed (git commit -s -S)
  • README.md and stdlib-docs are updated if user-facing behavior or functions changed

🤖 Generated with Claude Code

@ferenc-csaky ferenc-csaky modified the milestone: 0.12.6 Nov 17, 2026
Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
@ferenc-csaky
ferenc-csaky merged commit fe8ca51 into main Nov 17, 2026
13 checks passed
@ferenc-csaky
ferenc-csaky deleted the fix/issue-451 branch November 17, 2026 18:20
ferenc-csaky added a commit that referenced this pull request Nov 17, 2026
Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
Co-authored-by: Ferenc Csaky <ferenc@datasqrl.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] kafka-safe subtask is reported idle at a fresh start while it delivers rows, so a temporal join's clock runs ahead of the backlog

2 participants