Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,8 @@ Set `scan.source-watermark.idle-advance-timeout` to advance the watermark conser
```

Idle advancement supports only the `topic` table option, not `topic-pattern`. Configuring `topic-pattern` with idle advancement will result in a failure.
The connector checks this through Kafka's `AdminClient` and requires permission to describe the topic configuration and its offsets, and on a fresh start from the committed offsets, also the consumer group.
It also needs the brokers to be reachable and the topic to use `LogAppendTime`.
The connector checks both through Kafka's `AdminClient` and requires permission to describe the topic configuration and its offsets, and on a fresh start from the committed offsets, also the consumer group.
If Kafka cannot be reached or the topic is not configured for `LogAppendTime`, idle advancement pauses rather than advancing unsafely.
Do not combine idle advancement with `scan.watermark.idle-timeout`. The connector ignores that option when idle advancement is enabled, because once Flink marks a partition idle, it stops letting the connector advance that partition's watermark.

Expand All @@ -341,11 +342,27 @@ A partition can be idle with records still unread, when the source is busy with
For an idle partition, the connector asks the broker for the end of its log, and the watermark advances only if the partition has been read to that end.
It then moves to the wall-clock time at which the broker was asked, minus the adaptive out-of-orderness delay and the safety margin, also when the partition's last record was read from a backlog, such as when a new job starts from the earliest offset.
An idle partition with unread records, or whose log end cannot be fetched, keeps the watermark its records give it.
An idle partition that has not produced a row yet is marked idle instead, which leaves it out of the source watermark until it does, but only once every partition of the source subtask that has not produced a row has been read to the end of its log.
After a restore, a partition can take longer than the timeout to produce its first row, and marking it idle meanwhile would let the subtask's other partitions carry the watermark past its records.
An idle partition that has not produced a row yet is marked idle instead, which leaves it out of the source watermark until it does.
That happens only once every partition of the source subtask that has not produced a row has been read to the end of its log.
After a restore, a partition can take longer than the timeout to produce its first row.
Marking it idle meanwhile would let the subtask's other partitions carry the watermark past its records.
Such a partition has been read to the end when its split started at the end of its log, or when its consumer has since reached that end, past any commit markers, and every record it returned has been read.
On a fresh start from the earliest or the committed offsets, the connector first looks up where each split starts.
A partition's first row moves its watermark to just below that row at once, without waiting for `scan.source-watermark.min-records` records, so downstream operators see the source subtask as active, and hold their watermark below the row, before any watermark that would pass it.
A partition's first row moves its watermark to just below that row at once, without waiting for `scan.source-watermark.min-records` records.
Downstream operators then see the source subtask as active, and hold their watermark below the row, before any watermark that would pass it.
A source subtask is reported idle only while none of its partitions has produced a row and each has been read to the end of its log, as far as the latest successful log-end fetch shows.
It is reported idle once that has lasted for the idle-advance timeout.
It is reported active again when a partition produces its first row, or before that when a new partition is assigned.
While it is idle, it does not ask the brokers.
While a subtask is idle, the watermark of a downstream operator follows the operator's other inputs, such as the source's other subtasks or the other table of a join, and it does not move back when the subtask turns active again.
So records that were written to a partition before the connector discovered it can still arrive late.
The connector looks for new partitions every `scan.topic-partition-discovery.interval`, 5 minutes by default.
Turning active again keeps the operator's watermark from moving further, but only if the source's other subtasks have not moved it yet.
Otherwise Flink leaves the subtask out of the operator's watermark until the subtask's own watermark has caught up with it.
A subtask is not reported idle, and none of its partitions is marked idle, before the source's enumerator, which assigns partitions to subtasks, has told it that its first partition discovery is done and assigned.
Until then the subtask may still be about to get a partition with a backlog: its first on a fresh start, or after a restore one that was added to the topic while the job was down.
Reported idle meanwhile, it would let downstream operators move their watermark past that backlog.
So until then it holds back the watermark of downstream operators, for as long as that first discovery takes.

The log ends of all of a source subtask's partitions are fetched together, at most once per broker-check TTL, which therefore also bounds how far an idle partition's watermark trails the wall clock: keep it close to the idle-advance timeout.
The safety margin must cover clock skew between the Kafka brokers and the TaskManagers, and the time a record takes to become readable after the broker timestamps it.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@
import java.util.List;
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.connector.source.Source;
import org.apache.flink.api.connector.source.SourceEvent;
import org.apache.flink.api.connector.source.SourceReader;
import org.apache.flink.api.connector.source.SourceReaderContext;
import org.apache.flink.api.connector.source.SplitEnumerator;
import org.apache.flink.api.connector.source.SplitEnumeratorContext;
import org.apache.flink.connector.kafka.source.split.KafkaPartitionSplit;

/**
Expand All @@ -28,6 +31,10 @@
* advancement needs these to tell whether the splits that have not produced a row have anything
* left to read. Flink assigns splits on the source task's thread, whose positions the generators
* read.
*
* <p>Also records when the enumerator has finished the reader's first assignment, which its
* enumerator, wrapped in an {@link InitialAssignmentSignalingEnumerator}, tells every reader: a
* reader is idle only from then on.
*/
@Internal
public final class AssignmentRecordingSource<T, EnumChkT>
Expand All @@ -37,6 +44,20 @@ public AssignmentRecordingSource(Source<T, KafkaPartitionSplit, EnumChkT> delega
super(delegate);
}

@Override
public SplitEnumerator<KafkaPartitionSplit, EnumChkT> createEnumerator(
SplitEnumeratorContext<KafkaPartitionSplit> enumContext) throws Exception {
return new InitialAssignmentSignalingEnumerator<>(enumContext, delegate::createEnumerator);
}

@Override
public SplitEnumerator<KafkaPartitionSplit, EnumChkT> restoreEnumerator(
SplitEnumeratorContext<KafkaPartitionSplit> enumContext, EnumChkT checkpoint)
throws Exception {
return new InitialAssignmentSignalingEnumerator<>(
enumContext, context -> delegate.restoreEnumerator(context, checkpoint));
}

@Override
public SourceReader<T, KafkaPartitionSplit> createReader(SourceReaderContext readerContext)
throws Exception {
Expand All @@ -57,5 +78,14 @@ public void addSplits(List<KafkaPartitionSplit> splits) {
KafkaReadPositions.assigned(split.getTopicPartition(), split.getStartingOffset()));
super.addSplits(splits);
}

@Override
public void handleSourceEvents(SourceEvent sourceEvent) {
if (sourceEvent instanceof InitialAssignmentDoneEvent) {
KafkaReadPositions.recordInitialAssignmentDone();
} else {
super.handleSourceEvents(sourceEvent);
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/*
* Copyright © 2026 DataSQRL (contact@datasqrl.com)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.datasqrl.flinkrunner.connector.kafka;

import java.io.IOException;
import java.util.List;
import javax.annotation.Nullable;
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.connector.source.SourceEvent;
import org.apache.flink.api.connector.source.SourceSplit;
import org.apache.flink.api.connector.source.SplitEnumerator;

/**
* An enumerator that forwards every call to the enumerator it wraps, so that a wrapper overrides
* only what it changes.
*
* <p>The wrapped enumerator comes from {@link #delegate()}, not from the constructor as in {@link
* ForwardingSource}: a wrapper may have to create it with a context of its own, which it can only
* build once it is constructed itself.
*/
@Internal
public abstract class ForwardingEnumerator<SplitT extends SourceSplit, EnumChkT>
implements SplitEnumerator<SplitT, EnumChkT> {

protected abstract SplitEnumerator<SplitT, EnumChkT> delegate();

@Override
public void start() {
delegate().start();
}

@Override
public void handleSplitRequest(int subtaskId, @Nullable String requesterHostname) {
delegate().handleSplitRequest(subtaskId, requesterHostname);
}

@Override
public void addSplitsBack(List<SplitT> splits, int subtaskId) {
delegate().addSplitsBack(splits, subtaskId);
}

@Override
public void addReader(int subtaskId) {
delegate().addReader(subtaskId);
}

@Override
public EnumChkT snapshotState(long checkpointId) throws Exception {
return delegate().snapshotState(checkpointId);
}

@Override
public void notifyCheckpointComplete(long checkpointId) throws Exception {
delegate().notifyCheckpointComplete(checkpointId);
}

@Override
public void notifyCheckpointAborted(long checkpointId) throws Exception {
delegate().notifyCheckpointAborted(checkpointId);
}

@Override
public void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) {
delegate().handleSourceEvent(subtaskId, sourceEvent);
}

@Override
public void close() throws IOException {
delegate().close();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
/*
* Copyright © 2026 DataSQRL (contact@datasqrl.com)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.datasqrl.flinkrunner.connector.kafka;

import static org.apache.flink.util.Preconditions.checkNotNull;

import java.util.Map;
import java.util.concurrent.Callable;
import java.util.function.BiConsumer;
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.connector.source.ReaderInfo;
import org.apache.flink.api.connector.source.SourceEvent;
import org.apache.flink.api.connector.source.SourceSplit;
import org.apache.flink.api.connector.source.SplitEnumeratorContext;
import org.apache.flink.api.connector.source.SplitsAssignment;
import org.apache.flink.metrics.groups.SplitEnumeratorMetricGroup;

/**
* An enumerator context that forwards every call to the context it wraps, so that a wrapper
* overrides only what it changes.
*/
@Internal
public abstract class ForwardingEnumeratorContext<SplitT extends SourceSplit>
implements SplitEnumeratorContext<SplitT> {

protected final SplitEnumeratorContext<SplitT> delegate;

protected ForwardingEnumeratorContext(SplitEnumeratorContext<SplitT> delegate) {
this.delegate = checkNotNull(delegate, "Delegate enumerator context must not be null.");
}

@Override
public SplitEnumeratorMetricGroup metricGroup() {
return delegate.metricGroup();
}

@Override
public void sendEventToSourceReader(int subtaskId, SourceEvent event) {
delegate.sendEventToSourceReader(subtaskId, event);
}

@Override
public void sendEventToSourceReader(int subtaskId, int attemptNumber, SourceEvent event) {
delegate.sendEventToSourceReader(subtaskId, attemptNumber, event);
}

@Override
public int currentParallelism() {
return delegate.currentParallelism();
}

@Override
public Map<Integer, ReaderInfo> registeredReaders() {
return delegate.registeredReaders();
}

@Override
public Map<Integer, Map<Integer, ReaderInfo>> registeredReadersOfAttempts() {
return delegate.registeredReadersOfAttempts();
}

@Override
public void assignSplits(SplitsAssignment<SplitT> newSplitAssignments) {
delegate.assignSplits(newSplitAssignments);
}

@Override
public void assignSplit(SplitT split, int subtask) {
delegate.assignSplit(split, subtask);
}

@Override
public void signalNoMoreSplits(int subtask) {
delegate.signalNoMoreSplits(subtask);
}

@Override
public <T> void callAsync(Callable<T> callable, BiConsumer<T, Throwable> handler) {
delegate.callAsync(callable, handler);
}

@Override
public <T> void callAsync(
Callable<T> callable,
BiConsumer<T, Throwable> handler,
long initialDelayMillis,
long periodMillis) {
delegate.callAsync(callable, handler, initialDelayMillis, periodMillis);
}

@Override
public void runInCoordinatorThread(Runnable runnable) {
delegate.runInCoordinatorThread(runnable);
}

@Override
public void setIsProcessingBacklog(boolean isProcessingBacklog) {
delegate.setIsProcessingBacklog(isProcessingBacklog);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright © 2026 DataSQRL (contact@datasqrl.com)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.datasqrl.flinkrunner.connector.kafka;

import org.apache.flink.annotation.Internal;
import org.apache.flink.api.connector.source.SourceEvent;

/**
* Tells a reader that the enumerator has finished its first partition discovery and has assigned
* the reader every split that discovery had for it, which may be none.
*/
@Internal
public record InitialAssignmentDoneEvent() implements SourceEvent {}
Loading
Loading