Repository navigation
Conversation
…h timestamp contamination, and parameter checks
Contributor
There was a problem hiding this comment.
📋 Review Summary
This PR fixes several important bugs in GroupIntoBatches, including updating timer watermark holds for out-of-order elements, persisting timerTs to prevent processing-time timer starvation, and preventing timestamp overflow on window expiration calculations. It also adds display data support, improves precondition checks, and fixes Javadoc formatting.
🔍 General Feedback
- The state management and watermark hold logic in
GroupIntoBatchesDoFnare clean and properly handle out-of-order elements without resetting the processing-time deadline. testOutOfOrderTimestampsInBatchcould be strengthened to directly verify the updated timer watermark hold (Right now, a batch size of 2 flushes immediately insideprocessElement, which clears the timer before the watermark advances).
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
… hold on window expiration
Contributor
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
storedBatchSize.read() == 1). If subsequent elements arrived with timestamps earlier than the first element (out-of-order within the window), the watermark hold remained pinned to the first element's timestamp, risking premature watermark advancement.GroupIntoBatchescomparescurrentMinBufferedTsagainstoldMinBufferedTsand updates the timer's output timestamp whenever an earlier timestamp arrives (num == 1 || currentMinBufferedTs < oldMinBufferedTs).timerTsState to Prevent Timer Starvation:getCurrentRelativeTime() + maxBufferingDurationon every update would push the timer deadline further out, causing timer starvation.ValueState<Long> timerTs, which had a bug where it was never actually written, to store the target firing timestamp calculated on the first element. When updating the watermark hold, the target firing time is preserved.window.maxTimestamp().plus(allowedLateness)when settingholdTimerto preventInstantarithmetic overflow when close toBoundedWindow.TIMESTAMP_MAX_VALUE.populateDisplayDataonGroupIntoBatches,GroupIntoBatches.WithShardedKey, andGroupIntoBatchesDoFnto exposebatchSize,batchSizeBytes, andmaxBufferingDurationin runner UIs (such as Dataflow).Preconditions.checkStatewithcheckArgumentwith informative error messages forbatchSize < Long.MAX_VALUEandbatchSizeBytes < Long.MAX_VALUE.testOutOfOrderTimestampsInBatchverifying watermark hold output timestamp matches the minimum timestamp of out-of-order elements.checkArgumentbounds andpopulateDisplayData.