Repository navigation
fix: widen the ClickHouse query bound until reads answer - #250
Merged
Merged
Conversation
A fixed 30 s bound on schema reads failed every attempt once ClickHouse took longer than that to answer, so nothing waiting on the schema ever landed and the gRPC push path never answered. - Queries now run under a shared bound that doubles after each timeout, up to the 300 s ClickHouse gives its own socket reads, and steps back down after a quick answer. - The inserter's schema reads share that bound with the query client. - clickhouse.request_timeout_ms defaults to 60000, the starting bound. - New test: schema reads answered slower than the request bound, and every pushed record still lands.
|
Released in v1.18.50 -- https://github.com/hyperi-io/dfe-loader/releases/tag/v1.18.50 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release run for #249 failed its Test job in the gRPC outage suite. The cause is the fixed 30 s bound #249 put on every query: once a schema read takes longer than that, every attempt times out, the table's schema never loads and the pushed records never land.
Reproduced deterministically on 4673b76 with the new test below, which delays every schema read by 3 s under a 2 s bound: 0 of 20 records accepted in 60 s, after 4 retries. With this change it passes in about 20 s.
What changes:
clickhouse.request_timeout_msnow defaults to 60000. It is the starting bound for queries and still bounds an insert waiting to hand over its next chunk.Tests:
every_record_lands_when_schema_reads_answer_slower_than_the_request_bound(tests/integration/grpc_outage.rs): red on 4673b76, green here.Through a full outage the widened bound can hold one flush cycle past Kafka's 300 s
max.poll.interval.ms. The consumer rejoins with the batch's offsets still uncommitted, so nothing is lost. CONFIGURATION.md says so.