Skip to content

fix: widen the ClickHouse query bound until reads answer - #250

Merged
catinspace-au merged 1 commit into
mainfrom
fix/clickhouse-timeouts-followup
Oct 12, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/clickhouse-timeouts-followup

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

The release run for #249 failed its Test job in the gRPC outage suite. The cause is the fixed 30 s bound #249 put on every query: once a schema read takes longer than that, every attempt times out, the table's schema never loads and the pushed records never land.

Reproduced deterministically on 4673b76 with the new test below, which delays every schema read by 3 s under a 2 s bound: 0 of 20 records accepted in 60 s, after 4 retries. With this change it passes in about 20 s.

What changes:

  • Queries run under one shared bound that doubles after each timeout, up to 300 s (the default ClickHouse gives its own socket reads and writes), and steps back down after a quick answer.
  • The inserter's schema reads share that bound with the query client, so one slow read widens the next one wherever it happens.
  • clickhouse.request_timeout_ms now defaults to 60000. It is the starting bound for queries and still bounds an insert waiting to hand over its next chunk.
  • Inserts keep their own bounds and still go back to the hold on a timeout.

Tests:

  • every_record_lands_when_schema_reads_answer_slower_than_the_request_bound (tests/integration/grpc_outage.rs): red on 4673b76, green here.
  • Unit tests for the bound: widens after a slow read, stops at the ceiling, narrows after a quick answer, a base above the ceiling is its own ceiling.

Through a full outage the widened bound can hold one flush cycle past Kafka's 300 s max.poll.interval.ms. The consumer rejoins with the batch's offsets still uncommitted, so nothing is lost. CONFIGURATION.md says so.

A fixed 30 s bound on schema reads failed every attempt once ClickHouse took longer than that to answer, so nothing waiting on the schema ever landed and the gRPC push path never answered.

- Queries now run under a shared bound that doubles after each timeout, up to the 300 s ClickHouse gives its own socket reads, and steps back down after a quick answer.
- The inserter's schema reads share that bound with the query client.
- clickhouse.request_timeout_ms defaults to 60000, the starting bound.
- New test: schema reads answered slower than the request bound, and every pushed record still lands.
@catinspace-au
catinspace-au merged commit b081924 into main Oct 12, 2026
17 checks passed
@catinspace-au
catinspace-au deleted the fix/clickhouse-timeouts-followup branch October 12, 2026 06:32
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant