Repository navigation
feat(metrics): commit-time lag gauges for ack and source - #256
Open
iskakaushik wants to merge 1 commit into
Open
iskakaushik wants to merge 1 commit into
iskakaushik wants to merge 1 commit into
Conversation
iskakaushik
force-pushed
the
kaushik/ack-commit-lag
branch
from
October 20, 2026 14:23
3d07545 to
05e2f1c
Compare
iskakaushik
marked this pull request as ready for review
October 20, 2026 15:50
8 tasks
serprex
force-pushed
the
kaushik/ack-commit-lag
branch
2 times, most recently
from
October 23, 2026 20:45
13e47b1 to
6d6a195
Compare
Carry the commit record's xact_time with the emitter ack frontier and export it as walshadow_emitter_ack_commit_timestamp_seconds, plus the newest drained commit as walshadow_source_last_commit_timestamp_seconds. Both are Unix seconds, NaN until the first commit, so replication lag is a plain subtraction with no PromQL rate window.
serprex
force-pushed
the
kaushik/ack-commit-lag
branch
from
October 23, 2026 20:51
6d6a195 to
dc5def9
Compare
serprex
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replication lag from
rate(source WAL) - rate(ack LSN)flips sign on bursty sources. This adds time-based gauges that need no rate window.walshadow_emitter_ack_commit_timestamp_seconds: commit time (Unix seconds, f64) of the newest transaction ClickHouse has acked. Carried on the ack collector's contiguous frontier, so it advances exactly withwalshadow_emitter_ack_lsn. Only final (publishing) seqs of a commit move it; aborts and bootstrap seqs carry no time and leave it unchanged.walshadow_source_last_commit_timestamp_seconds: newest commit time the reorder stage has drained from the source.Lag reads as
source_last_commit_ts - ack_commit_ts(stays correct when the source is idle) ortime() - ack_commit_ts.Notes:
xact_timeis the commit record's timestamp (PG epoch µs), converted to Unix seconds at scrape. The source gauge is taken at reorder's commit handling, not the pump, so it trails the pump by the decode queue.Tests: frontier advance/gating and untimed/partial seqs in
ack.rs; epoch conversion and NaN/value encoding inops/metrics.rs. fmt, clippy -D warnings, lib/bin tests pass in Docker; only the known root-onlycleanup_failure_keeps_pin_and_consumes_attemptfails.