Skip to content

fix: make each Insights metric measure what it claims - #374

Merged
Priyanshu-u07 merged 1 commit into
mainfrom
fix/insights-ttft-latency
Oct 2, 2026
Merged

Priyanshu-u07 merged 1 commit into
mainfrom
fix/insights-ttft-latency

Conversation

@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

The tile read "Avg Latency (TTFT)" and showed neither. coalesce(ttft_ms, latency_ms) gave time to first token for streaming requests and total duration for the rest, averaged together, so the number moved with the streaming share of traffic rather than with performance. More streaming made latency appear to improve.

Each expression now means its name. latency_ms is total duration, ttft_ms is averaged only over the rows that have one, and the throughput figures use duration rather than the inverse coalesce they had.

Both averages carry samples, the count of rows behind them. The issue asks for TTFT "alongside its coverage": without it, an average over two streaming requests in a hundred reads the same as one over a hundred.

The dashboard is in this PR too

Seven places claimed TTFT while showing something else. Fixing only the backend would have left the tile labelled TTFT while displaying pure latency, a larger number, which reads as a sudden regression.

Relabelled: the metric tile, the chart subtitle, the legend and series names, and two table headers. The tile subtitle now carries real TTFT with its coverage, and falls back to req/min when nothing streamed.

Two more coalesces lived in the frontend and are gone. The logs table showed ttft_ms ?? latency_ms under a "Latency" header, and the CSV export wrote the same into a column headed "Latency (ms)".

Two details the shape depends on

avg_ttft_ms on a timeseries bucket is null, not zero, when nothing streamed in that bucket. Zero would claim a first token arrived instantly, and anything averaging these would be diluted by every bucket with no
streaming traffic — this issue one layer up. avg_latency_ms stays non-nullable: every returned bucket has at least one request and so always has a latency.

ttft_ms is optional-chained in the tile. It is a new field, so a response from an older backend has none, and an unguarded access would take the tile down during a rolling deploy.

Closes #304

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
@Priyanshu-u07
Priyanshu-u07 merged commit 69ebe93 into main Oct 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The Insights TTFT metric averages TTFT with total latency

1 participant