Skip to content

fix(infra): allow traceql metrics queries beyond 3h - #3810

Merged
tasoo-oos merged 1 commit into
mainfrom
t3083-tempo-metrics-query-range
Oct 15, 2026
Merged

tasoo-oos merged 1 commit into
mainfrom
t3083-tempo-metrics-query-range

Conversation

@manamana32321

@manamana32321 manamana32321 commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Description

Grafana에서 3시간이 넘는 구간을 잡으면 트레이스로 만든 그래프가 비어 버리는 문제를 고칩니다.

트레이스 자체는 90일치가 저장돼 있고 검색도 잘 됩니다. 막히는 건 트레이스를 모아 그린 그래프(요청량·응답시간 추이)뿐인데, Tempo가 이 그래프만 "한 번에 3시간어치까지"로 제한하고 있었습니다. 우리가 이 값을 정해준 적이 없어서 Tempo 기본값이 그대로 쓰이던 상태입니다.

상한을 7일로 올립니다. 트레이스 검색 쪽에 이미 적용돼 있던 값과 같은 값입니다.

Additional context

보존 기간(90일)은 데이터를 얼마나 오래 들고 있을지를 정하는 값이고, 이번에 고친 상한은 한 번에 몇 시간어치를 그려줄지를 정하는 값입니다.

Before submitting the PR, please make sure you do the following

Closes TAS-3083

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • TraceQL metrics queries can now cover time ranges of up to seven days.

Tempo caps metrics queries at query_frontend.metrics.max_duration, which
defaults to 3h. The chart left query_frontend empty, so any TraceQL metrics
query over a longer range returned 400 even though blocks are retained for
90 days. Trace search was unaffected (its own limit is 168h).

Set the metrics limit to 168h to match trace search.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@manamana32321 manamana32321 self-assigned this Oct 4, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a5eefd45-84e6-4a79-9cd9-7323048099db
📥 Commits

Reviewing files that changed from the base of the PR and between fa36506 and 22489af.

📒 Files selected for processing (1)
  • infra/k8s/monitoring/tempo/values.yaml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Tempo Helm values now set the query frontend metrics maximum duration to 168 hours.

Changes

Tempo metrics query duration

Layer / File(s) Summary
Set metrics query duration
infra/k8s/monitoring/tempo/values.yaml
The query frontend metrics configuration sets the maximum query duration to 168 hours.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Suggested reviewers: keulma

Merge Risk: ⚪ Minimal · up to 22489

The configuration is positioned to raise the metrics-query limit to seven days. No identified issue prevents merging after normal checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 22489

Seven-day metrics queries can increase load on trace monitoring and its storage. The change does not add credentials, privileges, or an external access path, and it is reversible through configuration. Effective workload limits and runtime behavior remain unverified.

Retained concerns

  • Low · reliability · inferred: Allowing seven-day metrics queries increases the potential work initiated by existing Grafana query users against Tempo and its trace store. This can affect monitoring availability, although exhaustion is not demonstrated and effective workload containment remains unverified.
Security review details

Security Blast Radius

  • inferred — The supported exposure is greater query work against each affected environment’s Tempo service and backing trace store. The evidence does not establish new access to other tenants, services, credentials, or data stores, nor does it establish cluster-wide exhaustion.

Security Findings and Attack Paths

  • inferred — An existing user able to query the Grafana Tempo datasource can request a larger metrics-query interval, potentially increasing backend work. This is a workload-abuse hypothesis, not a verified denial-of-service finding: capacity, repetition limits, cancellation behavior, and actual impact remain unverified.

Trust Boundaries and Controls

  • observed — Both Grafana environments configure GitHub authentication with an organization restriction and proxy Tempo requests to an internal service. These are existing controls, not new protections from this PR. Rendered authentication defaults and direct Tempo network reachability were not fully verified.

Hardening Proposals

  • proposed — Before production propagation, validate the effective duration setting and workload controls, exercise representative long-range queries in stage, and define a saturation or latency threshold for reverting or lowering the cap. This is a validation proposal, not evidence that controls are absent.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: increasing the TraceQL metrics query duration limit beyond 3 hours.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@tasoo-oos tasoo-oos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tasoo-oos
tasoo-oos added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit d696131 Oct 15, 2026
48 checks passed
@tasoo-oos
tasoo-oos deleted the t3083-tempo-metrics-query-range branch October 15, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants