Skip to content

feat(telemetry): an intake outcome counter and implement step seconds #4067

Open

Description

@dan-petty

Problem

Intake has no counter of placed, skipped, failed or duplicate-flagged candidates. The implement job has no timer for the scoped checks, the publish step and the watch of the PR checks. The scoped checks take about 3.5 minutes. The watch takes up to 3600 s. Nobody can see which of them is slow.

Expected

Intake. Add the counter devops_cli_roadmap_intake_candidates_total with the labels repo and result. The values of result are placed, skipped, failed and duplicate. apply_intake adds 1 for each candidate. Series count: 4 per repo.

Implement. Use the histogram devops_cli_roadmap_step_seconds of the issue #4066, with row set to implement. Add three StepName members: checks, publish and watch. The implement job times:

  • checks: the scoped checks (_scoped_checks);
  • publish: the push and the draft PR;
  • watch: the watch of the PR checks (watch_pull).

result is ok or error. Series count: 3 steps x 2 results = 6 per repo, times the histogram buckets.

Document both series in docs/TELEMETRY.md. State the series counts there.

The implement Job runs in a runner pod. Check that the Job sends its samples to the collector before it ends. If it does not, the pull request says so and times the steps in the Service from the log of the Job instead.

No live model run (AGENTS.md rule 13).

Acceptance criteria

  • uv run pytest tests/telemetry/test_roadmap_intake_implement_metrics.py -k intake_outcome - intake emits one outcome counter for each candidate, with its result.
  • uv run pytest tests/telemetry/test_roadmap_intake_implement_metrics.py -k implement_step - the scoped checks, the publish step and the watch each emit one histogram sample with row=implement.
  • uv run pytest tests/telemetry/test_roadmap_intake_implement_metrics.py -k step_names - the step values are members of StepName.
  • grep -q 'devops_cli_roadmap_intake_candidates_total' docs/TELEMETRY.md - the documentation lists the new counter (it prints nothing today).

Every command above fails on the unchanged code. The pull request cites the fail-to-pass run (devops ci fail-to-pass --base <ref>).

Feasibility

I read these files at origin/main: intake.py (apply_intake), implement_run.py (_scoped_checks, _push, _checks), implement_pull.py (watch_pull), telemetry/instruments.py and docs/TELEMETRY.md. I did not run the code. I did not check how a runner pod sends telemetry. That is why the pull request must check it.

The change makes no GitHub request.

Existing tool considered

telemetry/instruments.py (emit) and tool_metrics.py as the pattern. No new library.

Cost

Build: about 45k tokens and 2.5 h. Keep: about 3k tokens per quarter. Fix: about 10k tokens and 0.5 h.

Related

Part of #3027. #3069, #3650, #3655, #3076. Order: after the issues #4061 and #4066.

Activity

  1. added
    type/featureNew user-facing functionality or capability addition
    scope/telemetryOpenTelemetry tracing spans, Prometheus metrics, and Jaeger
    status/newStatus New: the item is not ready yet
    on Nov 7, 2026
  2. dan-petty commented on Nov 7, 2026

    @dan-petty
    OwnerAuthor

    Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.

    • Type: type/feature. The feat prefix of the title gives the type.
    • Scope: scope/telemetry. The (telemetry) scope of the title gives the scope.
    • Priority: P2-Medium. It improves observability of the implement job but does not block a release or fix a defect.
    • Value: Medium. Operators gain per-step timing and an intake outcome count, which aids debugging slow runs.
    • Effort: Medium. The body estimates about 2.5 hours of build work across intake, implement, and docs files.
    • Related: feat(telemetry): step seconds and a gate counter for the stage row #4066, which asks for a different change. Both use the histogram devops_cli_roadmap_step_seconds and the StepName enum; #4066 creates the instrument for stage, while this candidate adds checks, publish, and watch members and wires the implement job to emit samples.
  3. added
    priority/p2-mediumStandard priority feature work and architectural improvements
    on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mediumEffort Mediumpriority/p2-mediumStandard priority feature work and architectural improvementsscope/telemetryOpenTelemetry tracing spans, Prometheus metrics, and Jaegersource/agentFiled by an agentstatus/newStatus New: the item is not ready yettype/featureNew user-facing functionality or capability additionvalue/mediumValue Medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions