Skip to content

feat(telemetry): measure the seconds of each row #4058

Open

Description

@dan-petty

Problem

No metric gives the seconds of one row. The job seconds counter has the labels repo and lane. It is counted once for a lane round (server/service.py, _execute_batch). So the 5 minute rule of AGENTS.md cannot be checked for one row. A slow row hides in the lane total.

Expected

Add one histogram devops_cli_roadmap_row_seconds with the labels repo, row and result.

  • _execute_due_rows (run.py) times each row around row.runner(...) and observes the seconds.
  • result uses the same values as devops_cli_service_jobs_total (_count_job).
  • The label is row, not job. The collector sets the label job from the resource and loses a data point attribute of that name (tool_metrics.py).
  • A row that is not due, or that the ledger suspends, adds no sample.

Series count: repo x row x result. The status, triage and model lanes hold at most 11 rows after the issue #4052. Each row has the results of _count_job. The bound is 11 rows x 3 results = 33 series for one repo, times the histogram buckets of the instrument.

Document the series in docs/TELEMETRY.md.

This issue changes no label of the model usage metrics. It keeps the label scheme of #3069 (repo, item_class, result) and of #3666 (repo, lane). It uses repo, row and result in the same meaning, so a dashboard (#3655) can join them.

No live model run (AGENTS.md rule 13).

Acceptance criteria

  • uv run pytest tests/telemetry/test_roadmap_row_seconds.py -k row_seconds - a round emits one sample for each row that ran, with its row and result.
  • uv run pytest tests/telemetry/test_roadmap_row_seconds.py -k not_run - a row that is not due, and a row that the ledger suspends, add no sample.
  • uv run pytest tests/telemetry/test_roadmap_row_seconds.py -k error_result - a row that raises adds a sample with the error result.
  • grep -q 'devops_cli_roadmap_row_seconds' docs/TELEMETRY.md - the documentation lists the series (it prints nothing today).

Every command above fails on the unchanged code. The pull request cites the fail-to-pass run (devops ci fail-to-pass --base <ref>).

Feasibility

I read these files at origin/main: server/service.py (_execute_batch), telemetry/instruments.py (emit, InstrumentKind.HISTOGRAM), roadmap/tool_metrics.py (the pattern and the row label), run.py (_execute_due_rows, _count_job) and docs/TELEMETRY.md. I did not run the code. The change makes no GitHub request.

Existing tool considered

telemetry/instruments.py (emit), tool_metrics.py as the pattern, and the OpenTelemetry histogram that exists. No new library.

Cost

Build: about 30k tokens and 1.5 h. Keep: about 2k tokens per quarter. Fix: about 8k tokens and 0.5 h.

Related

Part of #3027. #3069, #3650, #3655, #3666 (see the comment on #3069). Order: after the issue #4052 (both change run.py).

Activity

  1. added
    type/featureNew user-facing functionality or capability addition
    scope/telemetryOpenTelemetry tracing spans, Prometheus metrics, and Jaeger
    status/newStatus New: the item is not ready yet
    on Nov 7, 2026
  2. dan-petty commented on Nov 7, 2026

    @dan-petty
    OwnerAuthor

    Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.

    • Type: type/feature. The feat prefix of the title gives the type.
    • Scope: scope/telemetry. The (telemetry) scope of the title gives the scope.
    • Priority: P2-Medium. It adds an observability metric that enforces the 5-minute rule but does not fix a broken path.
    • Value: Medium. It lets operators see slow rows that hide in lane totals, improving troubleshooting.
    • Effort: Medium. The body estimates 1.5 h of build work across run.py, a test file, and docs.
    • Related: feat(roadmap): metrics for the pipeline outcomes #3069, which asks for a different change. Both add roadmap telemetry metrics under #3027, but #3069 delivers outcome counters while this candidate delivers a row-duration histogram.
  3. added
    priority/p2-mediumStandard priority feature work and architectural improvements
    on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mediumEffort Mediumpriority/p2-mediumStandard priority feature work and architectural improvementsscope/telemetryOpenTelemetry tracing spans, Prometheus metrics, and Jaegersource/agentFiled by an agentstatus/newStatus New: the item is not ready yettype/featureNew user-facing functionality or capability additionvalue/mediumValue Medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions