Skip to content

fix(roadmap): a status read that fails fails its own row #4052

Open

Description

@dan-petty

Problem

The row metrics (run.py, _run_metrics_adapter) reads three sources. Each read has its own except Exception:

  • the project metrics (needs the clone);
  • the GitHub status page;
  • the Cloudflare status page.

A failed read is logged and skipped. The row counts success and stamps schedule.json. So the ledger, the counter devops_cli_service_jobs_total{row="metrics",result="error"} and the alert RoadmapServiceJobsFailing never see a failed collection.

The status page readers already raise ServiceStatusError on a failed read (telemetry/service_status.py). The row hides it.

Expected

Split the row into three rows. Remove the three except Exception blocks. A failed read raises, and the row of that source fails alone.

Row names:

  • metrics stays. It reads the project metrics and needs the clone.
  • github_status is new. It needs no clone.
  • cloudflare_status is new. It needs no clone.

The status lane order becomes metrics, implement, stage_ready, release_comments, github_status, cloudflare_status. The two new rows come last, so a slow status page never delays implement. The sentences that say implement follows metrics stay true.

Each row has its own entry in schedule.json, its own entry in the ledger file service-failures.json, and its own value of the row label. The label appears in devops_cli_service_jobs_total and devops_cli_service_job_suspended. Only metrics sends GitHub API requests, so its devops_cli_github_requests_total series do not change.

Change no interval. The two new rows use DEFAULT_ROADMAP_METRICS_INTERVAL_MINUTES. The three reads are the same calls under three rows.

Places that name the row list and change:

  • run.py: STATUS_LANE_ROWS, build_stub_table, and the docstrings.
  • k8s/README.md: item 17 of the Roadmap service list.
  • openspec/specs/roadmap-implement-job/spec.md (line 435): the requirement of the status lane order.
  • tests/roadmap/test_roadmap_run.py, tests/roadmap/test_roadmap_status_round.py, tests/roadmap/implement/test_roadmap_implement_submit.py (line 145) and tests/roadmap/stage/test_roadmap_stage_ready.py (line 407). Each lists the four row names.

Acceptance criteria

  • uv run pytest tests/roadmap/test_roadmap_status_rows.py -k source_failed - a failed GitHub status read counts result=error for the row github_status only. metrics and cloudflare_status still count success and stamp schedule.json.
  • uv run pytest tests/roadmap/test_roadmap_status_rows.py -k row_names - the status lane holds metrics, implement, stage_ready, release_comments, github_status and cloudflare_status in this order.
  • uv run pytest tests/roadmap/test_roadmap_status_rows.py -k ledger_per_row - three rounds of a failed Cloudflare read suspend cloudflare_status and no other row.
  • grep -q 'github_status' openspec/specs/roadmap-implement-job/spec.md && grep -q 'cloudflare_status' k8s/README.md - the spec and the service list name the new rows (both fail today).

Every command above fails on the unchanged code. The pull request cites the fail-to-pass run (devops ci fail-to-pass --base <ref>).

Feasibility

I read these files at origin/main: run.py (_run_metrics_adapter, STATUS_LANE_ROWS, build_stub_table), telemetry/service_status.py, server/counter_series.py (it reads STATUS_LANE_ROWS), k8s/README.md items 15 to 20 and the spec roadmap-implement-job. I searched for the row name metrics and listed the files that name it. I did not run the Service.

I searched k8s/monitoring for a row label selector and found none. So no alert or dashboard names a row.

Existing tool considered

The row table JobRow and the ledger of run.py. No library is needed.

Cost

Build: about 45k tokens and 2.5 h (two rows, the table, 4 test files, 3 documents). Keep: about 2k tokens per quarter. Fix: about 10k tokens and 0.5 h.

Related

Part of #3027. #3941 and #3996 (request cost of the metrics row only), #3666, #3655. Order: after the issue #4048. Both change run.py and k8s/README.md.

Activity

  1. added
    type/bugDefect, regression, or incorrect runtime behavior
    scope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs them
    status/newStatus New: the item is not ready yet
    on Nov 7, 2026
  2. dan-petty commented on Nov 7, 2026

    @dan-petty
    OwnerAuthor

    Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.

    • Type: type/bug. The fix prefix of the title gives the type.
    • Scope: scope/roadmap. The (roadmap) scope of the title gives the scope.
    • Priority: P2-Medium. Failed reads are silently swallowed, so the ledger and alerts never fire, but the service still delivers data.
    • Value: Medium. Restores correct failure counting and alerting for two status page reads without changing the data they deliver.
    • Effort: Medium. Split one row into three, update the lane order, four test files, and three documents; the candidate estimates about 2.5 h.
    • Related: fix(roadmap): one failed item does not suspend the stage row #3493, which asks for a different change. Both fix failure isolation in roadmap rows, but #3493 stops one failed item from suspending the stage_ready row, while the candidate stops one failed read from hiding the error inside the metrics row.
  3. added
    priority/p2-mediumStandard priority feature work and architectural improvements
    on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mediumEffort Mediumpriority/p2-mediumStandard priority feature work and architectural improvementsscope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs themsource/agentFiled by an agentstatus/newStatus New: the item is not ready yettype/bugDefect, regression, or incorrect runtime behaviorvalue/mediumValue Medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions