Fix stale/incorrect CLI claims in skills and AGENTS.md

[ndenny]

Summary

Live-tested all 8 claude-skills workflows, the 3 embedded setup skills (skills/embed/*.md), AGENTS.md, and the README examples against apimetrics-qc, and fixed every confirmed discrepancy between the docs and actual CLI/server behavior. Enum and command-availability claims were also cross-checked against the production spec, so the docs hold for both environments. One CLI behavior change: apimetrics onboard now prints the eight workflow skills as well as the setup guides.

Command-name / flag fixes

• list-call-results doesn't exist → list-results-by-call. Its flags are --from/--time/--result-category/--cursor/--limit (default 100, one page per call), not --since/--before; list-results likewise takes --from/--time.
• run-call doesn't exist → run-monitor <id> runs API, browser, and MCP monitors alike. It requires a body ({} minimum).
• list-slos/get-slo/create-slo/update-slo/delete-slo don't exist. SLOs are one per project: get-project-slo / update-project-slo (creates one if none exists) / delete-project-slo, scoped via include_tags/exclude_tags. apimetrics-slo-review is rewritten around this.
• list-schedules-for-call doesn't exist → list-schedules-by-call.
• The root --help example used create-monitor, which doesn't exist in either environment → a working create-call shorthand example (cli/cli.go).
• On QC, list-calls/create-call/get-call/update-call/delete-call are deprecated in favour of /api/3/monitors/ commands (list-api-monitors, create-api-monitor with a flat body, …). Production's spec still lists the v2 commands and has no v3 replacements, so the docs keep the v2 commands, explain the deprecation warning, name the replacements, and prefer list-monitors/get-monitor, which exist in both.

Result and query semantics

• result_category is PASS/SLOW/WARNING/FAIL (HTTP 4xx lands in WARNING, connection errors in FAIL) and is empty while a run is queued; meta.result carries QUEUED, then COMPLETE/HTTP_CLIENT_ERROR/CONNECTION_ERROR/…. Polling now waits on meta.result. --result-category silently ignores any other value and returns every category — documented wherever the flag appears.
• Result rows are {id, meta, timing} (meta.call_id, meta.start_time, timing.total ms); the response_time/test/created fields the docs named don't exist.
• get-result for ANALYST+ callers returns the full object by default (response headers/body, request method/URL/headers, DNS/TLS detail, timing); --mode full adds the request body, messages, and certificate detail. Below-ANALYST callers get meta and timing only.
• Query metric lists completed per monitor type (API/browser/MCP differ); omitting status includes failed runs; group_by: interval requires interval; project-wide DNS diagnostics ignore group_by: monitor.
• SLO period is DAY/WEEK/MONTH; get-project-slo returns platform defaults (created: null) when none is stored; get-call-passfail-total's top-level stat.total counts only uncategorised results, so sum stat.<CATEGORY>.total for the run count. Daily buckets are UTC.

Output and input

• -f paths are rooted at the response envelope, so they start with body., and must be single-quoted in zsh; a non-matching path prints nothing. The README's -f and -o table examples are fixed.
• CLI Shorthand values containing // must be quoted (url: '"https://…"'), and shorthand also reads stdin, so non-interactive callers need </dev/null or the command blocks.
• list-schedules/list-monitors return {data, meta}.
• list-auth-settings and monitor configs return credentials in clear text, so the audit and investigation workflows now project them with -f.

Monitor setup

• MCP monitors need a Streamable HTTP endpoint (usually /mcp); SSE-only /sse endpoints fail with 405. Every step needs timeout_ms and wait_ms — the old example returned 422.
• 422 is body validation (detail[].loc names the field) and 400 is a rejected value; quota exhaustion is a separate Project is out of quota error, not 422.
• create-call's meta accepts name/description/tags/workspace (domain is derived from the URL). set-call-conditions now has a working example and its source/operator vocabulary; projects carry default conditions (get-project-conditions).
• Creating a project does not grant the creator access; a follow-up create-project-access is required. add-call-to-schedule rejects invalid or other-project IDs with 400.
• Fixed the nonexistent --api-key flag and result.success/result.failure_reason shape in the embedded setup skills.

bulk (config-as-code)

• apimetrics:/<collection> isn't a resolvable scheme; the skill uses a real host+path, and the bulk help examples are fixed (including a --match example that matched nothing).
• bulk init/status/diff/pull work against /api/2/calls/, but bulk push of an unchanged checkout file fails — PUT rejects its read-only fields. The skill now reduces changed files to the writable fields (that PUT is a partial update), saves a baseline copy for rollback, and passes the same --project-id on every bulk command, because the checkout doesn't record its project.
• Documented, not fixed: bulk push prints Push complete. and exits 0 when uploads fail, bulk init crashes on a one-resource list, and --match resolves top-level fields only.

onboard

• AGENTS.md and the README said apimetrics onboard prints every workflow; it printed only the three setup guides. printSkills now also prints the eight SKILL.md files, and TestPrintSkillsIncludesEveryWorkflow checks every embedded file appears in its output.

Detail

How this was tested

Built apimetrics-qc from this branch and ran each workflow end to end in the contxt org's "Nick CLI test 1/2/3" projects: project creation and access grants, API/browser/MCP monitor creation (heredoc and shorthand), conditions, schedules, run-monitor on all three types, result polling and retrieval, SLO create/update/delete (restored afterwards), the query-* commands, and a full bulk init → edit → push → rollback cycle in an isolated project. Those test projects and monitors are kept for future testing. go test ./... passes.

Server-side behaviour found (not addressed here)

• create-project-access sets the grantee only from account_id, which defaults to the caller; email is an informational label. A body with email alone therefore grants the caller while recording someone else's email — ported legacy behaviour, but a footgun. The bootstrap skill now says so.
• get-result-content returned "Content not found" even for a result's own content_path.
• get-call-passfail-total's top-level stat.total counts only uncategorised results (deliberate parity with the legacy aggregation), so the run count is the sum of the per-category totals. The skills and AGENTS.md say so.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Several updated docs are internally inconsistent about result status field naming and one bulk init example contradicts its surrounding “URL/link” guidance, which could mislead users copying these workflows.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 4

Open (4)

• Align summary field name with the rest of the docs · New
• Use url-template for checkout path example · New
• Use result_category consistently for row status · New
• Remove conflicting meta.result_category naming · New

What changed in this PR

Updates the CLI’s agent-facing documentation (AGENTS.md, installable skills, and embedded onboarding skills) to remove stale command/flag/response-shape claims and align guidance with the current observed CLI surface, plus a small help-text tweak in the hidden bulk command group.

Changes:

• Fixes multiple incorrect command names/flags across skills (e.g., run-monitor vs run-call, list-results-by-call, --from/--time, per-project SLO commands).
• Updates embedded onboarding skills to remove the nonexistent --api-key recovery path and to use the correct result-status field naming.
• Replaces the bulk help example that previously used an unsupported apimetrics:/...-style address with a standard host/path example.

File	Description
skills/​embed/​setup-mcp-monitor.md	Updates polling/validation field name and removes nonexistent --api-key guidance.
skills/​embed/​setup-browser-monitor.md	Updates polling/validation field name and removes nonexistent --api-key guidance.
skills/​embed/​setup-api-monitor.md	Replaces nonexistent run-call/list-call-results usage with run-monitor/list-results-by-call and updates polling guidance.
skills/​claude-skills/​apimetrics-weekly-health-review/​SKILL.md	Aligns list/result commands and flags; updates SLO/list envelope guidance.
skills/​claude-skills/​apimetrics-slo-review/​SKILL.md	Reworks SLO model and commands to “one SLO per project” semantics.
skills/​claude-skills/​apimetrics-project-bootstrap/​SKILL.md	Updates project creation/access notes and run/result guidance.
skills/​claude-skills/​apimetrics-monitoring-estate-audit/​SKILL.md	Updates SLO and schedule-by-call guidance plus list flag corrections.
skills/​claude-skills/​apimetrics-incident-triage/​SKILL.md	Updates list-results flags and related guidance.
skills/​claude-skills/​apimetrics-failure-investigation/​SKILL.md	Updates list/run commands and result-detail guidance.
skills/​claude-skills/​apimetrics-config-as-code/​SKILL.md	Documents bulk init address behavior and updates recommended bulk init usage.
bulk/​commands.go	Updates the bulk group example string to use a standard host/path address format.
AGENTS.md	Updates command naming, list envelope notes, SLO model, and bulk-init guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟢 Approval recommended

The changes are documentation corrections plus one cosmetic example string; the verifiable claim (broken apimetrics:/ scheme) checks out against cli.FixAddress and the apimetrics config key, and the edits are internally consistent with prior feedback resolved.

Review effort: Balanced
Findings: None

Resolved since last review (4)

• Remove conflicting meta.result_category naming
• Use result_category consistently for row status
• Use url-template for checkout path example
• Align summary field name with the rest of the docs

[Copilot]

Copilot review overview

🟡 Changes recommended

The slo-review skill introduces an incorrect flag --apimetrics-project-id, whereas the CLI's actual global flag is --project-id, contradicting this PR's goal of removing inaccurate CLI claims.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Uses incorrect project override flag name · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

The updated bulk top-level help example still uses a potentially stale /monitors path and should be aligned with the repo’s other bulk init examples to avoid reintroducing misleading CLI help output.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Uses incorrect project override flag name

Previously missed (1)

In code that hasn't changed since last review

Top-level bulk help uses stale /monitors path

bulk/​commands.go:317

The top-level bulk command help example uses the path /monitors, but this endpoint/path isn't referenced anywhere else in the repo (and the bulk init subcommand examples use /users). Keeping /monitors here risks reintroducing a stale or misleading hint in CLI help text; consider aligning it with the bulk init examples.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The documentation's correctness depends on many command-name and behavioral claims that are generated at runtime from the live OpenAPI spec and cannot be independently verified from repository source, so a human familiar with the live CLI should confirm them.

Review effort: Balanced
Findings: None

[Copilot]

Copilot review overview

🟡 Changes recommended

One embedded-skill "hard rule" claims list-results-by-call works across API/browser/MCP monitors, contradicting the API-call-only scoping used everywhere else in this PR.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Restrict list-results-by-call to API calls only · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

The PR's core value is the factual accuracy of many CLI behavior, command-name, and flag claims that cannot be statically verified (the command tree is generated at runtime from a live OpenAPI spec and role-based server responses), so domain/human verification is warranted even though the two code edits are benign and internally consistent.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Restrict list-results-by-call to API calls only

[Copilot]

🔵 Needs a closer look

It is a large documentation PR whose many live-tested CLI/server behavior claims cannot be independently verified here, and it still reasserts list-results-by-call as cross-type against the command's own --help, so a human with environment access should confirm the claims before approval.

1 open finding

• Mislabels list-results-by-call as cross-monitor-type · New

🧠 Review effort: Balanced