Skip to content

Add analyst runbooks for 36 v1 templates - #93

Merged
cafzal merged 6 commits into
mainfrom
add-v1-template-runbooks
Jul 1, 2026
Merged

cafzal merged 6 commits into
mainfrom
add-v1-template-runbooks

Conversation

@cafzal

@cafzal cafzal commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

What

Adds paste-testable runbook.md walkthroughs to every v1 template that lacked one (36 templates), bringing the whole v1 catalog to parity with the existing runbooks (energy_grid_planning, telco_network_recovery, it-dependency-mapping, …).

Each runbook is the analyst-facing bridge between the ontology, the script, and a Director-of-Analytics-style user: a brief problem statement, a distilled chain diagram, and an ordered set of question-shaped prompts (each naming its /rai-* skill) with the response the chain produces. Single-reasoner templates use the compact format; multi-reasoner templates use the staged fenced format with a /rai-discovery routing step.

Coverage (36)

  • Prescriptive (21): ad_spend_allocation, campaign_roi, cell_tower_coverage, cicd_runner_allocation, demand_planning_temporal, diet, factory_production, financial_index_replication, hospital_staffing, money_laundering_motif_detection, network_flow_planning, pod_placement, product_configurator, production_planning, retail_markdown, shift_assignment, sprint_scheduling, supplier_reliability, supply_chain_transport, traveling_salesman, underwriting_audit, water_allocation
  • Graph (3): disease-outbreak-prevention, humanitarian-aid-supply-chain, wildlife-conservation-network
  • Rules (2): commercial_underwriting, shipment_compliance
  • Predictive / GNN (2): demand_forecasting, smoker_status_prediction
  • Multi-reasoner (7): book_slate_recommendation, fraud-detection, patient_cohort_recruitment, planogram_optimization, retail_planning, subscriber_retention, warehouse_allocation

Verification (folder-only, on Snowflake)

31 of the 36 runbooks were independently reproduced by a fresh, skill-loaded agent that had not seen the script. The test is deliberately clean-room: the agent gets only the template's data/, the runbook's pasteable prompts, and the named /rai-* skills — with no injected CSV headers, solver, problem type, seeds, or expected answers, so "matches the runbook" can't be circular. Every run executes on Snowflake (the engine customers use), and the headline numbers in each runbook are that run's real output.

The remaining 5 GNN-training templates (demand_forecasting, smoker_status_prediction, fraud-detection, subscriber_retention, retail_planning) are accepted via ground-truth script runs rather than the clean-room paste-test, since they require GPU training.

Folder-only validation corrected 11 runbooks whose stated figures depended on context not present in the pasteable prompts — a normalization convention (disease-outbreak, humanitarian: raw vs n−1-normalized weighted degree), an underspecified rule (underwriting_audit: pin the review rule so the deliberate-bug counts reproduce), or a solver-path-/training-dependent objective (ad_spend_allocation, demand_planning_temporal, financial_index_replication, hospital_staffing, pod_placement, planogram_optimization, supply_chain_transport, book_slate_recommendation). Each was softened or pinned to what an independent run reproduces.

Also brings six pre-existing runbooks up to the runbook standard (supply_chain_resilience, memory_supply_allocation, portfolio_balancing, datacenter_compute_allocation, bom-reachability, cybersecurity-attack-paths).

README fixes (also in this PR)

A /dev-templates-review pass across all v1 READMEs surfaced seven unambiguous, customer-facing defects, fixed here:

  • Broken Quickstart download URLs — private.relational.aidocs.relational.ai (cell_tower_coverage, financial_index_replication).
  • Quickstart unzip/cd names that didn't match the hyphenated .zip, breaking copy-paste (disease-outbreak-prevention, humanitarian-aid-supply-chain, wildlife-conservation-network).
  • Setup-breaking GRANT typo — FAVORITA.EXPERIMENTSFAVORITA_MINI.EXPERIMENTS (demand_forecasting).
  • Stray </content></invoke> tool markup rendering as literal text (memory_supply_allocation).

Deferred to a follow-up (not in this PR)

The same review flagged two larger bodies of work, kept out to keep this PR focused on runbooks:

  • Portfolio-wide README standardization (~45 READMEs): remove the body H1 (the gallery renders the title from front matter), add ## Model overview / ## Learn more / ## Support, convert ## Customize this template to the H3 subheadings, split ## Prerequisites into Access/Tools, trim oversized Quickstart "Expected output" blocks. The bulk of the portfolio predates this standard.
  • README "Expected output" vs script/runbook reconciliation (~12 templates): several READMEs print stale numbers that disagree with the shipped script (e.g. supply_chain_transport $5,080$1,444, retail_markdown $18,432.50$23,374.65, production_planning's stale 3-solve loop output, diet's basket, demand_planning_temporal's week count). These need per-script reruns.
  • Two script-level bugs seen while running templates end-to-end: fraud-detection ships FRAUD_DB where the data + EXPERIMENTS schema is FRAUD_DETECTION; hospital_staffing's script prints $1,728 where the README gives $1,116.

Adds paste-testable runbook.md walkthroughs to every v1 template that
lacked one, bringing the whole v1 catalog to parity with the existing
runbooks (energy_grid_planning, telco_network_recovery, etc.).

Each runbook is the analyst-facing bridge between the ontology, the
script, and a Director-of-Analytics-style user: a brief problem
statement, a distilled chain diagram, and an ordered set of
question-shaped prompts (each naming its /rai-* skill) with the response
the chain produces. Single-reasoner templates use the compact format;
multi-reasoner templates use the staged fenced format with a
/rai-discovery routing step.

Coverage (36): 21 prescriptive, 3 graph, 2 rules, 2 predictive (GNN),
7 multi-reasoner. Every runbook's headline numbers come from a real run
of the shipped script on Snowflake; 29 were additionally reproduced by a
fresh skill-loaded agent that had not seen the script. Numbers that are
solver-path- or training-dependent are framed to what an independent run
reproduces.
Per the dev-templates-review runbook checklist:
- disease-outbreak-prevention: anchor the graph prompt by its structural
  test (total risk across direct transfer links) instead of naming the
  algorithm ("weighted degree centrality").
- warehouse_allocation: name the property in the interpret Response
  ("Site.centrality scores from the graph step") instead of "from Stage 1".
Doc-only edits from the dev-templates-review runbook checklist (no numbers
changed — every figure reused was already present in the same runbook):
- supply_chain_resilience: add business lead-in + sequential-cascade preface,
  flesh out the build prompt to name concepts/relationships, add the solve-step
  headline plan.
- memory_supply_allocation: remove a stage-number reference from a prompt; trim
  over-dense Responses to lead with results.
- portfolio_balancing: add the cascade preface; trim Responses 6-8 of
  "why this works" prose.
- datacenter_compute_allocation: trim the step-5 Response config narration.
- bom-reachability: trim the Q4 Response methodology aside.
- cybersecurity-attack-paths: align the examine prompt ("...how many of each?").
Clean folder-only validation (data + prompts + skills, no injected
hints) surfaced seven runbooks whose stated figures depended on
context not present in the pasteable prompts. Softened or pinned each
to the reproducible result:

- ad_spend_allocation: drop full-budget/cap-binds claim; keep 9,740
  conversions, identical top-two levels, per-channel/campaign caps bind.
- demand_planning_temporal: drop start-grid-dependent costs/weeks; keep
  OPTIMAL, zero unmet, holding-only, rises with horizon.
- financial_index_replication: pin $10M portfolio value and define the
  equal-weight baseline in the prompts; soften TE and solve status.
- hospital_staffing: make the frontier/knee qualitative; drop
  interpretation-sensitive dollar/unmet figures.
- pod_placement: binding GPU node is an alternate-optimum detail.
- planogram_optimization: MiniZinc incumbent varies; keep structure.
- supply_chain_transport: keep reproducible LTL-vs-TL result.
Clean folder-only validation (data + prompts + skills, no injected
hints) of the last six runbooks. Two passed as written
(supplier_reliability, water_allocation); four needed correction:

- underwriting_audit: pin the rule under audit (manual review = senior)
  in the prompt, so the deliberate-bug counts (property b: 12
  counterexamples, property c: 8) are reproducible rather than
  dependent on the analyst guessing the review rule.
- disease-outbreak-prevention: report raw weighted-degree scores
  (260 / 218 / 188), which is what 'total risk-weighted connections'
  yields folder-only; the prior figures were normalized by n-1.
- humanitarian-aid-supply-chain: same — Central Warehouse weighted
  degree 794.81 raw, not the n-1-normalized 46.75.
- book_slate_recommendation: soften the specific objective value; it
  depends on the path-counting method, diversity grain, and
  explanation floor (all analyst choices), so state the stable
  structure instead of a single number.
Seven unambiguous customer-facing defects (a dev-templates-review pass
across all v1 READMEs surfaced these; the broader README
standardization backlog is deferred to a separate change):

- cell_tower_coverage, financial_index_replication: Quickstart download
  URL used private.relational.ai (404s); point at docs.relational.ai.
- disease-outbreak-prevention, humanitarian-aid-supply-chain,
  wildlife-conservation-network: Quickstart unzip/cd used underscored
  names that don't match the hyphenated .zip, breaking copy-paste.
- demand_forecasting: setup GRANTs targeted FAVORITA.EXPERIMENTS but the
  DDL creates FAVORITA_MINI.EXPERIMENTS, so setup failed.
- memory_supply_allocation: stray </content></invoke> tool markup at
  end of file rendered as literal text.
@cafzal
cafzal marked this pull request as ready for review July 1, 2026 21:32
@cafzal
cafzal merged commit 5ba1ea2 into main Jul 1, 2026
2 of 3 checks passed
@cafzal
cafzal deleted the add-v1-template-runbooks branch July 1, 2026 21:33

This branch had an error being deployed

1 failed deployment
Preview 96515a5c Deployed Jul 1, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant