Add analyst runbooks for 36 v1 templates - #93
Merged
Merged
Conversation
Adds paste-testable runbook.md walkthroughs to every v1 template that lacked one, bringing the whole v1 catalog to parity with the existing runbooks (energy_grid_planning, telco_network_recovery, etc.). Each runbook is the analyst-facing bridge between the ontology, the script, and a Director-of-Analytics-style user: a brief problem statement, a distilled chain diagram, and an ordered set of question-shaped prompts (each naming its /rai-* skill) with the response the chain produces. Single-reasoner templates use the compact format; multi-reasoner templates use the staged fenced format with a /rai-discovery routing step. Coverage (36): 21 prescriptive, 3 graph, 2 rules, 2 predictive (GNN), 7 multi-reasoner. Every runbook's headline numbers come from a real run of the shipped script on Snowflake; 29 were additionally reproduced by a fresh skill-loaded agent that had not seen the script. Numbers that are solver-path- or training-dependent are framed to what an independent run reproduces.
Per the dev-templates-review runbook checklist:
- disease-outbreak-prevention: anchor the graph prompt by its structural
test (total risk across direct transfer links) instead of naming the
algorithm ("weighted degree centrality").
- warehouse_allocation: name the property in the interpret Response
("Site.centrality scores from the graph step") instead of "from Stage 1".
Doc-only edits from the dev-templates-review runbook checklist (no numbers
changed — every figure reused was already present in the same runbook):
- supply_chain_resilience: add business lead-in + sequential-cascade preface,
flesh out the build prompt to name concepts/relationships, add the solve-step
headline plan.
- memory_supply_allocation: remove a stage-number reference from a prompt; trim
over-dense Responses to lead with results.
- portfolio_balancing: add the cascade preface; trim Responses 6-8 of
"why this works" prose.
- datacenter_compute_allocation: trim the step-5 Response config narration.
- bom-reachability: trim the Q4 Response methodology aside.
- cybersecurity-attack-paths: align the examine prompt ("...how many of each?").
Clean folder-only validation (data + prompts + skills, no injected hints) surfaced seven runbooks whose stated figures depended on context not present in the pasteable prompts. Softened or pinned each to the reproducible result: - ad_spend_allocation: drop full-budget/cap-binds claim; keep 9,740 conversions, identical top-two levels, per-channel/campaign caps bind. - demand_planning_temporal: drop start-grid-dependent costs/weeks; keep OPTIMAL, zero unmet, holding-only, rises with horizon. - financial_index_replication: pin $10M portfolio value and define the equal-weight baseline in the prompts; soften TE and solve status. - hospital_staffing: make the frontier/knee qualitative; drop interpretation-sensitive dollar/unmet figures. - pod_placement: binding GPU node is an alternate-optimum detail. - planogram_optimization: MiniZinc incumbent varies; keep structure. - supply_chain_transport: keep reproducible LTL-vs-TL result.
Clean folder-only validation (data + prompts + skills, no injected hints) of the last six runbooks. Two passed as written (supplier_reliability, water_allocation); four needed correction: - underwriting_audit: pin the rule under audit (manual review = senior) in the prompt, so the deliberate-bug counts (property b: 12 counterexamples, property c: 8) are reproducible rather than dependent on the analyst guessing the review rule. - disease-outbreak-prevention: report raw weighted-degree scores (260 / 218 / 188), which is what 'total risk-weighted connections' yields folder-only; the prior figures were normalized by n-1. - humanitarian-aid-supply-chain: same — Central Warehouse weighted degree 794.81 raw, not the n-1-normalized 46.75. - book_slate_recommendation: soften the specific objective value; it depends on the path-counting method, diversity grain, and explanation floor (all analyst choices), so state the stable structure instead of a single number.
Seven unambiguous customer-facing defects (a dev-templates-review pass across all v1 READMEs surfaced these; the broader README standardization backlog is deferred to a separate change): - cell_tower_coverage, financial_index_replication: Quickstart download URL used private.relational.ai (404s); point at docs.relational.ai. - disease-outbreak-prevention, humanitarian-aid-supply-chain, wildlife-conservation-network: Quickstart unzip/cd used underscored names that don't match the hyphenated .zip, breaking copy-paste. - demand_forecasting: setup GRANTs targeted FAVORITA.EXPERIMENTS but the DDL creates FAVORITA_MINI.EXPERIMENTS, so setup failed. - memory_supply_allocation: stray </content></invoke> tool markup at end of file rendered as literal text.
cafzal
marked this pull request as ready for review
July 1, 2026 21:32
This was referenced Jul 1, 2026
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds paste-testable
runbook.mdwalkthroughs to every v1 template that lacked one (36 templates), bringing the whole v1 catalog to parity with the existing runbooks (energy_grid_planning,telco_network_recovery,it-dependency-mapping, …).Each runbook is the analyst-facing bridge between the ontology, the script, and a Director-of-Analytics-style user: a brief problem statement, a distilled chain diagram, and an ordered set of question-shaped prompts (each naming its
/rai-*skill) with the response the chain produces. Single-reasoner templates use the compact format; multi-reasoner templates use the staged fenced format with a/rai-discoveryrouting step.Coverage (36)
Verification (folder-only, on Snowflake)
31 of the 36 runbooks were independently reproduced by a fresh, skill-loaded agent that had not seen the script. The test is deliberately clean-room: the agent gets only the template's
data/, the runbook's pasteable prompts, and the named/rai-*skills — with no injected CSV headers, solver, problem type, seeds, or expected answers, so "matches the runbook" can't be circular. Every run executes on Snowflake (the engine customers use), and the headline numbers in each runbook are that run's real output.The remaining 5 GNN-training templates (demand_forecasting, smoker_status_prediction, fraud-detection, subscriber_retention, retail_planning) are accepted via ground-truth script runs rather than the clean-room paste-test, since they require GPU training.
Folder-only validation corrected 11 runbooks whose stated figures depended on context not present in the pasteable prompts — a normalization convention (disease-outbreak, humanitarian: raw vs n−1-normalized weighted degree), an underspecified rule (underwriting_audit: pin the review rule so the deliberate-bug counts reproduce), or a solver-path-/training-dependent objective (ad_spend_allocation, demand_planning_temporal, financial_index_replication, hospital_staffing, pod_placement, planogram_optimization, supply_chain_transport, book_slate_recommendation). Each was softened or pinned to what an independent run reproduces.
Also brings six pre-existing runbooks up to the runbook standard (supply_chain_resilience, memory_supply_allocation, portfolio_balancing, datacenter_compute_allocation, bom-reachability, cybersecurity-attack-paths).
README fixes (also in this PR)
A
/dev-templates-reviewpass across all v1 READMEs surfaced seven unambiguous, customer-facing defects, fixed here:private.relational.ai→docs.relational.ai(cell_tower_coverage, financial_index_replication).unzip/cdnames that didn't match the hyphenated.zip, breaking copy-paste (disease-outbreak-prevention, humanitarian-aid-supply-chain, wildlife-conservation-network).FAVORITA.EXPERIMENTS→FAVORITA_MINI.EXPERIMENTS(demand_forecasting).</content></invoke>tool markup rendering as literal text (memory_supply_allocation).Deferred to a follow-up (not in this PR)
The same review flagged two larger bodies of work, kept out to keep this PR focused on runbooks:
## Model overview/## Learn more/## Support, convert## Customize this templateto the H3 subheadings, split## Prerequisitesinto Access/Tools, trim oversized Quickstart "Expected output" blocks. The bulk of the portfolio predates this standard.$5,080→$1,444, retail_markdown$18,432.50→$23,374.65, production_planning's stale 3-solve loop output, diet's basket, demand_planning_temporal's week count). These need per-script reruns.FRAUD_DBwhere the data + EXPERIMENTS schema isFRAUD_DETECTION; hospital_staffing's script prints$1,728where the README gives$1,116.