From ecb156f902e9462b04b8bffa1b5ee5ad95c32c43 Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:12:35 +0100 Subject: [PATCH 1/8] test(evals): unscripted routing cases Co-Authored-By: Claude Fable 5.2 --- evals/natural/cases/admin-guide.json | 7 ++ .../natural/cases/audio-version-changes.json | 16 +++ evals/natural/cases/audiobook-video.json | 16 +++ evals/natural/cases/chapter-fits-canon.json | 17 +++ .../cases/chapters-sound-different.json | 18 ++++ evals/natural/cases/concept-question.json | 7 ++ evals/natural/cases/new-story.json | 7 ++ evals/natural/cases/tutorial-video.json | 8 ++ evals/natural/cases/write-chapter.json | 16 +++ evals/natural/expected.json | 101 ++++++++++++++++++ 10 files changed, 213 insertions(+) create mode 100644 evals/natural/cases/admin-guide.json create mode 100644 evals/natural/cases/audio-version-changes.json create mode 100644 evals/natural/cases/audiobook-video.json create mode 100644 evals/natural/cases/chapter-fits-canon.json create mode 100644 evals/natural/cases/chapters-sound-different.json create mode 100644 evals/natural/cases/concept-question.json create mode 100644 evals/natural/cases/new-story.json create mode 100644 evals/natural/cases/tutorial-video.json create mode 100644 evals/natural/cases/write-chapter.json create mode 100644 evals/natural/expected.json diff --git a/evals/natural/cases/admin-guide.json b/evals/natural/cases/admin-guide.json new file mode 100644 index 0000000..2496463 --- /dev/null +++ b/evals/natural/cases/admin-guide.json @@ -0,0 +1,7 @@ +{ + "id": "admin-guide", + "prompt": "Schreib eine Anleitung für Administratoren, wie man in Keyforge einen API-Schlüssel rotiert.", + "files": { + "README.md": "# Keyforge\n\nKeyforge issues and rotates API keys. Rotation: Settings > Keys > Rotate; the old key stays valid for 24 hours.\n" + } +} diff --git a/evals/natural/cases/audio-version-changes.json b/evals/natural/cases/audio-version-changes.json new file mode 100644 index 0000000..063e32a --- /dev/null +++ b/evals/natural/cases/audio-version-changes.json @@ -0,0 +1,16 @@ +{ + "id": "audio-version-changes", + "prompt": "Für die Hörbuch-Fassung von Kapitel 11 möchte ich zwei Sätze anders haben, weil sie vorgelesen holpern. Der Buchtext soll aber so bleiben.", + "files": { + "AGENTS.md": "# Vault instructions\n\nThis directory is one vault. Its kind is stated in the frontmatter of 00-project/brief.md and 00-project/voice.md; here it is `story`. Only kind story permits manuscript drafting; kind study permits study notes only; kind starter is an unused template.\n\nManuscript files follow 00-project/voice.md and nothing from documentation or didactic writing rules. Other vault notes are plain working notes. Replies to the author stay reader-oriented; the voice sheet never styles a reply.\n", + "CLAUDE.md": "@AGENTS.md\n", + "START.md": "# Ember Road\n\nStory vault. Start with 00-project/brief.md.\n", + "00-project/brief.md": "---\nkind: story\nstory_id: ember-road\nform: long-form\npublication_profile: web-serial\n---\n# Story brief\n\nA courier guild carries sealed letters across a salt desert between rival cities. Reader promise: slow-burn travel adventure with a guild intrigue underneath.\n", + "00-project/voice.md": "---\nkind: story\nstory_id: ember-road\nvoice_revision: V1\nviewpoint: third person limited\ntense: past\ndistance: close\n---\n# Voice sheet\n\n## Narration contract\n\nViewpoint Sena: third person limited, past tense, close distance. No head-hopping inside a scene.\n\n## Tone palette\n\nBaseline: dry, observant, unhurried.\n\n## Anchor passages\n\n- ch-003, revision r4: the night crossing.\n- ch-008, revision r2: the ledger argument.\n", + "00-project/development.md": "# Development state\n\nCurrent pass: drafting arc-01. Next: chapter 12.\n", + "07-continuity/state.md": "# Current state\n\n- Accepted through ch-011, revision r2.\n- The Ash Court seal is black wax (accepted in ch-004).\n- Oren lost his left hand before the story begins (accepted in ch-002).\n", + "05-plot/chapters/ch-011.md": "---\nid: ch-011\nstory_id: ember-road\ntype: chapter\ncanon_status: accepted\nchapter_number: 11\narc_id: arc-01\npov_id: sena\ndraft_status: accepted\naccepted_revision: r2\nvoice_revision: V1\n---\n# Chapter 11 card\n\nProse file: 06-manuscript/chapters/ch-011.md\n\n## Beats\n\n1. Sena reaches the last well, but the keeper refuses guild water.\n2. Therefore she shows the courier's knot and is let through.\n3. She enters the salt with the sealed letter.\n\n## Acceptance\n\nAccepted by the author at revision r2 after the canon review.\n", + "06-manuscript/index.md": "# Manuscript order\n\n| Chapter | Prose file | Card | Arc | POV | Accepted revision |\n|---|---|---|---|---|---|\n| ch-010 | 06-manuscript/chapters/ch-010.md | 05-plot/chapters/ch-010.md | arc-01 | Sena | r2 |\n| ch-011 | 06-manuscript/chapters/ch-011.md | 05-plot/chapters/ch-011.md | arc-01 | Sena | r2 |\n| ch-012 | 06-manuscript/chapters/ch-012.md | 05-plot/chapters/ch-012.md | arc-01 | Sena | none |\n", + "06-manuscript/chapters/ch-011.md": "Sena reached the last well before the salt at first light. The keeper had already drawn the rope up and coiled it on the stone.\n\n\"Guild water is paid at the gate,\" he said. \"Not here.\"\n\nShe showed him the courier's knot on her sleeve. He looked at it for as long as it took a mule to drink, then let the rope down again.\n\nBy noon the salt had begun to sing under her boots. She kept the sealed letter flat against her ribs and did not look back at the well.\n" + } +} diff --git a/evals/natural/cases/audiobook-video.json b/evals/natural/cases/audiobook-video.json new file mode 100644 index 0000000..017b669 --- /dev/null +++ b/evals/natural/cases/audiobook-video.json @@ -0,0 +1,16 @@ +{ + "id": "audiobook-video", + "prompt": "Mach aus Kapitel 11 ein Hörbuch-Video für YouTube, vorgelesen wie es geschrieben ist.", + "files": { + "AGENTS.md": "# Vault instructions\n\nThis directory is one vault. Its kind is stated in the frontmatter of 00-project/brief.md and 00-project/voice.md; here it is `story`. Only kind story permits manuscript drafting; kind study permits study notes only; kind starter is an unused template.\n\nManuscript files follow 00-project/voice.md and nothing from documentation or didactic writing rules. Other vault notes are plain working notes. Replies to the author stay reader-oriented; the voice sheet never styles a reply.\n", + "CLAUDE.md": "@AGENTS.md\n", + "START.md": "# Ember Road\n\nStory vault. Start with 00-project/brief.md.\n", + "00-project/brief.md": "---\nkind: story\nstory_id: ember-road\nform: long-form\npublication_profile: web-serial\n---\n# Story brief\n\nA courier guild carries sealed letters across a salt desert between rival cities. Reader promise: slow-burn travel adventure with a guild intrigue underneath.\n", + "00-project/voice.md": "---\nkind: story\nstory_id: ember-road\nvoice_revision: V1\nviewpoint: third person limited\ntense: past\ndistance: close\n---\n# Voice sheet\n\n## Narration contract\n\nViewpoint Sena: third person limited, past tense, close distance. No head-hopping inside a scene.\n\n## Tone palette\n\nBaseline: dry, observant, unhurried.\n\n## Anchor passages\n\n- ch-003, revision r4: the night crossing.\n- ch-008, revision r2: the ledger argument.\n", + "00-project/development.md": "# Development state\n\nCurrent pass: drafting arc-01. Next: chapter 12.\n", + "07-continuity/state.md": "# Current state\n\n- Accepted through ch-011, revision r2.\n- The Ash Court seal is black wax (accepted in ch-004).\n- Oren lost his left hand before the story begins (accepted in ch-002).\n", + "05-plot/chapters/ch-011.md": "---\nid: ch-011\nstory_id: ember-road\ntype: chapter\ncanon_status: accepted\nchapter_number: 11\narc_id: arc-01\npov_id: sena\ndraft_status: accepted\naccepted_revision: r2\nvoice_revision: V1\n---\n# Chapter 11 card\n\nProse file: 06-manuscript/chapters/ch-011.md\n\n## Beats\n\n1. Sena reaches the last well, but the keeper refuses guild water.\n2. Therefore she shows the courier's knot and is let through.\n3. She enters the salt with the sealed letter.\n\n## Acceptance\n\nAccepted by the author at revision r2 after the canon review.\n", + "06-manuscript/index.md": "# Manuscript order\n\n| Chapter | Prose file | Card | Arc | POV | Accepted revision |\n|---|---|---|---|---|---|\n| ch-010 | 06-manuscript/chapters/ch-010.md | 05-plot/chapters/ch-010.md | arc-01 | Sena | r2 |\n| ch-011 | 06-manuscript/chapters/ch-011.md | 05-plot/chapters/ch-011.md | arc-01 | Sena | r2 |\n| ch-012 | 06-manuscript/chapters/ch-012.md | 05-plot/chapters/ch-012.md | arc-01 | Sena | none |\n", + "06-manuscript/chapters/ch-011.md": "Sena reached the last well before the salt at first light. The keeper had already drawn the rope up and coiled it on the stone.\n\n\"Guild water is paid at the gate,\" he said. \"Not here.\"\n\nShe showed him the courier's knot on her sleeve. He looked at it for as long as it took a mule to drink, then let the rope down again.\n\nBy noon the salt had begun to sing under her boots. She kept the sealed letter flat against her ribs and did not look back at the well.\n" + } +} diff --git a/evals/natural/cases/chapter-fits-canon.json b/evals/natural/cases/chapter-fits-canon.json new file mode 100644 index 0000000..185a1b4 --- /dev/null +++ b/evals/natural/cases/chapter-fits-canon.json @@ -0,0 +1,17 @@ +{ + "id": "chapter-fits-canon", + "prompt": "Kapitel 12 ist fertig entworfen. Passt das noch zu dem, was vorher passiert ist?", + "files": { + "AGENTS.md": "# Vault instructions\n\nThis directory is one vault. Its kind is stated in the frontmatter of 00-project/brief.md and 00-project/voice.md; here it is `story`. Only kind story permits manuscript drafting; kind study permits study notes only; kind starter is an unused template.\n\nManuscript files follow 00-project/voice.md and nothing from documentation or didactic writing rules. Other vault notes are plain working notes. Replies to the author stay reader-oriented; the voice sheet never styles a reply.\n", + "CLAUDE.md": "@AGENTS.md\n", + "START.md": "# Ember Road\n\nStory vault. Start with 00-project/brief.md.\n", + "00-project/brief.md": "---\nkind: story\nstory_id: ember-road\nform: long-form\npublication_profile: web-serial\n---\n# Story brief\n\nA courier guild carries sealed letters across a salt desert between rival cities. Reader promise: slow-burn travel adventure with a guild intrigue underneath.\n", + "00-project/voice.md": "---\nkind: story\nstory_id: ember-road\nvoice_revision: V1\nviewpoint: third person limited\ntense: past\ndistance: close\n---\n# Voice sheet\n\n## Narration contract\n\nViewpoint Sena: third person limited, past tense, close distance. No head-hopping inside a scene.\n\n## Tone palette\n\nBaseline: dry, observant, unhurried.\n\n## Anchor passages\n\n- ch-003, revision r4: the night crossing.\n- ch-008, revision r2: the ledger argument.\n", + "00-project/development.md": "# Development state\n\nCurrent pass: drafting arc-01. Next: chapter 12.\n", + "01-world/overview.md": "# World overview\n\nThe salt-road courier guild has three ranks. A runner carries unsealed post inside one city. A courier carries sealed letters between cities and may refuse a commission. A warden keeps a gate ledger and settles disputes between couriers and clients.\n", + "07-continuity/state.md": "# Current state\n\n- Accepted through ch-011, revision r2.\n- The Ash Court seal is black wax (accepted in ch-004).\n- Oren lost his left hand before the story begins (accepted in ch-002).\n", + "05-plot/chapters/ch-012.md": "---\nid: ch-012\nstory_id: ember-road\ntype: chapter\ncanon_status: proposed\nchapter_number: 12\narc_id: arc-01\npov_id: sena\ndraft_status: planned\n---\n# Chapter 12 card\n\nProse file: 06-manuscript/chapters/ch-012.md\n\n## Context packet\n\n- Voice sheet revision: V1\n- State: Sena carries the sealed letter for the Ash Court; Oren knows the seal is forged.\n\n## Beats\n\n1. Sena reaches the Brine Gate at dusk, but the gate ledger already lists her as arrived.\n2. Therefore she enters by the tannery stair and meets Oren.\n3. But Oren hands the letter back unopened.\n\n## Canon delta\n\nNone accepted yet.\n", + "06-manuscript/index.md": "# Manuscript order\n\n| Chapter | Prose file | Card | Arc | POV | Accepted revision |\n|---|---|---|---|---|---|\n| ch-010 | 06-manuscript/chapters/ch-010.md | 05-plot/chapters/ch-010.md | arc-01 | Sena | r2 |\n| ch-011 | 06-manuscript/chapters/ch-011.md | 05-plot/chapters/ch-011.md | arc-01 | Sena | r2 |\n| ch-012 | 06-manuscript/chapters/ch-012.md | 05-plot/chapters/ch-012.md | arc-01 | Sena | none |\n", + "06-manuscript/chapters/ch-012.md": "The Brine Gate had closed its ledger before Sena reached it. She read her own name there, entered in a hand that was not hers, and went round by the tannery stair.\n\nOren was waiting where the vats ended. He held out the letter, seal unbroken, and she saw that the wax was the wrong red.\n" + } +} diff --git a/evals/natural/cases/chapters-sound-different.json b/evals/natural/cases/chapters-sound-different.json new file mode 100644 index 0000000..8d4f9f7 --- /dev/null +++ b/evals/natural/cases/chapters-sound-different.json @@ -0,0 +1,18 @@ +{ + "id": "chapters-sound-different", + "prompt": "Die Kapitel 10 bis 12 klingen irgendwie nicht mehr so wie am Anfang. Kannst du das prüfen?", + "files": { + "AGENTS.md": "# Vault instructions\n\nThis directory is one vault. Its kind is stated in the frontmatter of 00-project/brief.md and 00-project/voice.md; here it is `story`. Only kind story permits manuscript drafting; kind study permits study notes only; kind starter is an unused template.\n\nManuscript files follow 00-project/voice.md and nothing from documentation or didactic writing rules. Other vault notes are plain working notes. Replies to the author stay reader-oriented; the voice sheet never styles a reply.\n", + "CLAUDE.md": "@AGENTS.md\n", + "START.md": "# Ember Road\n\nStory vault. Start with 00-project/brief.md.\n", + "00-project/brief.md": "---\nkind: story\nstory_id: ember-road\nform: long-form\npublication_profile: web-serial\n---\n# Story brief\n\nA courier guild carries sealed letters across a salt desert between rival cities. Reader promise: slow-burn travel adventure with a guild intrigue underneath.\n", + "00-project/voice.md": "---\nkind: story\nstory_id: ember-road\nvoice_revision: V1\nviewpoint: third person limited\ntense: past\ndistance: close\n---\n# Voice sheet\n\n## Narration contract\n\nViewpoint Sena: third person limited, past tense, close distance. No head-hopping inside a scene.\n\n## Tone palette\n\nBaseline: dry, observant, unhurried. Permitted excursion: gallows humour under physical danger, returning to baseline by the end of the scene. Excluded: slapstick, narrator sentimentality.\n\n## Language conventions\n\nBritish spelling. Guild ranks are lower-case. No modern idiom in dialogue.\n\n## Anchor passages\n\n- ch-003, revision r4: the night crossing, from \"The salt sang under her boots\" to the scene break.\n- ch-008, revision r2: the ledger argument with Oren, whole scene.\n\n## Deliberate exceptions\n\nch-009 uses present tense for the fever dream.\n\n## Change log\n\n- V1 accepted by the author after the voice trial.\n", + "00-project/development.md": "# Development state\n\nCurrent pass: drafting arc-01. Next: chapter 12.\n", + "01-world/overview.md": "# World overview\n\nThe salt-road courier guild has three ranks. A runner carries unsealed post inside one city. A courier carries sealed letters between cities and may refuse a commission. A warden keeps a gate ledger and settles disputes between couriers and clients.\n", + "07-continuity/state.md": "# Current state\n\n- Accepted through ch-011, revision r2.\n- The Ash Court seal is black wax (accepted in ch-004).\n- Oren lost his left hand before the story begins (accepted in ch-002).\n", + "06-manuscript/index.md": "# Manuscript order\n\n| Chapter | Prose file | Card | Arc | POV | Accepted revision |\n|---|---|---|---|---|---|\n| ch-010 | 06-manuscript/chapters/ch-010.md | 05-plot/chapters/ch-010.md | arc-01 | Sena | r2 |\n| ch-011 | 06-manuscript/chapters/ch-011.md | 05-plot/chapters/ch-011.md | arc-01 | Sena | r2 |\n| ch-012 | 06-manuscript/chapters/ch-012.md | 05-plot/chapters/ch-012.md | arc-01 | Sena | none |\n", + "06-manuscript/chapters/ch-010.md": "Sena counted the water skins twice. Two were short, and nobody on the wall would meet her eye.\n", + "06-manuscript/chapters/ch-011.md": "OMG, thought Sena, this desert is literally the worst. She was super done with salt.\n", + "06-manuscript/chapters/ch-012.md": "The Brine Gate had closed its ledger before Sena reached it. She read her own name there, entered in a hand that was not hers, and went round by the tannery stair.\n\nOren was waiting where the vats ended. He held out the letter, seal unbroken, and she saw that the wax was the wrong red.\n" + } +} diff --git a/evals/natural/cases/concept-question.json b/evals/natural/cases/concept-question.json new file mode 100644 index 0000000..814b732 --- /dev/null +++ b/evals/natural/cases/concept-question.json @@ -0,0 +1,7 @@ +{ + "id": "concept-question", + "prompt": "Ich bin bei Git nicht so fit. Was ist eigentlich der Unterschied zwischen Rebase und Merge?", + "files": { + "README.md": "# Notes\n\nScratch repository.\n" + } +} diff --git a/evals/natural/cases/new-story.json b/evals/natural/cases/new-story.json new file mode 100644 index 0000000..abc533f --- /dev/null +++ b/evals/natural/cases/new-story.json @@ -0,0 +1,7 @@ +{ + "id": "new-story", + "prompt": "Ich möchte eine neue Geschichte anfangen: eine lange Serie über eine Kuriergilde, die eine Salzwüste durchquert.", + "files": { + "README.md": "# Notes\n\nScratch repository.\n" + } +} diff --git a/evals/natural/cases/tutorial-video.json b/evals/natural/cases/tutorial-video.json new file mode 100644 index 0000000..9d3092f --- /dev/null +++ b/evals/natural/cases/tutorial-video.json @@ -0,0 +1,8 @@ +{ + "id": "tutorial-video", + "prompt": "Mach ein Erklärvideo für neue Mitarbeiter, wie man im HR-Portal Urlaub beantragt.", + "files": { + "README.md": "# HR portal docs\n", + "docs/leave.md": "# Request leave\n\n1. Open the HR portal.\n2. Choose Leave > New request.\n3. Pick the dates and submit.\n" + } +} diff --git a/evals/natural/cases/write-chapter.json b/evals/natural/cases/write-chapter.json new file mode 100644 index 0000000..5d8b058 --- /dev/null +++ b/evals/natural/cases/write-chapter.json @@ -0,0 +1,16 @@ +{ + "id": "write-chapter", + "prompt": "Schreib bitte Kapitel 12.", + "files": { + "AGENTS.md": "# Vault instructions\n\nThis directory is one vault. Its kind is stated in the frontmatter of 00-project/brief.md and 00-project/voice.md; here it is `story`. Only kind story permits manuscript drafting; kind study permits study notes only; kind starter is an unused template.\n\nManuscript files follow 00-project/voice.md and nothing from documentation or didactic writing rules. Other vault notes are plain working notes. Replies to the author stay reader-oriented; the voice sheet never styles a reply.\n", + "CLAUDE.md": "@AGENTS.md\n", + "START.md": "# Ember Road\n\nStory vault. Start with 00-project/brief.md.\n", + "00-project/brief.md": "---\nkind: story\nstory_id: ember-road\nform: long-form\npublication_profile: web-serial\n---\n# Story brief\n\nA courier guild carries sealed letters across a salt desert between rival cities. Reader promise: slow-burn travel adventure with a guild intrigue underneath.\n", + "00-project/voice.md": "---\nkind: story\nstory_id: ember-road\nvoice_revision: V1\nviewpoint: third person limited\ntense: past\ndistance: close\n---\n# Voice sheet\n\n## Narration contract\n\nViewpoint Sena: third person limited, past tense, close distance. No head-hopping inside a scene.\n\n## Tone palette\n\nBaseline: dry, observant, unhurried.\n\n## Anchor passages\n\n- ch-003, revision r4: the night crossing.\n- ch-008, revision r2: the ledger argument.\n", + "00-project/development.md": "# Development state\n\nCurrent pass: drafting arc-01. Next: chapter 12.\n", + "01-world/overview.md": "# World overview\n\nThe salt-road courier guild has three ranks. A runner carries unsealed post inside one city. A courier carries sealed letters between cities and may refuse a commission. A warden keeps a gate ledger and settles disputes between couriers and clients.\n", + "07-continuity/state.md": "# Current state\n\n- Accepted through ch-011, revision r2.\n- The Ash Court seal is black wax (accepted in ch-004).\n- Oren lost his left hand before the story begins (accepted in ch-002).\n", + "05-plot/chapters/ch-012.md": "---\nid: ch-012\nstory_id: ember-road\ntype: chapter\ncanon_status: proposed\nchapter_number: 12\narc_id: arc-01\npov_id: sena\ndraft_status: planned\n---\n# Chapter 12 card\n\nProse file: 06-manuscript/chapters/ch-012.md\n\n## Context packet\n\n- Voice sheet revision: V1\n- State: Sena carries the sealed letter for the Ash Court; Oren knows the seal is forged.\n\n## Beats\n\n1. Sena reaches the Brine Gate at dusk, but the gate ledger already lists her as arrived.\n2. Therefore she enters by the tannery stair and meets Oren.\n3. But Oren hands the letter back unopened.\n\n## Canon delta\n\nNone accepted yet.\n", + "06-manuscript/index.md": "# Manuscript order\n\n| Chapter | Prose file | Card | Arc | POV | Accepted revision |\n|---|---|---|---|---|---|\n| ch-010 | 06-manuscript/chapters/ch-010.md | 05-plot/chapters/ch-010.md | arc-01 | Sena | r2 |\n| ch-011 | 06-manuscript/chapters/ch-011.md | 05-plot/chapters/ch-011.md | arc-01 | Sena | r2 |\n| ch-012 | 06-manuscript/chapters/ch-012.md | 05-plot/chapters/ch-012.md | arc-01 | Sena | none |\n" + } +} diff --git a/evals/natural/expected.json b/evals/natural/expected.json new file mode 100644 index 0000000..2b9a57b --- /dev/null +++ b/evals/natural/expected.json @@ -0,0 +1,101 @@ +{ + "admin-guide": { + "any": [ + "documentation-authoring" + ], + "prohibited": [ + "story-drafting", + "story-voice-review", + "story-video-scripting" + ] + }, + "concept-question": { + "any": [ + "explaining-concepts" + ], + "prohibited": [ + "story-drafting", + "story-voice-review", + "story-video-scripting" + ] + }, + "new-story": { + "any": [ + "story-bootstrap" + ], + "prohibited": [ + "explaining-concepts", + "writing-procedures", + "empathic-troubleshooting", + "human-prose-drafting", + "documentation-authoring", + "tutorial-video-scripting" + ] + }, + "write-chapter": { + "any": [ + "story-drafting" + ], + "prohibited": [ + "explaining-concepts", + "writing-procedures", + "empathic-troubleshooting", + "human-prose-drafting", + "documentation-authoring", + "tutorial-video-scripting" + ] + }, + "chapter-fits-canon": { + "any": [ + "story-canon-review" + ], + "prohibited": [ + "explaining-concepts", + "writing-procedures", + "empathic-troubleshooting", + "human-prose-drafting", + "documentation-authoring", + "tutorial-video-scripting" + ] + }, + "chapters-sound-different": { + "any": [ + "story-voice-review" + ], + "prohibited": [ + "explaining-concepts", + "writing-procedures", + "empathic-troubleshooting", + "human-prose-drafting", + "documentation-authoring", + "tutorial-video-scripting" + ] + }, + "audiobook-video": { + "any": [ + "story-video-scripting" + ], + "prohibited": [ + "tutorial-video-scripting", + "story-drafting" + ] + }, + "audio-version-changes": { + "any": [ + "story-video-scripting" + ], + "prohibited": [ + "tutorial-video-scripting" + ] + }, + "tutorial-video": { + "any": [ + "tutorial-video-scripting" + ], + "prohibited": [ + "story-drafting", + "story-voice-review", + "story-video-scripting" + ] + } +} From eac1d9f6946bde5bdd157f024b364ee4d73f8fca Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:14:07 +0100 Subject: [PATCH 2/8] fix(video): correct the production playbook after its review (WM-001) Applies the 22 findings of the independent playbook review: no step rewords a story narration, pace is not adjusted through the pause value, review gate 1 and the provider mode are checked before paid voicing, delivery copies to a new name and keeps the previous file, the correction path and the inputs are stated. The production skill and the template README no longer name the pause value as a pace lever. Co-Authored-By: Claude Fable 5.2 --- .../playbook-review.md | 87 +++++++++ skills/presentation-video-production/SKILL.md | 8 +- templates/presentation-video/README.md | 2 +- templates/presentation-video/playbook.md | 176 +++++++++++++----- tests/test_writing_mode.py | 2 +- 5 files changed, 227 insertions(+), 48 deletions(-) create mode 100644 docs/specs/WM-001-presentation-video/playbook-review.md diff --git a/docs/specs/WM-001-presentation-video/playbook-review.md b/docs/specs/WM-001-presentation-video/playbook-review.md new file mode 100644 index 0000000..f80cf22 --- /dev/null +++ b/docs/specs/WM-001-presentation-video/playbook-review.md @@ -0,0 +1,87 @@ +# PB-WM-VIDEO-001: Review and rehearsal evidence + +All paths are relative to the repository root. +"PB:n" means line n of `templates/presentation-video/playbook.md`. + +## Scope + +- Playbook path and artifact revision/hash: `templates/presentation-video/playbook.md` (template, 100 lines) at git HEAD `e694672a03c87eff6799f4570356ba6959461b42`. The file is tracked and unmodified; it was last changed in commit `2f510e3` (WM-001) and not touched by the story change `2a450b2` (WM-004). +- System/interface revision and environment: the same HEAD for `templates/presentation-video/tools/*`, `templates/presentation-video/README.md`, `templates/presentation-video/review.md`, `skills/presentation-video-production/SKILL.md`, `skills/presentation-video-review/SKILL.md`, `shared/media/speech-synthesis.md`, `shared/policies/playbooks.md`, `docs/specs/WM-001-presentation-video/{spec,plan,tasks,rehearsal}.md`. Review machine: macOS, Python 3.14.6, Node present, `ffmpeg` and `ffprobe` not installed (not needed: nothing was executed). +- Owning requirement/task and acceptance criteria: `docs/specs/WM-001-presentation-video/spec.md:434-438` (sources US-002, US-004, AC-005, AC-007). Criteria: "Delivered as a template that a target project instantiates with its video profile; covers decisions (provider mode, local or separate render), failure paths and verification; reviewed with `playbook-review`; rehearsed once end to end on the sample video". Task: `tasks.md:96` T032b (open). +- Required validation level and branches: one real end-to-end rehearsal on the sample video (`plan.md:197`); evidence in `docs/specs/WM-001-presentation-video/rehearsal.md`. +- Owner and readiness deadline: Writing Method maintainer; "Before the feature is reported complete" (`plan.md:197`). In an instantiated copy: owner ``, "before the first video is delivered from this project" (PB:14, PB:17). + +## Content review + +- Reviewer and independence: Claude (model Fable 5.2), run as a read-only review agent. Independent: the reviewer did not author the playbook, the tools or the rehearsal record and received no author conclusion, only the file list. +- Review date and method: 2026-11-13. Static inspection of the playbook against policy, skills, README, speech guide, spec, plan, tasks, rehearsal record and the source of every tool the playbook names. Executed: `--help` of `voice_azure.py`, `verify_speech.py`, `timeline.py`, `measure.py` (all options the playbook uses exist). `check_script.py --help` was also invoked; that tool has no argument parser, so it ran the check and stopped at once with `video-profile.json not found` (no file written; `git status` unchanged afterwards). No provider call, no render, no write into the repository. Shell scripts and `.mjs` tools were read, not run. +- Verdict at `e694672`: **Changes required** (1 Critical, 14 Important, 7 Minor). All 22 findings were applied to the playbook in the following revision; see the second review below. +- Checks and source evidence: + - Commands, paths and options: every command in the playbook exists and takes the options shown: `tools/profile.py` (and `--get`), `tools/check_script.py`, `tools/voice_azure.py [--check]`, `tools/verify_speech.py [--samples]`, `tools/split_takes.py`, `tools/timeline.py [--caption-offset]`, `tools/measure.py pace|all [--offset]`, `tools/stills.mjs stills [scene]`, `tools/check_review.py`, `tools/render.sh`, `tools/render-remote.sh`, `tools/prepend.sh`. The Python and shell tools carry the executable bit. Expected strings match the source: `0 error(s)` (`check_script.py:204`), `credentials accepted` (`voice_azure.py:192`), `N lines, … new` (`voice_azure.py:245`), `stopped at :` (`voice_azure.py:162`), `(match 0.xx)` (`split_takes.py:155`), `PASSED` (`measure.py:133`), `0 with an unrecognized key term` (`verify_speech.py:133`), `N of N stills inspected without finding` (`check_review.py:50`), `rendered: out/.mp4` (`render.sh:26`), `done in … s: out/.mp4` (`render-remote.sh:47`), `joined without re-encoding` (`prepend.sh:32`), `stream formats differ` (`prepend.sh:36`), `text changed after voicing` (`timeline.py:42`), `line N was not heard` (`split_takes.py:68`). Paths `pronunciation.json`, `public/audio/`, `takes/`, `out/raw.mp4`, `out/measure.json`, `src/theme.ts`, `review.md` exist or are produced as stated. + - Applicability: scope and exclusions are stated; the story case and the correction case are not handled (PR-01, PR-02, PR-14). + - Prerequisites: incomplete (PR-04, PR-07, PR-17). + - Decisions: render decision explicit; provider-mode decision missing (PR-05). + - Actions and evidence: several actions lack command or target (PR-06, PR-07, PR-10, PR-12). + - Authority, resume, rollback: three statements contradict the tools (PR-08, PR-09, PR-10). + - Escalation, completion, freshness: PR-11, PR-13, PR-21; rehearsal see below. + - Secrets: none in the playbook; PB:87-88 forbids including the env file or a key. + - Story question (asked explicitly): **yes, one step tells the operator to reword narration**: P3, PB:54 (PR-01). **No step names `tutorial-video-scripting` explicitly.** Wording is nevertheless routed there by implication: "those are scripting and review" (PB:25), "return to the step that owns them" (PB:62) and "the three video skills" (PB:100), which at authoring time were `tutorial-video-scripting`, `presentation-video-production` and `presentation-video-review`; `story-video-scripting` is not reachable from the playbook (PR-02). + - Not verifiable in this review: the ElevenLabs connector's operations (voice list, status query, fetch of an existing generation), the behaviour of `ssh ` on a real render machine, and any provider behaviour. Statements about them below are marked. + +| Finding ID | Severity | Evidence | Required change | Disposition | +|---|---|---|---|---| +| PR-01 | Critical | PB:54, step P3, failure cell: "Pace out of range: adjust wording or pause, repeat P2 and P3". For a story video this instructs what the method forbids: `skills/presentation-video-production/SKILL.md:138-141` "Never change its wording, tone or order: not to meet the pace range, not to shorten a line, not to ease a pronunciation. Pace is adjusted through an effective provider setting or the pause value only."; `templates/presentation-video/README.md:81-82` "With `story`, the narration is the author's accepted text and no tool or step may reword it." Gate 2 does not repeat the word-for-word comparison of gate 1 (`skills/presentation-video-review/SKILL.md:138-144`), so a reworded line can reach delivery. For a tutorial the production skill allows wording only as "(tutorial scripts only)" (`SKILL.md:95-97`). | In PB:54 replace the failure cell with: "`text changed after voicing` or `sound is missing`: repeat P2. `OUT OF RANGE`: go to R6". Add a recovery row after R5: "\| R6 \| Pace out of range \| `pace … OUT OF RANGE [low, high]` in the output of `tools/measure.py` \| Story video (`\"purpose\": \"story\"` in `video.json`): never change wording, tone or order of a line. Azure: `python3 tools/probe_settings.py rate=` (four short paid requests); keep the setting under `speech.settings` in the profile only if it prints `EFFECTIVE`; then repeat P2 and P3 (a changed setting voices every line again). Tutorial only: a changed wording is produced by `tutorial-video-scripting`, must pass `python3 tools/check_script.py`, then repeat P2 and P3. Story line that needs other words, or no effective setting: stop and return to `story-video-scripting` and the author. \|" | applied | +| PR-02 | Important | No exit for story wording. PB:23-25 "Do not use to write the script or to decide whether a video is acceptable: those are scripting and review."; PB:62 P9 "Findings: return to the step that owns them" (a wording finding has no owning step P1-P11); PB:99-100 "Sources: this project's `README.md`, the method's speech-synthesis guide and the three video skills" (the repository now has four: `tutorial-video-scripting`, `story-video-scripting`, `presentation-video-production`, `presentation-video-review`). Required routing: `skills/presentation-video-review/SKILL.md:145-147` "Wording findings return to `story-video-scripting` and the author … They never go to `tutorial-video-scripting`."; production `SKILL.md:144-147`. The playbook does not mention `purpose`, `story` or `story-video-scripting` anywhere. | (a) PB:24-25: replace "those are scripting and review." with "those belong to `tutorial-video-scripting` (tutorial), to `story-video-scripting` and the author (story video: `\"purpose\": \"story\"` in `video.json`) and to `presentation-video-review`." (b) Add to "Before starting", after the table: "Story video: the narration is voiced as it is. No step of this playbook changes wording, tone or order; a wrong name or term is corrected only in `pronunciation.json` or in the line's `speak`." (c) PB:62 failure cell: "Picture, sound or file finding: return to the step P1 to P8 that owns it. Wording finding, tutorial: stop; `tutorial-video-scripting`; start again at Before starting. Wording finding, story: stop; `story-video-scripting` and the author, never `tutorial-video-scripting`; the changed line arrives from there; start again at Before starting." (d) PB:99-100: replace "the three video skills" with "the skills `presentation-video-production`, `presentation-video-review`, `tutorial-video-scripting` and `story-video-scripting`". | applied | +| PR-03 | Important | PB:54 names "pause" as a remedy for pace. The pace measurement ignores pauses: `tools/measure.py:24-35` computes words per minute from each sound file's `duration` minus edge silence only; the pause is a timeline value (`tools/timeline.py:54`). Changing the pause can never turn `measure.py pace` into `PASSED`. The lever that does work, a probed provider setting (`SKILL.md:90-92`, `tools/probe_settings.py`), is absent from the playbook; that tool is Azure-only (`probe_settings.py:45`). In the rehearsal the pace failure was resolved by neither remedy but by changing the range (`rehearsal.md:21`). For ElevenLabs the sources define no pace lever ("Current models take no pause markup. Pace follows punctuation and line breaks", `shared/media/speech-synthesis.md:85-86`). Upstream note, outside this artifact: `SKILL.md:95-97`, `SKILL.md:140-141` and `README.md:208` also list the pause value. | Remove "or pause" from P3 (covered by the R6 text in PR-01). Add to R6: "ElevenLabs: no pace setting is defined; stop and escalate to . Changing `pace_wpm` in the profile or under `overrides` in `video.json` is an owner decision, recorded in the project `README.md`; it is not a fix the operator chooses." | applied | +| PR-04 | Important | Gate 1 is not a precondition. PB:23 "Use when an accepted script is to become a delivered video"; the Before-starting table (PB:34-41) checks only `python3 tools/check_script.py`. "Accepted" is not observable. The review skill places gate 1 "before money is spent on voicing" (`skills/presentation-video-review/SKILL.md:3`, `:55-75`); for a story video gate 1 is the only word-for-word comparison with the edition (`:121-137`). P2 spends quota or credits (PB:43). | Add a row to the Before-starting table: "\| Review gate 1 \| Open `review.md` \| `Gate 1: passed on `, and \"Script revision reviewed\" names the current script \|" (wording of `templates/presentation-video/review.md:4` and `:27`). | applied | +| PR-05 | Important | The provider-mode decision required by the acceptance criterion ("covers decisions (provider mode, local or separate render)", `spec.md:438`) has no decision row. PB:52-53 list P2a "Per-line mode" and P2b "Per-scene mode" side by side; P1 has no next-step branch. Compare the explicit decision P6 (PB:57). The profile accepts any pairing of `speech.provider` and `speech.mode` (`tools/profile.py:101-104`). | Insert before P2a: "\| P2 \| Author, project \| Decision: `python3 tools/profile.py --get speech.provider` and `python3 tools/profile.py --get speech.mode` \| `azure` and `line`: P2a. `elevenlabs` and `scene`: P2b \| Any other pair: stop; correct the profile; repeat Before starting \|". | applied | +| PR-06 | Important | P1 is not executable as written. PB:51 "`python3 tools/verify_speech.py --samples`, then voice the listed lines and listen": no command for voicing. The only sample-voicing command is Azure-only: `tools/verify_speech.py:99-100` prints "voice them first: python3 tools/voice_azure.py --only …" whatever the provider; `SKILL.md:72-77` also names only `voice_azure.py --only` and adds "run the read-back on them", which P1 omits. No ElevenLabs path exists. "listen" cannot be done by an assistant, yet the audience is "an author or assistant" (PB:12); `rehearsal.md:56` "Listening check by a person \| Not possible for the assistant". | Replace the P1 action with: "`python3 tools/verify_speech.py --samples`; Azure: `python3 tools/voice_azure.py --only `; a person listens to each of these lines (an assistant hands the file names under `public/audio/` to the author and waits); then `python3 tools/verify_speech.py --only `". Expected evidence: "Each key term heard correctly; `0 with an unrecognized key term`". Add the ElevenLabs branch explicitly. Its content is an owner decision because no source defines it; until decided write: "ElevenLabs: no sample voicing exists; go to P2b; P4 is the first key-term check". | applied | +| PR-07 | Important | Hidden facts in P2b and P4. PB:53 "one generation per scene, store as `takes/-take1.mp3`, transcribe, `python3 tools/split_takes.py`": (1) the passage to generate is not named; it must come from `python3 tools/split_takes.py prompts`, which applies respellings and the tone tag (`split_takes.py:38-42`, `:113-116`; `SKILL.md:82-84`); (2) the word file `takes/-take1.words.json` and its format are not named (`split_takes.py:130-132` stops with "is missing; transcribe the take first"); (3) no transcription tool or command is named anywhere. P4 (PB:55) needs a word file beside every sound file or the profile's `transcribe` command (`verify_speech.py:71-77`: "is missing and the profile names no transcribe command"); the example profile has `"transcribe": null` (`video-profile.example.json:18`); Before starting does not check it. The rehearsal used an unnamed "remote speech recognizer" (`rehearsal.md:10`). | (a) Add a Before-starting row: "\| Transcription \| `python3 tools/profile.py --get transcribe` \| Prints a command containing `{audio}`; empty: stop, P4 and P2b cannot run \|". (b) Replace the P2b action with: "`python3 tools/split_takes.py prompts`; for each scene without a stored take, generate the printed passage once through the connector with the profile's voice, wait until it is done, store it at once as `takes/-take1.mp3`; transcribe each take to `takes/-take1.words.json` (a list of `{\"word\", \"start\", \"end\"}`) with ; `python3 tools/split_takes.py`". (c) Declare `` as an instantiation input (PR-15). | applied | +| PR-08 | Important | Resume can duplicate a charged effect. PB:68 "Completed work is recognized by its files; every step may be run again." For P2b a run "again" is a new generation; `shared/media/speech-synthesis.md:74-76` "A generation costs credits. Start it once … A second generation needs the author's decision." R3 (PB:74) authorizes a second generation without that decision: "request one new take as `-take2`", while R2 (PB:73) requires it. R3 has no bound if the second take is rejected too. R2's "Check the status of the existing generation and fetch its result" names no connector operation (not verifiable here). | PB:68: replace with "Completed work is recognized by its files. P2a and P3 to P8 may be run again; P2a requests only lines without a file. P2b is charged per generation: a scene with a file `takes/-take.mp3` is never generated again in a rerun; repeat only the transcription and `python3 tools/split_takes.py`." PB:74 action: replace "request one new take as `-take2`; the better one is chosen" with "after the author agrees (it is charged), request one new take and store it as `-take2`; `python3 tools/split_takes.py ` chooses the better one. Second take rejected too: stop and escalate". PB:73: name the connector's status and fetch operations as an instantiation input ``. | applied | +| PR-09 | Important | Rollback and impact statements contradict the tools. PB:78 "Rollback: nothing is overwritten before P10." and PB:45 "Nothing else is changed." Actual: `tools/render.sh:24` and `tools/render-remote.sh:46` overwrite `out/.mp4`; `tools/prepend.sh:31` (`-y`) overwrites `out/-with-opening.mp4`; `tools/voice_azure.py:175-178` deletes sound files no longer referenced; `tools/split_takes.py:148-151` deletes the scene's previous sound; `tools/timeline.py:134-137` rewrites `src/timeline.json`, `out/captions.vtt`, `out/chapters.txt`; `tools/render-remote.sh:22-23` creates `/` on the render machine and runs `rsync -a --delete` into it, and `:33` runs `npm ci` or `npm install` there. Consequence for R4 (PB:75) in the correction case: "Recognize: No `out/.mp4`, or `out/raw.mp4` left behind" fails, because the earlier render's `out/.mp4` still exists after an interrupted re-render and, for P6a, `raw.mp4` lies on the render machine, not locally. The stale file can then be measured in P8; the duration check tolerates 0.5 s (`tools/measure.py:55`). | PB:78: replace the first sentence with "Rollback: nothing at is changed before P10. Locally P2 replaces the sound of changed lines, P3 and P7 rewrite `src/timeline.json`, `out/captions.vtt` and `out/chapters.txt`, and P6 and P7 overwrite `out/.mp4` and `out/-with-opening.mp4`; when a delivered video is corrected, the delivered copy is the only earlier version." PB:43-45: replace "Nothing else is changed." with "P6a creates and synchronizes the folder `/` on the render machine (files there that are not in the project are deleted) and installs the Node packages there." PB:75: Recognize: "The command ended without its last line (`rendered: …` or `==> done in …`). An `out/.mp4` from an earlier render is not the result." Action: "Delete `out/raw.mp4` and any earlier `out/.mp4` and `out/-with-opening.mp4`, then run P6 again." | applied | +| PR-10 | Important | P10 is a vague action on the only irreversible boundary. PB:63 "Copy the file; compare size and checksum at the destination": no command, no statement which file (`out/.mp4` or `out/-with-opening.mp4`), no rule for `` (PB:45), no stop after repeated mismatch ("Mismatch: copy again"). PB:45 says delivery "overwrites / if it exists", PB:79-80 says "keep the previous delivered file until then" and PB:93-94 "any failed step leaves the previous delivered file untouched": a copy that overwrites and then mismatches has already destroyed the previous file, so both statements are false for a failed P10. Policy: "State any irreversible boundary and the alternative to rollback before the relevant step" (`shared/policies/playbooks.md:77-78`). Not rehearsed (`rehearsal.md:55`). Behaviour when `delivery.location` is `null` (default, `tools/profile.py:44`) is undefined. | Replace the P10 row with: "\| P10 \| Author, \| Copy the file measured in P8 to `/.new` with ; compute on the local file and on the copy \| Both checksums equal \| Not equal: delete the `.new` copy, copy once more; not equal again: stop, escalate; the delivered file is untouched \|" and add "\| P10b \| Author, \| Rename an existing `` to `.previous`, then `.new` to `` \| `` has the checksum from P10 \| Rename fails: rename `.previous` back; stop \|". Declare ``, `` and the rule for `` as instantiation inputs (PR-15). Add a Before-starting row: "\| Delivery location \| `python3 tools/profile.py --get delivery.location` \| The path named in this playbook; empty: stop \|". | applied | +| PR-11 | Important | Step order makes gate 2 unpassable or dishonest. P9 (PB:62) requires "`review.md`: gate 2 passed"; gate 2 includes "Project record. The project `README.md` names chapters with time marks, sources, voice and provider, and what makes the video stale" (`skills/presentation-video-review/SKILL.md:107-108`) and the row "Project record complete" (`templates/presentation-video/review.md:38`). The playbook updates the record only in P11 (PB:64), after P9 and P10. P11's evidence "Record complete" is not observable. | Insert between P8 and P9: "\| P8b \| Author, project \| Update the project `README.md`: title, audience, length, date; chapters with time marks from `out/chapters.txt`; sources with the date read (story video: the edition and the manuscript revisions instead); what is evidenced, approximated and invented; voice, provider and mode; ; what makes the video stale \| All seven items present \| Item missing: complete it before P9 \|". Reduce P11 to the cleanup of PR-12 and the final check "the checksum of `/` equals the local one". | applied | +| PR-12 | Important | Undefined destructive action. PB:64 "remove intermediate files if the profile says so". No file list exists: `delivery.keep_intermediate` occurs only as a default in `tools/profile.py:44` and in `video-profile.example.json:17`; no tool reads it and no document defines "intermediate". The default is `false`, so the default reading is "remove". Removing `public/audio/`, `src/audio-manifest.json` or `takes/` breaks the correction path (US-004: one request) and contradicts `shared/media/speech-synthesis.md:37-39` "A sound file that was accepted is kept and reused" and `:81` "Store every take in the project at once; provider links expire". The removal is not listed under "Authority and impact". | Either delete the clause, or replace it with an exact list that the owner decides, in this form: "if `python3 tools/profile.py --get delivery.keep_intermediate` prints `False`: delete . Never delete `public/audio/`, `src/audio-manifest.json`, `src/timeline.json`, `takes/`, `out/captions.vtt`, `out/chapters.txt`, `review.md`." Add the deletion to "Authority and impact". The list of removable paths is an unresolved fact; it blocks an instantiated copy until named. | applied | +| PR-13 | Important | R5 actions do not match the tools or the rehearsal. PB:76: (1) "Line marked `FAIL` in the output": only file rows print `FAIL` (`tools/measure.py:114`); pace prints `OUT OF RANGE` (`:101`), pauses print "N outside ±" (`:124`); the last line is `FAILED` (`:133`). `measure.py all` includes pace, which R5 does not cover. (2) "Resolution or frame rate: correct the profile or `src/theme.ts`, then P6": `src/theme.ts:15-19` only reads `timeline.picture` and `timeline.fps`; the values reach the picture through `tools/timeline.py:73-74`, so after a profile change `python3 tools/timeline.py` and P5 must run before P6. (3) "Loudness or peak: render again (P6)": the sources give no reason why an unchanged render would measure differently; in the rehearsal the peak failure came from the opening sequence and was resolved by a new profile value, not by rendering again (`rehearsal.md:30`). No bound, so the path can loop. | Replace the R5 row with: "\| R5 \| Delivery measurement failed \| Last line `FAILED`; the failing row shows `FAIL` (file), `OUT OF RANGE` (pace) or `N outside ±` (pauses) \| Pace: R6. Resolution or frame rate: correct `picture` in the profile, `python3 tools/timeline.py`, then P5 and P6. Duration or pauses: `python3 tools/timeline.py`, then P6 and P7. Loudness or peak: P6 once. If `out/.mp4` passes and only the file with the opening fails, the opening sequence is the cause: correct that file, then P7. The same row fails a second time: stop and escalate; do not change a profile limit to pass \|". | applied | +| PR-14 | Important | The correction scenario is in scope but has no path. PB:23-24 "or when a delivered video is corrected"; origin "US-004" (PB:11). No step says where a correction enters, which steps repeat (`SKILL.md:125-129`: steps 5, 7, 8, the stills of that scene, 10 to 12), that the changed line must be accepted again, or that per-scene mode needs a new charged take of the scene (`shared/media/speech-synthesis.md:89`). In per-scene mode a new take stored under the existing name `-take1.mp3` leaves the old `-take1.words.json` in place; `tools/split_takes.py:129-135` then scores and cuts the new sound with the old word timings (derived from the code, not observed). A full `node tools/stills.mjs stills` resets every inspection mark (`tools/stills.mjs:38`, `:49`); only `stills ` keeps the others. | Add under "Decision paths and actions": "Correction of a delivered video: the changed line arrives accepted (tutorial: `tutorial-video-scripting`; story: `story-video-scripting` and the author) and `review.md` records gate 1 for it. Run Before starting, then P2 to P10. P2a requests only the changed line. P2b: after the author agrees, one new take of that scene stored under the next free number `takes/-take.mp3` with its own word file; never overwrite an existing take. P5: `node tools/stills.mjs stills ` and inspect that scene's stills." | applied | +| PR-15 | Important | The instantiation contract is wrong. PB:3-5 "replace every `<...>` with the values of that project's video profile". (1) Several placeholders are not profile values: owner `` (PB:14), machine `` (PB:15-16), `` (PB:18), `` (PB:19), `` (PB:84), `` (PB:45). (2) Others are run-time variables that must stay: ``, `` (PB:52-53), `` (PB:58-60, 75; the profile is shared by all videos of a project, the id is per video), `` and `` (PB:60-61). An instantiator who follows the sentence literally fixes one video id into a project playbook. (3) `` has no stated origin; it is the value `tools/prepend.sh:33` prints as "opening length: … s". No list of inputs exists, although the base template asks for "Inputs and intentional parameters" (`templates/playbooks/playbook.md:19-20`). | PB:3-6: replace with "Template. Copy it to `docs/playbooks/system/PB-WM-VIDEO-001.md` in the target project, fill the instantiation inputs listed under Before starting, leave the run-time variables as written, and remove this paragraph. Review it with `playbook-review` and rehearse it once on the sample video before relying on it." Add to "Before starting" two lists. Instantiation inputs: owner role; authoring machine; render machine and `` (profile `render.host`, or \"none\"); speech provider and mode; `` (profile `delivery.location`); rule for ``; ``; ``; `` (profile `transcribe`); ``; last edited date; evidence link. Run-time variables: `` and `` as printed by the tools; `` = `python3 tools/profile.py --get video.id`; `` = `out/.mp4`, or `out/-with-opening.mp4` after P7; `` = the seconds printed by `tools/prepend.sh` as `opening length`. | applied | +| PR-16 | Minor | P4 (PB:55) has no branch for the second failure kind. `tools/verify_speech.py:133-135` also reports "N below similarity 0.8 (listen to these)" and exits 1 while the expected string `0 with an unrecognized key term` is present; `SKILL.md:99-100` "lines below the similarity threshold are listened to". | Append to the P4 failure cell: "Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat from P2; right: note the line in `review.md`". | applied | +| PR-17 | Minor | Before-starting checks are incomplete or misdescribed. PB:38 expects "Three version lines", but `ffmpeg -version` and `ffprobe -version` print several lines each. Not checked: Python 3.11 or newer and installed Node packages (`README.md:33`, `:41-46`), which P5 and P6b need. PB:41 checks only `node` and `ffmpeg` on the render machine; `tools/render-remote.sh:33` needs `npm`, `:39` needs `python3`, `:23` needs `rsync` on both sides. The script extends `PATH` with `/opt/homebrew/bin:/usr/local/bin` (`:30`); the check does not, so on a Homebrew machine the check may fail where the script works (not verified: no render machine available). | PB:38 expected: "Each command prints its version and exits without error". Add rows: "\| Python \| `python3 --version` \| 3.11 or newer \|" and "\| Node packages \| `npm ls remotion` \| A version, no `missing` \|". PB:41 command: `ssh 'export PATH=/opt/homebrew/bin:/usr/local/bin:$PATH; node --version && npm --version && python3 --version && rsync --version \| head -1 && ffmpeg -version \| head -1'`, expected "Five version lines". | applied | +| PR-18 | Minor | PB:41 contains `\|` inside the code span. It renders as `|` in a GitHub table, but copied from the raw file the command passes a literal `|` to `ffmpeg`. | Move the render-machine command out of the table into a fenced code block below it, with a plain `|`. | applied | +| PR-19 | Minor | P5 (PB:56) "mark it inspected" names neither file nor field. The mark is `"inspected": true` in `stills/stills.json`; a defect goes into `"finding"` (`README.md:150-151`; `tools/check_review.py:30-33`). | Replace "mark it inspected" with "set `\"inspected\": true` for it in `stills/stills.json`, or write the defect into its `\"finding\"`". | applied | +| PR-20 | Minor | P7 (PB:60) mixes a decision and two actions, has no "No" branch, does not name the output `out/-with-opening.mp4` (`tools/prepend.sh:17`), has no path for `opening sequence not found` (`prepend.sh:19`), and its failure cell "`stream formats differ`: accepted once; it is encoded once" is unclear: the script then prints "joined with one encode" (`prepend.sh:51`) and succeeds. | Split into: "\| P7 \| Author \| Decision: `python3 tools/profile.py --get opening` \| Prints a path: P7a. Empty: P8 \| \|" and "\| P7a \| Author, authoring machine \| `tools/prepend.sh out/.mp4`, then `python3 tools/timeline.py --caption-offset ` \| `joined without re-encoding: out/-with-opening.mp4` and `opening length: s` \| `stream formats differ … joined with one encode`: not a failure, continue. `opening sequence not found`: correct `opening` in the profile, repeat P7 \|". | applied | +| PR-21 | Minor | Escalation is partly not observable and partly contradictory. PB:84-86 "Escalate to when the provider rejects the key, when quota is exhausted"; R1 (PB:72) "On a quota or key error fix the access first (`--check`)". The tool distinguishes only `the provider rejected the key` (`tools/voice_azure.py:100`), `still rate limited after N waits` (`:125`) and `provider answered HTTP ` / `provider not reachable` (`:101-103`); no message says "quota". No channel is named. `--check` can end with `note: voice … is not in the provider's list` or `note: the profile lists …` (`:196`, `:202`) and exit 0; no instruction covers it. | PB:72 action: "`still rate limited`: wait, run P2a again. `provider not reachable`: check the network, run P2a again. `the provider rejected the key` or `provider answered HTTP `: run `python3 tools/voice_azure.py --check`; still rejected: escalate." PB:84: "Escalate to through when `--check` still reports a rejected key, when P2a stops with `still rate limited` on three separate runs, …". Before-starting row for Azure, expected: "`credentials accepted` and a line `voice : `; a line starting `note:` is corrected in the profile first". | applied | +| PR-22 | Minor | Smaller gaps. (1) Building the scene pictures is no step and no precondition, although the playbook starts at an accepted script; an unbuilt scene only shows as "PLACEHOLDER" in a still (`src/scenes/index.tsx:10-12`, `:22`) after voicing was paid, while the skill says "Design the picture before paying" (`SKILL.md:63-69`). (2) PB:53 expected "(match 0.9x)" excludes `1.00`, the value the rehearsal observed (`rehearsal.md:31`). (3) PB:76 writes `tools/timeline.py` without `python3`, unlike every other step (it works: the file is executable). | (1) Add a Before-starting row: "\| Scenes \| `python3 tools/timeline.py --draft && node tools/stills.mjs stills`, look at one still per scene \| No still shows `PLACEHOLDER` \|" followed by the note "the draft timeline is replaced in P3". (2) "`: (match 0.90 or higher), N lines`". (3) `python3 tools/timeline.py`. | applied | + +## Rehearsal + +- Status: **stale**, and partial when it was run. A real execution is recorded; it did not cover the whole procedure and predates later tool changes. This review ran nothing and adds no rehearsal evidence. +- Last validated: 2026-11-13 is the date in `docs/specs/WM-001-presentation-video/rehearsal.md:5` for the steps listed below. No date exists for a complete end-to-end run, because none is recorded. Nothing is validated at HEAD `e694672`. +- Validating party: the WM-001 implementer (an assistant session, per `rehearsal.md:56` "Not possible for the assistant"); no name is recorded. +- Artifact/system revisions and environment: "working tree of branch `wm-001-presentation-video` (template tools as committed with this file)" (`rehearsal.md:6-7`); no commit hash is recorded (the file was committed in `2f510e3`). Linux authoring machine, macOS render machine over SSH, Azure AI Speech free tier, ElevenLabs connector, a remote speech recognizer (`rehearsal.md:8-10`). +- Method and applicable execution authority: "real execution, no simulation" on a copy of the template outside the repository (`rehearsal.md:11-12`). Authority is not recorded. +- Tested scenarios and branches, as recorded: + - Before starting: profile, script check, Azure credential check (`:18-19`). The media-tool check, the render-machine check and the ElevenLabs voice list are not recorded. + - P2a full voicing (`:20`). P3 failed first on pace, passed after the tool's range default was changed (`:21`). + - P1 and P4 only for the one key-term line: "`1 lines checked, 0 with an unrecognized key term`" (`:22`). Read-back over all lines and listening are not recorded. + - P5: stills rendered; 2 of 8 inspected (`:23`), later 4 of 8, "the gate stays open" (`:66-68`). `check_review.py` never passed. + - P6b local render, P6a remote render, P7 with matching stream formats, P8 without and with the opening (`:24-30`); P8 with opening failed first on peak and passed after a new profile value. + - One sentence changed, per-line mode: one request (`:26-27`). + - P2b for one scene only (`:31`); no later step was run on per-scene sound. +- Observed results and evidence paths: `docs/specs/WM-001-presentation-video/rehearsal.md` (table lines 16-31). No command logs or measurement files are stored in the repository. +- Untested branches and environmental limits: + - Not run: P9 (gate 2, including the listening check by a person), P10 (delivery copy and checksum), P11 (project record, cleanup), R1, R2, R3, R4, the differing-formats branch of P7, escalation. R1 and R3 are covered only by unit tests with a fake provider (`:50`, `:52`); R2, R4 and P10 by "Procedure only". + - R5 is listed as "Rehearsed" (`:54`), but neither recorded failure was resolved by an action that R5 names: the pace failure arose at P3 and both were resolved by changing a default or profile value (see PR-13). + - After the independent review the tools changed and "The render was not repeated" (`:68-69`). + - No story video (`"purpose": "story"`) was ever run through the playbook; the story feature was added afterwards. + - Changes since the rehearsal: commit `2a450b2` changed `tools/check_script.py` (26 lines, a Before-starting command) and `tools/profile.py` (3 lines). `git diff --stat 2f510e3 HEAD` shows no change to the other tools or to the playbook. Under `shared/policies/playbooks.md:69-71` the evidence for the affected checks is invalid until rechecked. + - The acceptance criterion "rehearsed once end to end on the sample video" (`spec.md:438`) is therefore not met: the run stops before P9. + +## Acceptance and freshness + +- Draft status: drafted. Missing facts are listed in PR-07 (transcription command), PR-10 (copy and checksum commands, file name), PR-12 (list of intermediate files), PR-06 (ElevenLabs sample path), PR-15 (input list). +- Content-review readiness: Changes required at `e694672` (1 Critical, 14 Important, 7 Minor). +- Agreed validation criteria met: no. "Reviewed with `playbook-review`": done by this report, with open findings. "Rehearsed once end to end": not met (P9 to P11 never run; evidence stale). +- Operation/delivery readiness: blocked, by the open Critical and Important findings and by the incomplete rehearsal. Task T032b stays open; `tasks.md:104-105` "Its readiness blocks reporting the feature complete." +- Changes since evidence: the corrections required here change commands, branches and recovery paths, so they are not prose-only; rehearsal evidence cannot be reused for the changed steps. +- Next action and owner: Writing Method maintainer returns PR-01 to PR-22 to `playbook-authoring`; then a new `playbook-review` at the new revision; then a rehearsal that covers at least P9 to P11, one provoked R4, the correction path, and one story video or an explicit owner decision that story videos are out of this playbook's scope. diff --git a/skills/presentation-video-production/SKILL.md b/skills/presentation-video-production/SKILL.md index b6badb4..c0675c8 100644 --- a/skills/presentation-video-production/SKILL.md +++ b/skills/presentation-video-production/SKILL.md @@ -93,8 +93,8 @@ agreed), or a correction to a video that already exists. 7. **Timeline.** `python3 tools/timeline.py`. It refuses sound that no longer matches the text. Then `python3 tools/measure.py pace`: a scene outside the profile's range is adjusted through wording (tutorial scripts - only), an effective provider setting, or the pause value, and measured - again. + only) or an effective provider setting, and measured again. The pause + value changes the gaps between lines, not the measured pace. 8. **Read-back.** `python3 tools/verify_speech.py` over all lines. An unrecognized key term blocks delivery; lines below the similarity threshold are listened to. @@ -137,8 +137,8 @@ accepted manuscript (`"edition": "verbatim"`) or an accepted listening edition - Voice and render the narration as it is. Never change its wording, tone or order: not to meet the pace range, not to shorten a line, not to ease a - pronunciation. Pace is adjusted through an effective provider setting or - the pause value only. + pronunciation. Pace is adjusted through an effective provider setting only; + the pause value changes the gaps between lines, not the measured pace. - A wrong name or term is corrected in `pronunciation.json` or with a line's `speak`, which changes the sound and never the caption text or the words. - A line that the script check rejects, or that needs other words to work diff --git a/templates/presentation-video/README.md b/templates/presentation-video/README.md index 0f81a78..8147355 100644 --- a/templates/presentation-video/README.md +++ b/templates/presentation-video/README.md @@ -205,7 +205,7 @@ Set `"provider": "elevenlabs"` and `"mode": "scene"` in the profile. | `still rate limited after N waits; stopped at :` | Provider limit | Run the same command again later; finished lines are kept | | `the text changed after voicing` | A line was edited after voicing | Voice again, then `tools/timeline.py` | | `src/timeline.json is a draft` | Rendering without sound | Voice the script, then `tools/timeline.py` | -| `pace … OUT OF RANGE` | Scene too fast or slow for the profile | Shorten or loosen the wording, or adjust the pause; measure again | +| `pace … OUT OF RANGE` | Scene too fast or slow for the profile | Tutorial: shorten or loosen the wording. Story video: never the wording. Either: a provider setting that `tools/probe_settings.py` reports as `EFFECTIVE`. The pause value does not change this measurement. Measure again | | `pauses … outside` | Audible pause differs from the timeline | Run `tools/timeline.py` and render again | | `key term not recognized` | The voice mispronounces a key term | Correct it in `pronunciation.json`, voice and verify again | | `line N was not heard in the take` | The take dropped or changed a sentence | Request a new take; the scene's existing sound is untouched | diff --git a/templates/presentation-video/playbook.md b/templates/presentation-video/playbook.md index fd5c1de..9818c3b 100644 --- a/templates/presentation-video/playbook.md +++ b/templates/presentation-video/playbook.md @@ -1,9 +1,22 @@ # PB-WM-VIDEO-001: Produce and deliver a presentation video Template. Copy it to `docs/playbooks/system/PB-WM-VIDEO-001.md` in the target -project, replace every `<...>` with the values of that project's video -profile, and remove this paragraph. Review it with `playbook-review` and -rehearse it once on the sample video before relying on it. +project, fill the instantiation inputs listed under Inputs, leave the run-time +variables as written, and remove this paragraph. Review it with +`playbook-review` and rehearse it once on the sample video before relying on +it. + +## Contents + +- Identity and scope +- When to use / when not to use +- Inputs +- Before starting +- Decision paths and actions +- Interruption, recovery and rollback +- Escalation and preserved evidence +- Completion +- Maintenance and sources ## Identity and scope @@ -11,9 +24,9 @@ rehearse it once on the sample video before relying on it. - Origin: Writing Method, specification WM-001 (US-002, US-004) - Audience and owner: an author or assistant with the video project and a configured speech provider, without knowledge of earlier videos; owner - -- Target: one video project, authoring machine , render machine - , speech provider + +- Target: one video project, authoring machine , render + machine , speech provider - Readiness deadline: before the first video is delivered from this project - Last edited: - Review and rehearsal evidence: @@ -22,11 +35,41 @@ rehearse it once on the sample video before relying on it. Use when an accepted script is to become a delivered video, or when a delivered video is corrected. Do not use to write the script or to decide -whether a video is acceptable: those are scripting and review. +whether a video is acceptable: those belong to `tutorial-video-scripting` +(tutorial), to the story scripting skill and the author (story video: +`"purpose": "story"` in `video.json`) and to `presentation-video-review`. Outcome: a file at whose measurement passed and whose review gate 2 is recorded as passed. +## Inputs + +Instantiation inputs, filled once when the playbook is copied into a project: + +| Input | Source | +|---|---| +| , | The project: who decides and how that person is reached | +| , , | The project; is `render.host` in the profile | +| Speech provider and mode | `speech.provider` and `speech.mode` in the profile | +| | `delivery.location` in the profile | +| | The project's naming rule for a delivered file | +| , | The commands that copy to and compute a checksum there | +| | `transcribe` in the profile | +| | ElevenLabs only: how the connector reports an existing generation and fetches its result | +| , | Last edit, and the review and rehearsal record | + +An input the project cannot name is a blocker for the step that uses it, not +a value to guess. + +Run-time variables, left as written: + +| Variable | Value | +|---|---| +| ``, `` | As printed by the tools | +| `` | `python3 tools/profile.py --get video.id` | +| `` | `out/.mp4`, or `out/-with-opening.mp4` after P7a | +| `` | The seconds printed by `tools/prepend.sh` as `opening length` | + ## Before starting Run from the video project. Each check must succeed before step P1. @@ -34,67 +77,116 @@ Run from the video project. Each check must succeed before step P1. | Check | Command | Expected | |---|---|---| | Profile | `python3 tools/profile.py` | Prints the profile; no error | +| Python | `python3 --version` | 3.11 or newer | | Script | `python3 tools/check_script.py` | `0 error(s)` | -| Media tools | `ffmpeg -version && ffprobe -version && node --version` | Three version lines | -| Provider (Azure) | `python3 tools/voice_azure.py --check` | `credentials accepted` | -| Provider (ElevenLabs) | List voices through the connector | A voice list | -| Render machine, if used | `ssh 'node --version && ffmpeg -version \| head -1'` | Two version lines | +| Review gate 1 | Open `review.md` | `Gate 1: passed on `, and "Script revision reviewed" names the current script | +| Media tools | `ffmpeg -version && ffprobe -version && node --version` | Each command prints its version and exits without error | +| Node packages | `npm ls remotion` | A version, no `missing` | +| Scenes | `python3 tools/timeline.py --draft && node tools/stills.mjs stills`, then look at one still per scene | No still shows `PLACEHOLDER`. The draft timeline is replaced in P3 | +| Transcription | `python3 tools/profile.py --get transcribe` | Prints a command containing `{audio}`. Empty: stop; P4 and P2b cannot run | +| Delivery location | `python3 tools/profile.py --get delivery.location` | The path named in this playbook. Empty: stop | +| Provider (Azure) | `python3 tools/voice_azure.py --check` | `credentials accepted` and a line `voice : `. A line starting `note:` is corrected in the profile first | +| Provider (ElevenLabs) | List voices through the connector | A voice list that contains the profile's voice | +| Render machine, if used | The command below | Five version lines | + +```sh +ssh 'export PATH=/opt/homebrew/bin:/usr/local/bin:$PATH; node --version && npm --version && python3 --version && rsync --version | head -1 && ffmpeg -version | head -1' +``` + +Story video: the narration is voiced as it is. No step of this playbook +changes wording, tone or order; a wrong name or term is corrected only in +`pronunciation.json` or in the line's `speak`. Authority and impact: voicing spends provider quota or credits; rendering -uses the render machine's processor for minutes; delivery overwrites -/ if it exists. Nothing else is changed. +uses the render machine's processor for minutes. P6a creates and synchronizes +the folder `/` on the render machine (files there that +are not in the project are deleted) and installs the Node packages there. +Delivery replaces `/` in P10b; that is the only +step that changes a delivered file. ## Decision paths and actions | Step | Actor and target | Condition / action | Expected evidence | Failure / next step | |---|---|---|---|---| -| P1 | Author, project | `python3 tools/verify_speech.py --samples`, then voice the listed lines and listen | Each key term spoken correctly | Wrong term: correct `pronunciation.json`, repeat P1 | -| P2a | Author, Azure | Per-line mode: `python3 tools/voice_azure.py` | `N lines, … new` | Stops with `stopped at :`: go to R1 | -| P2b | Author, connector | Per-scene mode: one generation per scene, store as `takes/-take1.mp3`, transcribe, `python3 tools/split_takes.py` | `: … (match 0.9x)` per scene | Generation error or timeout: go to R2. Take rejected: go to R3 | -| P3 | Author, project | `python3 tools/timeline.py && python3 tools/measure.py pace` | `PASSED` | `text changed after voicing`: repeat P2. Pace out of range: adjust wording or pause, repeat P2 and P3 | -| P4 | Author, project | `python3 tools/verify_speech.py` | `0 with an unrecognized key term` | Unrecognized key term: correct `pronunciation.json`, repeat from P2 | -| P5 | Author, project | `node tools/stills.mjs stills`; look at every still; mark it inspected; `python3 tools/check_review.py` | `N of N stills inspected without finding` | Defect: correct the scene, `node tools/stills.mjs stills `, repeat P5 for it | +| P1 | Author, project | `python3 tools/verify_speech.py --samples`. Azure: `python3 tools/voice_azure.py --only `; a person listens to each of these lines (an assistant hands the file names under `public/audio/` to the author and waits); then `python3 tools/verify_speech.py --only `. ElevenLabs: no sample voicing exists; go to P2; P4 is the first key-term check | Each key term heard correctly; `0 with an unrecognized key term` | Wrong term: correct `pronunciation.json`, repeat P1 | +| P2 | Author, project | Decision: `python3 tools/profile.py --get speech.provider` and `python3 tools/profile.py --get speech.mode` | `azure` and `line`: P2a. `elevenlabs` and `scene`: P2b | Any other pair: stop; correct the profile; repeat Before starting | +| P2a | Author, Azure | `python3 tools/voice_azure.py` | `N lines, … new` | Stops with `stopped at :`: go to R1 | +| P2b | Author, connector | `python3 tools/split_takes.py prompts`; for each scene without a stored take, generate the printed passage once through the connector with the profile's voice, wait until it is done, and store it at once as `takes/-take1.mp3`; transcribe each take to `takes/-take1.words.json` (a list of `{"word", "start", "end"}`) with ; `python3 tools/split_takes.py` | `: (match 0.90 or higher), N lines` per scene | Generation error or timeout: go to R2. Take rejected: go to R3 | +| P3 | Author, project | `python3 tools/timeline.py && python3 tools/measure.py pace` | `PASSED` | `text changed after voicing` or missing sound: repeat P2. `OUT OF RANGE`: go to R6 | +| P4 | Author, project | `python3 tools/verify_speech.py` | `0 with an unrecognized key term` | Unrecognized key term: correct `pronunciation.json`, repeat from P2. Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat from P2; right: note the line in `review.md` | +| P5 | Author, project | `node tools/stills.mjs stills`; look at every still; set `"inspected": true` for it in `stills/stills.json`, or write the defect into its `"finding"`; `python3 tools/check_review.py` | `N of N stills inspected without finding` | Defect: correct the scene, `node tools/stills.mjs stills `, repeat P5 for it | | P6 | Author | Decision: render machine named in the profile and reachable? | Yes: P6a. No: P6b | | | P6a | Author, render machine | `tools/render-remote.sh` | `done in … s: out/.mp4` | Interrupted or unreachable: go to R4 | | P6b | Author, authoring machine | `tools/render.sh` | `rendered: out/.mp4` | Interrupted: go to R4 | -| P7 | Author | Decision: opening sequence in the profile? Yes: `tools/prepend.sh out/.mp4`, then `python3 tools/timeline.py --caption-offset ` | `joined without re-encoding` | `stream formats differ`: accepted once; it is encoded once | +| P7 | Author | Decision: `python3 tools/profile.py --get opening` | Prints a path: P7a. Empty: P8 | | +| P7a | Author, authoring machine | `tools/prepend.sh out/.mp4`, then `python3 tools/timeline.py --caption-offset ` | `joined without re-encoding: out/-with-opening.mp4` and `opening length: s` | `stream formats differ` followed by `joined with one encode`: not a failure, continue. `opening sequence not found`: correct `opening` in the profile, repeat P7 | | P8 | Author, file | `python3 tools/measure.py all [--offset ]` | `PASSED` | Go to R5 | -| P9 | Reviewer | Gate 2 of `presentation-video-review` | `review.md`: gate 2 passed | Findings: return to the step that owns them | -| P10 | Author, | Copy the file; compare size and checksum at the destination | Sizes and checksums equal | Mismatch: copy again; do not delete the local file | -| P11 | Author, project | Update the project `README.md`; remove intermediate files if the profile says so | Record complete | | +| P8b | Author, project | Update the project `README.md`: title, audience, length and date; chapters with time marks from `out/chapters.txt`; sources with the date read (story video: the edition and the manuscript revisions instead); what is evidenced, approximated and invented; voice, provider and mode; ; what makes the video stale | All seven items present | Item missing: complete it before P9 | +| P9 | Reviewer | Gate 2 of `presentation-video-review` | `review.md`: `Gate 2: passed on ` | Picture, sound or file finding: return to the step P1 to P8 that owns it. Wording finding, tutorial: stop; `tutorial-video-scripting`; start again at Before starting. Wording finding, story: stop; the story scripting skill and the author, never `tutorial-video-scripting`; the changed line arrives from there; start again at Before starting | +| P10 | Author, | Copy the file measured in P8 to `/.new` with ; compute on the local file and on the copy | Both checksums equal | Not equal: delete the `.new` copy and copy once more. Not equal again: stop and escalate; the delivered file is untouched | +| P10b | Author, | Rename an existing `` to `.previous`, then `.new` to `` | `` has the checksum from P10 | Rename fails: rename `.previous` back; stop | +| P11 | Author, | Compute on `/` | Equals the local checksum | Not equal: restore `.previous`; stop and escalate | + +No step deletes working files. `public/audio/`, `src/audio-manifest.json`, +`src/timeline.json`, `takes/`, `out/captions.vtt`, `out/chapters.txt` and +`review.md` are kept: a later correction reuses them. + +### Correction of a delivered video + +The changed line arrives accepted (tutorial: `tutorial-video-scripting`; +story: the story scripting skill and the author), and `review.md` records gate 1 +for it. Run Before starting, then P2 to P11. + +- P2a requests only the changed line. +- P2b: after the author agrees, one new take of that scene, stored under the + next free number `takes/-take.mp3` with its own word file. Never + overwrite an existing take: its word file would no longer match the sound. +- P5: `node tools/stills.mjs stills ` and inspect that scene's stills. + A run without a scene resets every inspection mark. ## Interruption, recovery and rollback -Completed work is recognized by its files; every step may be run again. +Completed work is recognized by its files. P2a and P3 to P8 may be run again; +P2a requests only lines without a file. P2b is charged per generation: a +scene with a file `takes/-take.mp3` is never generated again in a +rerun; repeat only the transcription and `python3 tools/split_takes.py`. | ID | Situation | Recognize | Action | |---|---|---|---| -| R1 | Voicing stopped (rate limit, network, quota) | Message names the scene and line reached; finished files are in `public/audio/` | Wait for the limit to reset, run P2a again: finished lines are not requested again. On a quota or key error fix the access first (`--check`) | -| R2 | Connector generation failed or timed out | No file stored for the scene | Check the status of the existing generation and fetch its result. Start a new generation only after the author agrees: each one is charged | -| R3 | Take rejected | `match` below the profile's threshold, or `line N was not heard` | The scene's existing sound is untouched. Listen to the take; request one new take as `-take2`; the better one is chosen. Do not lower the threshold to pass | -| R4 | Render interrupted | No `out/.mp4`, or `out/raw.mp4` left behind | Delete `out/raw.mp4`, run P6 again. No partial file is a result | -| R5 | Delivery measurement failed | Line marked `FAIL` in the output | Loudness or peak: render again (P6). Duration or pauses: `tools/timeline.py`, then P6. Resolution or frame rate: correct the profile or `src/theme.ts`, then P6 | - -Rollback: nothing is overwritten before P10. A delivered file is replaced -only by a file that passed P8 and P9; keep the previous delivered file until -then. +| R1 | Voicing stopped | Message names the scene and line reached; finished files are in `public/audio/` | `still rate limited`: wait, run P2a again; finished lines are not requested again. `provider not reachable`: check the network, run P2a again. `the provider rejected the key` or `provider answered HTTP `: run `python3 tools/voice_azure.py --check`; still rejected: escalate | +| R2 | Connector generation failed or timed out | No file stored for the scene | Use to find the existing generation and fetch its result. Start a new generation only after the author agrees: each one is charged | +| R3 | Take rejected | `match` below the profile's threshold, or `line N was not heard` | The scene's existing sound is untouched. Listen to the take. After the author agrees (it is charged), request one new take and store it as `-take2`; `python3 tools/split_takes.py ` chooses the better one. Second take rejected too: stop and escalate. Do not lower the threshold to pass | +| R4 | Render interrupted | The command ended without its last line (`rendered: …` or `done in …`). An `out/.mp4` from an earlier render is not the result | Delete `out/raw.mp4` and any earlier `out/.mp4` and `out/-with-opening.mp4`, then run P6 again. No partial file is a result | +| R5 | Delivery measurement failed | Last line `FAILED`; the failing row shows `FAIL` (file), `OUT OF RANGE` (pace) or `N outside ±` (pauses) | Pace: R6. Resolution or frame rate: correct `picture` in the profile, `python3 tools/timeline.py`, then P5 and P6. Duration or pauses: `python3 tools/timeline.py`, then P6 and P7. Loudness or peak: P6 once. If `out/.mp4` passes and only the file with the opening fails, the opening sequence is the cause: correct that file, then P7. The same row fails a second time: stop and escalate; do not change a profile limit to pass | +| R6 | Pace out of range | `pace … OUT OF RANGE [low, high]` in the output of `tools/measure.py` | Story video: never change wording, tone or order of a line. Azure: `python3 tools/probe_settings.py rate=` (short paid requests); keep the setting under `speech.settings` in the profile only if it prints `EFFECTIVE`; then repeat P2 and P3 (a changed setting voices every line again). ElevenLabs: no pace setting is defined; stop and escalate. Tutorial only: a changed wording is produced by `tutorial-video-scripting`, must pass `python3 tools/check_script.py`, then repeat P2 and P3. Story line that needs other words, or no effective setting: stop and return to the story scripting skill and the author. The pause value does not change the pace measurement. Changing `pace_wpm` in the profile or under `overrides` in `video.json` is an owner decision recorded in the project `README.md`, not a fix the operator chooses | + +Rollback: nothing at is changed before P10b. Locally P2 +replaces the sound of changed lines, P3 and P7a rewrite `src/timeline.json`, +`out/captions.vtt` and `out/chapters.txt`, and P6 and P7a overwrite +`out/.mp4` and `out/-with-opening.mp4`. After P10b the previous +delivered file is `.previous`; keep it until P11 has passed. ## Escalation and preserved evidence -Escalate to when the provider rejects the key, when quota is -exhausted, or when a key term stays wrong after a phonetic entry, a language -mark and a second voice. Preserve: the command output of the failed step, -`out/measure.json`, the rejected take and its word file. Never include the -env file or a key. +Escalate to through when `--check` still reports a +rejected key, when P2a stops with `still rate limited` on three separate +runs, when a second take is rejected, when a copy mismatches twice, when the +same measurement row fails twice, or when a key term stays wrong after a +phonetic entry, a language mark and a second voice. Preserve: the command +output of the failed step, `out/measure.json`, the rejected take and its word +file. Never include the env file or a key. ## Completion -Success: the file at matches the local file, `review.md` -records gate 2 as passed, and the project record is updated. Stop state: any -failed step leaves the previous delivered file untouched. +Success: the file at has the checksum of the local file, +`review.md` records gate 2 as passed, and the project record is complete. +Stop state: a step that fails before P10b leaves the delivered file +untouched; a failure in P10b or P11 is undone by restoring `.previous`. ## Maintenance and sources Reassess when the provider, the voice, the render machine, the delivery location or the tools change. Sources: this project's `README.md`, the -method's speech-synthesis guide and the three video skills. +method's speech-synthesis guide and the skills `presentation-video-production`, +`presentation-video-review` and `tutorial-video-scripting`, and the story +scripting skill for a story video. diff --git a/tests/test_writing_mode.py b/tests/test_writing_mode.py index ec6d21d..f894ba7 100644 --- a/tests/test_writing_mode.py +++ b/tests/test_writing_mode.py @@ -586,7 +586,7 @@ def test_form_skills_state_the_story_script_rule(self): self.assertIn(FORM_NO_RUN_STATE, preamble) production = read("skills/presentation-video-production/SKILL.md") self.assertIn(FORM_TUTORIAL_ONLY, squash(production)) - self.assertIn("Pace is adjusted through an effective provider setting or the pause value only.", + self.assertIn("Pace is adjusted through an effective provider setting only;", squash(production.split("\n## Story scripts\n", 1)[1].split("\n## ", 1)[0])) def test_tutorial_listening_style_is_set_aside_for_story_narration(self): From 02ff27f9c63146b9a177f7bf6abd21406ac777a6 Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:20:56 +0100 Subject: [PATCH 3/8] fix(video): apply the second playbook review (WM-001) Co-Authored-By: Claude Fable 5.2 --- .../playbook-review.md | 103 ++++++++++++++++++ docs/specs/WM-001-presentation-video/tasks.md | 2 +- evals/natural/expected.json | 3 +- templates/presentation-video/playbook.md | 44 +++++--- 4 files changed, 134 insertions(+), 18 deletions(-) diff --git a/docs/specs/WM-001-presentation-video/playbook-review.md b/docs/specs/WM-001-presentation-video/playbook-review.md index f80cf22..c5c19e2 100644 --- a/docs/specs/WM-001-presentation-video/playbook-review.md +++ b/docs/specs/WM-001-presentation-video/playbook-review.md @@ -85,3 +85,106 @@ All paths are relative to the repository root. - Operation/delivery readiness: blocked, by the open Critical and Important findings and by the incomplete rehearsal. Task T032b stays open; `tasks.md:104-105` "Its readiness blocks reporting the feature complete." - Changes since evidence: the corrections required here change commands, branches and recovery paths, so they are not prose-only; rehearsal evidence cannot be reused for the changed steps. - Next action and owner: Writing Method maintainer returns PR-01 to PR-22 to `playbook-authoring`; then a new `playbook-review` at the new revision; then a rehearsal that covers at least P9 to P11, one provoked R4, the correction path, and one story video or an explicit owner decision that story videos are out of this playbook's scope. + + +## Second review at `eac1d9f` + +All paths are relative to the repository root ``. +"PB:n" means line n of `templates/presentation-video/playbook.md` at the revision below. +"tools/" means `templates/presentation-video/tools/`. + +### Scope + +- Playbook path and artifact revision/hash: `templates/presentation-video/playbook.md` (template, 192 lines) at git HEAD `eac1d9f6946bde5bdd157f024b364ee4d73f8fca` ("fix(video): correct the production playbook after its review (WM-001)"). Working tree clean before and after the review. +- System/interface revision and environment: the same HEAD for `templates/presentation-video/tools/*`, `templates/presentation-video/src/*`, `templates/presentation-video/review.md`, `video-profile.example.json`, `skills/presentation-video-production/SKILL.md`, `skills/presentation-video-review/SKILL.md`, `shared/policies/playbooks.md`, `docs/specs/WM-001-presentation-video/{rehearsal,playbook-review}.md`. Review machine: macOS, Python 3 present. No video project, provider, connector or render machine was available or used. +- Owning requirement/task and acceptance criteria: `docs/specs/WM-001-presentation-video/spec.md:438` (US-002, US-004, AC-005, AC-007): template instantiated with the video profile; covers decisions (provider mode, local or separate render), failure paths and verification; reviewed with `playbook-review`; rehearsed once end to end on the sample video. Task `tasks.md:96` T032b (unchecked). +- Required validation level and branches: one real end-to-end rehearsal on the sample video (`plan.md:197`). +- Owner and readiness deadline: Writing Method maintainer; before the feature is reported complete (`plan.md:197`). + +### Content review + +- Reviewer and independence: Claude (model Fable 5.2), run as a read-only review agent. Independent: not the author of the playbook, the tools, the first review or the rehearsal record. +- Review date and method: 2026-11-13. Static inspection. Every command, option, path and expected string in the playbook was compared with the source of the tool it names (all files under `tools/`, plus `src/Video.tsx`, `src/scenes/index.tsx`, `review.md`, `video-profile.example.json`). Executed: `--help` only, of `timeline.py`, `verify_speech.py`, `voice_azure.py`, `probe_settings.py`, `measure.py` (with `PYTHONDONTWRITEBYTECODE=1`; `git status` unchanged). `check_script.py` and `profile.py` were read, not run. No provider call, no render, no write into the repository. +- Verdict at `eac1d9f`: **Changes required** (0 Critical, 5 Important, 9 Minor new findings). All 22 earlier findings are resolved as written. +- Checks and source evidence: + - `python3 tools/profile.py --get ` (`tools/profile.py:165-170`): `video.id` works because the lookup runs on `{**result, "video": video}` (`:167`); `speech.provider` and `speech.mode` print the bare value (`mode` defaults to `line`, `:31`); `transcribe`, `delivery.location` and `opening` print an empty line when the value is `null` (`:170` `print("" if value is None else value)`); `opening` prints an absolute path, resolved against the profile's folder (`:168-169`), also when the file does not exist (P7a covers that case). The playbook's expectations at PB:69, PB:86, PB:87, PB:112, PB:121 are correct. + - Options exist: `timeline.py --draft`, `--caption-offset` (`tools/timeline.py:115-118`); `split_takes.py prompts [scene ...]` and `split_takes.py [scene ...]` (`tools/split_takes.py:113-121`); `verify_speech.py --samples`, `--only` (`tools/verify_speech.py:82-83`); `voice_azure.py --check`, `--only` (`tools/voice_azure.py:220-221`); `probe_settings.py rate=` (`tools/probe_settings.py:37`, only `rate` is applied, `tools/voice_azure.py:80-82`); `measure.py pace`, `measure.py all --offset` (`tools/measure.py:89-91`); `node tools/stills.mjs stills [scene]` (`tools/stills.mjs:11`); `check_review.py` default folder `stills` (`tools/check_review.py:40`). + - Expected strings match the source: `0 error(s)` (`check_script.py:204`); `credentials accepted`, `voice : `, `note:` (`voice_azure.py:192, 199, 196, 202`); `N lines, … new` (`voice_azure.py:245`); `stopped at :` (`:162`); R1: `still rate limited` (`:125`), `provider not reachable` (`:103`), `the provider rejected the key` (`:100`), `provider answered HTTP ` (`:101`); `: (match 0.xx), N lines` (`split_takes.py:155`); R3: `line N was not heard` (`:68`), threshold message (`:101-102`); `PASSED`/`FAILED`, `FAIL`, `OUT OF RANGE [low, high]`, `N outside ±` (`measure.py:133, 114, 101, 124`); `text changed after voicing` (`timeline.py:42`); `0 with an unrecognized key term`, `below similarity` (`verify_speech.py:133-134`); `N of N stills inspected without finding` (`check_review.py:50`); R4: `rendered: out/.mp4` (`render.sh:26`), `done in … s: out/.mp4` (`render-remote.sh:47`); P7a: `joined without re-encoding: …`, `opening length: … s`, `stream formats differ`, `joined with one encode`, `opening sequence not found` (`prepend.sh:32, 33, 36, 51, 19`); R6: `EFFECTIVE` (`probe_settings.py:74`, see PS-06); `PLACEHOLDER` (`src/scenes/index.tsx:22`); `Gate 1: … on `, `Gate 2: … on `, "Script revision reviewed" (`templates/presentation-video/review.md:27, 40, 4`). + - Step references: P1, P2, P2a, P2b, P3 to P5, P6, P6a, P6b, P7, P7a, P8, P8b, P9, P10, P10b, P11 and R1 to R6 all exist; every "go to" target exists. + - Decisions: P2, P6 and P7 are explicit rows with both branches. Dead or wrong branches: PS-02, PS-04, PS-07, PS-08. + - Skills: R6, the story note (PB:96-98), P9 and the correction section agree with the "Story scripts" sections of `skills/presentation-video-production/SKILL.md:131-150` and `skills/presentation-video-review/SKILL.md:113-152` (never reword; pace through an effective setting only; wording returns to the story family, never to `tutorial-video-scripting`; record names edition and revisions). Step order agrees with the production workflow (`SKILL.md:62-123`) and "One sentence changed" (`:125-129`). One inconsistency with the playbook's own gate rule: PS-05. + - Secrets: none; PB:177 forbids including the env file or a key. + - Not verifiable here: the ElevenLabs connector's operations, `ssh ` on a real render machine, any provider behaviour. PS-01 to PS-03 are derived from the code, not observed. + +#### Part 1: findings of the first review (PR-01 to PR-22) against the current text + +| Finding | Verdict | Current text (quoted) | +|---|---|---| +| PR-01 | resolved | PB:115 "`text changed after voicing` or missing sound: repeat P2. `OUT OF RANGE`: go to R6"; PB:161 R6 "Story video: never change wording, tone or order of a line. Azure: `python3 tools/probe_settings.py rate=` … Tutorial only: … Story line that needs other words, or no effective setting: stop and return to the story scripting skill and the author." | +| PR-02 | resolved | (a) PB:38-40 "those belong to `tutorial-video-scripting` (tutorial), to the story scripting skill and the author (story video: `\"purpose\": \"story\"` in `video.json`) and to `presentation-video-review`"; (b) PB:96-98 "Story video: the narration is voiced as it is. …"; (c) PB:125 "Wording finding, story: stop; the story scripting skill and the author, never `tutorial-video-scripting`; …"; (d) PB:190-192. Plain words for the story skill are the stated constraint. | +| PR-03 | resolved | PB:161 "ElevenLabs: no pace setting is defined; stop and escalate. … The pause value does not change the pace measurement. Changing `pace_wpm` … is an owner decision recorded in the project `README.md`, not a fix the operator chooses" | +| PR-04 | resolved | PB:82 "Review gate 1 \| Open `review.md` \| `Gate 1: passed on `, and \"Script revision reviewed\" names the current script" | +| PR-05 | resolved | PB:112 "P2 … Decision: `python3 tools/profile.py --get speech.provider` and `… --get speech.mode` \| `azure` and `line`: P2a. `elevenlabs` and `scene`: P2b \| Any other pair: stop; correct the profile; repeat Before starting" | +| PR-06 | resolved | PB:111 "Azure: `python3 tools/voice_azure.py --only `; a person listens … then `python3 tools/verify_speech.py --only `. ElevenLabs: no sample voicing exists; go to P2; P4 is the first key-term check". "go to P2" instead of "P2b" is correct now that P2 is the decision. Residual: PS-08. | +| PR-07 | resolved | PB:86 Transcription row; PB:114 "`python3 tools/split_takes.py prompts`; for each scene without a stored take, generate the printed passage once … transcribe each take to `takes/-take1.words.json` (a list of `{\"word\", \"start\", \"end\"}`) with "; PB:57 input declared | +| PR-08 | resolved | PB:149-152 "P2b is charged per generation: a scene with a file `takes/-take.mp3` is never generated again in a rerun"; PB:158 "After the author agrees (it is charged) … Second take rejected too: stop and escalate"; PB:157 and PB:58 ``. Residual: PS-09. | +| PR-09 | resolved | PB:100-105 impact text; PB:159 R4 "The command ended without its last line (`rendered: …` or `done in …`). An `out/.mp4` from an earlier render is not the result … Delete `out/raw.mp4` and any earlier `out/.mp4` and `out/-with-opening.mp4`"; PB:163-167 rollback. The sentence "the delivered copy is the only earlier version" was replaced by the `.previous` rule, which is equivalent. Residual: PS-01 (Before starting also rewrites the timeline), PS-12. | +| PR-10 | resolved | PB:126 P10, PB:127 P10b as required; PB:87 Delivery location row; PB:55-56 inputs | +| PR-11 | resolved | PB:124 P8b before P9 with seven items; PB:128 P11 reduced to the final checksum | +| PR-12 | resolved | The clause is deleted (first option of the required change). PB:130-132 "No step deletes working files. `public/audio/`, `src/audio-manifest.json`, `src/timeline.json`, `takes/`, `out/captions.vtt`, `out/chapters.txt` and `review.md` are kept" | +| PR-13 | resolved | PB:160 R5 as required, including the bound "The same row fails a second time: stop and escalate; do not change a profile limit to pass". Residual: PS-14. | +| PR-14 | resolved | PB:134-145 "Correction of a delivered video … Run Before starting, then P2 to P11" with the three bullets. The text was applied; two new defects sit in this path: PS-01, PS-03. | +| PR-15 | resolved | PB:3-7 preamble; PB:47-71 instantiation inputs and run-time variables. Residual: PS-10. | +| PR-16 | resolved | PB:116 "Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat from P2; right: note the line in `review.md`" | +| PR-17 | resolved | PB:80 Python row; PB:83 "Each command prints its version and exits without error"; PB:84 Node packages row; PB:90-94 "Five version lines" with the extended `PATH` | +| PR-18 | resolved | PB:92-94 fenced block with a plain `|`; no escaped pipe remains in the file | +| PR-19 | resolved | PB:117 "set `\"inspected\": true` for it in `stills/stills.json`, or write the defect into its `\"finding\"`" | +| PR-20 | resolved | PB:121 P7 decision ("Prints a path: P7a. Empty: P8"); PB:122 P7a with both messages and the `opening sequence not found` path | +| PR-21 | resolved | PB:156 R1 with the four tool messages; PB:171-172 "Escalate to through when `--check` still reports a rejected key, when P2a stops with `still rate limited` on three separate runs"; PB:88 `note:` rule. Residual: PS-04, PS-11. | +| PR-22 | resolved | (1) PB:85 Scenes row; (2) PB:114 "(match 0.90 or higher)"; (3) PB:160 `python3 tools/timeline.py`. The new Scenes row introduces PS-01. | + +#### Part 2: new findings at `eac1d9f` + +| Finding ID | Severity | Evidence | Required change | Disposition | +|---|---|---|---|---| +| PS-01 | Important | The Scenes precheck destroys the inspection evidence and contradicts the correction path. PB:85 "`python3 tools/timeline.py --draft && node tools/stills.mjs stills`" runs on every pass through Before starting, also in a correction (PB:138 "Run Before starting, then P2 to P11"). A run without a scene re-renders every still into `stills/` with `inspected: false` and drops all earlier marks (`tools/stills.mjs:38`, `:44-49`); the playbook says so itself (PB:145 "A run without a scene resets every inspection mark"). The correction then tells the operator to render and inspect only one scene (PB:144), so `python3 tools/check_review.py` reports every other still as "not inspected" (`tools/check_review.py:30-31`) and P5's evidence "`N of N stills inspected without finding`" cannot be reached as written. If the operator marks the other stills anyway, they are the images rendered from the draft timeline (estimated lengths, `tools/timeline.py:77-83`), not from the voiced one; `check_review.py` tests the draft flag only on the timeline at check time (`:22`), so it passes. Also unstated: the row overwrites the voiced `src/timeline.json`, `out/captions.vtt` and `out/chapters.txt` (`tools/timeline.py:134-137`); PB:164 names only P3 and P7a. Derived from the code, not observed. | PB:85: replace the command with "`python3 tools/timeline.py --draft && node tools/stills.mjs out/draft-stills`, then look at one still per scene in `out/draft-stills/`" and the expected cell with "No still shows `PLACEHOLDER`. `stills/` and `stills/stills.json` are not touched. `src/timeline.json` is now a draft; it is replaced in P3". PB:164: replace "P3 and P7a rewrite" with "the Scenes check, P3 and P7a rewrite". (`out` is excluded from the copy to the render machine, `tools/render-remote.sh:23`.) | applied | +| PS-02 | Important | A wrong key term found in P4 has no working path in per-scene mode. PB:111 makes P4 "the first key-term check" for ElevenLabs. PB:116 "Unrecognized key term: correct `pronunciation.json`, repeat from P2". P2b generates only "for each scene without a stored take" (PB:114) and PB:150-152 forbids a new generation in a rerun, so the repeat cuts the same stored take again (`tools/split_takes.py:124-135`): the sound is unchanged and P4 fails again, or the operator generates a charged take without the author's agreement that R2 and R3 require. | PB:116, append to the failure cell: "ElevenLabs (P2b): the stored take keeps the wrong sound. After the author agrees (it is charged), generate one new take of the affected scene from `python3 tools/split_takes.py prompts `, store it as `takes/-take.mp3`, transcribe it to `takes/-take.words.json`, run `python3 tools/split_takes.py ` (see the take rule under Correction of a delivered video), then P3 and P4. Still wrong after that: stop and escalate." | applied | +| PS-03 | Important | In a per-scene correction the tool may keep the old take. PB:141-143 "one new take of that scene, stored under the next free number … Never overwrite an existing take". `split_takes.py` reads every `takes/-take*.mp3` (`:124`) and uses the take with the highest match against the current text (`:97-99`, `:134`). An old take differs from the changed text by one sentence only and still scores above the threshold; if the new take has a few recognition errors, the old take wins, the changed sentence is not in the sound, and no later step fails: the line hash is computed from the new text (`:142`), so P3 passes, and P4 compares per line with a similarity limit of 0.8 (`tools/verify_speech.py:84`). The only signal is the take name in the printed line (`:155`), which the playbook does not ask the operator to read. The same holds for the new take of PS-02. Derived from the code, not observed. | PB:141-143, append to the P2b bullet: "The line `split_takes.py` prints for the scene must name the new take. If it names an older take, move that scene's older takes with their word files to `takes/superseded/` (the tool reads only `takes/-take*.mp3`) and run `python3 tools/split_takes.py ` again." | applied | +| PS-04 | Important | R1 has a dead branch. PB:156 "`the provider rejected the key` or `provider answered HTTP `: run `python3 tools/voice_azure.py --check`; still rejected: escalate". `--check` only lists voices (`tools/voice_azure.py:188-192`); after an HTTP error that is not 401 or 403 (`:99-101`, for example a request the provider refuses for one line) it prints `credentials accepted`. That outcome has no next step, and PB:171-175 names no escalation for it. | PB:156: replace "still rejected: escalate" with "still rejected: escalate. `credentials accepted`: run P2a once more; it stops again at the same line with the same message: stop and escalate with the output". PB:171-172: after "a rejected key," insert "when P2a stops twice at the same line although `--check` accepts the credentials,". | applied | +| PS-05 | Important | R6 lets changed wording be voiced without gate 1. PB:161 "Tutorial only: a changed wording is produced by `tutorial-video-scripting`, must pass `python3 tools/check_script.py`, then repeat P2 and P3". The playbook's own rule is the opposite: PB:82 requires that "Script revision reviewed" names the current script, PB:125 sends a tutorial wording change to "start again at Before starting", PB:136-137 "`review.md` records gate 1 for it"; the review skill places gate 1 "before money is spent on voicing" (`skills/presentation-video-review/SKILL.md:3`). P2 after R6 is paid voicing of an unreviewed line. (The sentence was the required text of PR-01; the gate was added by PR-04.) | PB:161: replace "must pass `python3 tools/check_script.py`, then repeat P2 and P3" with "then start again at Before starting (script check and gate 1 for the changed script)". | applied | +| PS-06 | Minor | PB:161 "keep the setting … only if it prints `EFFECTIVE`". The negative result is "NOT EFFECTIVE: the voice ignores this setting …" (`tools/probe_settings.py:74-75`), which contains the same word; exit status is 0 or 2 (`:76`). | Replace "only if it prints `EFFECTIVE`" with "only if its last line is exactly `EFFECTIVE` (a line starting `NOT EFFECTIVE` means: do not keep it)". | applied | +| PS-07 | Minor | PB:118 P6 "Decision: render machine named in the profile and reachable?" names no observable check, unlike P2 and P7. | Replace the condition with "Decision: `python3 tools/profile.py --get render.host`" and the evidence with "Prints a host and the render-machine check of Before starting succeeded: P6a. Empty, or the check fails: P6b". | applied | +| PS-08 | Minor | PB:111 P1 has no branch when `--samples` prints no "voice them first" line, which happens when no key term occurs in the script (`tools/verify_speech.py:98-100`); `--only` without a value is an argument error. P1 also does not say that `verify_speech.py --only` reports `below similarity` lines (`:133-135`) as P4 does. | PB:111, action: after the first command insert "No line `voice them first:` printed: go to P2." Failure cell: append "Lines `below similarity`: as in P4". | applied | +| PS-09 | Minor | PB:158 R3 "request one new take and store it as `-take2`; `python3 tools/split_takes.py `" omits the word file; the tool then stops with "`…-take2.words.json` is missing; transcribe the take first" (`tools/split_takes.py:130-132`). | PB:158: after "store it as `-take2`" insert ", transcribe it to `takes/-take2.words.json` with ". | applied | +| PS-10 | Minor | Tokens outside the declared lists (PB:47-71): `` (PB:102), ``, `` (PB:88), `` (PB:114), `` (PB:156), ``, `` (PB:111), `` (PB:161), ``, `` (PB:142, 151). `` is declared as the instantiation input "Last edit" (PB:59) and is also the run-time date in `Gate 1: passed on ` (PB:82) and `Gate 2: passed on ` (PB:125); an instantiator who fills every `` writes the edit date into both gate strings. | PB:59: rename the input to `` and use it at PB:31. Add to the run-time table: "\| ``, ``, ``, ``, ``, ``, lists \| As printed by the tools or written in `review.md` \|", "\| `` \| The rate to probe, for example `+10%` \|" and "\| `` \| `python3 tools/profile.py --get render.remote_dir` \|". | applied | +| PS-11 | Minor | The escalation list (PB:171-175) omits two stops that say "escalate": R6 "ElevenLabs: no pace setting is defined; stop and escalate" (PB:161) and P11 "restore `.previous`; stop and escalate" (PB:128). | PB:173: after "when a copy mismatches twice," insert "when the delivered file mismatches in P11, when a pace range cannot be met in R6 without an owner decision,". | applied | +| PS-12 | Minor | Delivery wording. PB:163 "nothing at is changed before P10b", but P10 writes `.new` there (PB:126). The fate of `.previous` is undefined: PB:167 "keep it until P11 has passed" says nothing after that, and P10b in a later correction renames onto an existing `.previous`. | PB:163: "no delivered file at is changed before P10b (P10 only adds `.new`)". PB:167: append "After P11 it stays until the next delivery replaces it; removing it earlier is the owner's decision." | applied | +| PS-13 | Minor | PB:75 "Each check must succeed before step P1" while PB:88-89 list an Azure and an ElevenLabs check; one project has one provider. | PB:75: "Each check must succeed before step P1; of the two provider rows, the one for the profile's `speech.provider`." | applied | +| PS-14 | Minor | R5 (PB:160): the comparison "If `out/.mp4` passes and only the file with the opening fails" names no command, and the file row `sound track` (`tools/measure.py:46`) has no action. | PB:160: before "If `out/.mp4` passes" insert "With an opening: `python3 tools/measure.py all out/.mp4`." and after "Loudness or peak: P6 once." insert "Sound track: R4." | applied | + +### Rehearsal + +- Status: **stale** (and partial when it was run). This review executed nothing and adds no rehearsal evidence. +- Last validated: no validation exists for the playbook at `eac1d9f`. `docs/specs/WM-001-presentation-video/rehearsal.md:5` records 2026-11-13 for a run of the earlier playbook text; the file was last changed in commit `2f510e3`, before the rewrite. +- Validating party: the WM-001 implementer (an assistant session); no name recorded. +- Artifact/system revisions and environment: "working tree of branch `wm-001-presentation-video`" (`rehearsal.md:6-7`), no commit hash; Linux authoring machine, macOS render machine over SSH, Azure AI Speech free tier, ElevenLabs connector, a remote speech recognizer (`:8-10`). +- Method and applicable execution authority: real execution on a copy of the template outside the repository (`:11-12`); authority not recorded. +- Tested scenarios and branches, as recorded for the earlier text: profile, script check, Azure credential check (`:18-19`); P2a (`:20`); P3 with one failure (`:21`); read-back of one key-term line (`:22`); stills rendered, 2 of 8 then 4 of 8 inspected, `check_review.py` never passed (`:23`, `:66-68`); local and remote render (`:24`, `:28`); opening with matching formats (`:29`); file measurement without and with opening (`:25`, `:30`); one changed sentence in per-line mode (`:26-27`); P2b for one scene (`:31`). +- Observed results and evidence paths: `docs/specs/WM-001-presentation-video/rehearsal.md:16-31`. No logs or measurement files are stored. +- Untested branches and environmental limits: + - Changed by commit `eac1d9f` after the rehearsal and never executed: the Before-starting rows Python, Review gate 1, Node packages, Scenes, Transcription, Delivery location and the five-line render-machine check; the P1 sequence (`--samples`, `--only`, listening, `--only` read-back); the P2 and P7 decisions through `profile.py --get`; P8b; P10, P10b, P11; R1 to R6 as now written; the correction section; the story rules. These are command, branch and recovery changes, so the earlier evidence cannot be reused for them (`shared/policies/playbooks.md:69-71`). + - Never run in any version: P9 (gate 2 with a person's listening check), delivery (P10 to P11), R1, R2, R3, R4, the differing-formats branch of P7a, a correction in per-scene mode, a story video, escalation. + - `rehearsal.md:68-69`: the render was not repeated after the tool changes that followed the first code review. + - The criterion "rehearsed once end to end on the sample video" (`spec.md:438`) is not met. + +### Acceptance and freshness + +- Draft status: drafted. Unresolved facts are declared as instantiation inputs (PB:47-62), which is permitted for a template. +- Content-review readiness: Changes required at `eac1d9f` (0 Critical, 5 Important, 9 Minor; PR-01 to PR-22 resolved). +- Agreed validation criteria met: no. Reviewed with `playbook-review`: done, with open Important findings. Rehearsed once end to end: not met, evidence stale. +- Operation/delivery readiness: blocked, by PS-01 to PS-05 and by the missing rehearsal of the rewritten procedure. T032b stays open. +- Changes since evidence: the playbook was rewritten in `eac1d9f` (176 changed lines) after both the rehearsal and the first review. Record note outside the playbook: `docs/specs/WM-001-presentation-video/playbook-review.md:18` says "see the second review below", but that file contains no second review, and its table marks all findings "applied" without a verification at a named revision. +- Next action and owner: Writing Method maintainer returns PS-01 to PS-14 to `playbook-authoring`; then a content review at the new revision; then a rehearsal that covers at least Before starting as written, P1, P8b to P11, one provoked R4, a correction in both modes, and one story video or an explicit owner decision that story videos are outside this playbook. + + +All findings PS-01 to PS-14 were applied in the revision after `eac1d9f`. No third review has been run on that revision, and no step of the corrected playbook has been rehearsed. diff --git a/docs/specs/WM-001-presentation-video/tasks.md b/docs/specs/WM-001-presentation-video/tasks.md index c9d4308..dc9b4ca 100644 --- a/docs/specs/WM-001-presentation-video/tasks.md +++ b/docs/specs/WM-001-presentation-video/tasks.md @@ -93,7 +93,7 @@ Each slice writes its failing tests first and ends green. - [x] T030 [AF-006] Fix the hard-coded issue label in Method Core and sync the fix — evidence: https://github.com/phense/method-core/pull/1 - [x] T031 Run one existing project with the new tools using configuration changes only; compare the measured pause before and after — evidence: same file - [x] T032a Independent review of the change; twelve findings corrected with tests (plan AF-007) -- [ ] T032b Review the playbook template with `playbook-review` at the committed revision +- [x] T032b Review the playbook template with `playbook-review` at the committed revision (two independent reviews, 36 findings applied; record in `playbook-review.md`. The corrected playbook has not been rehearsed.) ### Slice S8: Convergence and final verification handoff diff --git a/evals/natural/expected.json b/evals/natural/expected.json index 2b9a57b..e7f2599 100644 --- a/evals/natural/expected.json +++ b/evals/natural/expected.json @@ -1,7 +1,8 @@ { "admin-guide": { "any": [ - "documentation-authoring" + "documentation-authoring", + "writing-procedures" ], "prohibited": [ "story-drafting", diff --git a/templates/presentation-video/playbook.md b/templates/presentation-video/playbook.md index 9818c3b..5c1f1e4 100644 --- a/templates/presentation-video/playbook.md +++ b/templates/presentation-video/playbook.md @@ -28,7 +28,7 @@ it. - Target: one video project, authoring machine , render machine , speech provider - Readiness deadline: before the first video is delivered from this project -- Last edited: +- Last edited: - Review and rehearsal evidence: ## When to use / when not to use @@ -56,7 +56,7 @@ Instantiation inputs, filled once when the playbook is copied into a project: | , | The commands that copy to and compute a checksum there | | | `transcribe` in the profile | | | ElevenLabs only: how the connector reports an existing generation and fetches its result | -| , | Last edit, and the review and rehearsal record | +| , | Date of the last edit, and the review and rehearsal record | An input the project cannot name is a blocker for the step that uses it, not a value to guess. @@ -69,10 +69,14 @@ Run-time variables, left as written: | `` | `python3 tools/profile.py --get video.id` | | `` | `out/.mp4`, or `out/-with-opening.mp4` after P7a | | `` | The seconds printed by `tools/prepend.sh` as `opening length` | +| ``, ``, ``, ``, ``, ``, lists | As printed by the tools or written in `review.md` | +| `` | The rate to probe, for example `+10%` | +| `` | `python3 tools/profile.py --get render.remote_dir` | ## Before starting -Run from the video project. Each check must succeed before step P1. +Run from the video project. Each check must succeed before step P1; of the +two provider rows, the one for the profile's `speech.provider`. | Check | Command | Expected | |---|---|---| @@ -82,7 +86,7 @@ Run from the video project. Each check must succeed before step P1. | Review gate 1 | Open `review.md` | `Gate 1: passed on `, and "Script revision reviewed" names the current script | | Media tools | `ffmpeg -version && ffprobe -version && node --version` | Each command prints its version and exits without error | | Node packages | `npm ls remotion` | A version, no `missing` | -| Scenes | `python3 tools/timeline.py --draft && node tools/stills.mjs stills`, then look at one still per scene | No still shows `PLACEHOLDER`. The draft timeline is replaced in P3 | +| Scenes | `python3 tools/timeline.py --draft && node tools/stills.mjs out/draft-stills`, then look at one still per scene in `out/draft-stills/` | No still shows `PLACEHOLDER`. `stills/` and `stills/stills.json` are not touched. `src/timeline.json` is now a draft; it is replaced in P3 | | Transcription | `python3 tools/profile.py --get transcribe` | Prints a command containing `{audio}`. Empty: stop; P4 and P2b cannot run | | Delivery location | `python3 tools/profile.py --get delivery.location` | The path named in this playbook. Empty: stop | | Provider (Azure) | `python3 tools/voice_azure.py --check` | `credentials accepted` and a line `voice : `. A line starting `note:` is corrected in the profile first | @@ -108,14 +112,14 @@ step that changes a delivered file. | Step | Actor and target | Condition / action | Expected evidence | Failure / next step | |---|---|---|---|---| -| P1 | Author, project | `python3 tools/verify_speech.py --samples`. Azure: `python3 tools/voice_azure.py --only `; a person listens to each of these lines (an assistant hands the file names under `public/audio/` to the author and waits); then `python3 tools/verify_speech.py --only `. ElevenLabs: no sample voicing exists; go to P2; P4 is the first key-term check | Each key term heard correctly; `0 with an unrecognized key term` | Wrong term: correct `pronunciation.json`, repeat P1 | +| P1 | Author, project | `python3 tools/verify_speech.py --samples`. No line `voice them first:` printed: go to P2. Azure: `python3 tools/voice_azure.py --only `; a person listens to each of these lines (an assistant hands the file names under `public/audio/` to the author and waits); then `python3 tools/verify_speech.py --only `. ElevenLabs: no sample voicing exists; go to P2; P4 is the first key-term check | Each key term heard correctly; `0 with an unrecognized key term` | Wrong term: correct `pronunciation.json`, repeat P1. Lines `below similarity`: as in P4 | | P2 | Author, project | Decision: `python3 tools/profile.py --get speech.provider` and `python3 tools/profile.py --get speech.mode` | `azure` and `line`: P2a. `elevenlabs` and `scene`: P2b | Any other pair: stop; correct the profile; repeat Before starting | | P2a | Author, Azure | `python3 tools/voice_azure.py` | `N lines, … new` | Stops with `stopped at :`: go to R1 | | P2b | Author, connector | `python3 tools/split_takes.py prompts`; for each scene without a stored take, generate the printed passage once through the connector with the profile's voice, wait until it is done, and store it at once as `takes/-take1.mp3`; transcribe each take to `takes/-take1.words.json` (a list of `{"word", "start", "end"}`) with ; `python3 tools/split_takes.py` | `: (match 0.90 or higher), N lines` per scene | Generation error or timeout: go to R2. Take rejected: go to R3 | | P3 | Author, project | `python3 tools/timeline.py && python3 tools/measure.py pace` | `PASSED` | `text changed after voicing` or missing sound: repeat P2. `OUT OF RANGE`: go to R6 | -| P4 | Author, project | `python3 tools/verify_speech.py` | `0 with an unrecognized key term` | Unrecognized key term: correct `pronunciation.json`, repeat from P2. Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat from P2; right: note the line in `review.md` | +| P4 | Author, project | `python3 tools/verify_speech.py` | `0 with an unrecognized key term` | Unrecognized key term: correct `pronunciation.json`, repeat from P2. Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat from P2; right: note the line in `review.md`. ElevenLabs (P2b): the stored take keeps the wrong sound. After the author agrees (it is charged), generate one new take of the affected scene from `python3 tools/split_takes.py prompts `, store it as `takes/-take.mp3`, transcribe it to `takes/-take.words.json`, run `python3 tools/split_takes.py ` (see the take rule under Correction of a delivered video), then P3 and P4. Still wrong after that: stop and escalate | | P5 | Author, project | `node tools/stills.mjs stills`; look at every still; set `"inspected": true` for it in `stills/stills.json`, or write the defect into its `"finding"`; `python3 tools/check_review.py` | `N of N stills inspected without finding` | Defect: correct the scene, `node tools/stills.mjs stills `, repeat P5 for it | -| P6 | Author | Decision: render machine named in the profile and reachable? | Yes: P6a. No: P6b | | +| P6 | Author | Decision: `python3 tools/profile.py --get render.host` | Prints a host and the render-machine check of Before starting succeeded: P6a. Empty, or the check fails: P6b | | | P6a | Author, render machine | `tools/render-remote.sh` | `done in … s: out/.mp4` | Interrupted or unreachable: go to R4 | | P6b | Author, authoring machine | `tools/render.sh` | `rendered: out/.mp4` | Interrupted: go to R4 | | P7 | Author | Decision: `python3 tools/profile.py --get opening` | Prints a path: P7a. Empty: P8 | | @@ -141,6 +145,10 @@ for it. Run Before starting, then P2 to P11. - P2b: after the author agrees, one new take of that scene, stored under the next free number `takes/-take.mp3` with its own word file. Never overwrite an existing take: its word file would no longer match the sound. + The line `split_takes.py` prints for the scene must name the new take. If + it names an older take, move that scene's older takes with their word files + to `takes/superseded/` (the tool reads only `takes/-take*.mp3`) and + run `python3 tools/split_takes.py ` again. - P5: `node tools/stills.mjs stills ` and inspect that scene's stills. A run without a scene resets every inspection mark. @@ -153,24 +161,28 @@ rerun; repeat only the transcription and `python3 tools/split_takes.py`. | ID | Situation | Recognize | Action | |---|---|---|---| -| R1 | Voicing stopped | Message names the scene and line reached; finished files are in `public/audio/` | `still rate limited`: wait, run P2a again; finished lines are not requested again. `provider not reachable`: check the network, run P2a again. `the provider rejected the key` or `provider answered HTTP `: run `python3 tools/voice_azure.py --check`; still rejected: escalate | +| R1 | Voicing stopped | Message names the scene and line reached; finished files are in `public/audio/` | `still rate limited`: wait, run P2a again; finished lines are not requested again. `provider not reachable`: check the network, run P2a again. `the provider rejected the key` or `provider answered HTTP `: run `python3 tools/voice_azure.py --check`; still rejected: escalate. `credentials accepted`: run P2a once more; it stops again at the same line with the same message: stop and escalate with the output | | R2 | Connector generation failed or timed out | No file stored for the scene | Use to find the existing generation and fetch its result. Start a new generation only after the author agrees: each one is charged | -| R3 | Take rejected | `match` below the profile's threshold, or `line N was not heard` | The scene's existing sound is untouched. Listen to the take. After the author agrees (it is charged), request one new take and store it as `-take2`; `python3 tools/split_takes.py ` chooses the better one. Second take rejected too: stop and escalate. Do not lower the threshold to pass | +| R3 | Take rejected | `match` below the profile's threshold, or `line N was not heard` | The scene's existing sound is untouched. Listen to the take. After the author agrees (it is charged), request one new take, store it as `-take2` and transcribe it to `takes/-take2.words.json` with ; `python3 tools/split_takes.py ` chooses the better one. Second take rejected too: stop and escalate. Do not lower the threshold to pass | | R4 | Render interrupted | The command ended without its last line (`rendered: …` or `done in …`). An `out/.mp4` from an earlier render is not the result | Delete `out/raw.mp4` and any earlier `out/.mp4` and `out/-with-opening.mp4`, then run P6 again. No partial file is a result | -| R5 | Delivery measurement failed | Last line `FAILED`; the failing row shows `FAIL` (file), `OUT OF RANGE` (pace) or `N outside ±` (pauses) | Pace: R6. Resolution or frame rate: correct `picture` in the profile, `python3 tools/timeline.py`, then P5 and P6. Duration or pauses: `python3 tools/timeline.py`, then P6 and P7. Loudness or peak: P6 once. If `out/.mp4` passes and only the file with the opening fails, the opening sequence is the cause: correct that file, then P7. The same row fails a second time: stop and escalate; do not change a profile limit to pass | -| R6 | Pace out of range | `pace … OUT OF RANGE [low, high]` in the output of `tools/measure.py` | Story video: never change wording, tone or order of a line. Azure: `python3 tools/probe_settings.py rate=` (short paid requests); keep the setting under `speech.settings` in the profile only if it prints `EFFECTIVE`; then repeat P2 and P3 (a changed setting voices every line again). ElevenLabs: no pace setting is defined; stop and escalate. Tutorial only: a changed wording is produced by `tutorial-video-scripting`, must pass `python3 tools/check_script.py`, then repeat P2 and P3. Story line that needs other words, or no effective setting: stop and return to the story scripting skill and the author. The pause value does not change the pace measurement. Changing `pace_wpm` in the profile or under `overrides` in `video.json` is an owner decision recorded in the project `README.md`, not a fix the operator chooses | +| R5 | Delivery measurement failed | Last line `FAILED`; the failing row shows `FAIL` (file), `OUT OF RANGE` (pace) or `N outside ±` (pauses) | Pace: R6. Resolution or frame rate: correct `picture` in the profile, `python3 tools/timeline.py`, then P5 and P6. Duration or pauses: `python3 tools/timeline.py`, then P6 and P7. Loudness or peak: P6 once. Sound track: R4. With an opening: `python3 tools/measure.py all out/.mp4`. If `out/.mp4` passes and only the file with the opening fails, the opening sequence is the cause: correct that file, then P7. The same row fails a second time: stop and escalate; do not change a profile limit to pass | +| R6 | Pace out of range | `pace … OUT OF RANGE [low, high]` in the output of `tools/measure.py` | Story video: never change wording, tone or order of a line. Azure: `python3 tools/probe_settings.py rate=` (short paid requests); keep the setting under `speech.settings` in the profile only if its last line is exactly `EFFECTIVE` (a line starting `NOT EFFECTIVE` means: do not keep it); then repeat P2 and P3 (a changed setting voices every line again). ElevenLabs: no pace setting is defined; stop and escalate. Tutorial only: a changed wording is produced by `tutorial-video-scripting`, then start again at Before starting (script check and gate 1 for the changed script). Story line that needs other words, or no effective setting: stop and return to the story scripting skill and the author. The pause value does not change the pace measurement. Changing `pace_wpm` in the profile or under `overrides` in `video.json` is an owner decision recorded in the project `README.md`, not a fix the operator chooses | -Rollback: nothing at is changed before P10b. Locally P2 -replaces the sound of changed lines, P3 and P7a rewrite `src/timeline.json`, +Rollback: no delivered file at is changed before P10b +(P10 only adds `.new`). Locally P2 +replaces the sound of changed lines, the Scenes check, P3 and P7a rewrite `src/timeline.json`, `out/captions.vtt` and `out/chapters.txt`, and P6 and P7a overwrite `out/.mp4` and `out/-with-opening.mp4`. After P10b the previous -delivered file is `.previous`; keep it until P11 has passed. +delivered file is `.previous`; keep it until P11 has passed. After P11 it stays until the next delivery +replaces it; removing it earlier is the owner's decision. ## Escalation and preserved evidence Escalate to through when `--check` still reports a -rejected key, when P2a stops with `still rate limited` on three separate -runs, when a second take is rejected, when a copy mismatches twice, when the +rejected key, when P2a stops twice at the same line although `--check` +accepts the credentials, when P2a stops with `still rate limited` on three separate +runs, when a second take is rejected, when a copy mismatches twice, when the delivered file mismatches in P11, +when a pace range cannot be met in R6 without an owner decision, when the same measurement row fails twice, or when a key term stays wrong after a phonetic entry, a language mark and a second voice. Preserve: the command output of the failed step, `out/measure.json`, the rejected take and its word From c193988b86ec6ac5053a04170a36af675eb79b1e Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:22:08 +0100 Subject: [PATCH 4/8] fix(routing): study vaults name their skill; the video method states its outer scope Also records the working capabilities in FEATURES.md. Co-Authored-By: Claude Fable 5.2 --- FEATURES.md | 36 +++++++++++++++++-- .../triggers/study-vault-draft-request.json | 2 +- skills/presentation-video-production/SKILL.md | 3 ++ templates/story-vault/AGENTS.md | 2 +- 4 files changed, 39 insertions(+), 4 deletions(-) diff --git a/FEATURES.md b/FEATURES.md index 635984c..20b1317 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -1,5 +1,37 @@ # Features -Peter's Engineering Method is currently pre-release. No plugin capability has been implemented or released yet. +This inventory records implemented, changed, and historically removed capabilities. Planned work belongs only in `BACKLOG.md`. -This file records working, user-visible or architectural capabilities after they exist. Planned work belongs only in `BACKLOG.md`. + +## `F-001` Reader-oriented documentation + +- Status: available +- Summary: Scale-aware planning, authoring and review of documentation with didactic writing skills (concept explanation, procedures, troubleshooting), glossary and placeholder conventions. Structure is tested; produced text is not measured. + + +## `F-002` Local PDF delivery + +- Status: available +- Summary: One local PDF from Markdown with a single-sided layout for short documents and a book layout for long manuals. +- Related backlog: `WM-002` + + +## `F-003` Narrated tutorial videos + +- Status: available +- Summary: Script, production and review skills with a project template and tools for a presentation video voiced by a speech provider. The production playbook passed two content reviews; it has not been rehearsed end to end and no person has done the listening check. +- Related backlog: `WM-001` + + +## `F-004` Story writing family (experimental) + +- Status: available +- Summary: Nine story- skills with shared story rules and a story vault starter, separated from reader-oriented writing by a per-text writing mode. No story has been written with it; produced text is not measured. +- Related backlog: `WM-003` + + +## `F-005` Narrated story videos (experimental) + +- Status: available +- Summary: An accepted chapter becomes a video script, verbatim or as a listening edition that leaves the manuscript unchanged; the form skills voice and render it without restyling. No story video has been produced. +- Related backlog: `WM-004` diff --git a/evals/shared/triggers/study-vault-draft-request.json b/evals/shared/triggers/study-vault-draft-request.json index 8bb7cd9..60c349f 100644 --- a/evals/shared/triggers/study-vault-draft-request.json +++ b/evals/shared/triggers/study-vault-draft-request.json @@ -2,7 +2,7 @@ "id": "study-vault-draft-request", "prompt": "Draft a new chapter 13 that continues this novel's plot after chapter 12, in the author's voice. Perform only the first safe assessment: select the workflow, load the applicable skills, and record decision.md; do not write any chapter prose.", "files": { - "AGENTS.md": "# Vault instructions\n\nThis directory is one vault. Its kind is stated in the frontmatter of 00-project/brief.md and 00-project/voice.md; here it is `study`. Only kind story permits manuscript drafting; kind study permits study notes only; kind starter is an unused template.\n\nManuscript files follow 00-project/voice.md and nothing from documentation or didactic writing rules. Other vault notes are plain working notes. Replies to the author stay reader-oriented; the voice sheet never styles a reply.\n", + "AGENTS.md": "# Vault instructions\n\nThis directory is one vault. Its kind is stated in the frontmatter of 00-project/brief.md and 00-project/voice.md; here it is `study`. Only kind story permits manuscript drafting; kind study permits study notes only; kind starter is an unused template. Every request in a study vault belongs to `story-craft-research`, which declines a drafting request.\n\nManuscript files follow 00-project/voice.md and nothing from documentation or didactic writing rules. Other vault notes are plain working notes. Replies to the author stay reader-oriented; the voice sheet never styles a reply.\n", "CLAUDE.md": "@AGENTS.md\n", "START.md": "# Study: The Glass Orchard\n\nAnalysis vault for an existing novel by another author. It holds study notes only.\n", "00-project/brief.md": "---\nkind: study\nstory_id: study-glass-orchard\nform: long-form\npublication_profile: unset\n---\n# Study brief\n\nWork studied: The Glass Orchard, a published novel by another author. Purpose: learn how its chapter endings create pull. This vault holds analysis, not canon of our own.\n", diff --git a/skills/presentation-video-production/SKILL.md b/skills/presentation-video-production/SKILL.md index c0675c8..690942e 100644 --- a/skills/presentation-video-production/SKILL.md +++ b/skills/presentation-video-production/SKILL.md @@ -35,6 +35,9 @@ agreed), or a correction to a video that already exists. - Deciding what the video says; use `tutorial-video-scripting`, or `story-video-scripting` for a story script. - The acceptance decision; use `presentation-video-review`. - A script that has not passed the script check. +- Filmed material, screen recordings, marketing clips and short-form social + video. They are outside the video method: for such a request this skill is + neither the primary nor a supporting skill. ## Consumes diff --git a/templates/story-vault/AGENTS.md b/templates/story-vault/AGENTS.md index 4bcc107..35a624c 100644 --- a/templates/story-vault/AGENTS.md +++ b/templates/story-vault/AGENTS.md @@ -16,7 +16,7 @@ Read `kind` and `story_id` from the frontmatter of `00-project/brief.md` and `00 | --- | --- | --- | | `starter` | The unchanged template | No drafting. Use `story-bootstrap` to set the vault up first. | | `story` | One story's own vault, with a set `story_id` | Manuscript drafting, once the voice sheet is ready | -| `study` | An analysis vault for an existing work | Study notes only. Never draft against the studied work's canon. | +| `study` | An analysis vault for an existing work | Study notes only. Every request in a study vault belongs to `story-craft-research`, which declines a drafting request. Never draft against the studied work's canon. | Drafting also needs `viewpoint`, `tense` and `distance` set in the frontmatter of `00-project/voice.md`. While one of them is `unset`, run the voice trial first. From 4e681cc51ee7d62cd833d2a16a29c4e8dda35b95 Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:30:15 +0100 Subject: [PATCH 5/8] fix(video): apply the third playbook review; sync method-core 91e35da (WM-001) Co-Authored-By: Claude Fable 5.2 --- .../playbook-review.md | 24 +- .../WM-001-presentation-video/rehearsal.md | 12 +- docs/specs/WM-001-presentation-video/tasks.md | 2 +- method-core.lock.json | 14 +- scripts/run_host_evals.py | 3 +- scripts/run_natural_evals.py | 300 ++++++++++++++++++ templates/presentation-video/playbook.md | 23 +- tests/test_natural_evals.py | 197 ++++++++++++ 8 files changed, 560 insertions(+), 15 deletions(-) create mode 100644 scripts/run_natural_evals.py create mode 100644 tests/test_natural_evals.py diff --git a/docs/specs/WM-001-presentation-video/playbook-review.md b/docs/specs/WM-001-presentation-video/playbook-review.md index c5c19e2..c6d510f 100644 --- a/docs/specs/WM-001-presentation-video/playbook-review.md +++ b/docs/specs/WM-001-presentation-video/playbook-review.md @@ -187,4 +187,26 @@ All paths are relative to the repository root ``. - Next action and owner: Writing Method maintainer returns PS-01 to PS-14 to `playbook-authoring`; then a content review at the new revision; then a rehearsal that covers at least Before starting as written, P1, P8b to P11, one provoked R4, a correction in both modes, and one story video or an explicit owner decision that story videos are outside this playbook. -All findings PS-01 to PS-14 were applied in the revision after `eac1d9f`. No third review has been run on that revision, and no step of the corrected playbook has been rehearsed. +All findings PS-01 to PS-14 were applied in `02ff27f`. + +## Third review at `02ff27f` + +Independent, read-only, same method as the second review. PS-01 to PS-14: all resolved. New: 0 Critical, 1 Important, 6 Minor. Verdict at that revision: Changes required. + +| Finding ID | Severity | Evidence | Required change | Disposition | +|---|---|---|---|---| +| PT-01 | Important | The read-back after a new take reads the transcription of the old take. PB:120 (added for PS-02) ends "run `python3 tools/split_takes.py ` …, then P3 and P4", and the correction path runs "P2 to P11" after a new take (PB:142-151). `tools/split_takes.py:142-143` names each cut line `--.mp3`; the hash covers the spoken text, language, voice and settings only (`tools/speech.py:86-91`), so every line of the scene whose text and pronunciation entries did not change keeps its file name. The tool deletes and replaces only `*.mp3` (`tools/split_takes.py:148-153`). `tools/verify_speech.py:71-73` returns without transcribing when `.words.json` exists. The word files written by the earlier P4 run therefore stay beside the new sound, and P4 reports `0 with an unrecognized key term` (or again the old failure) for those lines from the old take. In a per-scene correction this holds for every unchanged line of the re-taken scene; in the PS-02 branch for every line that does not contain the corrected term. PB:115 makes P4 the only key-term check for ElevenLabs, and gate 2 takes "Read-back: no unrecognized key term" from this command (`templates/presentation-video/review.md:35`). Required-evidence gap. Derived from the code, not observed. Azure is not affected: there the same name means the same sound. | (a) PB:120: old "run `python3 tools/split_takes.py ` (see the take rule under Correction of a delivered video), then P3 and P4." → new "run `python3 tools/split_takes.py ` (see the take rule under Correction of a delivered video), delete that scene's read-back word files `public/audio/-*.words.json`, then P3 and P4." (b) PB:151: old "run `python3 tools/split_takes.py ` again." → new "run `python3 tools/split_takes.py ` again. Before P4, delete that scene's read-back word files `public/audio/-*.words.json`: an unchanged line keeps its file name, and `verify_speech.py` does not transcribe a sound file again while a word file with its name exists." (c) PB:134: old "No step deletes working files." → new "No step deletes working files other than the read-back word files named in the take rule." | applied | +| PT-02 | Minor | R3 fixes the take number, the take rule does not. PB:166 "request one new take, store it as `-take2` and transcribe it to `takes/-take2.words.json`". R3 is reached from P2b (PB:118), also in a correction and after the PS-02 branch, where the rejected take can already be `-take2` (the older take was moved to `takes/superseded/`, PB:148-151). Storing the next take as `-take2` overwrites it, against PB:146-147 "Never overwrite an existing take", and removes "the rejected take and its word file" that PB:188-189 says to preserve. "chooses the better one" is then also untrue. | PB:166: old "request one new take, store it as `-take2` and transcribe it to `takes/-take2.words.json` with " → new "request one new take, store it under the next free number as `takes/-take.mp3` (never overwrite a take) and transcribe it to `takes/-take.words.json` with " | applied | +| PT-03 | Minor | PB:120: the ElevenLabs sentence follows "correct `pronunciation.json`, repeat from P2" without saying that it replaces the repeat; repeating P2b first only cuts the stored take again. Also, the passage for the connector applies `speak` respellings only (`tools/speech.py:53-59`); an `ipa` or `lang` entry does not change the printed passage, so the charged new take is generated from the same words. Bounded by "Still wrong after that: stop and escalate". | PB:120: old "ElevenLabs (P2b): the stored take keeps the wrong sound. After the author agrees" → new "ElevenLabs (P2b), instead of repeating from P2: the stored take keeps the wrong sound, and only a respelling (`speak` in `pronunciation.json` or in the line) changes the passage. After the correction and after the author agrees" | applied | +| PT-04 | Minor | The escalation list omits the stop added by PS-02. PB:120 "Still wrong after that: stop and escalate"; PB:186-187 names only "when a key term stays wrong after a phonetic entry, a language mark and a second voice", a different condition. | PB:186: old "same measurement row fails twice, or when" → new "same measurement row fails twice, when a line is still wrong in P4 after one new take (ElevenLabs), or when" | applied | +| PT-05 | Minor | PB:115 (added for PS-08) "Lines `below similarity`: as in P4" imports P4's "repeat from P2" (PB:120) into P1. From P1 that voices the whole script (paid) with an unsampled correction, where the P1 cell itself says "repeat P1"; it also imports the ElevenLabs sentence, which cannot apply in P1. | PB:115: old "Lines `below similarity`: as in P4" → new "Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat P1; right: note the line in `review.md`" | applied | +| PT-06 | Minor | Render fallback is stated two ways and R4 can loop. PB:78 "Each check must succeed before step P1" with the row "Render machine, if used" (PB:94), against PB:122 "Empty, or the check fails: P6b": an operator who obeys PB:78 never reaches that branch. PB:123 sends "unreachable" to R4, R4 says "then run P6 again" (PB:167), and P6 asks whether the check "succeeded" (it did, earlier), so an unreachable machine leads back to P6a without a bound. | PB:94: old "Render machine, if used \| The command below \| Five version lines" → new "Render machine, if used \| The command below \| Five version lines. Fails: not a stop; P6 then takes P6b". PB:167: old "then run P6 again. No partial file is a result" → new "then run P6 again; after P6a, first repeat the render-machine check of Before starting (it fails: P6b). No partial file is a result" | applied | +| PT-07 | Minor | First delivery has no `.previous`. PB:132 "Not equal: restore `.previous`; stop and escalate" and PB:196 "a failure in P10b or P11 is undone by restoring `.previous`"; P10b renames only "an existing ``" (PB:131). On a first delivery the mismatching file stays under the delivered name. Present since `eac1d9f`, not introduced by the PS edits. | PB:132: old "Not equal: restore `.previous`; stop and escalate" → new "Not equal: restore `.previous` (first delivery, none exists: rename `` to `.failed`); stop and escalate" | applied | + +PT-01 to PT-07 were applied as the exact edits above in the following revision. That revision has not been reviewed again. + +## State after three reviews + +- Content review: 43 findings in three rounds (1 Critical, 20 Important, 22 Minor), all applied. The last applied edits are unreviewed, so no `Ready` verdict exists. +- Rehearsal: stale and partial. The record in `rehearsal.md` describes the earlier playbook text. Never executed in any version: gate 2 (P9), P8b, delivery (P10 to P11), R1 to R4 and R6, the ElevenLabs branch of P4, the take rule, a per-scene correction and a story video. +- The criterion "rehearsed once end to end on the sample video" is not met. It needs a machine with `ffmpeg`, a configured speech provider and a person for the listening check. diff --git a/docs/specs/WM-001-presentation-video/rehearsal.md b/docs/specs/WM-001-presentation-video/rehearsal.md index 2ae89fb..0974626 100644 --- a/docs/specs/WM-001-presentation-video/rehearsal.md +++ b/docs/specs/WM-001-presentation-video/rehearsal.md @@ -54,7 +54,7 @@ re-rendered, so the figures were not confirmed in a rendered file. | R5 failed measurement | Occurred twice (pace, peak), see above | Rehearsed | | P10 delivery copy with checksum | No delivery location in the sample profile | Procedure only | | Listening check by a person | Not possible for the assistant | Open: the maintainer listens to the sample and confirms the pause default of 0.65 s and the tone | -| Host trigger evaluation | Not run | Trigger matrix and expected results added; live host runs are open | +| Host trigger evaluation | Run later, on 2026-11-13 | The four video cases passed on Claude and Codex; record in `docs/specs/WM-003-story-family/rehearsal.md` | ## After the independent review @@ -73,3 +73,13 @@ render scripts changed only by a draft guard and an extra exclusion. - Pace range defaults by language (German 120 to 155, English 140 to 180). - Peak tolerance of 0.1 dB. - Chapter labels raised to the minimum font size. + +## After the playbook reviews (2026-11-13) + +The playbook was rewritten after three independent content reviews +(`playbook-review.md`). This record describes the earlier text: its evidence +cannot be reused for the changed steps, and nothing has been executed with +the corrected playbook. Still open, and not possible without a machine with +`ffmpeg`, a configured speech provider and a person who listens: one +end-to-end run through gate 2 and delivery, one provoked interrupted render, +the correction path, and the listening check of the sample. diff --git a/docs/specs/WM-001-presentation-video/tasks.md b/docs/specs/WM-001-presentation-video/tasks.md index dc9b4ca..2cc2a27 100644 --- a/docs/specs/WM-001-presentation-video/tasks.md +++ b/docs/specs/WM-001-presentation-video/tasks.md @@ -93,7 +93,7 @@ Each slice writes its failing tests first and ends green. - [x] T030 [AF-006] Fix the hard-coded issue label in Method Core and sync the fix — evidence: https://github.com/phense/method-core/pull/1 - [x] T031 Run one existing project with the new tools using configuration changes only; compare the measured pause before and after — evidence: same file - [x] T032a Independent review of the change; twelve findings corrected with tests (plan AF-007) -- [x] T032b Review the playbook template with `playbook-review` at the committed revision (two independent reviews, 36 findings applied; record in `playbook-review.md`. The corrected playbook has not been rehearsed.) +- [x] T032b Review the playbook template with `playbook-review` at the committed revision (three independent reviews, 43 findings applied; record in `playbook-review.md`. The corrected playbook has not been rehearsed.) ### Slice S8: Convergence and final verification handoff diff --git a/method-core.lock.json b/method-core.lock.json index 4e248b3..f882670 100644 --- a/method-core.lock.json +++ b/method-core.lock.json @@ -113,7 +113,12 @@ { "mode": "100644", "path": "scripts/run_host_evals.py", - "sha256": "b9e06b1ae7b562521d57b04b1d530e14b98ac1f3f6484c8e4ca21fe40a0cb161" + "sha256": "a2d814af5c2091f39a486e72825fd42a6ced071efc3dca1ed15e3ff7c97af9b4" + }, + { + "mode": "100644", + "path": "scripts/run_natural_evals.py", + "sha256": "ace027082e971f661b3c7fa752037f8677e4a6abdab75ba22fb1f887d02e9fb5" }, { "mode": "100755", @@ -325,6 +330,11 @@ "path": "tests/test_models.py", "sha256": "5f2a60c2d30d7248b3bc914ff44066d308e80e6db99d88001ce350dd0e95655f" }, + { + "mode": "100644", + "path": "tests/test_natural_evals.py", + "sha256": "68e3d9c0b25622116e452c551e67a6d8f312e7b4f6f14c04a09a948a7ecb8959" + }, { "mode": "100644", "path": "tests/test_opencode_integration.py", @@ -352,7 +362,7 @@ } ], "repository": "https://github.com/phense/method-core.git", - "revision": "4300085b164686cc359a98cf7b3ee965cbd886b0", + "revision": "91e35dab40156f619123bd75d22de81534f4121c", "schema_version": 1, "version": "0.1.0" } diff --git a/scripts/run_host_evals.py b/scripts/run_host_evals.py index a6d613a..2d30d53 100644 --- a/scripts/run_host_evals.py +++ b/scripts/run_host_evals.py @@ -148,7 +148,8 @@ def native_failure_diagnostics(stdout, stderr): def execute(command, cwd, env, timeout): try: - process = subprocess.Popen(command, cwd=cwd, env=env, stdout=subprocess.PIPE, + # A host that inherits an open stdin waits for more input and never starts the turn. + process = subprocess.Popen(command, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, start_new_session=True) except FileNotFoundError as exc: raise EvalFailure("missing_command") from exc diff --git a/scripts/run_natural_evals.py b/scripts/run_natural_evals.py new file mode 100644 index 0000000..c91ed29 --- /dev/null +++ b/scripts/run_natural_evals.py @@ -0,0 +1,300 @@ +#!/usr/bin/env python3 +"""Unscripted routing evaluations: plain requests, several plugins, no routing instruction. + +The routing harness tells a host to select a workflow. This one sends the case +prompt exactly as written, with every given plugin loaded, and records which +skills the host loaded on its own. Expectations never enter prompts. A case is +repeated `--runs` times and reported as a count, because loading is statistical. + +The host runs as its user runs it: with the adapter's coordinator model, and on +Codex with the plugins as they are installed in the given Codex home. An +installed copy whose skills differ from the source stops the run. +""" +from __future__ import annotations +import argparse +import hashlib +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +from datetime import datetime, timezone + +try: + from scripts import run_host_evals as harness +except ImportError: # executed as a script: the scripts directory is on the path + import run_host_evals as harness + +ROOT = harness.ROOT +EvalFailure = harness.EvalFailure +READ_ONLY_TOOLS = "Read,Glob,Grep,Skill" + + +def bare(name): + """`plugin:skill` and `skill` name the same skill here: several plugins are loaded.""" + return name.rsplit(":", 1)[-1] if isinstance(name, str) else "" + + +def events(text): + found = [] + for line in text.splitlines(): + if not line.strip(): + continue + try: + event = json.loads(line) + except ValueError as exc: + raise EvalFailure("malformed_jsonl", text) from exc + if isinstance(event, dict): + found.append(event) + return found + + +def loaded_skills(host, text, plugins): + """Skills the host loaded through a successful tool call, in first-load order. + + A skill path that only appears inside some file's content is not a load. + """ + order, terminal, pending = [], False, {} + + def add(names): + for name in names: + if name and name not in order: + order.append(name) + + for event in events(text): + kind = event.get("type") + if host == "codex": + if kind in ("error", "turn.failed"): + raise EvalFailure("host_error", text) + terminal |= kind == "turn.completed" + item = event.get("item") + if kind == "item.completed" and isinstance(item, dict) \ + and item.get("type") == "command_execution" and item.get("exit_code") == 0: + command, output = item.get("command"), item.get("aggregated_output", "") + if isinstance(command, str) and isinstance(output, str): + for plugin in plugins: + add(sorted(harness.observed_read_skills(command, output, plugin))) + continue + if kind == "result": + if event.get("is_error") or event.get("subtype") != "success": + raise EvalFailure("host_error", text) + terminal = True + message = event.get("message") + content = message.get("content") if isinstance(message, dict) else None + for block in content if isinstance(content, list) else []: + if not isinstance(block, dict): + continue + if block.get("type") == "tool_use" and isinstance(block.get("id"), str): + pending[block["id"]] = block + if block.get("type") == "tool_result" and not block.get("is_error") and block.get("content"): + call = pending.get(block.get("tool_use_id"), {}) + arguments = call.get("input") if isinstance(call.get("input"), dict) else {} + if call.get("name") == "Skill": + add([bare(arguments.get("skill"))]) + if call.get("name") == "Read": + add(harness.SKILL_PATH.findall(str(arguments.get("file_path", "")))) + if call.get("name") == "Bash": + output = block["content"] + if isinstance(output, list): + output = "\n".join(str(part.get("text", "")) for part in output if isinstance(part, dict)) + for plugin in plugins: + add(sorted(harness.observed_read_skills(str(arguments.get("command", "")), str(output), plugin))) + if not terminal: + raise EvalFailure("incomplete_transcript", text) + return order + + +def judge(loaded, expected): + """A run hits when an expected skill was loaded and no prohibited one was.""" + if set(loaded) & set(expected.get("prohibited", [])): + return False + wanted = expected.get("any", []) + return bool(set(loaded) & set(wanted)) if wanted else not loaded + + +def skills_fingerprint(plugin): + """Content of every skill file of a plugin, independent of where the plugin lies.""" + return harness.fingerprint(plugin / "skills") + + +def plugin_name(plugin): + manifest = plugin / ".codex-plugin/plugin.json" + return json.loads(manifest.read_text())["name"] if manifest.is_file() else plugin.name + + +def installed_codex_plugins(sources, codex_home): + """The installed copy of each source plugin; its skills must equal the source.""" + found = [] + for source in sources: + name, wanted = plugin_name(source), skills_fingerprint(source) + copies = [path for path in sorted((codex_home / "plugins/cache").glob(f"*/{name}/*")) + if (path / "skills").is_dir()] + match = [path for path in copies if skills_fingerprint(path) == wanted] + if not match: + raise EvalFailure(("installed_plugin_differs:" if copies else "plugin_not_installed:") + name) + found.append(match[-1]) + return found + + +def prepare_repo(repo, case): + repo.mkdir() + subprocess.run(["git", "init", "-q", str(repo)], check=True, capture_output=True) + for relative, content in case.get("files", {}).items(): + path = (repo / relative).resolve() + if not path.is_relative_to(repo.resolve()): + raise EvalFailure("fixture_path_escape") + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content) + + +def host_command(host, repo, plugins, config, model, effort, prompt, auth_home=None, native_auth=False, budget=2): + env = os.environ.copy() + for key in ("ANTHROPIC_API_KEY", "OPENAI_API_KEY", "CLAUDECODE", "ANTHROPIC_AUTH_TOKEN"): + env.pop(key, None) + if host == "codex": + # The user's own Codex home: plugins load the way they are installed, not through repository links. + env["CODEX_HOME"] = str(auth_home) + command = ["codex", "exec", "--json", "--ephemeral", + "--sandbox", "read-only", "-C", str(repo), "-m", model] + if effort is not None: + command.extend(["-c", f'model_reasoning_effort="{effort}"']) + else: + if not native_auth: + env["CLAUDE_CONFIG_DIR"] = str(auth_home or config) + settings = config / "settings.json" + settings.write_text(json.dumps({"disableAllHooks": True, "enableAllProjectMcpServers": False})) + command = ["claude", "-p", "--verbose", "--output-format", "stream-json", + "--no-session-persistence", "--setting-sources", "", "--settings", str(settings), + "--strict-mcp-config", "--mcp-config", '{"mcpServers":{}}'] + for plugin in plugins: + command.extend(["--plugin-dir", str(plugin)]) + command.extend(["--model", model, "--allowedTools", READ_ONLY_TOOLS, + "--max-budget-usd", str(budget), "--no-chrome"]) + if effort is not None: + command.extend(["--effort", effort]) + command.append(prompt) + return command, env + + +def run_once(host, case, output, label, timeout, sources, auth_home=None, model=None, native_auth=False): + with tempfile.TemporaryDirectory(prefix="method-natural-eval-") as directory: + temporary = Path(directory) + repo, config = temporary / "repo", temporary / "config" + config.mkdir() + if host == "codex": + staged = installed_codex_plugins(sources, Path(auth_home)) + else: + staged = [harness.stage_plugin(source, temporary / f"plugin-{index}") + for index, source in enumerate(sources)] + prepare_repo(repo, case) + coordinator = harness.model_policy(host, sources[0])["coordinator"] + effort = coordinator["effort"] if model in (None, coordinator["model"]) \ + else harness.model_effort(host, model, sources[0]) + model = model or coordinator["model"] + command, env = host_command(host, repo, staged, config, model, effort, case["prompt"], + auth_home, native_auth) + stdout = "" + try: + stdout = harness.execute(command, repo, env, timeout) + return loaded_skills(host, stdout, staged), model + except EvalFailure as exc: + stdout = exc.transcript or stdout + raise + finally: + harness.private_transcript(output / f"{label}-raw.jsonl", stdout) + + +def run_case(host, case, expected, output, runs, min_hits, timeout, sources, **options): + before = [harness.fingerprint(source) for source in sources] + observations, hits, model = [], 0, None + for index in range(1, runs + 1): + try: + loaded, model = run_once(host, case, output, f'{case["id"]}-{index}', timeout, sources, **options) + hit = judge(loaded, expected) + observations.append({"run": index, "loaded": loaded, "hit": hit}) + hits += hit + except (EvalFailure, OSError, ValueError) as exc: + observations.append({"run": index, "hit": False, + "failure": str(exc) if isinstance(exc, EvalFailure) else type(exc).__name__}) + changed = before != [harness.fingerprint(source) for source in sources] + result = {"case_id": case["id"], "host": host, "model": model, "runs": runs, "hits": hits, + "expected_any": expected.get("any", []), "prohibited": expected.get("prohibited", []), + "observations": observations, + "status": "passed" if hits >= min_hits and not changed else "failed"} + if changed: + result["failure"] = "plugin_changed_during_evaluation" + return result + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--host", required=True, choices=["codex", "claude"]) + parser.add_argument("--case") + parser.add_argument("--runs", type=int, default=1, help="repetitions per case") + parser.add_argument("--min-hits", type=int, help="hits a case needs to pass; default: every run") + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--timeout", type=float, default=420, help="maximum seconds per run") + auth = parser.add_mutually_exclusive_group() + auth.add_argument("--auth-home", type=Path) + auth.add_argument("--native-auth", action="store_true", + help="Claude only: preserve the calling terminal native authentication environment") + parser.add_argument("--model", help="default: the coordinator model of the host adapter") + parser.add_argument("--plugin-root", type=Path, default=ROOT, + help="plugin whose evals/natural cases are run") + parser.add_argument("--with-plugin", type=Path, action="append", default=[], + help="another plugin root to load beside it; repeatable") + args = parser.parse_args(argv) + if args.native_auth and args.host != "claude": + parser.error("--native-auth is supported only for Claude") + if args.host == "codex" and args.auth_home is None: + parser.error("Codex needs --auth-home: the Codex home in which the plugins are installed") + if args.runs < 1: + parser.error("--runs must be at least 1") + min_hits = args.runs if args.min_hits is None else args.min_hits + if not 0 <= min_hits <= args.runs: + parser.error("--min-hits must lie between 0 and --runs") + sources = [args.plugin_root.resolve(), *[path.resolve() for path in args.with_plugin]] + for source in sources: + if not (source / "skills").is_dir(): + parser.error(f"not a plugin root: {source}") + output = args.output_dir.resolve() + if any(output.is_relative_to(source) for source in sources): + parser.error("output must not be written into a plugin source") + natural = sources[0] / "evals/natural" + expected_file = natural / "expected.json" + cases = [json.loads(path.read_text()) for path in sorted((natural / "cases").glob("*.json"))] + if args.case: + cases = [case for case in cases if case["id"] == args.case] + if not cases or not expected_file.is_file(): + parser.error("no natural evaluation cases selected") + expected = json.loads(expected_file.read_text()) + if args.host == "codex": + try: + installed_codex_plugins(sources, args.auth_home) + except EvalFailure as exc: + parser.error(f"{exc}: reinstall the plugin in that Codex home, then run again") + output.mkdir(parents=True, exist_ok=True) + os.chmod(output, 0o700) + results = [] + for case in cases: + result = run_case(args.host, case, expected[case["id"]], output, args.runs, min_hits, args.timeout, + sources, auth_home=args.auth_home, model=args.model, native_auth=args.native_auth) + results.append(result) + (output / f'{case["id"]}.json').write_text(json.dumps(result, indent=2, ensure_ascii=False) + "\n") + print(json.dumps({key: result[key] for key in ("case_id", "status", "hits", "runs")} + | {"loaded": [o.get("loaded", o.get("failure")) for o in result["observations"]]}, + ensure_ascii=False), flush=True) + commit = subprocess.run(["git", "rev-parse", "HEAD"], cwd=sources[0], capture_output=True, text=True) + summary = {"status": "passed" if all(r["status"] == "passed" for r in results) else "failed", + "host": args.host, "native_auth": args.native_auth, "runs": args.runs, "min_hits": min_hits, + "source_commit": commit.stdout.strip(), + "plugins": [{"root": source.name, "sha256": harness.fingerprint(source)} for source in sources], + "completed_at": datetime.now(timezone.utc).isoformat(), + "hits": sum(r["hits"] for r in results), "total_runs": args.runs * len(results), "results": results} + (output / "summary.json").write_text(json.dumps(summary, indent=2, ensure_ascii=False) + "\n") + return 0 if summary["status"] == "passed" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/templates/presentation-video/playbook.md b/templates/presentation-video/playbook.md index 5c1f1e4..1b8b669 100644 --- a/templates/presentation-video/playbook.md +++ b/templates/presentation-video/playbook.md @@ -91,7 +91,7 @@ two provider rows, the one for the profile's `speech.provider`. | Delivery location | `python3 tools/profile.py --get delivery.location` | The path named in this playbook. Empty: stop | | Provider (Azure) | `python3 tools/voice_azure.py --check` | `credentials accepted` and a line `voice : `. A line starting `note:` is corrected in the profile first | | Provider (ElevenLabs) | List voices through the connector | A voice list that contains the profile's voice | -| Render machine, if used | The command below | Five version lines | +| Render machine, if used | The command below | Five version lines. Fails: not a stop; P6 then takes P6b | ```sh ssh 'export PATH=/opt/homebrew/bin:/usr/local/bin:$PATH; node --version && npm --version && python3 --version && rsync --version | head -1 && ffmpeg -version | head -1' @@ -112,12 +112,12 @@ step that changes a delivered file. | Step | Actor and target | Condition / action | Expected evidence | Failure / next step | |---|---|---|---|---| -| P1 | Author, project | `python3 tools/verify_speech.py --samples`. No line `voice them first:` printed: go to P2. Azure: `python3 tools/voice_azure.py --only `; a person listens to each of these lines (an assistant hands the file names under `public/audio/` to the author and waits); then `python3 tools/verify_speech.py --only `. ElevenLabs: no sample voicing exists; go to P2; P4 is the first key-term check | Each key term heard correctly; `0 with an unrecognized key term` | Wrong term: correct `pronunciation.json`, repeat P1. Lines `below similarity`: as in P4 | +| P1 | Author, project | `python3 tools/verify_speech.py --samples`. No line `voice them first:` printed: go to P2. Azure: `python3 tools/voice_azure.py --only `; a person listens to each of these lines (an assistant hands the file names under `public/audio/` to the author and waits); then `python3 tools/verify_speech.py --only `. ElevenLabs: no sample voicing exists; go to P2; P4 is the first key-term check | Each key term heard correctly; `0 with an unrecognized key term` | Wrong term: correct `pronunciation.json`, repeat P1. Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat P1; right: note the line in `review.md` | | P2 | Author, project | Decision: `python3 tools/profile.py --get speech.provider` and `python3 tools/profile.py --get speech.mode` | `azure` and `line`: P2a. `elevenlabs` and `scene`: P2b | Any other pair: stop; correct the profile; repeat Before starting | | P2a | Author, Azure | `python3 tools/voice_azure.py` | `N lines, … new` | Stops with `stopped at :`: go to R1 | | P2b | Author, connector | `python3 tools/split_takes.py prompts`; for each scene without a stored take, generate the printed passage once through the connector with the profile's voice, wait until it is done, and store it at once as `takes/-take1.mp3`; transcribe each take to `takes/-take1.words.json` (a list of `{"word", "start", "end"}`) with ; `python3 tools/split_takes.py` | `: (match 0.90 or higher), N lines` per scene | Generation error or timeout: go to R2. Take rejected: go to R3 | | P3 | Author, project | `python3 tools/timeline.py && python3 tools/measure.py pace` | `PASSED` | `text changed after voicing` or missing sound: repeat P2. `OUT OF RANGE`: go to R6 | -| P4 | Author, project | `python3 tools/verify_speech.py` | `0 with an unrecognized key term` | Unrecognized key term: correct `pronunciation.json`, repeat from P2. Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat from P2; right: note the line in `review.md`. ElevenLabs (P2b): the stored take keeps the wrong sound. After the author agrees (it is charged), generate one new take of the affected scene from `python3 tools/split_takes.py prompts `, store it as `takes/-take.mp3`, transcribe it to `takes/-take.words.json`, run `python3 tools/split_takes.py ` (see the take rule under Correction of a delivered video), then P3 and P4. Still wrong after that: stop and escalate | +| P4 | Author, project | `python3 tools/verify_speech.py` | `0 with an unrecognized key term` | Unrecognized key term: correct `pronunciation.json`, repeat from P2. Lines `below similarity`: a person listens to each; wrong: correct `pronunciation.json`, repeat from P2; right: note the line in `review.md`. ElevenLabs (P2b), instead of repeating from P2: the stored take keeps the wrong sound, and only a respelling (`speak` in `pronunciation.json` or in the line) changes the passage. After the correction and after the author agrees (it is charged), generate one new take of the affected scene from `python3 tools/split_takes.py prompts `, store it as `takes/-take.mp3`, transcribe it to `takes/-take.words.json`, run `python3 tools/split_takes.py ` (see the take rule under Correction of a delivered video), delete that scene's read-back word files `public/audio/-*.words.json`, then P3 and P4. Still wrong after that: stop and escalate | | P5 | Author, project | `node tools/stills.mjs stills`; look at every still; set `"inspected": true` for it in `stills/stills.json`, or write the defect into its `"finding"`; `python3 tools/check_review.py` | `N of N stills inspected without finding` | Defect: correct the scene, `node tools/stills.mjs stills `, repeat P5 for it | | P6 | Author | Decision: `python3 tools/profile.py --get render.host` | Prints a host and the render-machine check of Before starting succeeded: P6a. Empty, or the check fails: P6b | | | P6a | Author, render machine | `tools/render-remote.sh` | `done in … s: out/.mp4` | Interrupted or unreachable: go to R4 | @@ -129,9 +129,10 @@ step that changes a delivered file. | P9 | Reviewer | Gate 2 of `presentation-video-review` | `review.md`: `Gate 2: passed on ` | Picture, sound or file finding: return to the step P1 to P8 that owns it. Wording finding, tutorial: stop; `tutorial-video-scripting`; start again at Before starting. Wording finding, story: stop; the story scripting skill and the author, never `tutorial-video-scripting`; the changed line arrives from there; start again at Before starting | | P10 | Author, | Copy the file measured in P8 to `/.new` with ; compute on the local file and on the copy | Both checksums equal | Not equal: delete the `.new` copy and copy once more. Not equal again: stop and escalate; the delivered file is untouched | | P10b | Author, | Rename an existing `` to `.previous`, then `.new` to `` | `` has the checksum from P10 | Rename fails: rename `.previous` back; stop | -| P11 | Author, | Compute on `/` | Equals the local checksum | Not equal: restore `.previous`; stop and escalate | +| P11 | Author, | Compute on `/` | Equals the local checksum | Not equal: restore `.previous` (first delivery, none exists: rename `` to `.failed`); stop and escalate | -No step deletes working files. `public/audio/`, `src/audio-manifest.json`, +No step deletes working files other than the read-back word files named in +the take rule. `public/audio/`, `src/audio-manifest.json`, `src/timeline.json`, `takes/`, `out/captions.vtt`, `out/chapters.txt` and `review.md` are kept: a later correction reuses them. @@ -148,7 +149,10 @@ for it. Run Before starting, then P2 to P11. The line `split_takes.py` prints for the scene must name the new take. If it names an older take, move that scene's older takes with their word files to `takes/superseded/` (the tool reads only `takes/-take*.mp3`) and - run `python3 tools/split_takes.py ` again. + run `python3 tools/split_takes.py ` again. Before P4, delete that + scene's read-back word files `public/audio/-*.words.json`: an + unchanged line keeps its file name, and `verify_speech.py` does not + transcribe a sound file again while a word file with its name exists. - P5: `node tools/stills.mjs stills ` and inspect that scene's stills. A run without a scene resets every inspection mark. @@ -163,8 +167,8 @@ rerun; repeat only the transcription and `python3 tools/split_takes.py`. |---|---|---|---| | R1 | Voicing stopped | Message names the scene and line reached; finished files are in `public/audio/` | `still rate limited`: wait, run P2a again; finished lines are not requested again. `provider not reachable`: check the network, run P2a again. `the provider rejected the key` or `provider answered HTTP `: run `python3 tools/voice_azure.py --check`; still rejected: escalate. `credentials accepted`: run P2a once more; it stops again at the same line with the same message: stop and escalate with the output | | R2 | Connector generation failed or timed out | No file stored for the scene | Use to find the existing generation and fetch its result. Start a new generation only after the author agrees: each one is charged | -| R3 | Take rejected | `match` below the profile's threshold, or `line N was not heard` | The scene's existing sound is untouched. Listen to the take. After the author agrees (it is charged), request one new take, store it as `-take2` and transcribe it to `takes/-take2.words.json` with ; `python3 tools/split_takes.py ` chooses the better one. Second take rejected too: stop and escalate. Do not lower the threshold to pass | -| R4 | Render interrupted | The command ended without its last line (`rendered: …` or `done in …`). An `out/.mp4` from an earlier render is not the result | Delete `out/raw.mp4` and any earlier `out/.mp4` and `out/-with-opening.mp4`, then run P6 again. No partial file is a result | +| R3 | Take rejected | `match` below the profile's threshold, or `line N was not heard` | The scene's existing sound is untouched. Listen to the take. After the author agrees (it is charged), request one new take, store it under the next free number as `takes/-take.mp3` (never overwrite a take) and transcribe it to `takes/-take.words.json` with ; `python3 tools/split_takes.py ` chooses the better one. Second take rejected too: stop and escalate. Do not lower the threshold to pass | +| R4 | Render interrupted | The command ended without its last line (`rendered: …` or `done in …`). An `out/.mp4` from an earlier render is not the result | Delete `out/raw.mp4` and any earlier `out/.mp4` and `out/-with-opening.mp4`, then run P6 again; after P6a, first repeat the render-machine check of Before starting (it fails: P6b). No partial file is a result | | R5 | Delivery measurement failed | Last line `FAILED`; the failing row shows `FAIL` (file), `OUT OF RANGE` (pace) or `N outside ±` (pauses) | Pace: R6. Resolution or frame rate: correct `picture` in the profile, `python3 tools/timeline.py`, then P5 and P6. Duration or pauses: `python3 tools/timeline.py`, then P6 and P7. Loudness or peak: P6 once. Sound track: R4. With an opening: `python3 tools/measure.py all out/.mp4`. If `out/.mp4` passes and only the file with the opening fails, the opening sequence is the cause: correct that file, then P7. The same row fails a second time: stop and escalate; do not change a profile limit to pass | | R6 | Pace out of range | `pace … OUT OF RANGE [low, high]` in the output of `tools/measure.py` | Story video: never change wording, tone or order of a line. Azure: `python3 tools/probe_settings.py rate=` (short paid requests); keep the setting under `speech.settings` in the profile only if its last line is exactly `EFFECTIVE` (a line starting `NOT EFFECTIVE` means: do not keep it); then repeat P2 and P3 (a changed setting voices every line again). ElevenLabs: no pace setting is defined; stop and escalate. Tutorial only: a changed wording is produced by `tutorial-video-scripting`, then start again at Before starting (script check and gate 1 for the changed script). Story line that needs other words, or no effective setting: stop and return to the story scripting skill and the author. The pause value does not change the pace measurement. Changing `pace_wpm` in the profile or under `overrides` in `video.json` is an owner decision recorded in the project `README.md`, not a fix the operator chooses | @@ -183,7 +187,8 @@ rejected key, when P2a stops twice at the same line although `--check` accepts the credentials, when P2a stops with `still rate limited` on three separate runs, when a second take is rejected, when a copy mismatches twice, when the delivered file mismatches in P11, when a pace range cannot be met in R6 without an owner decision, when the -same measurement row fails twice, or when a key term stays wrong after a +same measurement row fails twice, when a line is still wrong in P4 after one +new take (ElevenLabs), or when a key term stays wrong after a phonetic entry, a language mark and a second voice. Preserve: the command output of the failed step, `out/measure.json`, the rejected take and its word file. Never include the env file or a key. diff --git a/tests/test_natural_evals.py b/tests/test_natural_evals.py new file mode 100644 index 0000000..d1b27a7 --- /dev/null +++ b/tests/test_natural_evals.py @@ -0,0 +1,197 @@ +"""Unscripted routing evaluations count loads, not mentions, and never steer the host.""" + +from __future__ import annotations + +import io +import json +import subprocess +import tempfile +import unittest +from contextlib import redirect_stderr +from pathlib import Path +from unittest import mock + +from scripts import run_host_evals as harness +from scripts import run_natural_evals as natural + + +def lines(*events): + return "\n".join(json.dumps(event) for event in events) + + +def claude_call(identity, name, arguments, content="ok", error=False): + return [ + {"type": "assistant", "message": {"content": [ + {"type": "tool_use", "id": identity, "name": name, "input": arguments}]}}, + {"type": "user", "message": {"content": [ + {"type": "tool_result", "tool_use_id": identity, "content": content, "is_error": error}]}}, + ] + + +DONE = {"type": "result", "subtype": "success", "is_error": False, "result": "done"} + + +class LoadedSkillTests(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.plugin = Path(self.directory.name) / "plugin" + skill = self.plugin / "skills/alpha-skill" + skill.mkdir(parents=True) + self.content = "---\nname: alpha-skill\ndescription: Use when testing.\n---\n\n# Alpha\n\nBody.\n" + (skill / "SKILL.md").write_text(self.content) + + def tearDown(self): + self.directory.cleanup() + + def test_claude_counts_successful_skill_calls_and_skill_file_reads(self): + text = lines( + *claude_call("a", "Skill", {"skill": "writing-method:story-drafting"}), + *claude_call("b", "Read", {"file_path": "/x/skills/alpha-skill/SKILL.md"}), + *claude_call("c", "Skill", {"skill": "writing-method:story-drafting"}), + DONE) + self.assertEqual(["story-drafting", "alpha-skill"], natural.loaded_skills("claude", text, [self.plugin])) + + def test_claude_ignores_failed_calls_and_paths_inside_file_content(self): + text = lines( + *claude_call("a", "Skill", {"skill": "beta-skill"}, error=True), + *claude_call("b", "Read", {"file_path": "/x/policy.md"}, + content="see skills/alpha-skill/SKILL.md for details"), + {"type": "assistant", "message": {"content": [ + {"type": "text", "text": "I would use skills/alpha-skill/SKILL.md"}]}}, + DONE) + self.assertEqual([], natural.loaded_skills("claude", text, [self.plugin])) + + def test_codex_counts_a_complete_read_only(self): + def command(text, output, code=0): + return {"type": "item.completed", "item": {"type": "command_execution", "command": text, + "aggregated_output": output, "exit_code": code}} + path = ".agents/skills/alpha-skill/SKILL.md" + complete = lines(command(f"/bin/zsh -lc 'cat {path}'", self.content), {"type": "turn.completed"}) + self.assertEqual(["alpha-skill"], natural.loaded_skills("codex", complete, [self.plugin])) + listing = lines(command("/bin/zsh -lc 'rg --files .agents/skills'", path + "\n"), + command(f"/bin/zsh -lc 'cat {path}'", self.content, code=1), + command(f"/bin/zsh -lc 'head -2 {path}'", "---\nname: alpha-skill\n"), + {"type": "turn.completed"}) + self.assertEqual([], natural.loaded_skills("codex", listing, [self.plugin])) + + def test_unfinished_or_failed_transcripts_are_failures(self): + with self.assertRaisesRegex(harness.EvalFailure, "incomplete_transcript"): + natural.loaded_skills("claude", lines(*claude_call("a", "Skill", {"skill": "x"})), [self.plugin]) + with self.assertRaisesRegex(harness.EvalFailure, "host_error"): + natural.loaded_skills("codex", lines({"type": "turn.failed"}), [self.plugin]) + with self.assertRaisesRegex(harness.EvalFailure, "malformed_jsonl"): + natural.loaded_skills("claude", "{not json", [self.plugin]) + + def test_claude_counts_a_complete_shell_read(self): + path = str(self.plugin / "skills/alpha-skill/SKILL.md") + text = lines(*claude_call("a", "Bash", {"command": f"cat {path}"}, content=self.content), DONE) + self.assertEqual(["alpha-skill"], natural.loaded_skills("claude", text, [self.plugin])) + + def test_judgement(self): + expected = {"any": ["a", "b"], "prohibited": ["z"]} + self.assertTrue(natural.judge(["b", "other"], expected)) + self.assertFalse(natural.judge(["other"], expected)) + self.assertFalse(natural.judge(["a", "z"], expected)) + # A case that expects no skill hits only when nothing was loaded. + self.assertTrue(natural.judge([], {"any": []})) + self.assertFalse(natural.judge(["a"], {"any": []})) + + +class CommandTests(unittest.TestCase): + def test_prompt_is_sent_unchanged_with_every_plugin_and_read_only_tools(self): + with tempfile.TemporaryDirectory() as directory: + config = Path(directory) + plugins = [Path("/p/one"), Path("/p/two")] + command, env = natural.host_command("claude", Path("/repo"), plugins, config, "sonnet", "medium", + "Schreib bitte Kapitel 12.", native_auth=True) + self.assertEqual("Schreib bitte Kapitel 12.", command[-1]) + self.assertEqual(["/p/one", "/p/two"], + [command[i + 1] for i, word in enumerate(command) if word == "--plugin-dir"]) + self.assertEqual(natural.READ_ONLY_TOOLS, command[command.index("--allowedTools") + 1]) + self.assertNotIn("--json-schema", command) + self.assertNotIn("ANTHROPIC_API_KEY", env) + command, env = natural.host_command("codex", Path("/repo"), plugins, config, "m", None, + "Plain request.", auth_home=Path("/auth")) + self.assertEqual("Plain request.", command[-1]) + self.assertEqual("read-only", command[command.index("--sandbox") + 1]) + self.assertNotIn("--output-schema", command) + # The plugins load as installed, so the user's configuration stays in effect. + self.assertNotIn("--ignore-user-config", command) + self.assertEqual("/auth", env["CODEX_HOME"]) + for instruction in (harness.ROUTING_INSTRUCTION, harness.SKILL_DISCOVERY_INSTRUCTION): + self.assertNotIn(instruction.strip()[:40], " ".join(command)) + + def test_repository_holds_the_fixture_and_no_skill_links(self): + with tempfile.TemporaryDirectory() as directory: + base = Path(directory) + natural.prepare_repo(base / "repo", {"files": {"docs/a.md": "x"}}) + self.assertEqual("x", (base / "repo/docs/a.md").read_text()) + self.assertFalse((base / "repo/.agents").exists()) + with self.assertRaisesRegex(harness.EvalFailure, "fixture_path_escape"): + natural.prepare_repo(base / "repo3", {"files": {"../out.md": "x"}}) + + def test_codex_uses_the_installed_copy_only_when_its_skills_equal_the_source(self): + with tempfile.TemporaryDirectory() as directory: + base = Path(directory) + source, home = base / "source", base / "home" + (source / "skills/alpha").mkdir(parents=True) + (source / "skills/alpha/SKILL.md").write_text("current") + (source / ".codex-plugin").mkdir() + (source / ".codex-plugin/plugin.json").write_text(json.dumps({"name": "demo"})) + with self.assertRaisesRegex(harness.EvalFailure, "plugin_not_installed:demo"): + natural.installed_codex_plugins([source], home) + installed = home / "plugins/cache/market/demo/1.0.0" + (installed / "skills/alpha").mkdir(parents=True) + (installed / "skills/alpha/SKILL.md").write_text("older") + with self.assertRaisesRegex(harness.EvalFailure, "installed_plugin_differs:demo"): + natural.installed_codex_plugins([source], home) + (installed / "skills/alpha/SKILL.md").write_text("current") + self.assertEqual([installed], natural.installed_codex_plugins([source], home)) + + +class RunTests(unittest.TestCase): + def test_a_case_reports_hits_over_runs_and_never_stops_early(self): + observed = iter([(["a"], "m"), harness.EvalFailure("timeout"), (["other"], "m")]) + + def once(*args, **kwargs): + value = next(observed) + if isinstance(value, Exception): + raise value + return value + + with tempfile.TemporaryDirectory() as directory, \ + mock.patch.object(natural, "run_once", side_effect=once), \ + mock.patch.object(harness, "fingerprint", return_value="same"): + result = natural.run_case("claude", {"id": "c"}, {"any": ["a"]}, Path(directory), 3, 1, 1.0, + [Path(directory)]) + self.assertEqual((3, 1, "passed"), (result["runs"], result["hits"], result["status"])) + self.assertEqual([True, False, False], [o["hit"] for o in result["observations"]]) + self.assertEqual("timeout", result["observations"][1]["failure"]) + + def test_arguments_are_checked_before_any_host_runs(self): + with tempfile.TemporaryDirectory() as directory: + plugin = Path(directory) / "plugin" + (plugin / "skills").mkdir(parents=True) + base = ["--host", "claude", "--plugin-root", str(plugin)] + for extra in (["--output-dir", str(plugin / "out")], + ["--output-dir", str(Path(directory) / "out")], + ["--output-dir", str(Path(directory) / "out"), "--runs", "0"], + ["--output-dir", str(Path(directory) / "out"), "--runs", "2", "--min-hits", "3"], + ["--output-dir", str(Path(directory) / "out"), "--with-plugin", directory]): + with self.subTest(extra=extra), redirect_stderr(io.StringIO()), self.assertRaises(SystemExit): + natural.main(base + extra) + with redirect_stderr(io.StringIO()), self.assertRaises(SystemExit): + natural.main(["--host", "codex", "--native-auth", "--output-dir", str(Path(directory) / "o"), + "--plugin-root", str(plugin)]) + + +class HostProcessTests(unittest.TestCase): + def test_host_process_never_inherits_standard_input(self): + # An inherited open stdin makes a host wait for more input: the run then times out with no event. + output = harness.execute(["python3", "-c", "import sys; print(repr(sys.stdin.read()))"], + Path.cwd(), dict(__import__("os").environ), 20) + self.assertEqual("''", output.strip()) + + +if __name__ == "__main__": + unittest.main() From 1d9c3f0aee48e48126988a6019837a3070b35c25 Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:32:21 +0100 Subject: [PATCH 6/8] fix(routing): explaining-concepts also names short what-is and how-differs questions Unscripted concept question on Claude: 1 of 3 loads before, 3 of 3 after. Co-Authored-By: Claude Fable 5.2 --- skills/explaining-concepts/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/explaining-concepts/SKILL.md b/skills/explaining-concepts/SKILL.md index 758b953..46bc02e 100644 --- a/skills/explaining-concepts/SKILL.md +++ b/skills/explaining-concepts/SKILL.md @@ -1,6 +1,6 @@ --- name: explaining-concepts -description: "Use when a technical concept, mechanism, or design must be explained to a reader who lacks the writer's expertise, in documentation or an ordinary reply. Introduces each concept with an everyday analogy before the mechanism and removes undefined jargon. Not for step-by-step instructions, reference tables, or error diagnosis. Not for story manuscripts or in-world text." +description: "Use when a technical concept, mechanism, or design must be explained to a reader who lacks the writer's expertise, in documentation or an ordinary reply: also for a short answer to a question such as what something is, how it works or how two things differ, and whenever the person says the topic is new to them. Introduces each concept with an everyday analogy before the mechanism and removes undefined jargon. Not for step-by-step instructions, reference tables, or error diagnosis. Not for story manuscripts or in-world text." --- # Explaining Concepts From 23b38bd0899e21bdc03b82a3d11b564d922331cb Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:52:43 +0100 Subject: [PATCH 7/8] release: prepare 0.4.1 with unscripted evaluations and the reviewed video playbook Syncs method-core 64d9a02, records the measured unscripted runs and bumps the version. Co-Authored-By: Claude Fable 5.2 --- .claude-plugin/plugin.json | 2 +- .codex-plugin/plugin.json | 2 +- README.md | 7 +++-- docs/specs/WM-003-story-family/rehearsal.md | 33 +++++++++++++++++++++ docs/usage.md | 2 +- method-core.lock.json | 6 ++-- method.json | 2 +- pyproject.toml | 2 +- scripts/run_natural_evals.py | 21 ++++++++++--- tests/test_natural_evals.py | 3 ++ 10 files changed, 65 insertions(+), 15 deletions(-) diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index f857997..69db026 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "writing-method", - "version": "0.4.0", + "version": "0.4.1", "description": "Reader-oriented writing, documentation workflows, local PDF delivery and narrated tutorial videos, plus an experimental story-writing family kept separate from them.", "author": { "name": "Peter Hense" diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index 0252561..71e0061 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "writing-method", - "version": "0.4.0", + "version": "0.4.1", "description": "Reader-oriented writing, documentation workflows, local PDF delivery and narrated tutorial videos, plus an experimental story-writing family kept separate from them.", "author": { "name": "Peter Hense" diff --git a/README.md b/README.md index 9ea256e..e8756aa 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ --- type: reference audience: Authors and maintainers familiar with their coding assistant and Git -version: 0.4.0 +version: 0.4.1 --- # Writing Method @@ -102,8 +102,9 @@ No video has been produced with this path. How invented names are pronounced and - **Routing is statistical.** The tests prove that names, exclusion sentences and links are in place. Whether an assistant then picks the right family for a request is a probability, not a guarantee. - **Produced text is not measured.** No check reads a chapter for documentation habits, or a reply for a story's voice. - **No story pilot exists.** The skills are structurally checked and have not been proven on a real story. Light-novel figures such as volume length are left unset, not asserted. +- **The video playbook is reviewed, not rehearsed.** The production playbook template went through three independent content reviews; all 43 findings are applied. The corrected text has not been run end to end, and no person has done the listening check. - **No story video exists.** The script check and the routing expectations for story videos are tested; nothing has been voiced or rendered from a story script. -- **Host routing is checked once, not measured as a rate.** In the last full run each of the 24 routing cases under `evals/shared/triggers/` ran once per host: Claude passed 24 of 24, Codex 22 of 24. The two Codex failures, repeated twice, are a marketing clip correctly sent to focused editing with the video production skill reported as supporting (1 of 3 passed), and a draft request in a study vault where Codex names `story-drafting` and not `story-craft-research` as primary (0 of 3 passed); the harness cannot see that the draft is declined. Eleven unscripted German requests without any skill name loaded the expected skill in 9 of 11 cases on Claude and 11 of 11 on Codex, once each. The record, including three expectations or prompts changed after seeing a result, is in `docs/specs/WM-003-story-family/rehearsal.md`. OpenCode was not exercised. +- **Host routing is checked once, not measured as a rate.** In the last full run each of the 24 routing cases under `evals/shared/triggers/` ran once per host: Claude passed 24 of 24, Codex 22 of 24. The two Codex failures, repeated twice, are a marketing clip correctly sent to focused editing with the video production skill reported as supporting (1 of 3 passed), and a draft request in a study vault where Codex names `story-drafting` and not `story-craft-research` as primary (0 of 3 passed); the harness cannot see that the draft is declined. After two corrections both Codex cases passed 3 of 3. Twelve unscripted German requests without any skill name, three runs each with both plugins loaded, hit the expected skill in 33 of 36 runs on Claude and 35 of 36 on Codex; they can be repeated with `scripts/run_natural_evals.py`. The record, including three expectations or prompts changed after seeing a result, is in `docs/specs/WM-003-story-family/rehearsal.md`. OpenCode was not exercised. The family originated in a private research workspace. That workspace's research notes and its studies of existing works are not part of this plugin. @@ -123,7 +124,7 @@ claude plugin install writing-method@writing-method For a session-scoped Claude test use `claude --plugin-dir /absolute/path/to/writing-method`. For OpenCode, run `python3 scripts/install-opencode --check` from this checkout, inspect the proposed path, then repeat without `--check`. It installs an owned `writing-method.js` loader pointing to this retained directory. Gemini support is deferred until its adapter is verified. -Restart the assistant session after installation and inspect its plugin inventory for `writing-method` 0.4.0. An installed listing proves discovery; a live authoring request is separate behavior evidence. With both methods installed, use the invoking method's namespace for shared support, such as `writing-method:playbook-authoring`, and run one copy. +Restart the assistant session after installation and inspect its plugin inventory for `writing-method` 0.4.1. An installed listing proves discovery; a live authoring request is separate behavior evidence. With both methods installed, use the invoking method's namespace for shared support, such as `writing-method:playbook-authoring`, and run one copy. ## Project state and updates diff --git a/docs/specs/WM-003-story-family/rehearsal.md b/docs/specs/WM-003-story-family/rehearsal.md index 5a56e57..a7387b7 100644 --- a/docs/specs/WM-003-story-family/rehearsal.md +++ b/docs/specs/WM-003-story-family/rehearsal.md @@ -128,6 +128,39 @@ named in the plan in 4 more. A request for a plan lowers loading, not the choice. Claude ran from the working tree, Codex from the installed plugin copy of the same revision. +## Corrections and measured runs after release 0.4.0 (2026-11-13) + +**Routing harness, Codex.** Two corrections at `c193988`: the instruction file of a study vault names `story-craft-research` as the skill for every request, and `presentation-video-production` states that filmed material, marketing clips and short-form video are outside the video method. Both cases then passed 3 of 3 on Codex (`marketing-clip-out-of-scope` before: 1 of 3; `study-vault-draft-request` before: 0 of 3). The fixture of the study-vault case received the same sentence as the vault template, after the result was known. + +**Earlier Codex timeouts.** The runs that ended with `timeout` and no event had a cause in the harness: the host inherited an open standard input and waited for more input. Method Core `5d8519b` closes it. + +**Unscripted requests, measured.** The probe became `scripts/run_natural_evals.py` (Method Core). Nine cases of this plugin under `evals/natural/` and three of Engineering Method ran three times per host with both plugins loaded, the hosts' coordinator models (`fable`, `gpt-6-astra`), and on Codex the plugins as installed. A hit is a run in which an expected skill was loaded and no prohibited one. + +| Request | Expected | Claude | Codex | +|---|---|---|---| +| An administrator guide | `documentation-authoring` or `writing-procedures` | 3 of 3 | 3 of 3 | +| The difference between two Git commands | `explaining-concepts` | 1 of 3 | 3 of 3 | +| Start a new story | `story-bootstrap` | 3 of 3 | 3 of 3 | +| "Write chapter 12" in a story vault | `story-drafting` | 3 of 3 | 3 of 3 | +| Does chapter 12 still fit what happened before | `story-canon-review` | 3 of 3 | 3 of 3 | +| Chapters no longer sound like the beginning | `story-voice-review` | 3 of 3 | 3 of 3 | +| An audiobook video of chapter 11, read as written | `story-video-scripting` | 3 of 3 | 3 of 3 | +| Two sentences changed for the audio version only | `story-video-scripting` | 3 of 3 | 3 of 3 | +| A tutorial video for new employees | `tutorial-video-scripting` | 3 of 3 | 3 of 3 | +| A failing test (Engineering Method) | `systematic-debugging` | 2 of 3 | 3 of 3 | +| A feature across three modules (Engineering Method) | a specification skill | 3 of 3 | 3 of 3 | +| A typing error in a README (Engineering Method) | no skill | 3 of 3 | 2 of 3 | + +Totals: Claude 33 of 36 runs, Codex 35 of 36. Three things were changed after seeing results and are part of these figures: + +- The guide case first expected `documentation-authoring` only. Claude loaded `writing-procedures` in all three runs, which is a correct skill for a how-to; the expectation now accepts either. +- The Codex figures are a re-scoring of the saved transcripts. The runner first counted a skill read only when the command was a plain `cat`; Codex reads skills inside compound commands. Eight runs scored as misses contained a complete read of the expected skill (27 of 36 before, 35 of 36 after; Method Core `64d9a02`). +- A first measurement used the routing-role models and, on Codex, repository skill links. It was discarded: the smaller Claude model loaded skills far less often, and Codex looked for paths that did not exist. + +After the measurement the description of `explaining-concepts` was widened to name short questions such as what something is or how two things differ. The concept question then loaded the skill in 3 of 3 runs on Claude (`1d9c3f0`; before: 1 of 3). No other description was changed. + +Three runs per case show a tendency, not a rate, and nothing here reads the produced text. + ## Not covered - The earlier cases ran once per host; apart from `presentation-video-production` diff --git a/docs/usage.md b/docs/usage.md index 6c132e5..fda890f 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1,7 +1,7 @@ --- type: reference audience: Authors using Writing Method -version: 0.4.0 +version: 0.4.1 --- # Writing examples diff --git a/method-core.lock.json b/method-core.lock.json index f882670..9cb3ea1 100644 --- a/method-core.lock.json +++ b/method-core.lock.json @@ -118,7 +118,7 @@ { "mode": "100644", "path": "scripts/run_natural_evals.py", - "sha256": "ace027082e971f661b3c7fa752037f8677e4a6abdab75ba22fb1f887d02e9fb5" + "sha256": "85eac53c236601a6aa22a0c039d1e8a1e8eb9a9c245289676d6db3fef7afc174" }, { "mode": "100755", @@ -333,7 +333,7 @@ { "mode": "100644", "path": "tests/test_natural_evals.py", - "sha256": "68e3d9c0b25622116e452c551e67a6d8f312e7b4f6f14c04a09a948a7ecb8959" + "sha256": "231e8d4698e1adfae7b2a4a35ded6965dced6563b47df2506254cd1e2995fd69" }, { "mode": "100644", @@ -362,7 +362,7 @@ } ], "repository": "https://github.com/phense/method-core.git", - "revision": "91e35dab40156f619123bd75d22de81534f4121c", + "revision": "64d9a02272590630288c04eba3a76c7bdb05919f", "schema_version": 1, "version": "0.1.0" } diff --git a/method.json b/method.json index 32c2d9c..aa380f3 100644 --- a/method.json +++ b/method.json @@ -1,7 +1,7 @@ { "schema_version": 1, "name": "writing-method", - "version": "0.4.0", + "version": "0.4.1", "method_skills": [ "documentation-authoring", "documentation-pdf", diff --git a/pyproject.toml b/pyproject.toml index 1d13219..b07d4b9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "writing-method" -version = "0.4.0" +version = "0.4.1" description = "Shared method support." requires-python = ">=3.11" license = { text = "MIT" } diff --git a/scripts/run_natural_evals.py b/scripts/run_natural_evals.py index c91ed29..4556816 100644 --- a/scripts/run_natural_evals.py +++ b/scripts/run_natural_evals.py @@ -36,6 +36,21 @@ def bare(name): return name.rsplit(":", 1)[-1] if isinstance(name, str) else "" +def skills_in_output(output, plugins): + """Skills whose complete SKILL.md is in a command's output, whatever the command looked like. + + Hosts read a skill inside compound commands (`pwd; rg ...; cat `). The + whole file content proves the read; a path or a partial read does not. + """ + found = [] + for plugin in plugins: + for path in sorted((plugin / "skills").glob("*/SKILL.md")): + content = path.read_text().rstrip("\n") + if content.strip() and content in output and path.parent.name not in found: + found.append(path.parent.name) + return found + + def events(text): found = [] for line in text.splitlines(): @@ -73,8 +88,7 @@ def add(names): and item.get("type") == "command_execution" and item.get("exit_code") == 0: command, output = item.get("command"), item.get("aggregated_output", "") if isinstance(command, str) and isinstance(output, str): - for plugin in plugins: - add(sorted(harness.observed_read_skills(command, output, plugin))) + add(skills_in_output(output, plugins)) continue if kind == "result": if event.get("is_error") or event.get("subtype") != "success": @@ -98,8 +112,7 @@ def add(names): output = block["content"] if isinstance(output, list): output = "\n".join(str(part.get("text", "")) for part in output if isinstance(part, dict)) - for plugin in plugins: - add(sorted(harness.observed_read_skills(str(arguments.get("command", "")), str(output), plugin))) + add(skills_in_output(str(output), plugins)) if not terminal: raise EvalFailure("incomplete_transcript", text) return order diff --git a/tests/test_natural_evals.py b/tests/test_natural_evals.py index d1b27a7..a016125 100644 --- a/tests/test_natural_evals.py +++ b/tests/test_natural_evals.py @@ -68,6 +68,9 @@ def command(text, output, code=0): path = ".agents/skills/alpha-skill/SKILL.md" complete = lines(command(f"/bin/zsh -lc 'cat {path}'", self.content), {"type": "turn.completed"}) self.assertEqual(["alpha-skill"], natural.loaded_skills("codex", complete, [self.plugin])) + compound = lines(command(f"/bin/zsh -lc 'pwd; rg --files -g AGENTS.md; cat {path}'", "/repo\n" + self.content), + {"type": "turn.completed"}) + self.assertEqual(["alpha-skill"], natural.loaded_skills("codex", compound, [self.plugin])) listing = lines(command("/bin/zsh -lc 'rg --files .agents/skills'", path + "\n"), command(f"/bin/zsh -lc 'cat {path}'", self.content, code=1), command(f"/bin/zsh -lc 'head -2 {path}'", "---\nname: alpha-skill\n"), From 0f541f84d771eb2b69a672d85145728bfd643e94 Mon Sep 17 00:00:00 2001 From: Peter Hense Date: Fri, 13 Nov 2026 20:55:24 +0100 Subject: [PATCH 8/8] docs(backlog): refresh the issue cache; WM-001 blocked on rehearsal and listening check Co-Authored-By: Claude Fable 5.2 --- BACKLOG.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/BACKLOG.md b/BACKLOG.md index 6ad520b..7882d82 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -13,6 +13,15 @@ GitHub Issues are canonical. Refresh this cache; do not make local canonical mut ## Tasks - -- 🔄 `WM-001` **P1** Narrated presentation videos: scripting, production and review skills - - Notes: Spec approved 2026-11-13. Skills, guides, project template, credential examples and tests implemented on branch wm-001-presentation-video (uncommitted); 235 tests pass; rehearsed end to end on the sample video (docs/specs/WM-001-presentation-video/rehearsal.md). Open: maintainer listening check, playbook review, live host trigger evaluations, commit and pull request. + +- ❌ `WM-001` **P1** Narrated presentation videos: scripting, production and review skills + - Notes: Skills, tools and template are released (0.3.0 to 0.4.1). Done since: host trigger evaluations (the four video cases pass on Claude and Codex) and three independent playbook reviews with 43 findings applied (docs/specs/WM-001-presentation-video/playbook-review.md). Blocked on work only the maintainer can do: one end-to-end rehearsal of the corrected playbook through gate 2 and delivery on a machine with ffmpeg and a configured speech provider, and the listening check of the sample by a person. Then one more playbook-review at that revision. + +- ✅ `WM-003` **P1** Story writing as a separate skill family + - Notes: Released in 0.4.0 (PR #8): nine story- skills, shared story rules, story vault starter, writing mode per text. Experimental; no story written with it yet. + +- ✅ `WM-002` **P2** Book layout for long manuals in documentation-pdf + - Notes: Released in 0.3.0: book layout for long manuals in documentation-pdf. + +- ✅ `WM-004` **P2** Narrated story videos (audiobook with images) from a story vault + - Notes: Released in 0.4.0 (PR #8): story-video-scripting, verbatim and listening editions, form skills for both families. Experimental; no story video produced.