From 1d4e578b7af100bc855f6969fab4f9ee5768bf5e Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 16:57:50 -0600 Subject: [PATCH 001/405] docs(proposals): long-form media design - one approach, measured levers, quality-first defaults Design for making long pieces (chains, cuts, long audio) easy for agents: a playbook skill, per-shot-type defaults, per-segment checks with bounded re-rolls, a shared long-take bench, and a verdict for every open chain and long-media idea (#1187-#1197 and the parked chain family). Co-Authored-By: Claude Opus 5.5 --- docs/proposals/long-form-media.md | 330 ++++++++++++++++++++++++++++++ 1 file changed, 330 insertions(+) create mode 100644 docs/proposals/long-form-media.md diff --git a/docs/proposals/long-form-media.md b/docs/proposals/long-form-media.md new file mode 100644 index 000000000..ffbf466f5 --- /dev/null +++ b/docs/proposals/long-form-media.md @@ -0,0 +1,330 @@ +# Long-form media: one approach, measured levers, quality-first defaults + +_Design, 2026-11-03. Written by `claude-opus-5-5` (anthropic) with Don. This is the +design a plan is built from; the plan and its stage issues follow it._ + +## Intent + +An agent asked for a long piece (a minute, five minutes, an episode, a music video, +a long narrated walk) treats the model's 15-second clip as a render detail, not a +limit. It blocks and storyboards the piece, gives each scene a length that suits the +story, picks how each scene gets past the clip ceiling, and handles what goes wrong +while it runs. The person asking never has to know what a seam, a guide or an anchor +is. + +**Why now.** Don's own long takes (the constitution walk) worked, but smaller tests +and many community reports show the same struggles: drift that builds segment by +segment, visible seams, a bad segment that forces a full re-roll, and a soundtrack +that breaks at every join. Many ComfyUI packs each solve one slice of this +(roycho87/NerdNotes and geiru_context_suite, Smite79/MiniMax-H3-LongVideos, H3MD +Director, PrefixStream, Kinoloom, leonardofrito/Directors-Cut). dw should be the +best-in-breed version: those ideas folded into measured defaults and agent judgment, +not a shelf of switches. + +**Philosophy (Don's):** common things made easy, while the uncommon remains possible. +Defaults change wherever the evidence supports it; every knob stays. + +## Goals + +1. **One playbook an agent reads before planning anything longer than one clip.** For + each scene it says whether to cut, chain, bridge or render one long take; which + continuity mode and drift counters fit which kind of shot (static, moving camera, + dialogue, walk, montage); how the soundtrack stays continuous; what each lever + costs; and the evidence behind each rule. +2. **Planning carries length and continuity.** The storyboard, script-to-video and + series-episodes skills give every scene a target length and a continuity choice, + so an agent's plan states its levers and its cost before it spends. +3. **Defaults that do the right thing.** A plain request for a long take gets the + measured-best continuity, the drift counters that suit its shot type, and a + quality-first tier, without the requester naming any of them. A chain can be asked + for in seconds rather than segment counts. +4. **Handled during the run.** A chain checks each segment as it finishes, re-rolls a + segment that fails its check within a retry budget quoted up front, and a finished + chain can have one bad segment repaired in place without redrawing the rest. +5. **Quality against speed is one dial.** Defaults favour quality. The playbook says + how to move the dial toward speed and what each step costs in quality. Any speedup + that measures with no marked quality loss becomes the default. +6. **Every idea measured on one bench.** The eleven ideas filed 2026-11-03 (#1187 to + #1197), the parked chain ideas (#1030, #1032, #1033, #1044, #1067, #1150, #1169), + the open speed and LoRA candidates (#980, the anti-burn trial from #1151) and the + shipped counters (#1166) are scored on the same long-take bench. Winners are built + or become defaults; losers are recorded with their numbers and closed. +7. **Verifiable without code.** Every stage lands with acceptance intent a tester can + check over MCP alone, plus a design record, so the harnest tester can verify after + this work is declared done. + +## Non-goals + +- **Kandinsky 6.** Out of scope (Don: not impressed). Its skill keeps saying "cut". +- Training anything (LoRAs, lifters, distilled models). +- Code from licence-restricted sources. GPL, AGPL and no-licence repos contribute + concepts only. MIT sources may contribute code with attribution in + `docs/ATTRIBUTIONS.md`. +- A general video editor. Timeline export (#1197) hands a cut to Resolve; dw does + not grow transitions, effects or ripple editing. +- New closed-model dependencies. + +## Anti-goals (failure even if it technically works) + +- **A pile of opt-in switches only an expert would know to set.** If an idea wins, it + becomes default behaviour for the shots it suits, or it is chosen by the playbook + from the shot type an agent already records. An idea that only helps when hand-tuned + is documented as an uncommon knob, not promoted. +- A default that measures better on one scene and worse on another, shipped + everywhere. Defaults are by shot type, or not at all. +- A long piece whose cost surprises the requester: every automatic re-roll and every + quality-tier choice is inside the quoted estimate. +- Rules in the playbook that were never measured on our stack, stated as rules. + Community claims are marked as such until the bench confirms them. + +## Constraints + +- **Families in scope:** MiniMax H3 (video and audio), LTX-2.5 (video and audio), + MiniMax Music 3 (music), the TTS templates (`generate-speech`, `voice-clone`), FLUX 3 + Klein for boards. The domains around them: storyboarding and blocking, + script-to-video, series-episodes, assembly and scoring, cinematography. +- **Model knowledge lives in `plugins/dw/` and the catalog, never in engine code** + (CLAUDE.md). A per-shot-type default is a template default or a playbook rule, or an + engine behaviour driven by something the workflow declares or the engine measures, + never a hard-coded model fact. +- **Skill size caps.** The H3 skill is at its byte cap (12 KiB). The playbook is a new + skill with its own budget, and the family skills point to it rather than growing. + `tests/test_plugin_skills.py` pins skill names and budgets. +- **Security rules** in CLAUDE.md apply to every new argument that names media + (`dw/locations.py` policy, validators). +- **Release:** PR #1198 (0.19.0, develop to master) is open. This work lives on + `feat/long-form-media` and its stage branches, and merges to `develop` only after + #1198 merges, so the release is not disturbed. Until then the bench runs on its own + server instance (see *Decided by Don*). +- **Hardware:** lem (two RTX 3090, 24 GB each, shared host RAM). The bench numbers are + lem numbers; MPS gets the ordering, not the minutes. + +## Decided by Don + +| Decision | Ruling | +|---|---| +| Who builds, deploys and bench-tests | This session and its subagents, on lem, including deploys. The harnest tester takes over after this work is declared done. | +| GPU budget | As many lem hours as needed, both cards. | +| Calls of judgement | Decide by measurement where the numbers settle it. Ask Don when unsure, unable to decide, or when a question is pure taste; bring a recommendation either way. | +| Defaults and templates | May change wherever the evidence supports it. Keep every knob. | +| Scope | H3, LTX-2.5, long audio (songs, narration, ambience beds) and the related domains. Not Kandinsky 6. | +| Run-time re-rolls | Allowed: a segment that fails its check may be re-rolled automatically, within a retry budget stated in the up-front quote. | +| Quality against speed | Defaults favour quality, with instructions for moving the dial. Speedups with no marked quality loss become defaults. | +| Cost ceiling for `standard` | Up to about 6x today's H3 chain default (4-step turbo, int4, 960x544, ~2 min per segment) per minute of output. That leaves room for the base model at 1344x768 (~12 min per segment), which the dialogue rule already needs. `best` has no ceiling. | +| Bench server during the 0.19.0 release window | A separate bench `dw.serve` on lem from a git worktree of this work's branch, on its own port and its own card. The main server stays on `develop` for the harnest tester and the release preflight. Both cards go to the bench once #1198 merges. | +| Approvals and labels | Approving this design and its plan covers every stage, engine and syntax changes and default moves included. Stage issues are filed at hand-off, so the loop cannot race the build. This session closes losing ideas itself, with their numbers. | +| Done | The playbook, the defaults and the run-time behaviours have shipped, and every idea has a measured verdict. A large build that passes its gate (for example #1188 hybrid windows or #1194 performer replacement) may land as its own follow-up feature after the hand-off. | + +## The approach + +### 1. The lever map (what the playbook teaches) + +The playbook is organised as a decision per scene, then per seam, then per run. + +**Per piece.** Split the piece into scenes the way a storyboard would (a scene is one +place and one continuous moment). Give each scene a target length from the story, +not from the model. The model ceiling only decides how a scene is rendered. + +**Per scene, the rendering shape:** + +| Shape | When | What exists today | +|---|---|---| +| One clip | ≤ 14.4 s (H3), ≤ ~17 s (LTX at 768) | the family templates | +| Cuts | the scene changes place, time or angle, or a story beat allows a cut | `dialogue-short`, `music-video`, `shots-batch` + `assemble-and-score` | +| Chain | one action, line or camera move must cross the ceiling unbroken | `minimax/chained-segments`, `chain-video-continuity`, `chain-matched-*`; `ltx2/chained-segments`, `ltx2/extend-clip` | +| Bridge / lead-in | a cut must land on a fixed frame | `first-and-last-frame`, `last-frame-only` (joins.md) | +| One long render | a continuous take with no seams at all | not built (#1188 is the candidate) | + +**Per chain, by shot type** (the cell values are what the bench decides; the rows are +the shot types the playbook classifies): + +| Shot type | Continuity | Drift counters | Evidence today | +|---|---|---|---| +| Static or locked-off | guide | latent anchor (full or partial) + ? | #1175 static: anchor best, B+C cleanest | +| Moving camera (arc, dolly, track) | guide | guide noise; anchor weak or off | #1175 arc: anchor fails, guide noise only arm that doesn't hurt | +| Dialogue / voice across seams | guide with `carry_audio`, or `last_segment` | ? | guide A/B held seams better (h3-guides-complete) | +| Walk / steady subject motion | guide | ? | NerdNotes walk (community) | +| Driven by a soundtrack | `match_audio` chains | ? | the matched templates | + +The shot type comes from what the agent already records per shot (cinematography's +`move`: `static` against a camera move; whether the shot has dialogue). Where the +engine can measure it more reliably than the agent can declare it (segment 1's +motion), the bench decides whether "auto" means measured. + +**Per seam, during the run:** the check each segment must pass, what a failure looks +like (`analyze_seams` jump and look ratios, texture-band drift against segment 1, +`face_drift` against the cast portrait, `check-script` for lines), and the repair +ladder: re-roll that segment, then repair it in place between its neighbours, then +make the seam a cut. + +**Per piece, the soundtrack:** one bed under cut scenes (`assemble-and-score`, +`find_loop_bed`), voice carried across chain seams (`carry_audio`), music longer than +one Music 3 track (300 s practical ceiling; the join recipe), narration longer than +one TTS take (`voice-clone`'s ~95 s context; takes joined), and replacing or adding a +take's sound without redrawing its picture (#1192, if it wins). + +### 2. The quality-against-speed rubric + +Every recommendation the playbook makes carries a tier. The tiers are named once and +used by every family: + +- **draft:** approve blocking, timing, head count and line delivery. Cheapest path + whose staging carries to the final (the H3 *Ramp* rule: a draft only carries on its + own path). +- **standard:** the default for a deliverable. Quality-first, at a cost the bench shows + is reasonable for a long piece. +- **best:** the highest measured quality, cost no object. + +Rules: + +- Defaults are **standard**, chosen quality-first. Where today's default is a speed + choice (for example the H3 chain template's 4-step turbo LoRA at int4 and 960x544, + or the turbo LoRA's "digital" audio on dialogue, #984/#992), the bench decides + whether it moves. +- One argument moves the dial on a template (a `tier`-like variable), with the + existing per-argument knobs still exposed underneath. How a template maps one + variable onto several arguments is the builder's to design within the workflow + syntax. +- A speedup is "lossless" when the bench shows no marked loss: staging and identity + held on at least 3 seeds, PSNR in the "changed numerics, same shot" band or better + against the slow run (the workflow guide's bands), and no drift or seam measure + worse by more than its rerun noise. Lossless speedups become defaults for the tiers + they were measured at (sparse attention already is for H3). Lossy ones are listed as + dial steps with their measured cost. +- The playbook states the dial in plain terms: what each step down saves per minute of + output, and what it costs. +- **Audio under the lossless rule** (_builder's call, settled in the design review_). + Claude cannot judge timbre, lip-sync feel or room sound by ear, so the bench measures + what it can: level steps and continuity at seams, high-band energy against the slow + run (the measure #991 used for the turbo "digital" sound), `pcm_sha256` identity + where audio should be unchanged, and `check-script` for lines. A change that is + lossless on these measures but may change timbre is not called lossless by Claude: + it goes to Don's ear with a recommendation and the job ids to listen to. A small + objective audio comparison measure may be added to the tooling if the bench needs + one. +- **Skill budgets** (_builder's note_): `script-to-video` has almost no room left under + its margin, so the per-scene length and continuity fields go in its `references/` + files, not its `SKILL.md`. + +### 3. Engine and template behaviour (what makes it easy) + +These are the behaviours the bench may promote. Each one is built only if its +evaluation says so, and the mechanism is the builder's. + +1. **Length in seconds.** A chain can be asked for by duration; segment count and + frame snap follow (today a chain is `segments` or `match_audio`). +2. **Continuity default.** `minimax/chained-segments` defaults to the continuity the + bench measures best (today `last_frame`, although guide measured better at ~40% + more time; that default was left as Don's call and is now the bench's). +3. **Counters by shot type.** The chain picks its drift counters from the shot type, + declared or measured, using the per-shot-type table above. Explicit settings win. +4. **Per-segment check and bounded re-roll.** After each segment, the chain measures it + (seam and drift measures that need no human), and a segment that fails is re-rolled + with a new seed up to a budget the validate estimate includes. The run record says + which segments were re-rolled and why. The thresholds come from the bench's rerun + noise, not from guesses. + - _Builder's call (settled in the design review):_ on by default at `standard` and + `best`, off at `draft`; a default budget of one re-roll per segment, quoted in the + estimate as a worst case. When a segment's budget runs out, the chain keeps the + best-measured attempt, finishes, and flags that segment in the run record and the + job's warnings. It never restructures the run by itself: repair in place and + turning the seam into a cut are the agent's next rungs, after the chain is done. +5. **Repair in place** (#1169): re-render one segment of a finished chain with its + head held to the segment before and its tail held to the segment after. +6. **LTX parity.** The same playbook and, where the bench supports it, the same + behaviours for LTX-2.5 chains (its conditioning is different; `extend-clip` + conditions on a whole clip, not one frame). +7. **Audio continuity defaults.** The soundtrack of a long piece is continuous by + default: one bed under cuts, held voice across chain seams, and long music and + narration joined without audible seams. + +### 4. The bench + +A fixed set of long takes, run with fixed seeds through the same measures, kept as +stored workflows on lem so any later session (the regression agent included) can +rerun it. It replaces one-off probe protocols. + +**Scenes:** + +| Id | Scene | Family | Why | +|---|---|---|---| +| S1 | static locked-off backdrop (stripes and shapes) with a subject in front | H3 | the drift worst case; `probe-1166`'s static scene | +| S2 | moving-camera arc around a statue | H3 | where the anchor fails; `probe-1166`'s arc | +| S3 | a subject talking while walking, voice held across seams | H3 | dialogue and motion together, the constitution-walk shape | +| S4 | a scored montage of cut scenes with one music bed | H3 + Music 3 | the cut path and the soundtrack | +| S5 | silent action continuing across seams | LTX-2.5 | LTX chains and extend | +| S6 | a long narration under cut boards or shots | TTS + Music 3 | long audio joins | + +Each chained scene runs at a length that exposes drift (about ten segments, ~40 s) +and, once a candidate wins, at a two-minute length to confirm it holds. + +**Measures** (all reachable over MCP, so the tester can repeat them): texture bands on +a subject-free crop against segment 1 at the first, middle and last segments; +`analyze_seams` jump and look ratios and look steps; `face_drift` against the cast +portrait where a face is held; `check-script` on lines; seam level steps and +continuity of the soundtrack; wall time and peak VRAM per segment; frames at fixed +moments for eye judgement. Each measure's rerun noise is taken first (the same seed +twice) so a difference can be called real. + +**Verdict rule:** an arm wins a scene when it beats the control on that scene's +primary measure by more than rerun noise and is not worse on any other measure by +more than noise, confirmed on a second seed. Taste-only differences go to Don with a +recommendation. + +### 5. Evaluating the ideas + +Every idea goes through the bench before it is built or promoted. Cheap mechanisms +first; a structural rewrite only when the cheap ones leave a measured gap. + +| Group | Ideas | First gate | +|---|---|---| +| Counters on today's chains | #1190 anchor strength, #1187 keystone as keyframe (capstone second), #1189 hand-over back-off, #1032 sharpest of last N, #1033 and #1191 part 1 seam grade, #1191 parts 2-4, the anti-burn LoRA on continuation segments (#1151's trial), #1196 TST | S1, S2, S3 at ~40 s against today's default and today's best | +| Carry without the pixel round trip | #1150 latent tail, #1044 latent carry + crossfade, #1030 decode once | only if the counters leave a seam or drift gap on S1-S3 | +| One long render | #1188 hybrid windows | a spike to establish feasibility on the diffusers H3 pipeline, then the bench; built only if it beats the best chain | +| Running and repair | per-segment check + re-roll (new), #1169 repair in place, #1194 performer replacement | S3 with an injected bad segment; #1194 on a cut with a replaced character | +| Audio | #1192 audio-only pass, long music joins, long narration joins | S4, S6 | +| Speed | #1195 SelfLift, #980 FastH3 Omni, cache settings at long-take tiers, LTX speed levers | the lossless rule in section 2, on S1-S3 | +| Agent knowledge | #1193 skill probes, the playbook itself | two seeds per claim; a cold-session drill of the playbook | +| Delivery | #1197 timeline export | Resolve import of a bench cut | + +#1067 (region colour-match, plan approved, parked) is folded into the seam-grade +evaluation where it overlaps. #1149 (cinematic add-ons) and #1186 (short-line split) +stay out of this design. + +### 6. Agent knowledge (where the playbook lives) + +- A new cross-cutting skill in `plugins/dw/` for long-form pieces (the playbook above), + named in `plugins/dw/README.md` and CLAUDE.md and pinned by + `tests/test_plugin_skills.py`, like the other composition skills. +- `storyboard`, `script-to-video` and `series-episodes` gain a per-scene target length + and continuity choice and point to the playbook; `minimax-h3` and `ltx-2.5` replace + their "longer" lines with a pointer and keep only family-specific rules; + `minimax-music3` and the TTS templates point to the long-audio section. +- Every rule cites the bench record it came from. Community claims stay marked until + measured. + +### 7. Hand-off to the harnest tester + +- A design record in `docs/proposals/complete/` when the work is done, in the house + format (what shipped, what the bench found, what was cut and what would bring it + back). +- A feature issue with the plan, and one stage issue per stage, each with: what was + built, how to observe it over MCP, the acceptance intent (the tester writes cases + from it), the bench job ids and stored workflow names to rerun, and any + `breaking-change` note where a default moved. +- A short list for Don of the taste calls left open, each with a recommendation. + +## Left to the builder + +- The playbook skill's name and file layout within the plugin's conventions and + budgets. +- How a template exposes the quality dial within the workflow syntax. +- The engine mechanisms of each behaviour in section 3, including how a chain measures + a segment, how a re-roll budget enters the estimate, and how "auto" shot type is + measured, if the bench says it should be. +- The bench's exact prompts, crops, seeds and stored-workflow names, kept stable once + chosen. +- Stage order inside each group, and which ideas are closed on paper after the first + gate shows them dominated. +- Which existing knobs get renamed or folded, provided every one stays reachable and a + renamed one is marked `breaking-change`. From ab5e4e2ad58a51807e046ce099afe123ae79af78 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 16:59:31 -0600 Subject: [PATCH 002/405] docs(proposals): long-form media - standard is the quality-per-minute knee; speed allowances for the bench Co-Authored-By: Claude Opus 5.5 --- docs/proposals/long-form-media.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/proposals/long-form-media.md b/docs/proposals/long-form-media.md index ffbf466f5..d9c6ab38d 100644 --- a/docs/proposals/long-form-media.md +++ b/docs/proposals/long-form-media.md @@ -110,7 +110,8 @@ Defaults change wherever the evidence supports it; every knob stays. | Scope | H3, LTX-2.5, long audio (songs, narration, ambience beds) and the related domains. Not Kandinsky 6. | | Run-time re-rolls | Allowed: a segment that fails its check may be re-rolled automatically, within a retry budget stated in the up-front quote. | | Quality against speed | Defaults favour quality, with instructions for moving the dial. Speedups with no marked quality loss become defaults. | -| Cost ceiling for `standard` | Up to about 6x today's H3 chain default (4-step turbo, int4, 960x544, ~2 min per segment) per minute of output. That leaves room for the base model at 1344x768 (~12 min per segment), which the dialogue rule already needs. `best` has no ceiling. | +| Cost ceiling for `standard` | Up to about 6x today's H3 chain default (4-step turbo, int4, 960x544, ~2 min per segment) per minute of output is the hard ceiling, not the target. Don may reconsider quality-first if a 2-minute take costs ~6 hours, so `standard` is the knee of the bench's quality-per-minute curve: the cheapest setting whose quality is within noise of the next step up. If that knee still puts a 2-minute take above about 3 hours on one card, Don sees the numbers before it becomes the default. `best` has no ceiling. | +| Speed during testing | Make reasonable speed choices for the bench wherever they don't change what a measurement is meant to show: shorter chains, a draft tier, a smaller canvas, both cards in parallel, and reusing segment 1 across arms. Each such choice is validated once, by checking that the effect being measured reproduces at the cheaper setting, before arms rely on it. The same trade-off applies to the final defaults. | | Bench server during the 0.19.0 release window | A separate bench `dw.serve` on lem from a git worktree of this work's branch, on its own port and its own card. The main server stays on `develop` for the harnest tester and the release preflight. Both cards go to the bench once #1198 merges. | | Approvals and labels | Approving this design and its plan covers every stage, engine and syntax changes and default moves included. Stage issues are filed at hand-off, so the loop cannot race the build. This session closes losing ideas itself, with their numbers. | | Done | The playbook, the defaults and the run-time behaviours have shipped, and every idea has a measured verdict. A large build that passes its gate (for example #1188 hybrid windows or #1194 performer replacement) may land as its own follow-up feature after the hand-off. | From fbe77daca0fa85df6f5cddb7d6d6bd0bf2962240 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:06:47 -0600 Subject: [PATCH 003/405] docs(plans): long-form media implementation plan Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-11-03-long-form-media.md | 611 ++++++++++++++++++ 1 file changed, 611 insertions(+) create mode 100644 docs/superpowers/plans/2026-11-03-long-form-media.md diff --git a/docs/superpowers/plans/2026-11-03-long-form-media.md b/docs/superpowers/plans/2026-11-03-long-form-media.md new file mode 100644 index 000000000..e6808bdd6 --- /dev/null +++ b/docs/superpowers/plans/2026-11-03-long-form-media.md @@ -0,0 +1,611 @@ +# Long-form Media Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make long pieces (chained takes, cut scenes, long music and narration) easy for an agent and good by default, by measuring every lever on one bench, building and promoting the winners, and teaching agents the result in one playbook skill. + +**Architecture:** A bench (stored long-take workflows, a driver script and a second `dw.serve` on lem) scores every arm the same way. The engine work is confined to the chain (`dw/pipeline_processors/chain.py` and small new modules beside it), the H3 anchor, a new workflow-level `presets` mechanism for the quality dial, and a few tasks. Each idea passes a measured gate before it is built past a spike or promoted to a default. The agent-facing result is a new `long-form` skill plus pointers from the existing skills. + +**Tech Stack:** Python 3.12, PyTorch, HuggingFace diffusers modular pipelines (MiniMax-H3, LTX-2.5), the dw workflow engine, aiohttp server, pytest (xdist), the dw Claude Code plugin (Markdown skills), `gh` for GitHub, lem (2x RTX 3090) over ssh. + +**Spec:** `docs/proposals/long-form-media.md` (approved 2026-11-03). Executors read both. + +## Global Constraints + +- Branch: all work on `feat/long-form-media` (stage branches off it are fine). Merge to `develop` only after PR #1198 (0.19.0) merges. Never push to `master`. +- Bench server: a second `dw.serve` on lem from a git worktree of this branch, its own port (8766), its own card, its own workspace (`/home/don/bench-workspace`). The main server stays on `develop` for the harnest tester and the release preflight until #1198 merges. +- Kandinsky 6 is out of scope. +- Licences: GPL, AGPL and no-licence sources contribute concepts only. MIT sources may contribute code with an entry in `docs/ATTRIBUTIONS.md`. Every idea that ships credits its source there. +- Model knowledge lives in `plugins/dw/` and the catalog, never in engine code (CLAUDE.md). Compare backends with `get_device_type()`. +- Security: every new argument that names media goes through `dw/locations.py` (`PIPELINE_MEDIA_ARGUMENTS` for pipeline arguments) and `dw/security.py` validators. Never `eval`, `exec` or `shell=True`. +- Defaults may change where the bench supports it; every knob stays reachable. A renamed or moved knob is labelled `breaking-change` at hand-off. +- `standard` tier: the knee of the quality-per-minute curve, hard ceiling about 6x today's H3 chain default per output minute; if the knee puts a 2-minute take above about 3 hours on one card, stop and show Don the numbers before making it the default. +- Bench speed: cheaper bench settings (shorter chains, draft tier, smaller canvas, both cards, a shared segment 1) are allowed once a check shows the effect being measured reproduces at that setting. +- Run-time re-roll: on at `standard` and `best`, off at `draft`, default budget 1 per segment, quoted in the estimate as a worst case; on exhaustion keep the best-measured attempt, finish, flag the segment. Never restructure the run. +- Audio: Claude never calls a timbre change lossless. Anything that may change timbre goes to Don's ear with job ids and a recommendation. +- Judgement: decide by measurement; ask Don when unsure, unable to decide, or for pure taste, always with a recommendation. Collect those in `bench/records/for-don.md`. +- Skill budgets: `tests/test_plugin_skills.py` pins skills. The H3 skill is at its byte cap and `script-to-video`'s SKILL.md has almost no room: new text goes in the new skill or in `references/` files. +- Tests: `venv/bin/python -m pytest` locally (xdist `-n auto`); name files to run in one process. The full backend suite and the arch ratchet (`scripts/arch_metrics.py --check docs/stabilization/baseline.json`) pass before any merge. +- Commits: `feat(chain): # - ...` style with a scope, never a bare `fix: #N` headline (it auto-closes parents). End every message with `Co-Authored-By: Claude Opus 5.5 `. +- Approvals: Don's approval of the spec and this plan covers every stage. Stage issues are filed at hand-off (Task 30), not before. This session closes losing ideas itself, with numbers. +- Done: playbook, defaults and run-time behaviours shipped, and every idea in the spec's section 5 table has a measured verdict. A large build that passes its gate may become its own follow-up feature. + +## Review Focus + +- **A chain with a mid-run re-roll that also carries audio:** the re-rolled segment's audio replaces the failed attempt's on the timeline; nothing doubles, and seam crossfades are recomputed against the kept attempt. Test in Task 13. +- **`duration_seconds` that is not a whole number of segments:** the last segment snaps up to the frame grid and the stitched video is trimmed to the asked duration exactly (frame-rounded), with the soundtrack trimmed with it. Test in Task 12. +- **A `presets` value the caller overrides partly:** a caller passing `tier: "draft"` and `num_inference_steps: 30` gets 30 steps and the rest of the draft bundle. Test in Task 18. +- **A segment-check threshold on a deliberate cut or a static locked-off shot:** a seam whose shot is `hard_cut` is never checked, and a static shot whose typical motion is near zero does not fail its jump check from division by a tiny typical delta (the measure's own floor applies). Test in Task 13. +- **An anchor strength of 0 or 1:** 0 is bit-identical to no anchor, 1 bit-identical to today's full match. Test in Task 6. + +--- + +## Phase 0: The bench + +### Task 1: Bench server on lem + +**Files:** +- Create: `scripts/bench_server.sh` +- Create: `bench/README.md` + +**Interfaces:** +- Produces: a running bench `dw.serve` at `http://lem:8766` (token from `BENCH_TOKEN`), workspace `/home/don/bench-workspace`, serving the worktree `/home/don/dw-bench` checked out at the branch named on the command line. `scripts/bench_server.sh {start|stop|status|deploy }`. + +- [ ] **Step 1: Read lem's current state.** Run `ssh lem 'ps aux | grep -i "dw.serve\|python" | grep -v grep; docker ps; nvidia-smi --query-gpu=index,memory.used --format=csv; free -g'`. Record which server is live (native or the Docker container from 2026-11-01), which `--devices` it uses, and the free host RAM. +- [ ] **Step 2: Write `scripts/bench_server.sh`.** `deploy `: `git -C ~/diffusers-workflow worktree add ~/dw-bench ` the first time, else fetch and fast-forward the worktree. `start`: from `~/dw-bench`, `nohup ~/diffusers-workflow/venv/bin/python -m dw.serve --host 0.0.0.0 --port 8766 --devices "${BENCH_DEVICES:-cuda:1}" --token "$BENCH_TOKEN" --workspace ~/bench-workspace --examples-dir ~/dw-bench/workflows > ~/dw-bench.log 2>&1 & disown`, then poll `/api/health` for 120 s. `stop`: SIGTERM the process whose command line has `--port 8766`, then SIGKILL after 30 s (lem's server does not always exit on TERM). `status`: health JSON plus the worktree's commit. Running `python -m dw.serve` from inside the worktree imports the worktree's `dw` before the editable install. +- [ ] **Step 3: Free the bench card on the main server if it holds it.** If the main server runs workers on both cards, restart it with `--devices cuda:0` the way it was launched (the Docker container's `DW_DEVICES` in `docker/.env`, or `scripts/deploy.sh` with the systemd unit). Check with `nvidia-smi` that nothing of the main server is resident on cuda:1. If the auto-mode classifier refuses the restart, ask Don once to run it. +- [ ] **Step 4: Check host RAM.** Two servers do not share the host-RAM hold. From Step 1's `free -g` and the `host_ram_need` the main server projects for an H3 chain (`GET /api/jobs/` of a recent chain), confirm both servers' largest jobs fit together. If they don't, the bench runs only while the main server is idle, and `bench/README.md` says so. +- [ ] **Step 5: Deploy and start.** `scripts/bench_server.sh deploy feat/long-form-media && scripts/bench_server.sh start`. Expected: health answers `{"status":"ok", ...}` with `device` cuda and the worktree's commit. +- [ ] **Step 6: Write `bench/README.md`.** What the bench is, the server's port, card, workspace and token variable, how to deploy a branch to it, the host-RAM rule from Step 4, and that the main server is untouched. +- [ ] **Step 7: Commit.** `git add scripts/bench_server.sh bench/README.md && git commit -m "chore(bench): a second dw.serve on lem for the long-form bench"` + +### Task 2: Bench driver + +**Files:** +- Create: `bench/driver.py` +- Create: `bench/verdict.py` +- Test: `tests/test_bench_driver.py`, `tests/test_bench_verdict.py` + +**Interfaces:** +- Consumes: the bench server from Task 1; REST routes `POST /api/jobs`, `GET /api/jobs/{id}`, `POST /api/gallery/compare`, the assess route `api/gallery/{name}/assess` (as `dw_mcp/media.py` calls them). +- Produces: + - `Arm` dataclass: `name: str`, `scene: str`, `workflow: str` (path under `bench/workflows/`), `arguments: dict`, `seed: int`. + - `load_arms(path: str) -> list[Arm]`: reads a JSON list of arms. + - `submit(arm: Arm, server: str, token: str) -> str` (job id, arguments include the seed, `note` names the arm). + - `wait(job_id, server, token, timeout_s: int) -> dict` (the job detail). + - `measure(job: dict, scene: Scene, server, token) -> dict`: the measures in the spec section 4 for that scene, keyed by measure name. + - `Scene` dataclass: `name`, `crop: list[int]` (subject-free box), `moments: list[str]` (`"frame:N"` at segments 1, middle and last), `face_reference: str | None`, `lines: list[str] | None`. + - CLI: `python bench/driver.py run [--measure] [--out bench/records/.json]` and `python bench/driver.py report ...` (markdown tables to stdout). + - In `bench/verdict.py`: `noise_floor(a: dict, b: dict) -> dict` (per-measure absolute difference of two same-seed runs), and `verdict(control: dict, arm: dict, noise: dict, primary: str) -> str`, returning `"wins"`, `"loses"`, `"tie"` or `"mixed"` by the spec's rule: wins when the primary beats control by more than noise and no other measure is worse by more than noise. + +- [ ] **Step 1: Write the failing tests.** + +```python +def test_verdict_wins_only_past_noise_and_never_worse_elsewhere(): + noise = {"seam_look_mean": 0.2, "texture_fine_seg_last": 0.1} + control = {"seam_look_mean": 2.0, "texture_fine_seg_last": 1.0} + assert verdict(control, {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.05}, noise, "seam_look_mean") == "wins" + assert verdict(control, {"seam_look_mean": 1.9, "texture_fine_seg_last": 1.0}, noise, "seam_look_mean") == "tie" + assert verdict(control, {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.4}, noise, "seam_look_mean") == "mixed" + assert verdict(control, {"seam_look_mean": 2.6, "texture_fine_seg_last": 1.0}, noise, "seam_look_mean") == "loses" + +def test_noise_floor_is_the_absolute_difference_per_measure(): + assert noise_floor({"a": 1.0, "b": 3.0}, {"a": 1.25, "b": 2.0}) == {"a": 0.25, "b": 1.0} + +def test_load_arms_reads_every_field(tmp_path): ... # name, scene, workflow, arguments, seed round-trip + +def test_measure_calls_compare_with_the_scene_crop_and_texture_bands(monkeypatch): ... + # a fake HTTP layer records the compare payload: names=[output], crop=scene.crop, + # at=scene.moments, texture_bands=True; and the assess call with probe="analyze_seams" +``` + +Lower is better for every measure the driver reports; the driver negates any measure where higher is better before it reaches `verdict`. + +- [ ] **Step 2: Run them to verify they fail.** `venv/bin/python -m pytest tests/test_bench_driver.py tests/test_bench_verdict.py -v`. Expected: ImportError. +- [ ] **Step 3: Implement `bench/verdict.py` and `bench/driver.py`.** HTTP with `urllib.request` (no new dependency). The driver never fetches media files; every measure comes from the server's compare and assess routes, so the tester can repeat them over MCP. +- [ ] **Step 4: Run the tests.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(bench): driver and verdict rule for the long-form bench"` + +### Task 3: Bench scenes as workflows + +**Files:** +- Create: `bench/workflows/s1-static.json`, `s2-arc.json`, `s3-dialogue-walk.json`, `s4-montage.json`, `s5-ltx-action.json`, `s6-narration.json` +- Create: `bench/scenes.json` (one `Scene` per id, Task 2's shape) +- Create: `bench/arms/baseline.json` +- Test: `tests/test_bench_workflows.py` + +**Interfaces:** +- Consumes: Task 2's `Scene` and `Arm` shapes. +- Produces: six workflows that validate, each exposing the variables the arms vary: `continuity`, `guide_frames`, `segments` (later `duration_seconds`), `anchor`, `guide_noise`, `seed`, and the tier arguments (`width`, `height`, `num_inference_steps`, `weights_dtype`, LoRA name, `attention_backend`). + +- [ ] **Step 1: Write the failing test.** `test_every_bench_workflow_validates`: for each file in `bench/workflows/`, `dw.validate` on it returns no errors; `test_every_arm_names_a_scene_and_a_workflow_that_exist`. +- [ ] **Step 2: Run it.** Expected: FAIL (files missing). +- [ ] **Step 3: Write the scenes.** S1 and S2 copy `probe-1166`'s `drift-chain` workflow, prompts and assets (`GET /api/jobs/755a95b35aea/workflow` on the main server gives the arc's realized copy; the static one is in the first #1175 comparison comment on #1166), so the #1175 numbers are a known baseline. S3 is `templates/minimax/chained-segments` with `continuity: "guide"`, `carry_audio: true`, a cast portrait and a spoken line per segment through `prompts`. S4 is `templates/minimax/dialogue-short`-style cut scenes under one `templates/minimax/music` bed through `templates/assemble-and-score`. S5 is `templates/ltx2/chained-segments`. S6 is several `templates/voice-clone` takes and a Music 3 bed under `templates/animatic` boards. Upload any asset they need to the bench workspace with `POST /api/assets` and reference it as `asset:bench/...`. +- [ ] **Step 4: Write `bench/scenes.json`.** Crops chosen from each scene's own frames so the subject never enters them (S2's from #1175: `[0,0,240,150]`). Moments at segment 1's middle, the middle segment's middle and the last segment's middle. +- [ ] **Step 5: Write `bench/arms/baseline.json`.** The control arm per scene: today's template defaults, seed 1166. +- [ ] **Step 6: Run the test.** Expected: PASS. +- [ ] **Step 7: Commit.** `git commit -m "feat(bench): six long-take scenes and their baseline arms"` + +### Task 4: Noise floor and cheap-setting checks (lem run) + +**Files:** +- Create: `bench/records/noise-floor.md`, `bench/records/noise-floor.json` +- Create: `bench/arms/noise.json`, `bench/arms/cheap-checks.json` + +**Interfaces:** +- Consumes: Tasks 1-3. +- Produces: per scene, the noise floor dict every later `verdict` call uses, and the list of cheap bench settings proven safe. + +- [ ] **Step 1: Run each scene's control twice with the same seed.** `python bench/driver.py run bench/arms/noise.json --measure --out bench/records/noise-floor.json`. A plain rerun of a seeded workflow is served from the step cache, so the second run must disable it (a distinct inline `id`, or the cache-off setting the server documents in `docs/WORKER_GUIDE.md`). +- [ ] **Step 2: Compute and record the noise floor.** `noise_floor()` per scene, written as a table in `noise-floor.md`. +- [ ] **Step 3: Check the cheap settings.** For S1 and S2, rerun #1175's arms A, B and C at the cheap settings proposed for the bench (6 segments instead of 10; the draft tier; 960x544). The setting is safe when the ordering of A, B and C on the primary measures matches #1175's ordering at full settings. Record which settings passed in `noise-floor.md`. +- [ ] **Step 4: Commit the records.** `git commit -m "bench: noise floor and cheap-setting checks"` + +### Task 5: Baseline round (lem run) + +**Files:** +- Create: `bench/records/baseline.md`, `bench/records/baseline.json` +- Create: `bench/records/for-don.md` + +**Interfaces:** +- Produces: per scene, the control's measures, wall time per output minute and peak VRAM, plus today's best known arm on S1 and S2 (`anchor: "segment_1"` with `guide_noise` 0.4 on static; `guide_noise` 0.4 on arc). These are the numbers every later arm beats or doesn't. + +- [ ] **Step 1: Run the baseline and best-known arms** on all six scenes at the safe settings from Task 4. +- [ ] **Step 2: Write `baseline.md`.** One table per scene: measures, wall time, VRAM, frames at the three moments (job ids and output names, so anyone can open them). +- [ ] **Step 3: Start `for-don.md`** with any taste question the baseline already raises, each with a recommendation. +- [ ] **Step 4: Commit.** `git commit -m "bench: baseline round"` + +## Phase 1: Counters on today's chains + +Tasks 6 to 10 each build one cheap mechanism, off by default, then Task 11 measures them all together. A mechanism's default changes only in Task 11. + +### Task 6: `latent_anchor_strength` (#1190) + +**Files:** +- Modify: `dw/pipeline_processors/h3_anchor_stats.py` (`match_channel_statistics`) +- Modify: `dw/pipeline_processors/h3_anchor_steps.py` (reads the new input) +- Modify: `dw/pipeline_processors/h3_rules.py` (`LATENT_ANCHOR_STRENGTH_INPUT = "latent_anchor_strength"`, `anchor_strength_problem(value) -> str | None`) +- Modify: `dw/pipeline_processors/chain.py` (`ChainConfig.anchor_strength`, set on segments 2+ with the anchor) +- Modify: `dw/workflow_schema.json` (chain `anchor_strength`), `dw/guides.py` (validation hook beside `guide_noise_errors`) +- Test: `tests/test_h3_anchor.py`, `tests/test_chain.py` + +**Interfaces:** +- Produces: `match_channel_statistics(latents, anchor_mean, anchor_std, strength: float = 1.0)`; H3 call argument `latent_anchor_strength` (float in [0, 1], default 1); chain key `anchor_strength` (same range, only with `anchor: "segment_1"`). + +- [ ] **Step 1: Write the failing tests.** + +```python +def test_strength_one_is_todays_full_match(): + out = match_channel_statistics(x, mean_a, std_a, strength=1.0) + assert torch.equal(out, match_channel_statistics(x, mean_a, std_a)) + +def test_strength_zero_returns_the_latent_unchanged(): + assert torch.equal(match_channel_statistics(x, mean_a, std_a, strength=0.0), x) + +def test_half_strength_moves_each_channel_halfway(): + full = match_channel_statistics(x, mean_a, std_a) + half = match_channel_statistics(x, mean_a, std_a, strength=0.5) + assert torch.allclose(half, x + 0.5 * (full - x), atol=1e-5) + +def test_anchor_strength_problem_refuses_out_of_range_and_non_numbers(): + assert anchor_strength_problem(1.5) and anchor_strength_problem(-0.1) and anchor_strength_problem("a") + assert anchor_strength_problem(0.0) is None and anchor_strength_problem(1) is None + +def test_chain_anchor_strength_needs_anchor_segment_1(): ... # validate refuses it on a chain without the anchor, naming both keys +def test_chain_sets_strength_on_segments_after_the_first(): ... # FakePipeline sees latent_anchor_strength 0.5 on segments 2+, none on segment 1 +``` + +- [ ] **Step 2: Run them to verify they fail.** `venv/bin/python -m pytest tests/test_h3_anchor.py tests/test_chain.py -v -k strength`. Expected: FAIL. +- [ ] **Step 3: Implement.** The blend is `x + strength * (matched - x)` in float32, returned in the latent's dtype; strength 0 returns the input object itself (bit-identical). Validation through one rule, `anchor_strength_problem`, called by validate and by the run-time path (the `guide_noise` pattern, `h3_rules.guide_noise_problem`). +- [ ] **Step 4: Run the tests.** Expected: PASS. Then `venv/bin/python -m pytest tests/ -q` for the suite. +- [ ] **Step 5: Docs.** `docs/WORKFLOW_GUIDE.md` H3 anchor section and the chain table; `docs/ARCHITECTURE.md` H3 row unchanged unless an owner moved. +- [ ] **Step 6: Commit.** `git commit -m "feat(h3): #1190 - latent_anchor_strength, a partial statistics match"` + +### Task 7: `guide_backoff` (#1189) + +**Files:** +- Modify: `dw/pipeline_processors/chain.py` (`GuideContinuity.extract`, `ChainConfig`, a tail trim in `run_chain`) +- Modify: `dw/pipeline_processors/h3_rules.py` (`guide_chain_problems` checks it), `dw/workflow_schema.json` +- Test: `tests/test_chain.py`, `tests/test_guide_chain_validation.py` + +**Interfaces:** +- Produces: chain key `guide_backoff` (frames, a multiple of 17, default 0, guide continuity only). With backoff `b`, the carry is `frames[-(guide_frames + b):-b]` (audio sliced the same span), and every segment except the last loses its last `b` frames at assembly. `ChainConfig.tail_trim: int`. + +- [ ] **Step 1: Write the failing tests.** + +```python +def test_backoff_carries_the_frames_one_block_before_the_end(): ... + # 124-frame fake segments numbered by frame index; guide_frames 22, backoff 17: + # the carry holds frames 85..106 of the previous segment +def test_backoff_trims_every_segment_but_the_last(): ... + # 3 segments: stitched length == 124 + 2*(124-22) - 2*17, and the last segment keeps its tail +def test_backoff_slices_the_held_audio_to_the_same_span(): ... +def test_backoff_must_be_a_multiple_of_17_and_needs_guide_continuity(): ... # validate refuses 10, and 17 on last_frame +def test_backoff_zero_is_todays_chain(): ... # identical stitched frames and shots to a chain without the key +``` + +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement.** Shot records (`shots[i].num_frames`) reflect the trimmed length so `analyze_seams` measures the real seams. +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(chain): #1189 - guide_backoff hands over one block before the end"` + +### Task 8: `keystone` (#1187, first half) + +**Files:** +- Modify: `dw/pipeline_processors/chain.py` (`GuideContinuity.inject` and `LastFrameContinuity.inject` take a keystone frame), `dw/pipeline_processors/h3_rules.py`, `dw/workflow_schema.json` +- Test: `tests/test_chain.py` + +**Interfaces:** +- Produces: chain key `keystone`: `"none"` (default) or `"segment_1"`. With `"segment_1"`, every later segment's fl2va keyframe (`segment_argument`) is segment 1's frame 0 instead of the carried frame; the guide (guide continuity) is unchanged, and the keyframe region is already trimmed at assembly, so segment 1's frame never reaches the output. On `last_frame` continuity the keystone would replace the only carry, so it is refused there. + +- [ ] **Step 1: Write the failing tests.** `test_keystone_sets_every_later_keyframe_to_segment_1_frame_0`, `test_keystone_leaves_the_guide_as_the_carried_tail`, `test_keystone_is_refused_without_guide_continuity`, `test_keystone_none_is_todays_chain`. +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement.** +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(chain): #1187 - keystone: segment 1's frame 0 as every later keyframe"` + +The late-locked capstone (#1187's second half) needs a step-gated condition row; it is built only if Task 11 leaves the moving-camera gap open (Task 14). + +### Task 9: `carry_pick: sharpest` (#1032) + +**Files:** +- Create: `dw/pipeline_processors/chain_pick.py` +- Modify: `dw/pipeline_processors/chain.py` (`LastFrameContinuity.extract`, the head trim of the next segment), `dw/workflow_schema.json` +- Test: `tests/test_chain_pick.py`, `tests/test_chain.py` + +**Interfaces:** +- Produces: `sharpness(frame: numpy.ndarray) -> float` (variance of the Laplacian on luma) and `sharpest_index(frames: list, window: int) -> int` (index from the end, 0 = last frame) in `chain_pick.py`; chain key `carry_pick`: `{"sharpest": N}` with N in 1..8 (default absent = the last frame). Picking frame `-1-k` drops the last `k` frames of that segment at assembly (motion continuity: the next segment starts from the picked frame), and the job log names the picked frame per seam. + +- [ ] **Step 1: Write the failing tests.** `test_a_blurred_last_frame_loses_to_a_sharp_one_before_it`, `test_window_one_is_always_the_last_frame`, `test_the_segment_is_trimmed_after_the_picked_frame`, `test_the_pick_is_logged_per_seam`. +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement** with numpy only (a 3x3 Laplacian by array slicing). +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(chain): #1032 - carry_pick sharpest of the last N frames"` + +### Task 10: `look_hold`, the pixel drift set (#1191, #1033, #1067) + +**Files:** +- Create: `dw/pipeline_processors/chain_look.py` +- Modify: `dw/pipeline_processors/chain.py` (applies it per segment before the carry is extracted), `dw/workflow_schema.json` +- Test: `tests/test_chain_look.py`, `tests/test_chain.py` + +**Interfaces:** +- Produces in `chain_look.py`, all on `(n, h, w, 3)` uint8 arrays: + - `tone(frame) -> numpy.ndarray`: per-channel YCbCr quantiles of a pooled copy (17 points). + - `pickup_grade(frames, carried) -> frames`: match `frames[1]` to `carried` by quantile LUT, carry the same correction to the last frame, interpolate the LUT per frame. + - `ease_creep(frames, scene_look, tolerance=0.03) -> frames`: when the last frame's contrast or saturation exceeds the scene's by more than the tolerance, scale it down linearly from 1 at the first frame; never brighten, never touch a darker or paler shot. + - `ease_texture(frames, scene_texture, tolerance=0.03, max_cut=0.5) -> frames`: subtract up to `max_cut` of the faint, edge-masked luma detail at either end where it exceeds the scene's, interpolated across the shot. + - `precompensate(frame, boost) -> frame`: divide contrast and saturation by the measured boost (clamped to 1..1.25). + - Chain key `look_hold`: `true` (all four) or an object `{"pickup": bool, "creep": bool, "texture": bool, "precompensate": bool}`; default absent = off. The scene look and texture are segment 1's (the roughest of its first, middle and last frames for texture); the boost is the running mean of rendered-opening look over fed-frame look. +- Concepts from Smite79/MiniMax-H3-LongVideos (no-redistribution licence): no code, credit in `docs/ATTRIBUTIONS.md`. + +- [ ] **Step 1: Write the failing tests** on synthetic frames: `test_pickup_grade_matches_the_opening_to_the_carried_frame` (a frame tinted +10 in Cb comes back within 1 of the carried tone), `test_creep_never_brightens_and_never_touches_a_darker_shot`, `test_creep_brings_the_last_frame_to_the_scene_look_within_tolerance`, `test_texture_ease_leaves_edges_alone` (a sharp step edge survives, faint noise on a flat field drops by at most half), `test_precompensate_is_identity_at_boost_one`, `test_look_hold_off_is_todays_chain`. +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement** with numpy and torch only. Apply to the decoded segment frames before `continuity.extract`, so the carried frame is the corrected one, and to the stitched output. +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(chain): #1191 - look_hold: pickup grade, creep and texture ease-out, precompensated carry"` + +### Task 11: Counter round (lem run) + +**Files:** +- Create: `bench/arms/counters.json`, `bench/records/counters.md`, `bench/records/counters.json` +- Modify: `bench/records/for-don.md` + +**Interfaces:** +- Consumes: Tasks 4-10 deployed to the bench server (`scripts/bench_server.sh deploy feat/long-form-media && scripts/bench_server.sh stop && scripts/bench_server.sh start`). +- Produces: per shot type (static S1, moving S2, dialogue walk S3), the winning counter set, written as the table the playbook will quote, and the verdict for #1187 (keystone half), #1189, #1190, #1032, #1033, #1067, #1191 and the anti-burn LoRA. + +- [ ] **Step 1: Write the arms.** One arm per counter alone on S1, S2 and S3; `latent_anchor_strength` at 0.25, 0.5 and 0.75; the anti-burn LoRA (`minimax-h3/h3-antiburn` at 0.6) on every segment; then the pairwise combinations of every counter that won alone on a scene. +- [ ] **Step 2: Run round 1** (each alone), measure, compute verdicts with the noise floor. +- [ ] **Step 3: Run round 2** (combinations of the round-1 winners per scene) and a second seed for every winner. +- [ ] **Step 4: Confirm at length.** The best set per scene at two minutes. +- [ ] **Step 5: Record.** `counters.md`: a table per scene, the verdict per idea, the job ids, and frames at the three moments. Taste questions (look better against measure better) go to `for-don.md` with a recommendation. +- [ ] **Step 6: Set the defaults the evidence supports.** In `workflows/templates/minimax/chained-segments.json` (and the matched chains where the scene type applies): the continuity default, and the counters per shot type through Task 15's mechanism once it exists. Until then, record the intended defaults in `counters.md`. +- [ ] **Step 7: Commit.** `git commit -m "bench: counter round - verdicts for #1187a #1189 #1190 #1032 #1033 #1067 #1191"` + +## Phase 2: Running a long piece + +### Task 12: `duration_seconds` + +**Files:** +- Modify: `dw/pipeline_processors/chain.py` (`ChainConfig`, a pure `chain_segment_count(chain_definition, arguments) -> int | None`) +- Modify: `dw/workflow_schema.json` (chain `duration_seconds`; exactly one of `segments`, `match_audio`, `duration_seconds`) +- Modify: `dw/plan.py` / `dw/cost_scaling.py` (a chain's per-segment cost times `chain_segment_count`) +- Modify: `workflows/templates/minimax/chained-segments.json`, `workflows/templates/ltx2/chained-segments.json` (a `duration_seconds` variable; `segments` stays) +- Test: `tests/test_chain.py`, `tests/test_plan.py` (or the existing estimate test file) + +**Interfaces:** +- Produces: chain key `duration_seconds` (number > 0, or a `variable:` reference); `chain_segment_count(...)` used by the run and the estimate alike. + +- [ ] **Step 1: Write the failing tests.** `test_duration_plans_whole_segments_and_trims_to_the_asked_length` (40 s at 24 fps with 124-frame guide segments: the plan's frame total is ≥ 960, the stitched video is exactly 960 frames and its audio exactly 40 s), `test_exactly_one_of_segments_match_audio_duration`, `test_the_estimate_scales_by_the_segment_count_duration_implies`, `test_duration_with_frame_snap_snaps_the_last_segment_up`. +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement** by reusing `plan_segments(total_frames, segment_frames, head_trim, frame_snap)` with `total_frames = round(duration_seconds * fps)`, and the existing `match_audio` trim path for the overshoot. +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(chain): duration_seconds - ask for a chain by its length"` + +### Task 13: Per-segment check and bounded re-roll + +**Files:** +- Create: `dw/pipeline_processors/chain_check.py` +- Modify: `dw/pipeline_processors/chain.py` (the segment loop), `dw/workflow_schema.json`, `dw/plan.py` (worst-case retries in the estimate) +- Modify: `dw/events.py` only if a new warning kind needs registering +- Test: `tests/test_chain_check.py`, `tests/test_chain.py` + +**Interfaces:** +- Consumes: Task 4's noise floor (to set default thresholds), `dw/tasks/assess.py` seam arithmetic (`_seam_video`, `_seam_look`) and `dw/texture_bands.py` on in-memory frames (`assess._in_memory_thumbs`). +- Produces: + - `SegmentCheck` dataclass: `jump_ratio_max: float`, `look_ratio_max: float`, `texture_ratio_max: float | None` (fine band against segment 1 on the whole frame), `face: None` (reserved). + - `check_segment(previous_frames, frames, segment_1_frames, check: SegmentCheck) -> CheckResult` with `CheckResult(passed: bool, measures: dict, failed: list[str])`. + - Chain keys: `check`: `true` (the default thresholds), `false`, or an object with `SegmentCheck`'s fields; `retries`: integer 0..3. Template defaults: `check: true` and `retries: 1` at `standard` and `best`, `retries: 0` at `draft` (wired through Task 18's presets). + - Run record: each shot gains `attempts` (int) and, when its check failed on every attempt, `check_failed: [measure names]`; the job emits one `warning` per such segment naming the measures and the kept attempt. +- Re-roll: a failed segment is re-run with a fresh seed (the step's seed plus the attempt number times a fixed prime), from the same carry; the kept attempt is the one with the fewest failed measures, then the lowest jump ratio. Its audio and frames replace the failed attempt's; crossfades are computed against the kept attempt. A seam marked `hard_cut` is never checked. + +- [ ] **Step 1: Write the failing tests.** + +```python +def test_a_jump_over_the_threshold_fails_and_names_the_measure(): ... +def test_a_static_shot_with_near_zero_motion_does_not_fail_on_jump(): ... # typical delta under the assess floor +def test_hard_cut_seams_are_never_checked(): ... +def test_a_failed_segment_is_rerolled_with_a_new_seed_from_the_same_carry(): ... # FakePipeline sees seed+prime, carry unchanged +def test_the_kept_attempts_audio_replaces_the_failed_one(): ... # stitched audio length and seam crossfade match the kept attempt +def test_budget_exhausted_keeps_the_best_attempt_and_warns(): ... +def test_retries_zero_never_rerolls(): ... +def test_the_estimate_quotes_the_worst_case_retries(): ... # plan.estimate minutes == per-segment * segments * (1 + retries) +``` + +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement.** Default thresholds: the 99th percentile of each measure across the noise-floor runs of S1-S3, plus margin, written as constants in `chain_check.py` with a comment citing `bench/records/noise-floor.md`. +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Bench it (lem).** S3 with a segment made to fail (a prompt that breaks staging on segment 4 only, through `prompts`): the check catches it, the re-roll fixes or flags it. Record in `bench/records/check.md`. +- [ ] **Step 6: Commit.** `git commit -m "feat(chain): per-segment check with a bounded re-roll"` + +### Task 14: Repair in place (#1169) and the late capstone (#1187, second half) + +**Files:** +- Create: `workflows/templates/minimax/repair-segment.json` (if a pure workflow can express it) +- Create: `bench/records/repair.md` +- Modify (only if Step 2 says engine work is needed): `dw/pipeline_processors/h3_rules.py` (a tail guide ending past `num_frames`), `dw/pipeline_processors/h3_guide_steps.py` + +**Interfaces:** +- Produces: a template that re-renders segment `k` of a finished chain with segment `k-1`'s last `guide_frames` as a frame-0 guide and segment `k+1`'s first `guide_frames` as a guide ending at `num_frames`, then rejoins with `concat_videos` and `pair_audio` (the guide frames trimmed off both ends). Arguments: `chain_output` (an `output:` of the chain), `segment` (1-based), `prompt`, `seed`. + +- [ ] **Step 1: Check whether a pure workflow can do it.** Per-segment files exist with `save_segments` + `keep_segments`; `trim_video`, `concat_videos` and the existing guides cover the rest. `h3_rules.guide_end_problem` already accepts a guide that ends exactly at the end of the render. +- [ ] **Step 2: Build the template** (or, if a guide cannot sit at the tail, the smallest engine change that allows it, with tests in `tests/test_h3_guides.py`). +- [ ] **Step 3: Bench it.** On Task 13's S3 run with a flagged segment: repair it, measure `analyze_seams` at both of the repaired segment's seams against the chain's other seams. +- [ ] **Step 4: The capstone gate.** If Task 11 left S2 (moving camera) without a winning counter, spike #1187's late capstone: a one-frame condition row past the end of the render from segment 1's frame `num_frames - 5`, entering only below a sigma threshold. Record the spike's verdict in `repair.md`; build it only if it wins S2. +- [ ] **Step 5: Commit.** `git commit -m "feat(h3): #1169 - repair one chain segment in place between its neighbours"` + +### Task 15: Counters chosen by shot type + +**Files:** +- Modify: `dw/pipeline_processors/chain.py` (`ChainConfig`), `dw/workflow_schema.json` +- Modify: the H3 chain templates +- Test: `tests/test_chain.py` + +**Interfaces:** +- Produces: chain key `shot`: `"static" | "moving" | "dialogue" | "auto"`; the template maps each to Task 11's winning counter set through its own variables (the mapping is template data, not engine code, per CLAUDE.md). `"auto"` is built only if Task 11 shows that segment 1's measured motion (`analyze_shots`' typical delta) separates S1 from S2 reliably; then the chain measures segment 1 and picks the static or moving set. Explicit counter keys always win over the shot type. + +- [ ] **Step 1: Write the failing tests.** `test_shot_static_applies_the_templates_static_counters`, `test_explicit_counter_keys_win_over_the_shot_type`, and, if auto is built, `test_auto_picks_static_for_a_still_segment_1_and_moving_for_a_panning_one` on synthetic frames. +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement.** +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Bench it.** S1, S2 and S3 with `shot` set (and `auto`), against their Task 11 best arms: within noise. +- [ ] **Step 6: Commit.** `git commit -m "feat(chain): shot type picks the drift counters"` + +### Task 16: Performer replacement gate (#1194) + +**Files:** +- Create: `bench/records/performer-replacement.md` + +- [ ] **Step 1: Spike.** On one cut of S4: a per-frame person mask (check the catalog's segmentation models and SAM availability under diffusers/transformers on lem, with licences), a regional edit on ref2va (`templates/minimax/regional-edit`) at `refine_strength` 1.0 inside the mask with the replacement portrait as the identity reference and `hold_audio` the source. +- [ ] **Step 2: Measure.** `face_drift` against the replacement portrait, `compare_outputs` outside the mask (unchanged), the seam at the mask edge by eye. +- [ ] **Step 3: Verdict.** If it works, it becomes a follow-up feature (a person-track task and a cut-detection task) filed at hand-off; if not, close #1194 with the numbers. +- [ ] **Step 4: Commit the record.** + +## Phase 3: Structural continuity (gated) + +### Task 17: Latent carry family and one long render (#1150, #1044, #1030, #1188) + +**Files:** +- Create: `bench/records/structural.md` + +- [ ] **Step 1: The gate.** From Tasks 11-15: does the best chain per shot type still leave a seam or drift gap larger than noise against segment 1 on any scene? If not, close #1150, #1044 and #1030 with the numbers, and skip to Step 4. +- [ ] **Step 2: Latent carry spike** (#1150's stages A and B as its plan comment designs them, on a scratch flag): measure against the best chain on the scene with the gap. +- [ ] **Step 3: Verdict for #1150, #1044, #1030.** A winner becomes a follow-up feature with its own plan, filed at hand-off. +- [ ] **Step 4: Hybrid windows spike (#1188).** On the diffusers H3 modular pipeline: can a denoise block run the transformer on a token sub-range (video rows plus their 40 Hz audio rows, rotary positions kept) per step? Build the smallest version that renders S1 at about 40 s in windows, with no keystone, and measure it against the best chain. Record feasibility, VRAM, wall time and the measures. +- [ ] **Step 5: Verdict for #1188.** A win becomes a follow-up feature filed at hand-off; a loss or an infeasible spike closes it with the record. +- [ ] **Step 6: Commit the record.** + +## Phase 4: The quality dial and speed + +### Task 18: `presets`, one variable that moves several + +**Files:** +- Create: `dw/presets.py` +- Modify: `dw/workflow_schema.json` (top-level `presets`), `dw/workflow.py:356` and `dw/validation.py:901` (call `apply_presets` before `set_variables`), `dw/validation.py` (static checks) +- Test: `tests/test_presets.py` + +**Interfaces:** +- Produces: top-level workflow key `presets`: `{"": {"": {"": , ...}, ...}}`. `apply_presets(definition: dict, arguments: dict, variables: dict) -> None` sets, for each selector, the chosen bundle's values on every variable the caller did not pass. Static checks: every bundle key is a declared variable; the selector's default is one of its bundle names; a caller value for the selector outside the bundle names is refused at `arguments.` naming the allowed values. + +- [ ] **Step 1: Write the failing tests.** + +```python +def test_the_bundle_sets_the_variables_the_caller_did_not_pass(): ... +def test_a_caller_value_beats_the_bundle(): ... # tier=draft with num_inference_steps=30 -> 30 steps, rest of draft +def test_the_default_bundle_applies_with_no_arguments(): ... +def test_an_unknown_selector_value_is_refused_naming_the_values(): ... +def test_a_bundle_naming_an_undeclared_variable_is_a_validation_error(): ... +def test_a_workflow_without_presets_is_unchanged(): ... +``` + +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Implement.** +- [ ] **Step 4: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 5: Docs.** `docs/WORKFLOW_GUIDE.md` (a *Presets* section next to *What a variable is allowed to be*); the MCP surface passes the schema through, so check `scripts/surface_snapshot.py` and the OpenAPI snapshot. +- [ ] **Step 6: Commit.** `git commit -m "feat(workflow): presets - one variable selects a bundle of others"` + +### Task 19: Speed levers round (lem run) + +**Files:** +- Create: `bench/records/speed.md` + +- [ ] **Step 1: The candidates.** For H3 at each tier: `attention_backend` (sparse top-k, already default; sage_hub), the `cache` settings (`first_block` 0.05 at 20+ steps; MagCache 0.06 on a 29-step call), the quantized-weight cache, FastH3 Omni (#980), SelfLift (#1195: spike a mid-schedule half-canvas lift on a scratch flag first), TST (#1196: spike the per-head correction as an attention processor on dense attention first). For LTX-2.5: the levers its skill lists. +- [ ] **Step 2: Lossless test** per the spec: 3 seeds, staging and identity held, PSNR in the "same shot" band or better against the slow run, no drift or seam measure worse than noise. Audio measured as the spec says; a possible timbre change goes to `for-don.md`. +- [ ] **Step 3: Promote.** Lossless levers become template defaults at the tiers they were measured at. Lossy ones are recorded as dial steps with their cost. +- [ ] **Step 4: Verdicts** for #980, #1195 and #1196: built (a spike that won becomes a stage here, with its own tests in the TDD form above) or closed with numbers. +- [ ] **Step 5: Commit.** + +### Task 20: Tiers on the long-form templates + +**Files:** +- Modify: `workflows/templates/minimax/chained-segments.json`, `chain-video-continuity.json`, `chain-matched-to-audio.json`, `chain-matched-and-aligned.json`, `dialogue-short.json`, `music-video.json`, `workflows/templates/ltx2/chained-segments.json`, `extend-clip.json` +- Create: `bench/records/tiers.md` +- Test: `tests/test_bench_workflows.py` (a template's `tier` bundles name declared variables), the existing template tests + +**Interfaces:** +- Consumes: Task 18 `presets`; Tasks 11, 13 and 19's winners. +- Produces: a `tier` variable (`draft`, `standard`, `best`; default `standard`) on each template, through `presets`, setting canvas, weights, LoRA, steps, attention, cache, counters, `check` and `retries`. + +- [ ] **Step 1: Measure the quality-per-minute curve** on S1-S3 (and S5 for LTX) across the candidate settings; pick `standard` at the knee. +- [ ] **Step 2: Apply the cost rule.** If `standard` puts a 2-minute take above about 3 hours on one card, stop and put the numbers in `for-don.md` with a recommendation; wait for Don before making it the default. +- [ ] **Step 3: Write the bundles** and update each template's `cost` entry and `cost_drivers` to the new default (measured on the bench). +- [ ] **Step 4: Run the tests.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(templates): quality tiers on the long-form templates"` + +## Phase 5: Audio + +### Task 21: Audio-only pass (#1192) + +**Files:** +- Create: `dw/pipeline_processors/h3_audio_pass.py` (if the gate in Step 1 passes) +- Modify: `dw/pipeline_processors/h3_rules.py` (`source_video` with neither a mask nor `hold_audio` means the audio pass), `dw/pipeline_processors/h3_mask_steps.py` or a new block for the all-rows-kept case +- Test: `tests/test_h3_audio_pass.py` + +**Interfaces:** +- Produces: an H3 call with `source_video` and no `mask_*` and no `hold_audio` keeps every video row (the regional edit's kept-row write-back with an all-kept mask) and generates the audio from the prompt and its `` lines; the output is the source's own pixels with the new track. Optional `listen_scale` (1, 0.5, 0.25) runs the kept rows at a smaller canvas. + +- [ ] **Step 1: Gate.** Read #984's plan v3 (kept on the issue) for the locked-video-latent block design and check it against the regional edit blocks. If an all-kept mask through the existing blocks works, build on it; else the smallest new block. +- [ ] **Step 2: Write the failing tests:** `test_the_output_frames_are_the_sources_own`, `test_listen_scale_half_runs_the_kept_rows_at_half_canvas`, `test_a_mask_or_hold_audio_keeps_todays_meaning`. +- [ ] **Step 3: Run them.** Expected: FAIL. +- [ ] **Step 4: Implement.** +- [ ] **Step 5: Run the tests and the suite.** Expected: PASS. +- [ ] **Step 6: Bench it.** S4's silent shots and a re-voice of one S3 segment; `pcm`/level measures plus job ids in `for-don.md` for Don's ear (Claude cannot judge it). A template `templates/minimax/add-sound` if Don's verdict is positive. +- [ ] **Step 7: Commit.** + +### Task 22: Long music and long narration + +**Files:** +- Create: `workflows/templates/narration-long.json`, `workflows/templates/minimax/music-long.json` (if the gates pass) +- Create: `dw/tasks/text_takes.py` (only if no existing task splits text into takes) +- Create: `bench/records/audio.md` +- Test: `tests/test_text_takes.py` + +**Interfaces:** +- Produces: `split_takes(text: str, max_seconds: float, words_per_second: float) -> list[str]` (sentence boundaries, never mid-sentence; a sentence longer than one take is split at its last comma before the limit) as a task `split_takes`; `narration-long` speaks each take with `voice-clone` under one seed and voice and joins them with `concat_audio` (or the existing join task) with a measured gap; `music-long` writes a piece longer than one Music 3 track (300 s) as sections joined with a crossfade at musical boundaries (`analyze_beats`, `join_into_song`). + +- [ ] **Step 1: Check what exists.** `list_tasks` on the bench server for text splitting and audio joins; reuse before writing. +- [ ] **Step 2: Write the failing tests** for `split_takes` (no take over the limit; no take ends mid-sentence unless the sentence alone is over it; the takes rejoin to the text). +- [ ] **Step 3: Run them.** Expected: FAIL. **Step 4: Implement.** **Step 5: Run them.** Expected: PASS. +- [ ] **Step 6: Bench S6** (narration over 3 minutes, music over 5): seam level steps, `check-script` on the narration, and job ids in `for-don.md` for listening. +- [ ] **Step 7: Commit.** + +## Phase 6: LTX-2.5 parity + +### Task 23: LTX long takes + +**Files:** +- Create: `bench/records/ltx.md` +- Modify: `workflows/templates/ltx2/chained-segments.json`, `extend-clip.json` (defaults the round supports) + +- [ ] **Step 1: Round on S5.** `last_frame` chain against `extend-clip` conditioned on the whole previous segment, the `look_hold` set (Task 10, pixel-only, so family-independent), `carry_pick` (Task 9), the check and re-roll (Task 13), and `duration_seconds`. +- [ ] **Step 2: Defaults.** What wins becomes the LTX templates' defaults through `tier`. +- [ ] **Step 3: Record and commit.** + +## Phase 7: The agent layer + +### Task 24: The `long-form` playbook skill + +**Files:** +- Create: `plugins/dw/skills/long-form/SKILL.md`, `references/levers.md`, `references/quality-and-speed.md`, `references/run-and-repair.md`, `references/audio.md` +- Modify: `plugins/dw/README.md`, `CLAUDE.md` (the plugin paragraph's skill list), `.claude-plugin/marketplace.json` only if it lists skills +- Test: `tests/test_plugin_skills.py` (the new skill's catalog names exist, its references are linked, its budget holds) + +**Interfaces:** +- Consumes: every `bench/records/*.md` verdict; Tasks 12, 13, 14, 15, 18, 20, 21, 22, 23 behaviours. +- Produces: the decision per piece, scene, seam and run from the spec's section 1, the tier dial from section 2 with measured costs per output minute, and every rule citing the bench record it came from. Community claims marked as such. + +- [ ] **Step 1: Write the failing test** `test_long_form_skill_is_named_in_the_readme_and_claude_md`, and add `long-form` to the skill lists the existing tests pin. +- [ ] **Step 2: Run it.** Expected: FAIL. +- [ ] **Step 3: Write the skill** inside its budget. The description says when it applies: any request longer than one clip, a scene to block, a long take, a series, a music video, long music or narration. +- [ ] **Step 4: Run `tests/test_plugin_skills.py`.** Expected: PASS. +- [ ] **Step 5: Commit.** `git commit -m "feat(plugin): long-form - the playbook for pieces longer than one clip"` + +### Task 25: Pointers and planning fields in the other skills + +**Files:** +- Modify: `plugins/dw/skills/storyboard/SKILL.md`, `plugins/dw/skills/script-to-video/references/shot-document.md` (per-scene `length_seconds` and `continuity`), `plugins/dw/skills/series-episodes/SKILL.md`, `plugins/dw/skills/minimax-h3/SKILL.md` and `references/chains.md`, `plugins/dw/skills/ltx-2.5/SKILL.md`, `plugins/dw/skills/minimax-music3/SKILL.md` +- Test: `tests/test_plugin_skills.py` + +- [ ] **Step 1: Write the failing tests:** the shot document names `length_seconds` and `continuity`; minimax-h3 and ltx-2.5 point to `long-form` for anything longer than one clip; every budget holds. +- [ ] **Step 2: Run them.** Expected: FAIL. +- [ ] **Step 3: Edit the skills.** Replace each family's "longer" guidance with a pointer, keep only family-specific rules, and move the community drift notes in `chains.md` into the playbook with their measured status. +- [ ] **Step 4: Run the tests.** Expected: PASS. +- [ ] **Step 5: Commit.** + +### Task 26: Skill claims (#1193) + +**Files:** +- Create: `docs/proposals/audits/2026-11-longvideos-claims.md` +- Modify: the skill references each verdict belongs in + +- [ ] **Step 1: Run the five claims** as #1193 lists them, two seeds per arm, T2VA/Ref2VA turbo at 960x544, on the bench server. +- [ ] **Step 2: Record and land.** A claim that held on both seeds becomes a rule in its reference; the rest go under *Probed and not a rule*. +- [ ] **Step 3: Commit.** + +### Task 27: `export_timeline` (#1197) + +**Files:** +- Create: `dw/tasks/timeline_export.py` +- Modify: the task registry (`register_command`, as other tasks do), `docs/TASKS.md` +- Test: `tests/test_timeline_export.py` + +**Interfaces:** +- Produces: task `export_timeline` with arguments `video` (a joined output with `shots`), `sources` (optional per-shot source files; default the shots' own spans of `video`), `name`; writes `.otioz` (zip: `version.txt`, `content.otio`, media) next to its outputs. OTIO JSON written directly (`Timeline.1`, `Stack.1`, `Track.1`, `Clip.1`, `Gap.1`, `ExternalReference.1`, `TimeRange.1`, `RationalTime.1`), rates as exact rationals. Concept and schema shape from leonardofrito/Directors-Cut (MIT), credited in `docs/ATTRIBUTIONS.md`. + +- [ ] **Step 1: Write the failing tests:** the bundle has the three members; clip `source_range`s and gaps sum to the video's duration; a 23.976 fps video writes rate 24000/1001; a media path is confined by `validate_output_path`. +- [ ] **Step 2: Run them.** Expected: FAIL. **Step 3: Implement.** **Step 4: Run them.** Expected: PASS. +- [ ] **Step 5: Check Resolve import** is out of reach here (no Resolve on lem): ask Don in `for-don.md` to open one bench cut's `.otioz` in Resolve 18.5+. +- [ ] **Step 6: Commit.** + +### Task 28: Cold-session drill + +**Files:** +- Create: `bench/records/drill.md` + +- [ ] **Step 1: Run a fresh Claude session** with only the dw MCP pointed at the bench server and the plugin from this branch (the procedure in memory `cold-session-mcp-tests`), asked for: a 90-second single take of a walk-and-talk; a 3-minute piece of four scenes with a score; a 5-minute narration over boards. +- [ ] **Step 2: Score the transcripts:** did it read `long-form` first, pick the shape per scene the playbook says, quote cost with retries, use the tier dial, and handle a flagged segment? Fix the playbook where it misled, and rerun. +- [ ] **Step 3: Commit the record and any skill fixes.** + +## Phase 8: Close-out and hand-off + +### Task 29: Merge, deploy and the record + +**Files:** +- Create: `docs/proposals/complete/long-form-media-complete.md` +- Modify: `docs/proposals/todo.md`, `docs/ATTRIBUTIONS.md`, `docs/ARCHITECTURE.md` (rows for `chain_check.py`, `chain_look.py`, `chain_pick.py`, `presets.py`) + +- [ ] **Step 1: Wait for #1198 to merge** (`gh pr view 1198 --json state`). Then merge `feat/long-form-media` to `develop` with `--no-ff`. +- [ ] **Step 2: Verify:** `venv/bin/python -m pytest tests/ -q`, the UI checks if the UI changed, `scripts/arch_metrics.py --check docs/stabilization/baseline.json`, the OpenAPI and surface snapshots. All green before pushing. +- [ ] **Step 3: Push `develop` and deploy the main lem server** with `scripts/deploy.sh develop`; return the main server to both cards and stop the bench server (`scripts/bench_server.sh stop`; remove the worktree). +- [ ] **Step 4: Copy the bench to the main server** for the tester: the six bench workflows saved into a workspace `bench` there (`save_workflow`), with their assets (`copy_to_workspace` or `upload_asset`). +- [ ] **Step 5: Write the design record** in the house format (`complete/h3-guide-chain-drift-complete.md` is the model): what shipped, the bench's verdicts with job ids, what was cut and what would bring it back. +- [ ] **Step 6: Commit and push.** + +### Task 30: GitHub hand-off + +- [ ] **Step 1: File the feature issue** "Long-form media" (`feature`, `owner:tester`, `backend:shared`, `target:lem`) linking the design and the record, with the plan as its plan comment (``) and `status:plan-approved` added by Don's approval of this plan (ask Don to add the label, since only Don may). +- [ ] **Step 2: File one `stage` issue per shipped behaviour** (Tasks 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24-25, 27), each with: what was built; how to observe it over MCP (tool calls, workflow names in the `bench` workspace, arguments); the acceptance intent the tester writes cases from; the bench job ids that showed it; `breaking-change` where a default moved. Labels `stage`, `owner:tester`, `status:fixed-pending-verify`, `target:lem`. +- [ ] **Step 3: Close every idea with its verdict.** For each of #1187-#1197, #1030, #1032, #1033, #1044, #1067, #1150, #1169: a comment with the bench numbers and the stage that shipped it (closed as completed by the tester through the stage) or the reason it lost (closed `not planned` now). A winner too large for this effort gets a follow-up feature issue (`idea`, `owner:lead`). +- [ ] **Step 4: Post `bench/records/for-don.md`** as a comment on the feature issue: every taste call left open, each with a recommendation. +- [ ] **Step 5: Save a memory** of the outcome and the hand-off state. From 9fe1d70f414fd1f566b665963d6c85f9e3a99206 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:06:55 -0600 Subject: [PATCH 004/405] docs(plans): long-form media - name the real audio join task Co-Authored-By: Claude Opus 5.5 --- docs/superpowers/plans/2026-11-03-long-form-media.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-11-03-long-form-media.md b/docs/superpowers/plans/2026-11-03-long-form-media.md index e6808bdd6..d53153fc3 100644 --- a/docs/superpowers/plans/2026-11-03-long-form-media.md +++ b/docs/superpowers/plans/2026-11-03-long-form-media.md @@ -502,7 +502,7 @@ def test_a_workflow_without_presets_is_unchanged(): ... - Test: `tests/test_text_takes.py` **Interfaces:** -- Produces: `split_takes(text: str, max_seconds: float, words_per_second: float) -> list[str]` (sentence boundaries, never mid-sentence; a sentence longer than one take is split at its last comma before the limit) as a task `split_takes`; `narration-long` speaks each take with `voice-clone` under one seed and voice and joins them with `concat_audio` (or the existing join task) with a measured gap; `music-long` writes a piece longer than one Music 3 track (300 s) as sections joined with a crossfade at musical boundaries (`analyze_beats`, `join_into_song`). +- Produces: `split_takes(text: str, max_seconds: float, words_per_second: float) -> list[str]` (sentence boundaries, never mid-sentence; a sentence longer than one take is split at its last comma before the limit) as a task `split_takes`; `narration-long` speaks each take with `voice-clone` under one seed and voice and joins them with `crossfade_audio` (`dw/tasks/joins.py`) with a measured gap; `music-long` writes a piece longer than one Music 3 track (300 s) as sections joined with a crossfade at musical boundaries (`analyze_beats`, `join_into_song`). - [ ] **Step 1: Check what exists.** `list_tasks` on the bench server for text splitting and audio joins; reuse before writing. - [ ] **Step 2: Write the failing tests** for `split_takes` (no take over the limit; no take ends mid-sentence unless the sentence alone is over it; the takes rejoin to the text). From 9c5bc17b6e412df7bc9bb11e78f09070c7949c0e Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:41:38 -0600 Subject: [PATCH 005/405] docs(plans): long-form media - #1199 filed, ideas parked under it Co-Authored-By: Claude Opus 5.5 --- docs/superpowers/plans/2026-11-03-long-form-media.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-11-03-long-form-media.md b/docs/superpowers/plans/2026-11-03-long-form-media.md index d53153fc3..94f5b5453 100644 --- a/docs/superpowers/plans/2026-11-03-long-form-media.md +++ b/docs/superpowers/plans/2026-11-03-long-form-media.md @@ -604,7 +604,7 @@ def test_a_workflow_without_presets_is_unchanged(): ... ### Task 30: GitHub hand-off -- [ ] **Step 1: File the feature issue** "Long-form media" (`feature`, `owner:tester`, `backend:shared`, `target:lem`) linking the design and the record, with the plan as its plan comment (``) and `status:plan-approved` added by Don's approval of this plan (ask Don to add the label, since only Don may). +- [ ] **Step 1: Hand the feature issue over.** #1199 already exists (filed 2026-11-03, `owner:don` + `status:plan-approved`, Don's authorization), and the 18 folded ideas are parked under it as `owner:don` + `parked` + `status:plan-approved`. At hand-off: link the design record on #1199, swap it to `owner:tester`, and post the hand-off comment. - [ ] **Step 2: File one `stage` issue per shipped behaviour** (Tasks 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24-25, 27), each with: what was built; how to observe it over MCP (tool calls, workflow names in the `bench` workspace, arguments); the acceptance intent the tester writes cases from; the bench job ids that showed it; `breaking-change` where a default moved. Labels `stage`, `owner:tester`, `status:fixed-pending-verify`, `target:lem`. - [ ] **Step 3: Close every idea with its verdict.** For each of #1187-#1197, #1030, #1032, #1033, #1044, #1067, #1150, #1169: a comment with the bench numbers and the stage that shipped it (closed as completed by the tester through the stage) or the reason it lost (closed `not planned` now). A winner too large for this effort gets a follow-up feature issue (`idea`, `owner:lead`). - [ ] **Step 4: Post `bench/records/for-don.md`** as a comment on the feature issue: every taste call left open, each with a recommendation. From 5772dfa6524522951bb805a917489b624b985d91 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:44:47 -0600 Subject: [PATCH 006/405] chore(bench): a second dw.serve on lem for the long-form bench Co-Authored-By: Claude Opus 5.5 --- bench/README.md | 30 ++++++++++++++++ scripts/bench_server.sh | 76 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 bench/README.md create mode 100755 scripts/bench_server.sh diff --git a/bench/README.md b/bench/README.md new file mode 100644 index 000000000..4b21da3dc --- /dev/null +++ b/bench/README.md @@ -0,0 +1,30 @@ +# Long-form bench server + +A second `dw.serve` on lem for benching long video takes, so the main server (the automated tester's, port 8765) is never disturbed. + +| | | +|---|---| +| Port | 8766 (`http://lem:8766`) | +| Card | `cuda:1` (`BENCH_DEVICES` overrides); the main server runs on `cuda:0` only (systemd drop-in `~/.config/systemd/user/dw-serve.service.d/devices.conf`) | +| Workspace | `/home/don/bench-workspace` | +| Code | worktree `/home/don/dw-bench`, detached at `origin/`; the main checkout is never touched | +| Token | `BENCH_TOKEN`, else `/home/don/.dw-bench-token` (mode 600) | +| Log | `/home/don/dw-bench.log` | + +## Use + +```bash +# first time (the worktree does not exist yet): from any lem shell +ssh lem 'git -C ~/diffusers-workflow fetch origin feat/long-form-media && git -C ~/diffusers-workflow worktree add --detach ~/dw-bench origin/feat/long-form-media' +# afterwards +ssh lem 'bash ~/dw-bench/scripts/bench_server.sh deploy feat/long-form-media' +ssh lem 'bash ~/dw-bench/scripts/bench_server.sh start' # waits up to 120 s for /api/health +ssh lem 'bash ~/dw-bench/scripts/bench_server.sh status' +ssh lem 'bash ~/dw-bench/scripts/bench_server.sh stop' # SIGTERM, SIGKILL after 30 s +``` + +A code change needs `stop`, `deploy`, `start`: the server runs persistent workers. + +## Host RAM rule + +The two servers do not share the host-RAM hold. lem has 125 GB; see the Task 1 report for the measured budget. If a bench job and a main-server job could together exceed it, run the bench only while the main server is idle. diff --git a/scripts/bench_server.sh b/scripts/bench_server.sh new file mode 100755 index 000000000..6396aa750 --- /dev/null +++ b/scripts/bench_server.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# A second dw.serve on lem for the long-form bench. Runs ON lem: +# ssh lem 'bash ~/dw-bench/scripts/bench_server.sh {start|stop|status|deploy }' +# Port 8766, card cuda:1, workspace ~/bench-workspace. The main server (8765) is untouched. +# Token: $BENCH_TOKEN, else the file ~/.dw-bench-token. Cards: $BENCH_DEVICES (default cuda:1). +set -euo pipefail + +MAIN="$HOME/diffusers-workflow" +TREE="$HOME/dw-bench" +PORT=8766 +LOG="$HOME/dw-bench.log" + +token() { + if [ -n "${BENCH_TOKEN:-}" ]; then printf '%s' "$BENCH_TOKEN" + elif [ -r "$HOME/.dw-bench-token" ]; then tr -d '\n' < "$HOME/.dw-bench-token" + else echo "no BENCH_TOKEN and no ~/.dw-bench-token" >&2; exit 1; fi +} + +pid() { pgrep -f -- "dw.serve .*--port $PORT" | grep -v "^$$\$" | head -1 || true; } + +health() { curl -sf "http://127.0.0.1:$PORT/api/health" -H "Authorization: Bearer $(token)"; } + +cmd_deploy() { + local branch="${1:?usage: deploy }" + git -C "$MAIN" fetch origin "$branch" + if [ ! -d "$TREE" ]; then + git -C "$MAIN" worktree add --detach "$TREE" "origin/$branch" + else + git -C "$TREE" checkout --detach "origin/$branch" + fi + git -C "$TREE" log -1 --format='deployed %h %s' +} + +cmd_start() { + if [ -n "$(pid)" ]; then echo "already running (pid $(pid))"; return 0; fi + local tok; tok="$(token)" + mkdir -p "$HOME/bench-workspace" + cd "$TREE" + nohup "$MAIN/venv/bin/python" -m dw.serve --host 0.0.0.0 --port "$PORT" \ + --devices "${BENCH_DEVICES:-cuda:1}" --token "$tok" \ + --workspace "$HOME/bench-workspace" --examples-dir "$TREE/workflows" \ + > "$LOG" 2>&1 & + disown + for _ in $(seq 120); do + if health; then echo; return 0; fi + sleep 1 + done + echo "no health after 120 s; see $LOG" >&2 + return 1 +} + +cmd_stop() { + local p; p="$(pid)" + [ -z "$p" ] && { echo "not running"; return 0; } + kill -TERM "$p" + for _ in $(seq 30); do + kill -0 "$p" 2>/dev/null || { echo "stopped"; return 0; } + sleep 1 + done + kill -KILL "$p" 2>/dev/null || true + # the persistent workers are children; drop any left holding the bench card + echo "killed (SIGKILL after 30 s)" +} + +cmd_status() { + git -C "$TREE" log -1 --format='worktree %h %s' 2>/dev/null || echo "no worktree at $TREE" + health || echo "bench server not answering on :$PORT" +} + +case "${1:-}" in + deploy) shift; cmd_deploy "$@" ;; + start) cmd_start ;; + stop) cmd_stop ;; + status) cmd_status ;; + *) echo "usage: $0 {start|stop|status|deploy }" >&2; exit 2 ;; +esac From 18935d434071dd99c47396b07f0c28c1b4a9e804 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:45:50 -0600 Subject: [PATCH 007/405] docs(bench): host-RAM rule and reaching 8766 from the Mac Co-Authored-By: Claude Opus 5.5 --- bench/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/bench/README.md b/bench/README.md index 4b21da3dc..58738cbd3 100644 --- a/bench/README.md +++ b/bench/README.md @@ -27,4 +27,8 @@ A code change needs `stop`, `deploy`, `start`: the server runs persistent worker ## Host RAM rule -The two servers do not share the host-RAM hold. lem has 125 GB; see the Task 1 report for the measured budget. If a bench job and a main-server job could together exceed it, run the bench only while the main server is idle. +The two servers do not share the host-RAM hold. lem has 125 GB (122 GB available idle), but no recent H3 chain with a projected `host_ram_need` was on the main server to confirm that both servers' largest jobs fit together. Until that is measured, run the bench only while the main server is idle (`curl localhost:8765/api/jobs?status=running` on lem). + +## Reaching it from the Mac + +lem's firewall passes 8765 but not 8766 (a rule needs sudo: `sudo ufw allow 8766/tcp`, if ufw is the filter). Until then, tunnel: `ssh -f -N -L 18766:127.0.0.1:8766 lem`, then use `http://127.0.0.1:18766`. From e8080c5926fe801c7a16af7ff878db8ec7c240eb Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:47:32 -0600 Subject: [PATCH 008/405] fix(bench): stop takes the server's workers with it; deploy-safe script Co-Authored-By: Claude Opus 5.5 --- scripts/bench_server.sh | 54 ++++++++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/scripts/bench_server.sh b/scripts/bench_server.sh index 6396aa750..d106f0b34 100755 --- a/scripts/bench_server.sh +++ b/scripts/bench_server.sh @@ -13,12 +13,21 @@ LOG="$HOME/dw-bench.log" token() { if [ -n "${BENCH_TOKEN:-}" ]; then printf '%s' "$BENCH_TOKEN" elif [ -r "$HOME/.dw-bench-token" ]; then tr -d '\n' < "$HOME/.dw-bench-token" - else echo "no BENCH_TOKEN and no ~/.dw-bench-token" >&2; exit 1; fi + else echo "no bench token: set BENCH_TOKEN or create ~/.dw-bench-token" >&2; exit 1; fi +} + +# every descendant pid of $1, deepest first +descendants() { + local c + for c in $(pgrep -P "$1" || true); do descendants "$c"; echo "$c"; done } pid() { pgrep -f -- "dw.serve .*--port $PORT" | grep -v "^$$\$" | head -1 || true; } -health() { curl -sf "http://127.0.0.1:$PORT/api/health" -H "Authorization: Bearer $(token)"; } +health() { + local tok; tok="$(token)" || return 1 + curl -sf "http://127.0.0.1:$PORT/api/health" -H "Authorization: Bearer $tok" +} cmd_deploy() { local branch="${1:?usage: deploy }" @@ -40,9 +49,11 @@ cmd_start() { --devices "${BENCH_DEVICES:-cuda:1}" --token "$tok" \ --workspace "$HOME/bench-workspace" --examples-dir "$TREE/workflows" \ > "$LOG" 2>&1 & + local spid=$! disown for _ in $(seq 120); do if health; then echo; return 0; fi + kill -0 "$spid" 2>/dev/null || { echo "bench server exited; see $LOG" >&2; return 1; } sleep 1 done echo "no health after 120 s; see $LOG" >&2 @@ -52,14 +63,24 @@ cmd_start() { cmd_stop() { local p; p="$(pid)" [ -z "$p" ] && { echo "not running"; return 0; } - kill -TERM "$p" + # collect the server's workers first: a killed server orphans them + local all; all="$(descendants "$p") $p" + kill -TERM $all 2>/dev/null || true + local q alive for _ in $(seq 30); do - kill -0 "$p" 2>/dev/null || { echo "stopped"; return 0; } + alive="" + for q in $all; do kill -0 "$q" 2>/dev/null && alive="$alive $q"; done + [ -z "$alive" ] && break sleep 1 done - kill -KILL "$p" 2>/dev/null || true - # the persistent workers are children; drop any left holding the bench card - echo "killed (SIGKILL after 30 s)" + if [ -n "$alive" ]; then + kill -KILL $alive 2>/dev/null || true + echo "killed (SIGKILL after 30 s):$alive" + else + echo "stopped (server $p and its workers)" + fi + echo "gpu 1 compute apps (pid, used_memory):" + nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader -i 1 || true } cmd_status() { @@ -67,10 +88,15 @@ cmd_status() { health || echo "bench server not answering on :$PORT" } -case "${1:-}" in - deploy) shift; cmd_deploy "$@" ;; - start) cmd_start ;; - stop) cmd_stop ;; - status) cmd_status ;; - *) echo "usage: $0 {start|stop|status|deploy }" >&2; exit 2 ;; -esac +main() { + case "${1:-}" in + deploy) shift; cmd_deploy "$@" ;; + start) cmd_start ;; + stop) cmd_stop ;; + status) cmd_status ;; + *) echo "usage: $0 {start|stop|status|deploy }" >&2; exit 2 ;; + esac +} + +# everything above is parsed before main runs, so deploy can check out over this file +main "$@"; exit From 7b6143852d179b4d716add9219e43788b0cb1859 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:51:19 -0600 Subject: [PATCH 009/405] feat(bench): driver and verdict rule for the long-form bench Co-Authored-By: Claude Opus 5.5 --- bench/driver.py | 266 ++++++++++++++++++++++++++++++++++++ bench/verdict.py | 26 ++++ tests/test_bench_driver.py | 68 +++++++++ tests/test_bench_verdict.py | 19 +++ 4 files changed, 379 insertions(+) create mode 100644 bench/driver.py create mode 100644 bench/verdict.py create mode 100644 tests/test_bench_driver.py create mode 100644 tests/test_bench_verdict.py diff --git a/bench/driver.py b/bench/driver.py new file mode 100644 index 000000000..32062abc2 --- /dev/null +++ b/bench/driver.py @@ -0,0 +1,266 @@ +"""Long-form bench driver: submit arms to the bench dw.serve, wait, and +measure each output through the server's compare and assess routes (the +same routes the MCP tools call, so the tester can repeat any number). +Stdlib only; never fetches media files.""" + +import argparse +import json +import os +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass +from pathlib import Path + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) +from verdict import noise_floor, verdict # noqa: E402 + +DEFAULT_SERVER = "http://127.0.0.1:18766" +DEFAULT_TOKEN_FILE = ( + "/private/tmp/claude-501/-Users-don-src-dkackman-diffusers-workflow/" + "6fdf55cd-3550-46d6-9df0-2dc8bfbc47e0/scratchpad/bench-token" +) +MAX_COMPARE_BYTES = 4_000_000 +TERMINAL = ("succeeded", "failed", "cancelled") + + +@dataclass +class Arm: + name: str + scene: str + workflow: str # path under bench/workflows/ + arguments: dict + seed: int + + +@dataclass +class Scene: + name: str + crop: list + moments: list # "frame:N" at segments 1, middle and last + face_reference: str | None = None + lines: list | None = None + + +def load_arms(path): + data = json.loads(Path(path).read_text()) + return [ + Arm( + name=a["name"], + scene=a["scene"], + workflow=a["workflow"], + arguments=a.get("arguments") or {}, + seed=a["seed"], + ) + for a in data + ] + + +def load_scenes(path): + data = json.loads(Path(path).read_text()) + return { + s["name"]: Scene( + name=s["name"], + crop=s["crop"], + moments=s["moments"], + face_reference=s.get("face_reference"), + lines=s.get("lines"), + ) + for s in data + } + + +def http(method, path, server, token, payload=None, params=None): + """One JSON request; the single seam the tests fake.""" + url = server.rstrip("/") + path + if params: + url += "?" + urllib.parse.urlencode(params) + headers = {"Accept": "application/json"} + if token: + headers["Authorization"] = f"Bearer {token}" + data = None + if payload is not None: + data = json.dumps(payload).encode() + headers["Content-Type"] = "application/json" + request = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(request, timeout=120) as response: + return json.loads(response.read() or b"null") + except urllib.error.HTTPError as e: + raise RuntimeError(f"{method} {path}: {e.code} {e.read().decode()[:500]}") + + +def health(server, token): + return http("GET", "/api/health", server, token) + + +def submit(arm, server, token, workspace=None): + workflow = json.loads((HERE / "workflows" / arm.workflow).read_text()) + payload = { + "workflow": workflow, + "arguments": {**arm.arguments, "seed": arm.seed}, + "note": f"bench arm {arm.name} (scene {arm.scene}, seed {arm.seed})", + "acknowledged_cost": True, + } + params = {"workspace": workspace} if workspace else None + job = http("POST", "/api/jobs", server, token, payload, params) + return job["id"] + + +def wait(job_id, server, token, timeout_s, poll_s=15): + deadline = time.monotonic() + timeout_s + while True: + job = http("GET", f"/api/jobs/{job_id}", server, token) + if job.get("status") in TERMINAL: + return job + if time.monotonic() >= deadline: + raise TimeoutError(f"job {job_id} still {job.get('status')} after {timeout_s}s") + time.sleep(poll_s) + + +def output_name(job): + """The gallery name of the job's deliverable video: the last video file + in the manifest, `final` subfolder first.""" + kinds = job.get("output_kinds") or {} + videos = [ + (entry.get("subfolder") == "final", name) + for entry in job.get("manifest") or [] + for name in entry.get("files") or [] + if kinds.get(name, "video") == "video" + ] + finals = [name for final, name in videos if final] + best = (finals or [name for _, name in videos] or [None])[-1] + if best is None: + raise ValueError(f"job {job.get('id')} wrote no video") + return best + + +def _mean(values): + return sum(values) / len(values) if values else None + + +def _stats(out, values, name_mean, name_max): + values = [v for v in values if v is not None] + if values: + out[name_mean] = _mean(values) + out[name_max] = max(values) + + +def measure(job, scene, server, token, workspace=None): + name = output_name(job) + params = {"probe": "analyze_seams"} + if workspace: + params["workspace"] = workspace + seams = http( + "GET", f"/api/gallery/{urllib.parse.quote(name, safe='/')}/assess", + server, token, params=params, + ).get("seams") or [] + out = {} + _stats(out, [s.get("jump_ratio") for s in seams], "seam_jump_mean", "seam_jump_max") + _stats(out, [s.get("look_ratio") for s in seams], "seam_look_mean", "seam_look_max") + _stats( + out, [abs(s["level_step_db"]) for s in seams if s.get("level_step_db") is not None], + "seam_level_step_db_mean", "seam_level_step_db_max", + ) + payload = { + "names": [name], + "max_dimension": 384, + "max_total_bytes": MAX_COMPARE_BYTES, + "at": scene.moments, + "crop": scene.crop, + "texture_bands": True, + } + compared = http( + "POST", "/api/gallery/compare", server, token, payload, + {"workspace": workspace} if workspace else None, + ) + row = (compared.get("texture") or [{}])[0] + cells = [c.get("texture") for c in row.get("cells") or []] + if len(cells) >= 2 and cells[0] and cells[-1]: + for band in ("fine", "mid"): + first, last = cells[0]["luma"].get(band), cells[-1]["luma"].get(band) + if first and last is not None: + out[f"texture_{band}_seg_last"] = last / first + if job.get("started_at") is not None and job.get("finished_at") is not None: + out["wall_minutes"] = (job["finished_at"] - job["started_at"]) / 60.0 + return out + + +def run_arms(arms, scenes, server, token, do_measure, timeout_s, workspace=None): + ids = {arm.name: submit(arm, server, token, workspace) for arm in arms} + records = [] + for arm in arms: + job = wait(ids[arm.name], server, token, timeout_s) + record = {"arm": arm.name, "scene": arm.scene, "seed": arm.seed, + "job_id": ids[arm.name], "status": job.get("status"), + "error": job.get("error")} + if job.get("status") == "succeeded": + record["output"] = output_name(job) + if do_measure: + record["measures"] = measure(job, scenes[arm.scene], server, token, workspace) + records.append(record) + return records + + +def report(records): + lines = [] + keys = sorted({k for r in records for k in r.get("measures", {})}) + lines.append("| arm | scene | seed | status | " + " | ".join(keys) + " |") + lines.append("|" + "---|" * (4 + len(keys))) + for r in records: + m = r.get("measures", {}) + cells = [f"{m[k]:.3f}" if k in m else "" for k in keys] + lines.append(f"| {r['arm']} | {r['scene']} | {r['seed']} | {r['status']} | " + " | ".join(cells) + " |") + return "\n".join(lines) + + +def _token(args): + if args.token_file is None and os.environ.get("BENCH_TOKEN"): + return os.environ["BENCH_TOKEN"] + path = args.token_file or DEFAULT_TOKEN_FILE + try: + return Path(path).read_text().strip() + except OSError: + return os.environ.get("BENCH_TOKEN", "") + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--server", default=os.environ.get("BENCH_SERVER", DEFAULT_SERVER)) + parser.add_argument("--token-file") + parser.add_argument("--workspace") + sub = parser.add_subparsers(dest="command", required=True) + run = sub.add_parser("run", help="submit arms, wait, optionally measure") + run.add_argument("arms") + run.add_argument("--scenes", default=str(HERE / "scenes.json")) + run.add_argument("--measure", action="store_true") + run.add_argument("--out") + run.add_argument("--timeout", type=int, default=7200) + rep = sub.add_parser("report", help="markdown tables from records") + rep.add_argument("records", nargs="+") + args = parser.parse_args(argv) + if args.command == "report": + for path in args.records: + print(f"### {path}\n") + print(report(json.loads(Path(path).read_text()))) + print() + return 0 + token = _token(args) + records = run_arms( + load_arms(args.arms), load_scenes(args.scenes) if args.measure else {}, + args.server, token, args.measure, args.timeout, args.workspace, + ) + text = json.dumps(records, indent=2) + if args.out: + Path(args.out).parent.mkdir(parents=True, exist_ok=True) + Path(args.out).write_text(text) + else: + print(text) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bench/verdict.py b/bench/verdict.py new file mode 100644 index 000000000..257b9b664 --- /dev/null +++ b/bench/verdict.py @@ -0,0 +1,26 @@ +"""The bench's verdict rule. Every measure is lower-is-better (the driver +negates the ones where higher is better before it gets here).""" + + +def noise_floor(a, b): + """Per-measure absolute difference of two same-seed runs.""" + return {key: abs(a[key] - b[key]) for key in a if key in b} + + +def verdict(control, arm, noise, primary): + """`wins` when the primary beats control by more than its noise and no + other measure is worse by more than its noise; `loses` when the primary + is worse by more than noise; `mixed` when the primary wins but another + measure is worse past noise; `tie` otherwise.""" + gain = control[primary] - arm[primary] + floor = noise.get(primary, 0.0) + worse_elsewhere = any( + arm[key] - control[key] > noise.get(key, 0.0) + for key in control + if key != primary and key in arm + ) + if gain > floor: + return "mixed" if worse_elsewhere else "wins" + if -gain > floor: + return "loses" + return "mixed" if worse_elsewhere else "tie" diff --git a/tests/test_bench_driver.py b/tests/test_bench_driver.py new file mode 100644 index 000000000..4bd6a74cf --- /dev/null +++ b/tests/test_bench_driver.py @@ -0,0 +1,68 @@ +import json +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / "bench")) + +import driver # noqa: E402 + + +def test_load_arms_reads_every_field(tmp_path): + path = tmp_path / "arms.json" + path.write_text(json.dumps([{"name": "a", "scene": "s", "workflow": "w.json", + "arguments": {"prompt": "x"}, "seed": 7}])) + (arm,) = driver.load_arms(str(path)) + assert (arm.name, arm.scene, arm.workflow, arm.arguments, arm.seed) == ( + "a", "s", "w.json", {"prompt": "x"}, 7) + + +def test_submit_carries_seed_and_names_the_arm(monkeypatch, tmp_path): + seen = {} + + def fake(method, path, server, token, payload=None, params=None): + seen.update(method=method, path=path, payload=payload, token=token) + return {"id": "job-1"} + + monkeypatch.setattr(driver, "http", fake) + monkeypatch.setattr(driver, "HERE", tmp_path) + (tmp_path / "workflows").mkdir() + (tmp_path / "workflows" / "w.json").write_text('{"steps": []}') + arm = driver.Arm("armA", "s", "w.json", {"prompt": "x"}, 11) + assert driver.submit(arm, "http://h", "tok") == "job-1" + assert seen["path"] == "/api/jobs" + assert seen["payload"]["arguments"] == {"prompt": "x", "seed": 11} + assert "armA" in seen["payload"]["note"] + assert seen["payload"]["acknowledged_cost"] is True + + +def test_measure_calls_compare_with_the_scene_crop_and_texture_bands(monkeypatch): + calls = [] + cell = lambda fine, mid: {"texture": {"luma": {"fine": fine, "mid": mid}}} + + def fake(method, path, server, token, payload=None, params=None): + calls.append((method, path, payload, params)) + if path.endswith("/assess"): + return {"probe": "analyze_seams", "seams": [ + {"jump_ratio": 2.0, "look_ratio": 1.0, "level_step_db": -3.0}, + {"jump_ratio": 4.0, "look_ratio": 3.0, "level_step_db": 1.0}]} + return {"texture": [{"cells": [cell(10.0, 20.0), cell(8.0, 20.0), cell(5.0, 30.0)]}]} + + monkeypatch.setattr(driver, "http", fake) + scene = driver.Scene("s", [1, 2, 64, 64], ["frame:1", "frame:50", "frame:99"]) + job = {"id": "j", "manifest": [{"files": ["wf/v1/out.mp4"], "subfolder": "final"}], + "output_kinds": {"wf/v1/out.mp4": "video"}, "started_at": 0, "finished_at": 120} + m = driver.measure(job, scene, "http://h", "tok") + + assess = next(c for c in calls if c[1].endswith("/assess")) + assert assess[3]["probe"] == "analyze_seams" + compare = next(c for c in calls if c[1] == "/api/gallery/compare") + assert compare[2]["names"] == ["wf/v1/out.mp4"] + assert compare[2]["crop"] == [1, 2, 64, 64] + assert compare[2]["at"] == scene.moments + assert compare[2]["texture_bands"] is True + assert m["seam_jump_mean"] == 3.0 and m["seam_jump_max"] == 4.0 + assert m["seam_look_mean"] == 2.0 and m["seam_look_max"] == 3.0 + assert m["seam_level_step_db_max"] == 3.0 + assert m["texture_fine_seg_last"] == 0.5 and m["texture_mid_seg_last"] == 1.5 + assert m["wall_minutes"] == 2.0 diff --git a/tests/test_bench_verdict.py b/tests/test_bench_verdict.py new file mode 100644 index 000000000..dcab53b21 --- /dev/null +++ b/tests/test_bench_verdict.py @@ -0,0 +1,19 @@ +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "bench")) + +from verdict import noise_floor, verdict # noqa: E402 + + +def test_verdict_wins_only_past_noise_and_never_worse_elsewhere(): + noise = {"seam_look_mean": 0.2, "texture_fine_seg_last": 0.1} + control = {"seam_look_mean": 2.0, "texture_fine_seg_last": 1.0} + assert verdict(control, {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.05}, noise, "seam_look_mean") == "wins" + assert verdict(control, {"seam_look_mean": 1.9, "texture_fine_seg_last": 1.0}, noise, "seam_look_mean") == "tie" + assert verdict(control, {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.4}, noise, "seam_look_mean") == "mixed" + assert verdict(control, {"seam_look_mean": 2.6, "texture_fine_seg_last": 1.0}, noise, "seam_look_mean") == "loses" + + +def test_noise_floor_is_the_absolute_difference_per_measure(): + assert noise_floor({"a": 1.0, "b": 3.0}, {"a": 1.25, "b": 2.0}) == {"a": 0.25, "b": 1.0} From 2089e916b90b25b8c771fe3be009950ba9e1738a Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 17:54:22 -0600 Subject: [PATCH 010/405] fix(bench): two-sided texture drift, all bands, records written as they land Co-Authored-By: Claude Opus 5.5 --- bench/driver.py | 70 ++++++++++++++++++++++---------------- tests/test_bench_driver.py | 54 +++++++++++++++++++++++++++-- 2 files changed, 93 insertions(+), 31 deletions(-) diff --git a/bench/driver.py b/bench/driver.py index 32062abc2..4d77b88e3 100644 --- a/bench/driver.py +++ b/bench/driver.py @@ -5,6 +5,7 @@ import argparse import json +import math import os import sys import time @@ -19,10 +20,7 @@ from verdict import noise_floor, verdict # noqa: E402 DEFAULT_SERVER = "http://127.0.0.1:18766" -DEFAULT_TOKEN_FILE = ( - "/private/tmp/claude-501/-Users-don-src-dkackman-diffusers-workflow/" - "6fdf55cd-3550-46d6-9df0-2dc8bfbc47e0/scratchpad/bench-token" -) +DEFAULT_TOKEN_FILE = "~/.dw-bench-token" MAX_COMPARE_BYTES = 4_000_000 TERMINAL = ("succeeded", "failed", "cancelled") @@ -180,28 +178,43 @@ def measure(job, scene, server, token, workspace=None): row = (compared.get("texture") or [{}])[0] cells = [c.get("texture") for c in row.get("cells") or []] if len(cells) >= 2 and cells[0] and cells[-1]: - for band in ("fine", "mid"): - first, last = cells[0]["luma"].get(band), cells[-1]["luma"].get(band) - if first and last is not None: - out[f"texture_{band}_seg_last"] = last / first + # Two-sided: blur (ratio < 1) and crust (ratio > 1) both count as + # drift, so the measure is |ln(last / first)| and lower is better. + for band in ("fine", "mid", "coarse"): + luma = cells[0].get("luma") or {}, cells[-1].get("luma") or {} + first, last = luma[0].get(band), luma[1].get(band) + if first and last and first > 0 and last > 0: + out[f"texture_{band}_seg_last"] = abs(math.log(last / first)) if job.get("started_at") is not None and job.get("finished_at") is not None: out["wall_minutes"] = (job["finished_at"] - job["started_at"]) / 60.0 return out -def run_arms(arms, scenes, server, token, do_measure, timeout_s, workspace=None): +def run_arms(arms, scenes, server, token, do_measure, timeout_s, workspace=None, out_path=None): ids = {arm.name: submit(arm, server, token, workspace) for arm in arms} records = [] + + def flush(): + if out_path: + Path(out_path).parent.mkdir(parents=True, exist_ok=True) + Path(out_path).write_text(json.dumps(records, indent=2)) + for arm in arms: - job = wait(ids[arm.name], server, token, timeout_s) record = {"arm": arm.name, "scene": arm.scene, "seed": arm.seed, - "job_id": ids[arm.name], "status": job.get("status"), - "error": job.get("error")} - if job.get("status") == "succeeded": - record["output"] = output_name(job) - if do_measure: - record["measures"] = measure(job, scenes[arm.scene], server, token, workspace) + "job_id": ids[arm.name]} + try: + job = wait(ids[arm.name], server, token, timeout_s) + record["status"] = job.get("status") + record["error"] = job.get("error") + if job.get("status") == "succeeded": + record["output"] = output_name(job) + if do_measure: + record["measures"] = measure(job, scenes[arm.scene], server, token, workspace) + except Exception as exc: # one arm's failure must not lose the others + record.setdefault("status", "error") + record["error"] = f"{type(exc).__name__}: {exc}" records.append(record) + flush() return records @@ -218,13 +231,16 @@ def report(records): def _token(args): - if args.token_file is None and os.environ.get("BENCH_TOKEN"): + if args.token_file: + return Path(args.token_file).expanduser().read_text().strip() + if os.environ.get("BENCH_TOKEN"): return os.environ["BENCH_TOKEN"] - path = args.token_file or DEFAULT_TOKEN_FILE - try: - return Path(path).read_text().strip() - except OSError: - return os.environ.get("BENCH_TOKEN", "") + path = Path(DEFAULT_TOKEN_FILE).expanduser() + if path.is_file(): + return path.read_text().strip() + raise SystemExit( + f"no bench token: pass --token-file, set BENCH_TOKEN, or write {DEFAULT_TOKEN_FILE}" + ) def main(argv=None): @@ -251,14 +267,10 @@ def main(argv=None): token = _token(args) records = run_arms( load_arms(args.arms), load_scenes(args.scenes) if args.measure else {}, - args.server, token, args.measure, args.timeout, args.workspace, + args.server, token, args.measure, args.timeout, args.workspace, args.out, ) - text = json.dumps(records, indent=2) - if args.out: - Path(args.out).parent.mkdir(parents=True, exist_ok=True) - Path(args.out).write_text(text) - else: - print(text) + if not args.out: + print(json.dumps(records, indent=2)) return 0 diff --git a/tests/test_bench_driver.py b/tests/test_bench_driver.py index 4bd6a74cf..ce4bab202 100644 --- a/tests/test_bench_driver.py +++ b/tests/test_bench_driver.py @@ -1,10 +1,12 @@ import json +import math import sys from pathlib import Path ROOT = Path(__file__).resolve().parent.parent sys.path.insert(0, str(ROOT / "bench")) +import pytest # noqa: E402 import driver # noqa: E402 @@ -38,7 +40,8 @@ def fake(method, path, server, token, payload=None, params=None): def test_measure_calls_compare_with_the_scene_crop_and_texture_bands(monkeypatch): calls = [] - cell = lambda fine, mid: {"texture": {"luma": {"fine": fine, "mid": mid}}} + cell = lambda fine, mid, coarse=4.0: { + "texture": {"luma": {"fine": fine, "mid": mid, "coarse": coarse}}} def fake(method, path, server, token, payload=None, params=None): calls.append((method, path, payload, params)) @@ -64,5 +67,52 @@ def fake(method, path, server, token, payload=None, params=None): assert m["seam_jump_mean"] == 3.0 and m["seam_jump_max"] == 4.0 assert m["seam_look_mean"] == 2.0 and m["seam_look_max"] == 3.0 assert m["seam_level_step_db_max"] == 3.0 - assert m["texture_fine_seg_last"] == 0.5 and m["texture_mid_seg_last"] == 1.5 + assert m["texture_fine_seg_last"] == pytest.approx(math.log(2)) + assert m["texture_mid_seg_last"] == pytest.approx(math.log(1.5)) + assert m["texture_coarse_seg_last"] == 0.0 assert m["wall_minutes"] == 2.0 + + +def _texture_measure(monkeypatch, first, last): + cell = lambda v: {"texture": {"luma": {"fine": v, "mid": 1.0, "coarse": 1.0}}} + + def fake(method, path, server, token, payload=None, params=None): + if path.endswith("/assess"): + return {"seams": []} + return {"texture": [{"cells": [cell(first), cell(last)]}]} + + monkeypatch.setattr(driver, "http", fake) + scene = driver.Scene("s", None, ["frame:1", "frame:2"]) + job = {"manifest": [{"files": ["o.mp4"], "subfolder": "final"}], + "output_kinds": {"o.mp4": "video"}} + return driver.measure(job, scene, "http://h", "tok") + + +def test_texture_drift_is_two_sided(monkeypatch): + blur = _texture_measure(monkeypatch, 10.0, 5.0)["texture_fine_seg_last"] + crust = _texture_measure(monkeypatch, 10.0, 20.0)["texture_fine_seg_last"] + assert blur == pytest.approx(crust) and blur > 0 + assert _texture_measure(monkeypatch, 10.0, 10.0)["texture_fine_seg_last"] == 0.0 + assert "texture_fine_seg_last" not in _texture_measure(monkeypatch, 0.0, 5.0) + assert "texture_fine_seg_last" not in _texture_measure(monkeypatch, 10.0, None) + + +def test_run_arms_writes_each_record_as_it_lands(monkeypatch, tmp_path): + arms = [driver.Arm("a", "s", "w.json", {}, 1), driver.Arm("b", "s", "w.json", {}, 2)] + monkeypatch.setattr(driver, "submit", lambda arm, *a: f"id-{arm.name}") + out = tmp_path / "sub" / "rec.json" + seen = {} + + def fake_wait(job_id, *a): + if job_id == "id-b": + seen["after_first"] = json.loads(out.read_text()) + raise RuntimeError("poll died") + return {"status": "succeeded"} + + monkeypatch.setattr(driver, "wait", fake_wait) + monkeypatch.setattr(driver, "output_name", lambda job: "o.mp4") + records = driver.run_arms(arms, {}, "http://h", "tok", False, 1, out_path=str(out)) + assert [r["arm"] for r in seen["after_first"]] == ["a"] + saved = json.loads(out.read_text()) + assert [r["arm"] for r in saved] == ["a", "b"] + assert "poll died" in saved[1]["error"] and records == saved From acb5bbd59ae125a80e8ec09aa37ae9a7892ee875 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:01:28 -0600 Subject: [PATCH 011/405] feat(bench): six long-take scenes and their baseline arms Co-Authored-By: Claude Opus 5.5 --- bench/arms/baseline.json | 44 +++ bench/scenes.json | 101 ++++++ bench/workflows/s1-static.json | 197 ++++++++++ bench/workflows/s2-arc.json | 197 ++++++++++ bench/workflows/s3-dialogue-walk.json | 206 +++++++++++ bench/workflows/s4-montage.json | 501 ++++++++++++++++++++++++++ bench/workflows/s5-ltx-action.json | 178 +++++++++ bench/workflows/s6-narration.json | 180 +++++++++ tests/test_bench_workflows.py | 68 ++++ 9 files changed, 1672 insertions(+) create mode 100644 bench/arms/baseline.json create mode 100644 bench/scenes.json create mode 100644 bench/workflows/s1-static.json create mode 100644 bench/workflows/s2-arc.json create mode 100644 bench/workflows/s3-dialogue-walk.json create mode 100644 bench/workflows/s4-montage.json create mode 100644 bench/workflows/s5-ltx-action.json create mode 100644 bench/workflows/s6-narration.json create mode 100644 tests/test_bench_workflows.py diff --git a/bench/arms/baseline.json b/bench/arms/baseline.json new file mode 100644 index 000000000..8446b959f --- /dev/null +++ b/bench/arms/baseline.json @@ -0,0 +1,44 @@ +[ + { + "name": "baseline-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "baseline-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "baseline-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "baseline-s4", + "scene": "s4-montage", + "workflow": "s4-montage.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "baseline-s5", + "scene": "s5-ltx-action", + "workflow": "s5-ltx-action.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "baseline-s6", + "scene": "s6-narration", + "workflow": "s6-narration.json", + "arguments": {}, + "seed": 1166 + } +] diff --git a/bench/scenes.json b/bench/scenes.json new file mode 100644 index 000000000..8aca4c29a --- /dev/null +++ b/bench/scenes.json @@ -0,0 +1,101 @@ +[ + { + "name": "s1-static", + "crop": [ + 0, + 0, + 240, + 150 + ], + "moments": [ + "frame:62", + "frame:481", + "frame:990" + ], + "face_reference": "asset:bench/static-stripes.jpg" + }, + { + "name": "s2-arc", + "crop": [ + 0, + 0, + 240, + 150 + ], + "moments": [ + "frame:62", + "frame:481", + "frame:990" + ] + }, + { + "name": "s3-dialogue-walk", + "crop": [ + 0, + 0, + 240, + 150 + ], + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ], + "face_reference": "asset:bench/walker.png", + "lines": [ + "I told myself I would not come back here.", + "But the street looks exactly the way I left it.", + "Same lamps, same crooked stones, same bakery smell.", + "Funny how a place waits for you without asking.", + "Maybe that is what I was afraid of.", + "Well. Let's see if the door still sticks." + ] + }, + { + "name": "s4-montage", + "crop": [ + 0, + 0, + 240, + 150 + ], + "moments": [ + "frame:62", + "frame:186", + "frame:434" + ] + }, + { + "name": "s5-ltx-action", + "crop": [ + 0, + 0, + 240, + 150 + ], + "moments": [ + "frame:60", + "frame:180", + "frame:299" + ] + }, + { + "name": "s6-narration", + "crop": [ + 0, + 0, + 240, + 150 + ], + "moments": [ + "frame:48", + "frame:144", + "frame:240" + ], + "lines": [ + "The keeper climbed the stairs every evening, long before the storm.", + "By the time he reached the lamp, the sea had already gone dark.", + "Then the light turned, and the whole coast could see him." + ] + } +] diff --git a/bench/workflows/s1-static.json b/bench/workflows/s1-static.json new file mode 100644 index 000000000..dda9c2972 --- /dev/null +++ b/bench/workflows/s1-static.json @@ -0,0 +1,197 @@ +{ + "id": "BenchS1Static", + "description": "Long-form bench S1 (static): the #1175 control (arm A), reproduced exactly - same workflow, prompt, still, seed 1166 and arguments as probe-1166's drift-chain. #1175 drift probe: templates/minimax/chained-segments as shipped (turbo-4step-keyframe, fl2va) with guide continuity, plus two probe variables: `anchor` (chain.anchor, arm C) and `guide_noise` (H3 call argument, arm B; null is dropped before the call, so the default is today's call). Defaults are the control (arm A).", + "variables": { + "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous static shot. The shot opens exactly on : a young woman in a plain dark grey sweater stands centered in front of a flat painted wall of bold diagonal rainbow stripes with a black circle at upper left and a teal square at lower right. She breathes calmly, blinks, shifts her weight slightly and turns her head a little left and right, staying in place at the center of the frame. The painted wall stays exactly as it is, evenly lit by soft studio light. The camera is locked off on a tripod and never moves.\n\noverall_soundscape: Quiet studio room tone, the soft rustle of a wool sweater, calm breathing.\n\nnon_diegetic_music: None.", + "image": "asset:bench/static-stripes.jpg", + "num_frames": 124, + "segments": 10, + "continuity": "guide", + "guide_frames": 22, + "width": 960, + "height": 544, + "num_inference_steps": 5, + "video_shift": 6, + "audio_shift": 3, + "weights_dtype": "{int4}", + "seed": 1166, + "anchor": "none", + "guide_noise": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "attention_backend": "sparse" + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + } + }, + "seed": 1166, + "steps": [ + { + "name": "chained_keyframe_to_video_audio", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "attention_backend": "variable:attention_backend", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "fl2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "turbo", + "scale": 1 + } + ], + "chain": { + "segments": "variable:segments", + "continuity": "variable:continuity", + "guide_frames": "variable:guide_frames", + "anchor": "variable:anchor", + "segment_argument": "image", + "trim_frames": 2, + "crossfade_ms": 80, + "fps": 24, + "frame_snap": "constraint:num_frames" + }, + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "image": "variable:image", + "num_frames": "variable:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "guide_noise": "variable:guide_noise", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "final" + } + } + ] +} diff --git a/bench/workflows/s2-arc.json b/bench/workflows/s2-arc.json new file mode 100644 index 000000000..9dd057352 --- /dev/null +++ b/bench/workflows/s2-arc.json @@ -0,0 +1,197 @@ +{ + "id": "BenchS2Arc", + "description": "Long-form bench S2 (arc): the #1175 control (arm A), reproduced exactly - same workflow, prompt, still, seed 1166 and arguments as probe-1166's drift-chain. #1175 drift probe: templates/minimax/chained-segments as shipped (turbo-4step-keyframe, fl2va) with guide continuity, plus two probe variables: `anchor` (chain.anchor, arm C) and `guide_noise` (H3 call argument, arm B; null is dropped before the call, so the default is today's call). Defaults are the control (arm A).", + "variables": { + "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a bronze statue of a seated man on a stone plinth in a sunlit public garden, a red brick wall with ivy, tall cypress trees and a white gazebo behind it. The camera travels in a slow, steady orbit around the statue, arcing to the right at a constant speed and keeping the statue centered, so the brick wall, the cypresses, the gazebo and the garden behind it sweep past and change with the angle. Warm late-afternoon sunlight, leaves stirring gently in a light breeze.\n\noverall_soundscape: Outdoor garden ambience: light breeze in the leaves, distant birdsong, faint far-off traffic.\n\nnon_diegetic_music: None.", + "image": "asset:bench/arc-statue.jpg", + "num_frames": 124, + "segments": 10, + "continuity": "guide", + "guide_frames": 22, + "width": 960, + "height": 544, + "num_inference_steps": 5, + "video_shift": 6, + "audio_shift": 3, + "weights_dtype": "{int4}", + "seed": 1166, + "anchor": "none", + "guide_noise": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "attention_backend": "sparse" + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + } + }, + "seed": 1166, + "steps": [ + { + "name": "chained_keyframe_to_video_audio", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "attention_backend": "variable:attention_backend", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "fl2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "turbo", + "scale": 1 + } + ], + "chain": { + "segments": "variable:segments", + "continuity": "variable:continuity", + "guide_frames": "variable:guide_frames", + "anchor": "variable:anchor", + "segment_argument": "image", + "trim_frames": 2, + "crossfade_ms": 80, + "fps": 24, + "frame_snap": "constraint:num_frames" + }, + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "image": "variable:image", + "num_frames": "variable:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "guide_noise": "variable:guide_noise", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "final" + } + } + ] +} diff --git a/bench/workflows/s3-dialogue-walk.json b/bench/workflows/s3-dialogue-walk.json new file mode 100644 index 000000000..b497bc524 --- /dev/null +++ b/bench/workflows/s3-dialogue-walk.json @@ -0,0 +1,206 @@ +{ + "id": "BenchS3DialogueWalk", + "description": "Long-form bench S3 (dialogue walk): templates/minimax/chained-segments with continuity 'guide' and carry_audio, six segments, a cast portrait (asset:bench/walker.png) and one short spoken line per segment through `prompts`. Each line is sized to a segment (see plugins/dw/skills/minimax-h3/references/dialogue.md). `anchor` and `guide_noise` are the drift-probe knobs; their defaults are today's call.", + "variables": { + "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. She walks toward the camera down the cobblestone street, slow and steady, glancing up at the lamps. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] I told myself I would not come back here. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", + "image": "asset:bench/walker.png", + "num_frames": 124, + "segments": 6, + "continuity": "guide", + "guide_frames": 22, + "width": 960, + "height": 544, + "num_inference_steps": 5, + "video_shift": 6, + "audio_shift": 3, + "weights_dtype": "{int4}", + "seed": 1166, + "anchor": "none", + "guide_noise": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "attention_backend": "sparse", + "prompts": [ + "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. She walks toward the camera down the cobblestone street, slow and steady, glancing up at the lamps. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] I told myself I would not come back here. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", + "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. Her eyes move along the shopfronts as she walks on, the long olive coat swinging with each step. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] But the street looks exactly the way I left it. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", + "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. She slows for a moment, looks down at the stones, then lifts her chin and walks on. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] Same lamps, same crooked stones, same bakery smell. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", + "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. A faint smile crosses her face as she keeps walking toward the camera. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] Funny how a place waits for you without asking. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", + "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. Her expression softens and her pace eases, the golden light warm along her shoulder. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] Maybe that is what I was afraid of. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", + "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. She squares her shoulders and walks on, the camera keeping her centered in frame. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] Well. Let's see if the door still sticks. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None." + ] + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + } + }, + "seed": 1166, + "steps": [ + { + "name": "chained_keyframe_to_video_audio", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "attention_backend": "variable:attention_backend", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "fl2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "turbo", + "scale": 1 + } + ], + "chain": { + "segments": "variable:segments", + "continuity": "variable:continuity", + "guide_frames": "variable:guide_frames", + "anchor": "variable:anchor", + "segment_argument": "image", + "trim_frames": 2, + "crossfade_ms": 80, + "fps": 24, + "frame_snap": "constraint:num_frames", + "carry_audio": true + }, + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "image": "variable:image", + "num_frames": "variable:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "guide_noise": "variable:guide_noise", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "final" + } + } + ] +} diff --git a/bench/workflows/s4-montage.json b/bench/workflows/s4-montage.json new file mode 100644 index 000000000..e5b3d97cf --- /dev/null +++ b/bench/workflows/s4-montage.json @@ -0,0 +1,501 @@ +{ + "id": "BenchS4Montage", + "description": "Long-form bench S4 (montage): templates/minimax/dialogue-short's cast and four of its shots, cut and scored under one templates/minimax/music instrumental bed the way templates/assemble-and-score does it. Each shot is generated fresh from the same portraits, so it carries no chain drift; the bed is the one thing the cuts share.", + "summary": "Four cut H3 scenes under one Music 3 bed, cut and scored.", + "cost": [ + { + "device": "cuda", + "name": "RTX 3090", + "vram_gb": 24, + "minutes": 21.2 + } + ], + "cost_drivers": [ + "shots", + "num_inference_steps", + "width", + "height", + "weights_dtype", + "lora_model_name", + "attention_backend", + "cache", + "audio_duration" + ], + "vram_estimate": { + "base_gb": 16.0, + "bytes_per_voxel": 28.71, + "gb_per_reference": 1.0, + "voxel_variables": [ + "width", + "height", + "num_frames" + ], + "reason": "Ref2VA holds every reference's encoding resident beside the video latents, so the ceiling is a fixed base, a per-voxel term for the VAE decode's conv3d activation (width * height * num_frames), and a per-reference term. Classified against the 24 GB entry from the #479 field report (lem, RTX 3090 24 GB, int4 + 768p turbo LoRA, 9 steps, 2026-09-25/26) at 1344x768: 175 frames x 2 refs, 209 x 2 and 175 x 3 ran (projected 22.83, 23.77 and 23.83 GB); 209 x 3 OOM'd in reference encode needing a further 5.1 GB and 260 x 1 OOM'd at denoise step 1 needing a further 4.8 GB (projected 24.77 and 24.18 GB). The refused allocations are lower bounds, not peaks, so this is a classification with about 0.2 GB of margin either side, not a fitted curve. bytes_per_voxel is unchanged from #324's calibration (job 282172105da1); base_gb drops 0.3 GB now the references are counted separately. The cost entry's 42 min was that field run at 1344x768; re-priced to these 960x544 defaults it is 42 x 960*544 / (1344*768) = 21.2 (#866)" + }, + "variables": { + "character_a_portrait_prompt": "a 3/4-length studio portrait of a wiry man in his early 30s with short curly dark hair and expressive eyebrows, wearing a mustard-yellow sweater vest over a white button-down shirt, standing in a 1990s multi-camera sitcom apartment set with a sagging olive-green couch and a brick wall behind him, warm even studio lighting, sharp focus, photographic", + "character_b_portrait_prompt": "a 3/4-length studio portrait of a heavyset man in his mid 40s with a shaved head and a calm, heavy-lidded expression, wearing an open gray flannel shirt over a dark t-shirt, standing in a 1990s multi-camera sitcom apartment set with a sagging olive-green couch and a brick wall behind him, warm even studio lighting, sharp focus, photographic", + "shots": [ + { + "name": "cold_open", + "num_frames": 124, + "references": [ + { + "reference_type": "variable:subject_reference_type", + "from_previous_result": "draw_character_a" + }, + { + "reference_type": "variable:subject_reference_type", + "from_previous_result": "draw_character_b" + }, + { + "reference_type": "variable:voice_reference_type", + "from_file": "variable:character_a_voice" + }, + { + "reference_type": "variable:voice_reference_type", + "from_file": "variable:character_b_voice" + } + ], + "prompt": "subject_definitions:\n is the wiry man in , early 30s, short curly dark hair, mustard-yellow sweater vest over a white button-down shirt.\n is the heavyset man in , mid 40s, shaved head, open gray flannel shirt over a dark t-shirt.\n\nsummary:\n[reference generation] The target video is one shot of a 1990s multi-camera sitcom: reads on the couch while enters holding an empty ice cream carton and confronts him, in front of a live studio audience.\n\nretention_analysis:\n (appears in [Shot 1]): fully_preserved - the face, hair, sweater vest and proportions of are kept exactly, with the body newly animated to walk and speak.\n (appears in [Shot 1]): fully_preserved - the face, head, flannel shirt and proportions of are kept exactly, with the body newly posed seated and reading.\n\ndetailed_description:\nThe target video is in the style of a 1990s multi-camera sitcom, shot on studio video with warm, even, shadowless lighting. [Shot 1] A wide shot frames a living room set: a sagging olive-green couch centre frame in front of a brick wall with a window onto a painted city backdrop. sits at the right end of the couch, ankles crossed, reading a magazine and not looking up. strides in from frame left holding an empty pint-sized ice cream carton at chest height, stops beside the couch, holds the carton out toward and turns it upside down to show it is empty. (S1), a fast, nasal, high-pitched male tenor voice, tight with disbelief, says [English] Okay. Who finished the pistachio? keeps reading and does not react. A live studio audience laughs warmly as the carton turns over. The camera holds a static wide shot throughout, level with the set, the way a sitcom master camera holds a scene.\n\noverall_soundscape:\nA bright, close television-studio room tone. Quick footsteps on a hollow stage floor as enters, the papery rattle of the empty carton as it turns over, a soft page turn from the couch, and a warm burst of live studio-audience laughter that begins as the carton flips and settles under the end of the shot.\n\nnon_diegetic_music:\nN/A" + }, + { + "name": "deflect", + "num_frames": 124, + "references": [ + { + "reference_type": "variable:subject_reference_type", + "from_previous_result": "draw_character_b" + }, + { + "reference_type": "variable:voice_reference_type", + "from_file": "variable:character_b_voice" + } + ], + "prompt": "subject_definitions:\n is the heavyset man in , mid 40s, shaved head, calm and heavy-lidded, open gray flannel shirt over a dark t-shirt.\n\nsummary:\n[reference generation] The target video is one close-up shot of a 1990s multi-camera sitcom: answers an accusation without looking up from his magazine, in front of a live studio audience.\n\nretention_analysis:\n (appears in [Shot 1]): fully_preserved - the face, head, flannel shirt and proportions of are kept exactly, with the head and jaw newly animated to speak.\n\ndetailed_description:\nThe target video is in the style of a 1990s multi-camera sitcom, shot on studio video with warm, even, shadowless lighting. [Shot 1] A medium close-up frames from the chest up, seated at the end of an olive-green couch in front of a brick wall, holding an open magazine just below the frame line and reading it. He does not look up. (S1), a flat, low, unhurried male baritone voice, completely unbothered, says [English] Technically? The freezer has a defrost cycle. On the last word he turns a page. A live studio audience laughs as the line lands. The camera holds a static close-up throughout, the way a sitcom cross-shot camera holds a speaker.\n\noverall_soundscape:\nA bright, close television-studio room tone. A soft page turn, the faint creak of couch springs as settles, and a rolling wave of live studio-audience laughter that starts on the last word of the line and carries under the end of the shot.\n\nnon_diegetic_music:\nN/A" + }, + { + "name": "react", + "num_frames": 124, + "references": [ + { + "reference_type": "variable:subject_reference_type", + "from_previous_result": "draw_character_a" + }, + { + "reference_type": "variable:voice_reference_type", + "from_file": "variable:character_a_voice" + } + ], + "prompt": "subject_definitions:\n is the wiry man in , early 30s, short curly dark hair, expressive eyebrows, mustard-yellow sweater vest over a white button-down shirt.\n\nsummary:\n[reference generation] The target video is one close-up shot of a 1990s multi-camera sitcom: repeats an absurd excuse back in disbelief, holding up an empty ice cream carton, in front of a live studio audience.\n\nretention_analysis:\n (appears in [Shot 1]): fully_preserved - the face, hair, sweater vest and proportions of are kept exactly, with the head, jaw and arm newly animated to speak and gesture.\n\ndetailed_description:\nThe target video is in the style of a 1990s multi-camera sitcom, shot on studio video with warm, even, shadowless lighting. [Shot 1] A medium close-up frames from the chest up, standing beside a couch in front of a brick wall, an empty pint-sized ice cream carton raised in one hand at head height. His eyebrows are high and his mouth is half open before he speaks. (S1), a fast, nasal, high-pitched male tenor voice, tight with disbelief, cracking upward on the last word, says [English] The defrost cycle used a spoon? He shakes the empty carton once on the word spoon. A live studio audience erupts in a big laugh as the line lands. The camera holds a static close-up throughout, the way a sitcom cross-shot camera holds a speaker.\n\noverall_soundscape:\nA bright, close television-studio room tone. The hollow papery rattle of the empty carton as it shakes, and a big rising wave of live studio-audience laughter that breaks on the word spoon and rolls under the end of the shot.\n\nnon_diegetic_music:\nN/A" + }, + { + "name": "button", + "num_frames": 124, + "references": [ + { + "reference_type": "variable:subject_reference_type", + "from_previous_result": "draw_character_b" + }, + { + "reference_type": "variable:voice_reference_type", + "from_file": "variable:character_b_voice" + } + ], + "prompt": "subject_definitions:\n is the heavyset man in , mid 40s, shaved head, calm and heavy-lidded, open gray flannel shirt over a dark t-shirt.\n\nsummary:\n[reference generation] The target video is one close-up shot of a 1990s multi-camera sitcom: shrugs off the accusation with a deadpan line, in front of a live studio audience.\n\nretention_analysis:\n (appears in [Shot 1]): fully_preserved - the face, head, flannel shirt and proportions of are kept exactly, with the head, jaw and shoulders newly animated to speak and shrug.\n\ndetailed_description:\nThe target video is in the style of a 1990s multi-camera sitcom, shot on studio video with warm, even, shadowless lighting. [Shot 1] A medium close-up frames from the chest up, seated at the end of an olive-green couch in front of a brick wall, the magazine now closed in his lap. He looks up for the first time, holds a beat of eye contact just past the lens, and gives one slow shrug. (S1), a flat, low, unhurried male baritone voice, perfectly deadpan, says [English] I don't know its life. He goes back to the magazine. A live studio audience gives a big laugh with scattered applause as he looks back down. The camera holds a static close-up throughout, the way a sitcom cross-shot camera holds a speaker.\n\noverall_soundscape:\nA bright, close television-studio room tone. The soft slap of the magazine reopening, a couch-spring creak under the shrug, and a big wave of live studio-audience laughter with a few claps in it that breaks after the line and rolls under the end of the shot.\n\nnon_diegetic_music:\nN/A" + } + ], + "character_a_voice": null, + "character_b_voice": null, + "voice_reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3AudioReference", + "subject_reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3ImageReference", + "audio_bleed_ms": 1800, + "audio_bleed_gain_db": 0.0, + "seam_fade_ms": null, + "match_levels": null, + "match_levels_dbfs": null, + "width": 960, + "height": 544, + "num_inference_steps": 9, + "video_shift": 12.0, + "audio_shift": 3.0, + "weights_dtype": "{int4}", + "lora_scale": 1.0, + "lora_alpha": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors", + "lora_adapter_name": "turbo", + "seed": 1166, + "attention_backend": "sparse", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "cache": null, + "music_prompt": "A slow, tense instrumental score: low bowed cello and a soft pulsing synth under sparse piano notes, restrained, cinematic, no vocals.", + "music_lyrics": "[intro]\n[instrumental]\n[solo]\n[outro]", + "audio_duration": 30, + "sample_rate": 44100, + "fps": 24, + "score_start_frame": 0, + "total_frames": 496, + "score_gain": 1.0, + "world_gain": 1.8, + "world_fade_out_ms": 0, + "target_lufs": null, + "limit": false + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "MiniMax-H3's pipeline refuses a `width` or `height` that is not a multiple of 32, and only after the weights have loaded; refused here instead, with no 'snap', since rounding would be a silent change to the size asked for" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "MiniMax-H3's pipeline refuses a `width` or `height` that is not a multiple of 32, and only after the weights have loaded; refused here instead, with no 'snap', since rounding would be a silent change to the size asked for" + } + }, + "seed": "variable:seed", + "steps": [ + { + "name": "draw_character_a", + "pipeline": { + "configuration": { + "component_type": "ZImagePipeline", + "offload": "model" + }, + "from_pretrained_arguments": { + "model_name": "Tongyi-MAI/Z-Image-Turbo", + "torch_dtype": "torch.bfloat16", + "low_cpu_mem_usage": true + }, + "arguments": { + "prompt": "variable:character_a_portrait_prompt", + "num_inference_steps": 9, + "guidance_scale": 0.0, + "width": 768, + "height": 768 + } + }, + "result": { + "content_type": "image/jpeg", + "embed_metadata": true, + "subfolder": "intermediate", + "save": false + } + }, + { + "name": "draw_character_b", + "release_pipeline": true, + "pipeline": { + "configuration": { + "component_type": "ZImagePipeline", + "offload": "model" + }, + "from_pretrained_arguments": { + "model_name": "Tongyi-MAI/Z-Image-Turbo", + "torch_dtype": "torch.bfloat16", + "low_cpu_mem_usage": true + }, + "arguments": { + "prompt": "variable:character_b_portrait_prompt", + "num_inference_steps": 9, + "guidance_scale": 0.0, + "width": 768, + "height": 768 + } + }, + "result": { + "content_type": "image/jpeg", + "embed_metadata": true, + "subfolder": "intermediate", + "save": false + } + }, + { + "name": "shot", + "for_each": "variable:shots", + "release_pipeline": true, + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer_ref": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer_ref": { + "attention_backend": "variable:attention_backend", + "sparse_tau": "variable:sparse_tau", + "sparse_topk": "variable:sparse_topk", + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 1, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + }, + "cache": "variable:cache" + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "ref2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "variable:lora_adapter_name", + "scale": "variable:lora_scale", + "alpha": "variable:lora_alpha" + } + ], + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "item:prompt", + "references": "item:references", + "num_frames": "item:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "intermediate" + } + }, + { + "name": "music_bed", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "load_components": { + "dtype": "torch.bfloat16" + }, + "components_manager": { + "enable_auto_cpu_offload": true + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-Music3" + }, + "arguments": { + "prompt": "variable:music_prompt", + "lyrics": "variable:music_lyrics", + "audio_duration": "variable:audio_duration", + "output": "audios" + } + }, + "result": { + "content_type": "audio/mp3", + "sample_rate": 44100, + "save": false + }, + "release_pipeline": true + }, + { + "name": "edit", + "task": { + "command": "concat_videos", + "arguments": { + "videos": "gather:shot", + "fps": "variable:fps", + "sample_rate": "variable:sample_rate", + "audio_bleed_ms": "variable:audio_bleed_ms", + "audio_bleed_gain_db": "variable:audio_bleed_gain_db", + "seam_fade_ms": "variable:seam_fade_ms", + "match_levels": "variable:match_levels", + "match_levels_dbfs": "variable:match_levels_dbfs" + } + } + }, + { + "name": "world", + "task": { + "command": "resample_audio", + "arguments": { + "audio": "previous_result:edit", + "target_sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "world_faded", + "task": { + "command": "fade_audio", + "arguments": { + "audio": "previous_result:world", + "fade_out_ms": "variable:world_fade_out_ms", + "sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "soundtrack", + "task": { + "command": "slice_audio", + "arguments": { + "audio": "previous_result:music_bed", + "start_frame": "variable:score_start_frame", + "num_frames": "variable:total_frames", + "fps": "variable:fps" + } + } + }, + { + "name": "soundtrack_resampled", + "task": { + "command": "resample_audio", + "arguments": { + "audio": "previous_result:soundtrack", + "target_sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "mixed", + "task": { + "command": "mix_audio", + "arguments": { + "audios": [ + "previous_result:soundtrack_resampled", + "previous_result:world_faded" + ], + "gains": [ + "variable:score_gain", + "variable:world_gain" + ], + "sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "balanced", + "task": { + "command": "normalize_audio", + "arguments": { + "audio": "previous_result:mixed", + "peak_dbfs": -3.0, + "target_lufs": "variable:target_lufs", + "limit": "variable:limit", + "sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "film", + "task": { + "command": "pair_audio", + "arguments": { + "video": "previous_result:edit", + "audio": "previous_result:balanced", + "sample_rate": "variable:sample_rate", + "fit": "video" + } + }, + "result": { + "content_type": "video/mp4", + "fps": "variable:fps", + "subfolder": "final" + } + } + ] +} diff --git a/bench/workflows/s5-ltx-action.json b/bench/workflows/s5-ltx-action.json new file mode 100644 index 000000000..36f36aa2d --- /dev/null +++ b/bench/workflows/s5-ltx-action.json @@ -0,0 +1,178 @@ +{ + "id": "BenchS5LtxAction", + "description": "Long-form bench S5 (LTX action): templates/ltx2/chained-segments, three 121-frame segments of one continuous silent action (a courier cycling along a coastal road, from asset:bench/runner.png), chained by last frame. No speech anywhere; the sound is wind, sea and the bicycle.", + "summary": "A long LTX-2.5 clip with audio, chained from short segments that each start on the previous one's last frame.", + "cost_drivers": [ + "segments", + "num_frames", + "width", + "height" + ], + "variables": { + "transformer_weights_dtype": "{uint4}", + "text_encoder_weights_dtype": "{int8}", + "prompt": "The courier, a man in a blue shirt and black helmet with a yellow delivery box on the back, is pedaling steadily along the coastal road toward the right of frame, leaning slightly into the wind, the sea glittering beyond the low stone wall behind him as the camera tracks alongside at his speed. The only sounds are the whir of the chain, the hum of tires on asphalt and the steady rush of the wind and surf, with no speech and no music.", + "prompts": [ + "The courier, a man in a blue shirt and black helmet with a yellow delivery box on the back, is pedaling steadily along the coastal road toward the right of frame, leaning slightly into the wind, the sea glittering beyond the low stone wall behind him as the camera tracks alongside at his speed. The only sounds are the whir of the chain, the hum of tires on asphalt and the steady rush of the wind and surf, with no speech and no music.", + "The courier keeps pedaling at the same pace, the road running on along the shoreline, waves rolling in beyond the stone wall as the camera continues to track alongside at his speed. The only sounds are the whir of the chain, tires on asphalt and the wind and surf, with no speech and no music.", + "The courier pedals on without slowing, the road running straight ahead along the water, gulls wheeling over the waves as the camera continues to track alongside at his speed. The only sounds are the whir of the chain, tires on asphalt and the wind and surf, with no speech and no music." + ], + "negative_prompt": "constant:diffusers.pipelines.ltx2.utils.DEFAULT_NEGATIVE_PROMPT", + "width": 960, + "height": 544, + "frame_rate": 24.0, + "num_frames": 121, + "segments": 3, + "image": "asset:bench/runner.png", + "seed": 1166 + }, + "variable_constraints": { + "width": { + "modulus": 32, + "remainder": 0, + "reason": "LTX-2.5's pipelines refuse a width not divisible by 32 outright (ValueError, not a silent floor), so it is checked here before the chain's first segment load rather than after it" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "LTX-2.5's pipelines refuse a height not divisible by 32 outright (ValueError, not a silent floor), so it is checked here before the chain's first segment load rather than after it" + }, + "num_frames": { + "modulus": 8, + "remainder": 1, + "min_frames": 9, + "reason": "LTX-2.5's video VAE encodes 8 * n + 1 frames, so an off-grid count is not refused by the pipeline but floored to the grid below - a shorter clip than the one asked for. Refused here instead: no 'snap', because rounding a length the other way from the pipeline's own would be a second silent change" + } + }, + "seed": "variable:seed", + "steps": [ + { + "name": "chained_image_to_video", + "pipeline": { + "configuration": { + "component_type": "LTX2ImageToVideoPipeline", + "pre_load_modules": [ + "sdnq" + ], + "components": { + "transformer": { + "device": "cuda" + }, + "text_encoder": { + "group_offload": { + "offload_type": "leaf_level", + "use_stream": true + } + }, + "connectors": { + "group_offload": { + "offload_type": "leaf_level", + "use_stream": true + } + }, + "vae": { + "device": "cuda" + }, + "audio_vae": { + "device": "cuda" + }, + "vocoder": { + "device": "cuda" + }, + "duration_head": { + "device": "cuda" + } + }, + "vae": { + "enable_tiling": true + } + }, + "from_pretrained_arguments": { + "model_name": "Lightricks/LTX-2.5-Diffusers", + "torch_dtype": "torch.bfloat16" + }, + "transformer": { + "configuration": { + "component_type": "LTX2VideoTransformer3DModel", + "preserve_device_placement": true + }, + "quantization_config": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:transformer_weights_dtype", + "quantization_device": "cuda", + "return_device": "cuda", + "use_quantized_matmul": true, + "dequantize_fp32": false + } + }, + "from_pretrained_arguments": { + "model_name": "Lightricks/LTX-2.5-Diffusers", + "subfolder": "transformer", + "torch_dtype": "torch.bfloat16" + } + }, + "text_encoder": { + "configuration": { + "component_type": "transformers.Gemma4UnifiedForConditionalGeneration", + "preserve_device_placement": true + }, + "quantization_config": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:text_encoder_weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + "embed_vision", + "embed_audio", + "lm_head" + ] + } + }, + "from_pretrained_arguments": { + "model_name": "Lightricks/LTX-2.5-Diffusers", + "subfolder": "text_encoder", + "torch_dtype": "torch.bfloat16" + } + }, + "chain": { + "segments": "variable:segments", + "prompts": "variable:prompts", + "continuity": "last_frame", + "segment_argument": "image", + "trim_frames": 2, + "crossfade_ms": 80, + "save_segments": true + }, + "arguments": { + "prompt": "variable:prompt", + "negative_prompt": "variable:negative_prompt", + "image": "variable:image", + "width": "variable:width", + "height": "variable:height", + "num_frames": "variable:num_frames", + "frame_rate": "variable:frame_rate", + "sigmas": "constant:diffusers.pipelines.ltx2.utils.DISTILLED_SIGMA_VALUES", + "guidance_scale": 1.0, + "audio_guidance_scale": 1.0, + "stg_scale": 0.0, + "audio_stg_scale": 0.0, + "modality_scale": 1.0, + "audio_modality_scale": 1.0, + "output_type": "{np}" + } + }, + "result": { + "content_type": "video/mp4", + "fps": "variable:frame_rate", + "subfolder": "final" + } + } + ] +} diff --git a/bench/workflows/s6-narration.json b/bench/workflows/s6-narration.json new file mode 100644 index 000000000..4c78bc033 --- /dev/null +++ b/bench/workflows/s6-narration.json @@ -0,0 +1,180 @@ +{ + "id": "BenchS6Narration", + "summary": "Three narration takes and a Music 3 bed under three animatic boards.", + "description": "Long-form bench S6 (narration): three templates/generate-speech narration takes in the default voice (no consented cast voice clip is on the bench host, so templates/voice-clone is not used), a templates/minimax/music instrumental bed, and three storyboard stills held by templates/animatic. Narration and bed are mixed and paired under the boards.", + "cost_drivers": [ + "audio_duration" + ], + "variables": { + "take_1": "The keeper climbed the stairs every evening, long before the storm.", + "take_2": "By the time he reached the lamp, the sea had already gone dark.", + "take_3": "Then the light turned, and the whole coast could see him.", + "speech_model": "facebook/mms-tts-eng", + "music_prompt": "A slow, spare instrumental score: soft piano and a low sustained string pad, patient and melancholy, no vocals.", + "music_lyrics": "[intro]\n[instrumental]\n[outro]", + "audio_duration": 30, + "sample_rate": 44100, + "boards": [ + "asset:bench/board_1.png", + "asset:bench/board_2.png", + "asset:bench/board_3.png" + ], + "durations": [ + 4.0, + 4.0, + 4.0 + ], + "fps": 24.0, + "narration_gain": 1.0, + "bed_gain": 0.35, + "seed": 1166 + }, + "seed": "variable:seed", + "steps": [ + { + "name": "narrate_1", + "task": { + "command": "generate_speech", + "arguments": { + "text": "variable:take_1", + "model_name": "variable:speech_model" + } + } + }, + { + "name": "narrate_2", + "task": { + "command": "generate_speech", + "arguments": { + "text": "variable:take_2", + "model_name": "variable:speech_model" + } + } + }, + { + "name": "narrate_3", + "task": { + "command": "generate_speech", + "arguments": { + "text": "variable:take_3", + "model_name": "variable:speech_model" + } + } + }, + { + "name": "narration", + "task": { + "command": "crossfade_audio", + "arguments": { + "audios": [ + "previous_result:narrate_1", + "previous_result:narrate_2", + "previous_result:narrate_3" + ], + "crossfade_ms": 75, + "sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "narration_resampled", + "task": { + "command": "resample_audio", + "arguments": { + "audio": "previous_result:narration", + "target_sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "generate_music", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "load_components": { + "dtype": "torch.bfloat16" + }, + "components_manager": { + "enable_auto_cpu_offload": true + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-Music3" + }, + "arguments": { + "prompt": "variable:music_prompt", + "lyrics": "variable:music_lyrics", + "audio_duration": "variable:audio_duration", + "output": "audios" + } + }, + "result": { + "content_type": "audio/mp3", + "sample_rate": 44100, + "save": false + }, + "release_pipeline": true + }, + { + "name": "balanced", + "task": { + "command": "normalize_audio", + "arguments": { + "audio": "previous_result:generate_music", + "peak_dbfs": -3.0, + "sample_rate": 44100 + } + }, + "result": { + "content_type": "audio/mp3", + "sample_rate": 44100, + "save": false + } + }, + { + "name": "soundtrack", + "task": { + "command": "mix_audio", + "arguments": { + "audios": [ + "previous_result:narration_resampled", + "previous_result:balanced" + ], + "gains": [ + "variable:narration_gain", + "variable:bed_gain" + ], + "sample_rate": "variable:sample_rate" + } + } + }, + { + "name": "stills", + "task": { + "command": "stills_to_video", + "arguments": { + "images": "variable:boards", + "durations": "variable:durations", + "fps": "variable:fps" + } + } + }, + { + "name": "film", + "task": { + "command": "pair_audio", + "arguments": { + "video": "previous_result:stills", + "audio": "previous_result:soundtrack", + "fps": "variable:fps", + "fit": "video" + } + }, + "result": { + "content_type": "video/mp4", + "fps": "variable:fps", + "subfolder": "final" + } + } + ] +} diff --git a/tests/test_bench_workflows.py b/tests/test_bench_workflows.py new file mode 100644 index 000000000..f0634494e --- /dev/null +++ b/tests/test_bench_workflows.py @@ -0,0 +1,68 @@ +"""The long-form bench's six scenes: every workflow validates, and every arm +and scene names files that exist (bench/driver.py loads them).""" + +import json +import sys +from pathlib import Path + +import pytest + +BENCH = Path(__file__).resolve().parent.parent / "bench" +sys.path.insert(0, str(BENCH)) +import driver # noqa: E402 + +WORKFLOWS = sorted((BENCH / "workflows").glob("s*.json")) +IDS = ["s1-static", "s2-arc", "s3-dialogue-walk", "s4-montage", "s5-ltx-action", "s6-narration"] + + +def _errors(path): + from dw.workflow import Workflow + + definition = json.loads(path.read_text(encoding="utf-8")) + return Workflow(definition, None, str(path)).validation_errors({}) + + +def test_the_six_scene_workflows_exist(): + assert [p.stem for p in WORKFLOWS] == IDS + + +@pytest.mark.parametrize("path", WORKFLOWS, ids=lambda p: p.stem) +def test_every_bench_workflow_validates(path): + assert _errors(path) == [] + + +@pytest.mark.parametrize("path", WORKFLOWS, ids=lambda p: p.stem) +def test_chain_workflows_expose_the_arm_knobs(path): + variables = json.loads(path.read_text(encoding="utf-8"))["variables"] + assert "seed" in variables + if path.stem in ("s1-static", "s2-arc", "s3-dialogue-walk"): + for knob in ("continuity", "guide_frames", "segments", "anchor", "guide_noise", + "width", "height", "num_inference_steps", "weights_dtype", + "lora_model_name", "attention_backend"): + assert knob in variables, knob + + +def test_every_arm_names_a_scene_and_a_workflow_that_exist(): + arms = driver.load_arms(BENCH / "arms" / "baseline.json") + scenes = driver.load_scenes(BENCH / "scenes.json") + assert sorted(scenes) == IDS + assert sorted(a.scene for a in arms) == IDS + for arm in arms: + assert arm.scene in scenes + assert (BENCH / "workflows" / arm.workflow).is_file() + assert arm.seed == 1166 + + +def test_scenes_are_well_formed(): + for scene in driver.load_scenes(BENCH / "scenes.json").values(): + x0, y0, x1, y1 = scene.crop + assert 0 <= x0 < x1 and 0 <= y0 < y1 + assert len(scene.moments) == 3 + assert all(m.startswith("frame:") for m in scene.moments) + frames = [int(m.split(":")[1]) for m in scene.moments] + assert frames == sorted(frames) + + +def test_s2_reproduces_the_1175_arc_crop(): + scenes = driver.load_scenes(BENCH / "scenes.json") + assert scenes["s2-arc"].crop == [0, 0, 240, 150] From ff89fce10a0509ef19964f8c3d9062a0f2cf4ed3 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:06:00 -0600 Subject: [PATCH 012/405] fix(bench): S3 speaks a line per segment; crops marked provisional Co-Authored-By: Claude Opus 5.5 --- bench/driver.py | 2 ++ bench/scenes.json | 18 ++++++++++++------ bench/workflows/s3-dialogue-walk.json | 1 + bench/workflows/s5-ltx-action.json | 3 ++- tests/test_bench_workflows.py | 8 ++++++++ 5 files changed, 25 insertions(+), 7 deletions(-) diff --git a/bench/driver.py b/bench/driver.py index 4d77b88e3..3b48bf089 100644 --- a/bench/driver.py +++ b/bench/driver.py @@ -41,6 +41,7 @@ class Scene: moments: list # "frame:N" at segments 1, middle and last face_reference: str | None = None lines: list | None = None + crop_verified: bool = False def load_arms(path): @@ -66,6 +67,7 @@ def load_scenes(path): moments=s["moments"], face_reference=s.get("face_reference"), lines=s.get("lines"), + crop_verified=s.get("crop_verified", False), ) for s in data } diff --git a/bench/scenes.json b/bench/scenes.json index 8aca4c29a..580d48928 100644 --- a/bench/scenes.json +++ b/bench/scenes.json @@ -12,7 +12,8 @@ "frame:481", "frame:990" ], - "face_reference": "asset:bench/static-stripes.jpg" + "face_reference": "asset:bench/static-stripes.jpg", + "crop_verified": true }, { "name": "s2-arc", @@ -26,7 +27,8 @@ "frame:62", "frame:481", "frame:990" - ] + ], + "crop_verified": true }, { "name": "s3-dialogue-walk", @@ -49,7 +51,8 @@ "Funny how a place waits for you without asking.", "Maybe that is what I was afraid of.", "Well. Let's see if the door still sticks." - ] + ], + "crop_verified": false }, { "name": "s4-montage", @@ -63,7 +66,8 @@ "frame:62", "frame:186", "frame:434" - ] + ], + "crop_verified": false }, { "name": "s5-ltx-action", @@ -77,7 +81,8 @@ "frame:60", "frame:180", "frame:299" - ] + ], + "crop_verified": false }, { "name": "s6-narration", @@ -96,6 +101,7 @@ "The keeper climbed the stairs every evening, long before the storm.", "By the time he reached the lamp, the sea had already gone dark.", "Then the light turned, and the whole coast could see him." - ] + ], + "crop_verified": false } ] diff --git a/bench/workflows/s3-dialogue-walk.json b/bench/workflows/s3-dialogue-walk.json index b497bc524..b737659ef 100644 --- a/bench/workflows/s3-dialogue-walk.json +++ b/bench/workflows/s3-dialogue-walk.json @@ -164,6 +164,7 @@ } ], "chain": { + "prompts": "variable:prompts", "segments": "variable:segments", "continuity": "variable:continuity", "guide_frames": "variable:guide_frames", diff --git a/bench/workflows/s5-ltx-action.json b/bench/workflows/s5-ltx-action.json index 36f36aa2d..dd7824864 100644 --- a/bench/workflows/s5-ltx-action.json +++ b/bench/workflows/s5-ltx-action.json @@ -23,6 +23,7 @@ "frame_rate": 24.0, "num_frames": 121, "segments": 3, + "continuity": "last_frame", "image": "asset:bench/runner.png", "seed": 1166 }, @@ -144,7 +145,7 @@ "chain": { "segments": "variable:segments", "prompts": "variable:prompts", - "continuity": "last_frame", + "continuity": "variable:continuity", "segment_argument": "image", "trim_frames": 2, "crossfade_ms": 80, diff --git a/tests/test_bench_workflows.py b/tests/test_bench_workflows.py index f0634494e..6d8cc096e 100644 --- a/tests/test_bench_workflows.py +++ b/tests/test_bench_workflows.py @@ -66,3 +66,11 @@ def test_scenes_are_well_formed(): def test_s2_reproduces_the_1175_arc_crop(): scenes = driver.load_scenes(BENCH / "scenes.json") assert scenes["s2-arc"].crop == [0, 0, 240, 150] + + +def test_a_workflow_declaring_prompts_references_it(): + for path in sorted((BENCH / "workflows").glob("*.json")): + definition = json.loads(path.read_text()) + if "prompts" in (definition.get("variables") or {}): + steps = json.dumps(definition.get("steps")) + assert "variable:prompts" in steps, path.name From ce05f7b59ab78662fd8d05a8894990e36fd8057c Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:08:44 -0600 Subject: [PATCH 013/405] feat(bench): noise and cheap-check arms Co-Authored-By: Claude Opus 5.5 --- bench/arms/cheap-checks.json | 90 +++++++++++++++++++++++++++++++++++ bench/arms/noise.json | 92 ++++++++++++++++++++++++++++++++++++ bench/driver.py | 12 +++-- tests/test_bench_driver.py | 34 +++++++++++++ 4 files changed, 225 insertions(+), 3 deletions(-) create mode 100644 bench/arms/cheap-checks.json create mode 100644 bench/arms/noise.json diff --git a/bench/arms/cheap-checks.json b/bench/arms/cheap-checks.json new file mode 100644 index 000000000..9c4b6dbbb --- /dev/null +++ b/bench/arms/cheap-checks.json @@ -0,0 +1,90 @@ +[ + { + "name": "cheap-a-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "cheap-b-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "guide_noise": 0.4 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "cheap-c-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "anchor": "segment_1" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "cheap-a-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "cheap-b-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "guide_noise": 0.4 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "cheap-c-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "anchor": "segment_1" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + } +] \ No newline at end of file diff --git a/bench/arms/noise.json b/bench/arms/noise.json new file mode 100644 index 000000000..c4a2200e3 --- /dev/null +++ b/bench/arms/noise.json @@ -0,0 +1,92 @@ +[ + { + "name": "noise-a-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "noise-b-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": {}, + "seed": 1166, + "workflow_id_suffix": "-noise-b" + }, + { + "name": "noise-a-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "noise-b-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": {}, + "seed": 1166, + "workflow_id_suffix": "-noise-b" + }, + { + "name": "noise-a-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "noise-b-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": {}, + "seed": 1166, + "workflow_id_suffix": "-noise-b" + }, + { + "name": "noise-a-s4", + "scene": "s4-montage", + "workflow": "s4-montage.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "noise-b-s4", + "scene": "s4-montage", + "workflow": "s4-montage.json", + "arguments": {}, + "seed": 1166, + "workflow_id_suffix": "-noise-b" + }, + { + "name": "noise-a-s5", + "scene": "s5-ltx-action", + "workflow": "s5-ltx-action.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "noise-b-s5", + "scene": "s5-ltx-action", + "workflow": "s5-ltx-action.json", + "arguments": {}, + "seed": 1166, + "workflow_id_suffix": "-noise-b" + }, + { + "name": "noise-a-s6", + "scene": "s6-narration", + "workflow": "s6-narration.json", + "arguments": {}, + "seed": 1166 + }, + { + "name": "noise-b-s6", + "scene": "s6-narration", + "workflow": "s6-narration.json", + "arguments": {}, + "seed": 1166, + "workflow_id_suffix": "-noise-b" + } +] \ No newline at end of file diff --git a/bench/driver.py b/bench/driver.py index 3b48bf089..81798e58e 100644 --- a/bench/driver.py +++ b/bench/driver.py @@ -32,6 +32,8 @@ class Arm: workflow: str # path under bench/workflows/ arguments: dict seed: int + workflow_id_suffix: str = "" # distinct inline id forces a step-cache miss + moments: list | None = None # overrides the scene's moments for this arm @dataclass @@ -53,6 +55,8 @@ def load_arms(path): workflow=a["workflow"], arguments=a.get("arguments") or {}, seed=a["seed"], + workflow_id_suffix=a.get("workflow_id_suffix", ""), + moments=a.get("moments"), ) for a in data ] @@ -99,6 +103,8 @@ def health(server, token): def submit(arm, server, token, workspace=None): workflow = json.loads((HERE / "workflows" / arm.workflow).read_text()) + if arm.workflow_id_suffix: + workflow["id"] = workflow.get("id", Path(arm.workflow).stem) + arm.workflow_id_suffix payload = { "workflow": workflow, "arguments": {**arm.arguments, "seed": arm.seed}, @@ -149,7 +155,7 @@ def _stats(out, values, name_mean, name_max): out[name_max] = max(values) -def measure(job, scene, server, token, workspace=None): +def measure(job, scene, server, token, workspace=None, moments=None): name = output_name(job) params = {"probe": "analyze_seams"} if workspace: @@ -169,7 +175,7 @@ def measure(job, scene, server, token, workspace=None): "names": [name], "max_dimension": 384, "max_total_bytes": MAX_COMPARE_BYTES, - "at": scene.moments, + "at": moments or scene.moments, "crop": scene.crop, "texture_bands": True, } @@ -211,7 +217,7 @@ def flush(): if job.get("status") == "succeeded": record["output"] = output_name(job) if do_measure: - record["measures"] = measure(job, scenes[arm.scene], server, token, workspace) + record["measures"] = measure(job, scenes[arm.scene], server, token, workspace, arm.moments) except Exception as exc: # one arm's failure must not lose the others record.setdefault("status", "error") record["error"] = f"{type(exc).__name__}: {exc}" diff --git a/tests/test_bench_driver.py b/tests/test_bench_driver.py index ce4bab202..51f9e7cd0 100644 --- a/tests/test_bench_driver.py +++ b/tests/test_bench_driver.py @@ -116,3 +116,37 @@ def fake_wait(job_id, *a): saved = json.loads(out.read_text()) assert [r["arm"] for r in saved] == ["a", "b"] assert "poll died" in saved[1]["error"] and records == saved + + +def test_load_arms_reads_optional_suffix_and_moments(tmp_path): + path = tmp_path / "arms.json" + path.write_text(json.dumps([{"name": "a", "scene": "s", "workflow": "w.json", "seed": 7, + "workflow_id_suffix": "-noise-b", "moments": ["frame:1"]}])) + (arm,) = driver.load_arms(str(path)) + assert arm.workflow_id_suffix == "-noise-b" and arm.moments == ["frame:1"] + + +def test_submit_suffixes_the_workflow_id(monkeypatch, tmp_path): + seen = {} + monkeypatch.setattr(driver, "http", lambda m, p, s, t, payload=None, params=None: + seen.update(payload=payload) or {"id": "j"}) + monkeypatch.setattr(driver, "HERE", tmp_path) + (tmp_path / "workflows").mkdir() + (tmp_path / "workflows" / "w.json").write_text('{"id": "Wf", "steps": []}') + arm = driver.Arm("a", "s", "w.json", {}, 1, workflow_id_suffix="-noise-b") + driver.submit(arm, "http://h", "tok") + assert seen["payload"]["workflow"]["id"] == "Wf-noise-b" + + +def test_measure_uses_the_arm_moments_override(monkeypatch): + calls = [] + + def fake(method, path, server, token, payload=None, params=None): + calls.append(payload) + return {"seams": []} if path.endswith("/assess") else {"texture": []} + + monkeypatch.setattr(driver, "http", fake) + scene = driver.Scene("s", [0, 0, 8, 8], ["frame:1", "frame:2"]) + job = {"id": "j", "manifest": [{"files": ["o.mp4"], "subfolder": "final"}]} + driver.measure(job, scene, "http://h", "tok", moments=["frame:9"]) + assert calls[-1]["at"] == ["frame:9"] From f109d379878e7dc28c07ea7802cfb03096cef774 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:27:37 -0600 Subject: [PATCH 014/405] feat(h3): #1190 - latent_anchor_strength, a partial statistics match Co-Authored-By: Claude Opus 5.5 --- docs/WORKFLOW_GUIDE.md | 11 ++- dw/guides.py | 30 ++++++ dw/pipeline_processors/chain.py | 13 +++ dw/pipeline_processors/h3_anchor_stats.py | 11 ++- dw/pipeline_processors/h3_anchor_steps.py | 19 +++- dw/pipeline_processors/h3_rules.py | 26 ++++++ dw/pipeline_processors/pipeline_h3.py | 15 ++- dw/workflow_schema.json | 3 + tests/test_h3_anchor.py | 108 ++++++++++++++++++++++ 9 files changed, 229 insertions(+), 7 deletions(-) diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 2e2ddf9f7..2e76b0055 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -1872,18 +1872,23 @@ or a step's frames - VAE-encodes them on the render's canvas the way a guide is encoded, and takes each latent channel's mean and standard deviation over every position. After denoising and before decode, the generated video latent is moved onto them, channel by channel: `(x - mean_x) / std_x * std_a + mean_a`. The audio is never -touched. It is a full match, with no strength setting; without `latent_anchor` the +touched. By default it is a full match; `latent_anchor_strength` (a number in [0, 1], +default 1) blends it with the untouched latent, `x + strength * (matched - x)`, so 0.5 +moves each channel halfway and 0 changes nothing. Without `latent_anchor` the step is unchanged, bit for bit, and its job log says nothing about an anchor. A call that applies one logs a line naming its source, such as `segment 2/3: latent anchor applied - latent statistics from segment 1`. In a chain, `"anchor": "segment_1"` sets it for you: every segment after the first gets segment 1's frames as its `latent_anchor`, so the whole chain is held to the -first segment's look. Segment 1 itself is not corrected. +first segment's look. Segment 1 itself is not corrected. `"anchor_strength": 0.5` sets +`latent_anchor_strength` on those segments, for a chain that should hold its look +without flattening a deliberate change in light. | Rule | Value | |---|---| | `anchor` | `none` (default) or `segment_1`; anything else is refused, naming the values | +| `anchor_strength` | a number in [0, 1] (default 1, the full match); refused outside it, and refused without `anchor: "segment_1"`. A step's `latent_anchor_strength` is the same number, and needs a `latent_anchor` | | `segment_1` continuity | `last_frame` or `guide`; refused with any other | | where | MiniMax-H3 `t2va` or `fl2va`, where `guides` run; `latent_anchor` and `segment_1` are refused on `ref2va`, on a step that passes `references` (that call runs ref2va), and on a non-H3 step | | `segment_1` carry | refused with `segment_argument: "references"` - the later segments would run ref2va | @@ -2440,6 +2445,8 @@ joined into a single file: `last_frame` or `guide` continuity): every segment after the first is rescaled to segment 1's per-channel latent statistics before decode, holding the chain's colour and contrast (see *Anchoring a chain's colour*). +- `anchor_strength` — with `anchor: "segment_1"`, how far that match goes: 0 to 1, + default 1 (the full match). - `carry_frames` — with `last_segment`, bound the carry to the last N frames of the segment (the audio is cut to the same span). Unset carries the whole segment. - `carry_audio` — with `last_segment` or `guide`, whether the carried reference includes its diff --git a/dw/guides.py b/dw/guides.py index a88e9c264..c0c67f707 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -58,6 +58,8 @@ GUIDE_LIMIT, GUIDE_NOISE_INPUT, LATENT_ANCHOR_INPUT, + LATENT_ANCHOR_STRENGTH_INPUT, + anchor_strength_problem, RENDER_GRID, chain_anchor_problem, guide_chain_problems, @@ -331,6 +333,24 @@ def guide_lengths(pipeline, base_dir=None, probe=None): return lengths +def _strength_argument_errors(arguments, base, where): + """A `latent_anchor_strength` argument out of range, or with no + `latent_anchor` for it to set (#1190), as [{path, message}].""" + if not isinstance(arguments, dict): + return [] + strength = arguments.get(LATENT_ANCHOR_STRENGTH_INPUT) + if strength is None: + return [] + path = render_path(base + ("arguments", LATENT_ANCHOR_STRENGTH_INPUT)) + problem = anchor_strength_problem(strength) + if problem is None and arguments.get(LATENT_ANCHOR_INPUT) is None: + problem = ( + f"{LATENT_ANCHOR_STRENGTH_INPUT} sets how far {LATENT_ANCHOR_INPUT} " + f"goes, and the step has no {LATENT_ANCHOR_INPUT}" + ) + return [{"path": path, "message": f"{problem}{where}"}] if problem else [] + + def anchor_errors(workflow_definition, source_indices=None): """Every latent statistics anchor refused before the run, as [{path, message}]: a `latent_anchor` argument on a step that cannot take @@ -354,9 +374,19 @@ def anchor_errors(workflow_definition, source_indices=None): "message": f"{problem}{where}", } ) + errors += _strength_argument_errors(arguments, base, where) chain = pipeline.get("chain") if not isinstance(chain, dict): continue + strength = chain.get("anchor_strength") + problem = None if strength is None else anchor_strength_problem(strength) + if problem: + errors.append( + { + "path": render_path(base + ("chain", "anchor_strength")), + "message": f"{problem}{where}", + } + ) path = render_path(base + ("chain", "anchor")) problem = chain_anchor_problem(chain) if problem: diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index a13627b16..0297037e7 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -66,6 +66,8 @@ GUIDE_CONTINUITY, HOLD_AUDIO_INPUT, LATENT_ANCHOR_INPUT, + LATENT_ANCHOR_STRENGTH_INPUT, + anchor_strength_problem, chain_anchor_problem, guide_chain_problems, ) @@ -417,6 +419,8 @@ def _segment_arguments( continuity.inject(segment_arguments, carry, config.segment_argument) if anchor_frames is not None: segment_arguments[LATENT_ANCHOR_INPUT] = anchor_frames + if config.anchor_strength is not None: + segment_arguments[LATENT_ANCHOR_STRENGTH_INPUT] = config.anchor_strength return segment_arguments @@ -517,6 +521,15 @@ def __init__( if problem: raise ValueError(f"Chain {problem}") self.anchor = chain_definition.get("anchor", ANCHOR_NONE) + # How far each later segment's match goes (#1190); None is the full one + self.anchor_strength = chain_definition.get("anchor_strength") + problem = anchor_strength_problem(self.anchor_strength) if ( + self.anchor_strength is not None + ) else None + if problem: + raise ValueError(f"Chain {problem}") + if self.anchor_strength is not None: + self.anchor_strength = float(self.anchor_strength) self.segment_argument = chain_definition.get("segment_argument", "image") self.carry_frames = chain_definition.get("carry_frames", None) diff --git a/dw/pipeline_processors/h3_anchor_stats.py b/dw/pipeline_processors/h3_anchor_stats.py index 0c5fc984f..e209c6f83 100644 --- a/dw/pipeline_processors/h3_anchor_stats.py +++ b/dw/pipeline_processors/h3_anchor_stats.py @@ -24,10 +24,15 @@ def channel_statistics(latents): return mean, std -def match_channel_statistics(latents, anchor_mean, anchor_std): +def match_channel_statistics(latents, anchor_mean, anchor_std, strength=1.0): """`latents` (B, C, T, H, W) rescaled per channel to `anchor_mean` and `anchor_std`: `(x - mean_x) / std_x * std_a + mean_a`, computed in float32 - and returned in the latent's own dtype and device.""" + and returned in the latent's own dtype and device. + + `strength` in [0, 1] blends the match with the input, `x + strength * + (matched - x)`: 1 is the full match, 0 returns `latents` itself.""" + if strength == 0: + return latents if latents.dim() != 5: raise ValueError( f"latent_anchor works on a (batch, channel, frames, height, width) " @@ -43,4 +48,6 @@ def match_channel_statistics(latents, anchor_mean, anchor_std): f"render {values.shape[1]}" ) matched = (values - mean) / std.clamp_min(STD_FLOOR) * anchor_std + anchor_mean + if strength != 1.0: + matched = values + strength * (matched - values) return matched.to(latents.dtype) diff --git a/dw/pipeline_processors/h3_anchor_steps.py b/dw/pipeline_processors/h3_anchor_steps.py index 74bf386e1..3708cc799 100644 --- a/dw/pipeline_processors/h3_anchor_steps.py +++ b/dw/pipeline_processors/h3_anchor_steps.py @@ -20,7 +20,11 @@ from .h3_anchor_stats import channel_statistics, match_channel_statistics from .h3_guide_steps import fit_guide_frames -from .h3_rules import LATENT_ANCHOR_INPUT, snap_guide_length +from .h3_rules import ( + LATENT_ANCHOR_INPUT, + LATENT_ANCHOR_STRENGTH_INPUT, + snap_guide_length, +) logger = logging.getLogger("dw") @@ -76,6 +80,14 @@ def inputs(self): "video latent is rescaled to before decode." ), ), + InputParam( + name=LATENT_ANCHOR_STRENGTH_INPUT, + default=None, + description=( + "How far the rescale goes, a number in [0, 1]: 1 (or unset) is the full per-channel " + "match, 0 leaves the latent alone." + ), + ), InputParam(name="height", type_hint=int, required=True), InputParam(name="width", type_hint=int, required=True), ] @@ -93,7 +105,10 @@ def __call__(self, components, state): mean, std = anchor_statistics( components, frames, block_state.height, block_state.width ) - block_state.latents = match_channel_statistics(block_state.latents, mean, std) + strength = getattr(block_state, LATENT_ANCHOR_STRENGTH_INPUT, None) + block_state.latents = match_channel_statistics( + block_state.latents, mean, std, 1.0 if strength is None else float(strength) + ) self.set_block_state(state, block_state) return components, state diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index c74cb79a8..f67c36a94 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -212,6 +212,9 @@ def _full_frame_problems(arguments, canvas): # The latent statistics anchor's call argument (#1166; see ANCHOR_BLOCK) LATENT_ANCHOR_INPUT = "latent_anchor" +# How far the match goes (#1190): 1 (default) is the full per-channel match, 0 +# leaves the latent alone; a blend in between +LATENT_ANCHOR_STRENGTH_INPUT = "latent_anchor_strength" # The arguments that condition a step on a reference, a keyframe or a guide: the # tile plan has no place to put one, and each tile would need its own crop @@ -572,6 +575,24 @@ def guide_frame_problem(frame): ANCHOR_CONTINUITIES = ("last_frame", GUIDE_CONTINUITY) +def anchor_strength_problem(value): + """Why `value` cannot be a `latent_anchor_strength` / chain `anchor_strength`, + naming the range, or None. An unresolved reference is left to the run.""" + if isinstance(value, str) and references.is_ref(references.UNRESOLVED, value): + return None + if isinstance(value, bool) or not isinstance(value, (int, float)): + return ( + f"anchor strength is a number in [0, 1] - 1 is the full match, 0 " + f"none - and {value!r} is not a number" + ) + if not 0 <= value <= 1: # NaN fails it too + return ( + f"anchor strength is a number in [0, 1] - 1 is the full match, 0 " + f"none - and {value} is outside it" + ) + return None + + def chain_anchor_problem(chain): """The reason a chain block's `anchor` is refused for its own settings, or None: an unknown value (naming the values), or `segment_1` with a continuity @@ -587,6 +608,11 @@ def chain_anchor_problem(chain): f"{', '.join(CHAIN_ANCHOR_MODES)}" ) if anchor == ANCHOR_NONE: + if chain.get("anchor_strength") is not None: + return ( + "anchor_strength needs anchor 'segment_1' - it sets how far " + "the anchor's latent match goes, and anchor is 'none'" + ) return None continuity = chain.get("continuity", "last_frame") if isinstance(continuity, str) and references.is_ref( diff --git a/dw/pipeline_processors/pipeline_h3.py b/dw/pipeline_processors/pipeline_h3.py index 01c32223e..f51f93105 100644 --- a/dw/pipeline_processors/pipeline_h3.py +++ b/dw/pipeline_processors/pipeline_h3.py @@ -17,6 +17,7 @@ GUIDE_LIMIT, GUIDE_NOISE_INPUT, LATENT_ANCHOR_INPUT, + LATENT_ANCHOR_STRENGTH_INPUT, HELD_AUDIO_OUTPUT, HELD_AUDIO_RATE_OUTPUT, HOLD_AUDIO_INPUT, @@ -28,6 +29,7 @@ REGENERATE_MASK, SOURCE_VIDEO_INPUT, edits_region, + anchor_strength_problem, guide_noise_problem, mask_problems, media_problems, @@ -168,9 +170,18 @@ def _with_latent_anchor(self, arguments): frames of a video """ value = arguments.get(LATENT_ANCHOR_INPUT) + strength = arguments.get(LATENT_ANCHOR_STRENGTH_INPUT) if value is None: - return arguments + if strength is None: + return arguments + raise ValueError( + f"Step '{self.name}': {LATENT_ANCHOR_STRENGTH_INPUT} sets how far " + f"{LATENT_ANCHOR_INPUT} goes, and there is no {LATENT_ANCHOR_INPUT}" + ) where = f"Step '{self.name}': {LATENT_ANCHOR_INPUT}" + problem = None if strength is None else anchor_strength_problem(strength) + if problem: + raise ValueError(f"Step '{self.name}': {LATENT_ANCHOR_STRENGTH_INPUT}: {problem}") refusal = anchor_refusal(self.pipeline) if refusal: raise ValueError(f"{where}: {refusal}") @@ -181,6 +192,8 @@ def _with_latent_anchor(self, arguments): ) arguments = dict(arguments) arguments[LATENT_ANCHOR_INPUT] = prepared_anchor(where, value, self.base_dir) + if strength is not None: + arguments[LATENT_ANCHOR_STRENGTH_INPUT] = float(strength) # The one line that says the anchor applies to this call - without it # the job log shows only a block name (#1181) label = getattr(self, "segment_label", None) or self.name diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index a1fc0776f..02e3f17e0 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -344,6 +344,9 @@ "description": "Latent statistics anchor (MiniMax-H3 t2va and fl2va only, with continuity 'guide' or 'last_frame'). 'segment_1' passes segment 1's decoded frames to every later segment as its 'latent_anchor': each later segment's video latent is rescaled, per VAE latent channel, to segment 1's latent mean and std before decode, so colour and contrast stop drifting from segment to segment. Segment 1 itself is not corrected, and audio is never touched. 'none' (the default) anchors nothing; any other value, of any type, is refused naming the two. Accepts a 'variable:' reference.", "default": "none" }, + "anchor_strength": { + "description": "How far each later segment's latent statistics match goes, with anchor 'segment_1' (refused without it): a number from 0 to 1, default 1. 1 is the full per-channel mean/std match; 0 leaves the latent alone; 0.5 moves each channel halfway to segment 1's statistics, holding the look without flattening a deliberate change in light. Anything outside [0, 1], or not a number, is refused naming the range. Accepts a 'variable:' reference." + }, "segment_argument": { "description": "The pipeline argument the carry-over lands in - e.g. 'image' for image-to-video pipelines, or 'references' for reference-conditioned modular pipelines, where the carry-over is appended as an image or video reference.", "type": "string", diff --git a/tests/test_h3_anchor.py b/tests/test_h3_anchor.py index 1fff48cd2..fb32968fc 100644 --- a/tests/test_h3_anchor.py +++ b/tests/test_h3_anchor.py @@ -18,7 +18,9 @@ ANCHOR_AFTER, ANCHOR_BLOCK, LATENT_ANCHOR_INPUT, + LATENT_ANCHOR_STRENGTH_INPUT, TILE_CONFLICTS, + anchor_strength_problem, chain_anchor_problem, ) from tests.test_chain import FakePipeline, video_output @@ -57,6 +59,29 @@ def test_keeps_the_latents_dtype_and_shape(self): assert matched.dtype == torch.bfloat16 assert matched.shape == latents.shape + def test_strength_one_is_todays_full_match(self): + x = synthetic_latent() + mean_a, std_a = torch.tensor([0.5, -1.0, 2.0, 0.0]), torch.tensor([1.0, 2.0, 3.0, 4.0]) + out = match_channel_statistics(x, mean_a, std_a, strength=1.0) + assert torch.equal(out, match_channel_statistics(x, mean_a, std_a)) + + def test_strength_zero_returns_the_latent_unchanged(self): + x = synthetic_latent() + out = match_channel_statistics(x, torch.zeros(4), torch.ones(4), strength=0.0) + assert out is x + + def test_half_strength_moves_each_channel_halfway(self): + x = synthetic_latent() + mean_a, std_a = torch.tensor([0.5, -1.0, 2.0, 0.0]), torch.tensor([1.0, 2.0, 3.0, 4.0]) + full = match_channel_statistics(x, mean_a, std_a) + half = match_channel_statistics(x, mean_a, std_a, strength=0.5) + assert torch.allclose(half, x + 0.5 * (full - x), atol=1e-5) + + def test_partial_strength_keeps_the_latents_dtype(self): + x = synthetic_latent(dtype=torch.bfloat16) + out = match_channel_statistics(x, torch.zeros(4), torch.ones(4), strength=0.5) + assert out.dtype == torch.bfloat16 + def test_a_flat_channel_lands_on_the_anchors_mean(self): latents = synthetic_latent() latents[:, 0] = 7.0 @@ -100,6 +125,63 @@ def test_an_out_of_root_path_is_refused_at_run_time(self, monkeypatch, tmp_path) prepared_anchor("anchor", "/usr/share/videos/x.mp4", str(tmp_path)) +def test_anchor_strength_problem_refuses_out_of_range_and_non_numbers(): + assert anchor_strength_problem(1.5) and anchor_strength_problem(-0.1) + assert anchor_strength_problem("a") and anchor_strength_problem(True) + assert anchor_strength_problem(float("nan")) + assert "[0, 1]" in anchor_strength_problem(1.5) + assert anchor_strength_problem(0.0) is None and anchor_strength_problem(1) is None + assert anchor_strength_problem("variable:s") is None + + +def test_chain_anchor_strength_needs_anchor_segment_1(): + chain = {"segments": 2, "anchor_strength": 0.5} + (error,) = anchor_errors(definition(chain=chain)) + assert "anchor_strength" in error["message"] and "segment_1" in error["message"] + with pytest.raises(ValueError, match="anchor_strength.*segment_1"): + ChainConfig(chain, {}) + + +def test_chain_anchor_strength_out_of_range_is_refused_at_validate_and_run(): + chain = {"segments": 2, "anchor": "segment_1", "anchor_strength": 1.5} + (error,) = anchor_errors(definition(chain=chain)) + assert error["path"] == "steps[0].pipeline.chain.anchor_strength" + assert "[0, 1]" in error["message"] + with pytest.raises(ValueError, match=r"\[0, 1\]"): + ChainConfig(chain, {}) + + +def test_chain_anchor_strength_in_range_is_clean(): + chain = {"segments": 2, "anchor": "segment_1", "anchor_strength": 0.5} + assert anchor_errors(definition(chain=chain)) == [] + + +def test_a_strength_argument_without_an_anchor_is_refused(): + (error,) = anchor_errors(definition(latent_anchor_strength=0.5)) + assert error["path"] == "steps[0].pipeline.arguments.latent_anchor_strength" + assert "latent_anchor" in error["message"] + + +def test_a_strength_argument_out_of_range_is_refused(): + (error,) = anchor_errors(definition(latent_anchor="c.mp4", latent_anchor_strength=2)) + assert "[0, 1]" in error["message"] + + +def test_chain_sets_strength_on_segments_after_the_first(): + pipeline = FakePipeline(video_output) + chain = {"segments": 3, "anchor": "segment_1", "anchor_strength": 0.5} + run_chain(pipeline, chain, {"prompt": "p"}) + assert LATENT_ANCHOR_STRENGTH_INPUT not in pipeline.calls[0] + assert pipeline.calls[1][LATENT_ANCHOR_STRENGTH_INPUT] == 0.5 + assert pipeline.calls[2][LATENT_ANCHOR_STRENGTH_INPUT] == 0.5 + + +def test_no_strength_key_when_the_chain_sets_none(): + pipeline = FakePipeline(video_output) + run_chain(pipeline, {"segments": 2, "anchor": "segment_1"}, {"prompt": "p"}) + assert all(LATENT_ANCHOR_STRENGTH_INPUT not in call for call in pipeline.calls) + + class TestChainAnchor: def test_segments_after_the_first_get_segment_ones_frames(self): pipeline = FakePipeline(video_output) @@ -368,6 +450,32 @@ def test_takes_latent_anchor_as_an_input(self): assert LATENT_ANCHOR_INPUT in inputs +class TestStrengthStep: + def test_takes_the_strength_as_an_input(self): + inputs = {param.name for param in DwH3LatentAnchorStep().inputs} + assert LATENT_ANCHOR_STRENGTH_INPUT in inputs + + def test_the_step_blends_by_strength(self, monkeypatch): + from dw.pipeline_processors import h3_anchor_steps + + anchor_latent = synthetic_latent(seed=3) + monkeypatch.setattr( + h3_anchor_steps, + "anchor_statistics", + lambda components, frames, height, width: channel_statistics(anchor_latent), + ) + latents = synthetic_latent() + state = PipelineState() + state.set("latents", latents) + state.set(LATENT_ANCHOR_INPUT, np.zeros((5, 32, 32, 3), dtype=np.uint8)) + state.set(LATENT_ANCHOR_STRENGTH_INPUT, 0.5) + state.set("height", 32) + state.set("width", 32) + _, state = DwH3LatentAnchorStep()(None, state) + full = match_channel_statistics(latents, *channel_statistics(anchor_latent)) + assert torch.allclose(state.get("latents"), latents + 0.5 * (full - latents), atol=1e-5) + + class TestStep: def state(self, latents, anchor=None): state = PipelineState() From 3b1fc983b93d58eedcc991ac9d180dda45bf754e Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:30:35 -0600 Subject: [PATCH 015/405] style: ruff format and lint the long-form branch Co-Authored-By: Claude Opus 5.5 --- bench/driver.py | 71 +++++++++++++++------ dw/pipeline_processors/chain.py | 8 ++- dw/pipeline_processors/pipeline_h3.py | 4 +- tests/test_bench_driver.py | 89 +++++++++++++++++++++------ tests/test_bench_verdict.py | 45 ++++++++++++-- tests/test_bench_workflows.py | 25 ++++++-- tests/test_h3_anchor.py | 18 ++++-- 7 files changed, 206 insertions(+), 54 deletions(-) diff --git a/bench/driver.py b/bench/driver.py index 81798e58e..c95fdb3ed 100644 --- a/bench/driver.py +++ b/bench/driver.py @@ -17,7 +17,6 @@ HERE = Path(__file__).resolve().parent sys.path.insert(0, str(HERE)) -from verdict import noise_floor, verdict # noqa: E402 DEFAULT_SERVER = "http://127.0.0.1:18766" DEFAULT_TOKEN_FILE = "~/.dw-bench-token" @@ -104,7 +103,9 @@ def health(server, token): def submit(arm, server, token, workspace=None): workflow = json.loads((HERE / "workflows" / arm.workflow).read_text()) if arm.workflow_id_suffix: - workflow["id"] = workflow.get("id", Path(arm.workflow).stem) + arm.workflow_id_suffix + workflow["id"] = ( + workflow.get("id", Path(arm.workflow).stem) + arm.workflow_id_suffix + ) payload = { "workflow": workflow, "arguments": {**arm.arguments, "seed": arm.seed}, @@ -123,7 +124,9 @@ def wait(job_id, server, token, timeout_s, poll_s=15): if job.get("status") in TERMINAL: return job if time.monotonic() >= deadline: - raise TimeoutError(f"job {job_id} still {job.get('status')} after {timeout_s}s") + raise TimeoutError( + f"job {job_id} still {job.get('status')} after {timeout_s}s" + ) time.sleep(poll_s) @@ -160,16 +163,24 @@ def measure(job, scene, server, token, workspace=None, moments=None): params = {"probe": "analyze_seams"} if workspace: params["workspace"] = workspace - seams = http( - "GET", f"/api/gallery/{urllib.parse.quote(name, safe='/')}/assess", - server, token, params=params, - ).get("seams") or [] + seams = ( + http( + "GET", + f"/api/gallery/{urllib.parse.quote(name, safe='/')}/assess", + server, + token, + params=params, + ).get("seams") + or [] + ) out = {} _stats(out, [s.get("jump_ratio") for s in seams], "seam_jump_mean", "seam_jump_max") _stats(out, [s.get("look_ratio") for s in seams], "seam_look_mean", "seam_look_max") _stats( - out, [abs(s["level_step_db"]) for s in seams if s.get("level_step_db") is not None], - "seam_level_step_db_mean", "seam_level_step_db_max", + out, + [abs(s["level_step_db"]) for s in seams if s.get("level_step_db") is not None], + "seam_level_step_db_mean", + "seam_level_step_db_max", ) payload = { "names": [name], @@ -180,7 +191,11 @@ def measure(job, scene, server, token, workspace=None, moments=None): "texture_bands": True, } compared = http( - "POST", "/api/gallery/compare", server, token, payload, + "POST", + "/api/gallery/compare", + server, + token, + payload, {"workspace": workspace} if workspace else None, ) row = (compared.get("texture") or [{}])[0] @@ -198,7 +213,9 @@ def measure(job, scene, server, token, workspace=None, moments=None): return out -def run_arms(arms, scenes, server, token, do_measure, timeout_s, workspace=None, out_path=None): +def run_arms( + arms, scenes, server, token, do_measure, timeout_s, workspace=None, out_path=None +): ids = {arm.name: submit(arm, server, token, workspace) for arm in arms} records = [] @@ -208,8 +225,12 @@ def flush(): Path(out_path).write_text(json.dumps(records, indent=2)) for arm in arms: - record = {"arm": arm.name, "scene": arm.scene, "seed": arm.seed, - "job_id": ids[arm.name]} + record = { + "arm": arm.name, + "scene": arm.scene, + "seed": arm.seed, + "job_id": ids[arm.name], + } try: job = wait(ids[arm.name], server, token, timeout_s) record["status"] = job.get("status") @@ -217,7 +238,9 @@ def flush(): if job.get("status") == "succeeded": record["output"] = output_name(job) if do_measure: - record["measures"] = measure(job, scenes[arm.scene], server, token, workspace, arm.moments) + record["measures"] = measure( + job, scenes[arm.scene], server, token, workspace, arm.moments + ) except Exception as exc: # one arm's failure must not lose the others record.setdefault("status", "error") record["error"] = f"{type(exc).__name__}: {exc}" @@ -234,7 +257,11 @@ def report(records): for r in records: m = r.get("measures", {}) cells = [f"{m[k]:.3f}" if k in m else "" for k in keys] - lines.append(f"| {r['arm']} | {r['scene']} | {r['seed']} | {r['status']} | " + " | ".join(cells) + " |") + lines.append( + f"| {r['arm']} | {r['scene']} | {r['seed']} | {r['status']} | " + + " | ".join(cells) + + " |" + ) return "\n".join(lines) @@ -253,7 +280,9 @@ def _token(args): def main(argv=None): parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--server", default=os.environ.get("BENCH_SERVER", DEFAULT_SERVER)) + parser.add_argument( + "--server", default=os.environ.get("BENCH_SERVER", DEFAULT_SERVER) + ) parser.add_argument("--token-file") parser.add_argument("--workspace") sub = parser.add_subparsers(dest="command", required=True) @@ -274,8 +303,14 @@ def main(argv=None): return 0 token = _token(args) records = run_arms( - load_arms(args.arms), load_scenes(args.scenes) if args.measure else {}, - args.server, token, args.measure, args.timeout, args.workspace, args.out, + load_arms(args.arms), + load_scenes(args.scenes) if args.measure else {}, + args.server, + token, + args.measure, + args.timeout, + args.workspace, + args.out, ) if not args.out: print(json.dumps(records, indent=2)) diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index 0297037e7..8cbb1abf6 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -523,9 +523,11 @@ def __init__( self.anchor = chain_definition.get("anchor", ANCHOR_NONE) # How far each later segment's match goes (#1190); None is the full one self.anchor_strength = chain_definition.get("anchor_strength") - problem = anchor_strength_problem(self.anchor_strength) if ( - self.anchor_strength is not None - ) else None + problem = ( + anchor_strength_problem(self.anchor_strength) + if (self.anchor_strength is not None) + else None + ) if problem: raise ValueError(f"Chain {problem}") if self.anchor_strength is not None: diff --git a/dw/pipeline_processors/pipeline_h3.py b/dw/pipeline_processors/pipeline_h3.py index f51f93105..1b4719fce 100644 --- a/dw/pipeline_processors/pipeline_h3.py +++ b/dw/pipeline_processors/pipeline_h3.py @@ -181,7 +181,9 @@ def _with_latent_anchor(self, arguments): where = f"Step '{self.name}': {LATENT_ANCHOR_INPUT}" problem = None if strength is None else anchor_strength_problem(strength) if problem: - raise ValueError(f"Step '{self.name}': {LATENT_ANCHOR_STRENGTH_INPUT}: {problem}") + raise ValueError( + f"Step '{self.name}': {LATENT_ANCHOR_STRENGTH_INPUT}: {problem}" + ) refusal = anchor_refusal(self.pipeline) if refusal: raise ValueError(f"{where}: {refusal}") diff --git a/tests/test_bench_driver.py b/tests/test_bench_driver.py index 51f9e7cd0..75233ed8b 100644 --- a/tests/test_bench_driver.py +++ b/tests/test_bench_driver.py @@ -12,11 +12,27 @@ def test_load_arms_reads_every_field(tmp_path): path = tmp_path / "arms.json" - path.write_text(json.dumps([{"name": "a", "scene": "s", "workflow": "w.json", - "arguments": {"prompt": "x"}, "seed": 7}])) + path.write_text( + json.dumps( + [ + { + "name": "a", + "scene": "s", + "workflow": "w.json", + "arguments": {"prompt": "x"}, + "seed": 7, + } + ] + ) + ) (arm,) = driver.load_arms(str(path)) assert (arm.name, arm.scene, arm.workflow, arm.arguments, arm.seed) == ( - "a", "s", "w.json", {"prompt": "x"}, 7) + "a", + "s", + "w.json", + {"prompt": "x"}, + 7, + ) def test_submit_carries_seed_and_names_the_arm(monkeypatch, tmp_path): @@ -40,21 +56,33 @@ def fake(method, path, server, token, payload=None, params=None): def test_measure_calls_compare_with_the_scene_crop_and_texture_bands(monkeypatch): calls = [] - cell = lambda fine, mid, coarse=4.0: { - "texture": {"luma": {"fine": fine, "mid": mid, "coarse": coarse}}} + + def cell(fine, mid, coarse=4.0): + return {"texture": {"luma": {"fine": fine, "mid": mid, "coarse": coarse}}} def fake(method, path, server, token, payload=None, params=None): calls.append((method, path, payload, params)) if path.endswith("/assess"): - return {"probe": "analyze_seams", "seams": [ - {"jump_ratio": 2.0, "look_ratio": 1.0, "level_step_db": -3.0}, - {"jump_ratio": 4.0, "look_ratio": 3.0, "level_step_db": 1.0}]} - return {"texture": [{"cells": [cell(10.0, 20.0), cell(8.0, 20.0), cell(5.0, 30.0)]}]} + return { + "probe": "analyze_seams", + "seams": [ + {"jump_ratio": 2.0, "look_ratio": 1.0, "level_step_db": -3.0}, + {"jump_ratio": 4.0, "look_ratio": 3.0, "level_step_db": 1.0}, + ], + } + return { + "texture": [{"cells": [cell(10.0, 20.0), cell(8.0, 20.0), cell(5.0, 30.0)]}] + } monkeypatch.setattr(driver, "http", fake) scene = driver.Scene("s", [1, 2, 64, 64], ["frame:1", "frame:50", "frame:99"]) - job = {"id": "j", "manifest": [{"files": ["wf/v1/out.mp4"], "subfolder": "final"}], - "output_kinds": {"wf/v1/out.mp4": "video"}, "started_at": 0, "finished_at": 120} + job = { + "id": "j", + "manifest": [{"files": ["wf/v1/out.mp4"], "subfolder": "final"}], + "output_kinds": {"wf/v1/out.mp4": "video"}, + "started_at": 0, + "finished_at": 120, + } m = driver.measure(job, scene, "http://h", "tok") assess = next(c for c in calls if c[1].endswith("/assess")) @@ -74,7 +102,8 @@ def fake(method, path, server, token, payload=None, params=None): def _texture_measure(monkeypatch, first, last): - cell = lambda v: {"texture": {"luma": {"fine": v, "mid": 1.0, "coarse": 1.0}}} + def cell(v): + return {"texture": {"luma": {"fine": v, "mid": 1.0, "coarse": 1.0}}} def fake(method, path, server, token, payload=None, params=None): if path.endswith("/assess"): @@ -83,8 +112,10 @@ def fake(method, path, server, token, payload=None, params=None): monkeypatch.setattr(driver, "http", fake) scene = driver.Scene("s", None, ["frame:1", "frame:2"]) - job = {"manifest": [{"files": ["o.mp4"], "subfolder": "final"}], - "output_kinds": {"o.mp4": "video"}} + job = { + "manifest": [{"files": ["o.mp4"], "subfolder": "final"}], + "output_kinds": {"o.mp4": "video"}, + } return driver.measure(job, scene, "http://h", "tok") @@ -98,7 +129,10 @@ def test_texture_drift_is_two_sided(monkeypatch): def test_run_arms_writes_each_record_as_it_lands(monkeypatch, tmp_path): - arms = [driver.Arm("a", "s", "w.json", {}, 1), driver.Arm("b", "s", "w.json", {}, 2)] + arms = [ + driver.Arm("a", "s", "w.json", {}, 1), + driver.Arm("b", "s", "w.json", {}, 2), + ] monkeypatch.setattr(driver, "submit", lambda arm, *a: f"id-{arm.name}") out = tmp_path / "sub" / "rec.json" seen = {} @@ -120,16 +154,33 @@ def fake_wait(job_id, *a): def test_load_arms_reads_optional_suffix_and_moments(tmp_path): path = tmp_path / "arms.json" - path.write_text(json.dumps([{"name": "a", "scene": "s", "workflow": "w.json", "seed": 7, - "workflow_id_suffix": "-noise-b", "moments": ["frame:1"]}])) + path.write_text( + json.dumps( + [ + { + "name": "a", + "scene": "s", + "workflow": "w.json", + "seed": 7, + "workflow_id_suffix": "-noise-b", + "moments": ["frame:1"], + } + ] + ) + ) (arm,) = driver.load_arms(str(path)) assert arm.workflow_id_suffix == "-noise-b" and arm.moments == ["frame:1"] def test_submit_suffixes_the_workflow_id(monkeypatch, tmp_path): seen = {} - monkeypatch.setattr(driver, "http", lambda m, p, s, t, payload=None, params=None: - seen.update(payload=payload) or {"id": "j"}) + monkeypatch.setattr( + driver, + "http", + lambda m, p, s, t, payload=None, params=None: ( + seen.update(payload=payload) or {"id": "j"} + ), + ) monkeypatch.setattr(driver, "HERE", tmp_path) (tmp_path / "workflows").mkdir() (tmp_path / "workflows" / "w.json").write_text('{"id": "Wf", "steps": []}') diff --git a/tests/test_bench_verdict.py b/tests/test_bench_verdict.py index dcab53b21..eb6abcde9 100644 --- a/tests/test_bench_verdict.py +++ b/tests/test_bench_verdict.py @@ -9,11 +9,46 @@ def test_verdict_wins_only_past_noise_and_never_worse_elsewhere(): noise = {"seam_look_mean": 0.2, "texture_fine_seg_last": 0.1} control = {"seam_look_mean": 2.0, "texture_fine_seg_last": 1.0} - assert verdict(control, {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.05}, noise, "seam_look_mean") == "wins" - assert verdict(control, {"seam_look_mean": 1.9, "texture_fine_seg_last": 1.0}, noise, "seam_look_mean") == "tie" - assert verdict(control, {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.4}, noise, "seam_look_mean") == "mixed" - assert verdict(control, {"seam_look_mean": 2.6, "texture_fine_seg_last": 1.0}, noise, "seam_look_mean") == "loses" + assert ( + verdict( + control, + {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.05}, + noise, + "seam_look_mean", + ) + == "wins" + ) + assert ( + verdict( + control, + {"seam_look_mean": 1.9, "texture_fine_seg_last": 1.0}, + noise, + "seam_look_mean", + ) + == "tie" + ) + assert ( + verdict( + control, + {"seam_look_mean": 1.5, "texture_fine_seg_last": 1.4}, + noise, + "seam_look_mean", + ) + == "mixed" + ) + assert ( + verdict( + control, + {"seam_look_mean": 2.6, "texture_fine_seg_last": 1.0}, + noise, + "seam_look_mean", + ) + == "loses" + ) def test_noise_floor_is_the_absolute_difference_per_measure(): - assert noise_floor({"a": 1.0, "b": 3.0}, {"a": 1.25, "b": 2.0}) == {"a": 0.25, "b": 1.0} + assert noise_floor({"a": 1.0, "b": 3.0}, {"a": 1.25, "b": 2.0}) == { + "a": 0.25, + "b": 1.0, + } diff --git a/tests/test_bench_workflows.py b/tests/test_bench_workflows.py index 6d8cc096e..f28b1d699 100644 --- a/tests/test_bench_workflows.py +++ b/tests/test_bench_workflows.py @@ -12,7 +12,14 @@ import driver # noqa: E402 WORKFLOWS = sorted((BENCH / "workflows").glob("s*.json")) -IDS = ["s1-static", "s2-arc", "s3-dialogue-walk", "s4-montage", "s5-ltx-action", "s6-narration"] +IDS = [ + "s1-static", + "s2-arc", + "s3-dialogue-walk", + "s4-montage", + "s5-ltx-action", + "s6-narration", +] def _errors(path): @@ -36,9 +43,19 @@ def test_chain_workflows_expose_the_arm_knobs(path): variables = json.loads(path.read_text(encoding="utf-8"))["variables"] assert "seed" in variables if path.stem in ("s1-static", "s2-arc", "s3-dialogue-walk"): - for knob in ("continuity", "guide_frames", "segments", "anchor", "guide_noise", - "width", "height", "num_inference_steps", "weights_dtype", - "lora_model_name", "attention_backend"): + for knob in ( + "continuity", + "guide_frames", + "segments", + "anchor", + "guide_noise", + "width", + "height", + "num_inference_steps", + "weights_dtype", + "lora_model_name", + "attention_backend", + ): assert knob in variables, knob diff --git a/tests/test_h3_anchor.py b/tests/test_h3_anchor.py index fb32968fc..b6c5eb408 100644 --- a/tests/test_h3_anchor.py +++ b/tests/test_h3_anchor.py @@ -61,7 +61,10 @@ def test_keeps_the_latents_dtype_and_shape(self): def test_strength_one_is_todays_full_match(self): x = synthetic_latent() - mean_a, std_a = torch.tensor([0.5, -1.0, 2.0, 0.0]), torch.tensor([1.0, 2.0, 3.0, 4.0]) + mean_a, std_a = ( + torch.tensor([0.5, -1.0, 2.0, 0.0]), + torch.tensor([1.0, 2.0, 3.0, 4.0]), + ) out = match_channel_statistics(x, mean_a, std_a, strength=1.0) assert torch.equal(out, match_channel_statistics(x, mean_a, std_a)) @@ -72,7 +75,10 @@ def test_strength_zero_returns_the_latent_unchanged(self): def test_half_strength_moves_each_channel_halfway(self): x = synthetic_latent() - mean_a, std_a = torch.tensor([0.5, -1.0, 2.0, 0.0]), torch.tensor([1.0, 2.0, 3.0, 4.0]) + mean_a, std_a = ( + torch.tensor([0.5, -1.0, 2.0, 0.0]), + torch.tensor([1.0, 2.0, 3.0, 4.0]), + ) full = match_channel_statistics(x, mean_a, std_a) half = match_channel_statistics(x, mean_a, std_a, strength=0.5) assert torch.allclose(half, x + 0.5 * (full - x), atol=1e-5) @@ -163,7 +169,9 @@ def test_a_strength_argument_without_an_anchor_is_refused(): def test_a_strength_argument_out_of_range_is_refused(): - (error,) = anchor_errors(definition(latent_anchor="c.mp4", latent_anchor_strength=2)) + (error,) = anchor_errors( + definition(latent_anchor="c.mp4", latent_anchor_strength=2) + ) assert "[0, 1]" in error["message"] @@ -473,7 +481,9 @@ def test_the_step_blends_by_strength(self, monkeypatch): state.set("width", 32) _, state = DwH3LatentAnchorStep()(None, state) full = match_channel_statistics(latents, *channel_statistics(anchor_latent)) - assert torch.allclose(state.get("latents"), latents + 0.5 * (full - latents), atol=1e-5) + assert torch.allclose( + state.get("latents"), latents + 0.5 * (full - latents), atol=1e-5 + ) class TestStep: From b4d748435e87d0a4a28fd58cc3568ee68db2cb4b Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:33:19 -0600 Subject: [PATCH 016/405] feat(chain): #1189 - guide_backoff hands over one block before the end Co-Authored-By: Claude Opus 5.5 --- docs/ATTRIBUTIONS.md | 10 +++ docs/WORKFLOW_GUIDE.md | 2 + dw/guides.py | 8 +++ dw/pipeline_processors/chain.py | 47 +++++++++++++- dw/pipeline_processors/h3_rules.py | 48 ++++++++++++++ dw/workflow_schema.json | 8 +++ tests/test_chain.py | 96 ++++++++++++++++++++++++++++ tests/test_guide_chain_validation.py | 31 +++++++++ 8 files changed, 247 insertions(+), 3 deletions(-) diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md index d00ef8129..f4eb337b0 100644 --- a/docs/ATTRIBUTIONS.md +++ b/docs/ATTRIBUTIONS.md @@ -133,3 +133,13 @@ segment's per-channel mean and standard deviation before decode [#1166](https://github.com/dkackman/diffusers-workflow/issues/1166)). The diagnosis it answers, colour and contrast drifting from segment to segment of an H3 chain, was described by Lumifel and by roycho87 (NerdNotes). + +### [roycho87/geiru_context_suite](https://github.com/roycho87/geiru_context_suite) + +A set of ComfyUI nodes for long MiniMax H3 videos, under GPL-3.0, so only the +concept was taken, together with seitanism's context hold. It inspired +`guide_backoff` +([dw/pipeline_processors/chain.py](../dw/pipeline_processors/chain.py); +[#1189](https://github.com/dkackman/diffusers-workflow/issues/1189)): handing the +next segment the frames one block before the end and dropping the previous +segment's last block at assembly, so the join lands inside the held copy. diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 2e76b0055..337bdb7be 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -1858,6 +1858,7 @@ head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 | `guide_frames` | 22 (default) or 39 - a whole-latent guide length; anything else is refused | | where | MiniMax-H3 `t2va` or `fl2va`; refused on `ref2va`, with `references`, or on a non-H3 step - guides stay off ref2va | | `carry_frames` | refused - the carry is always the last `guide_frames` frames | +| `guide_backoff` | 0 (default) or a multiple of 17: the carry is the `guide_frames` frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs `num_frames` above P + back-off. Refused off `guide` continuity and with `match_audio` | | `trim_frames` | ignored (the trim is P), with a note in the job log | | `crossfade_ms` | ignored when `carry_audio` is true: the guide held the audio, so the seam is joined with only a declick | | `num_frames` | must exceed P, or the chain cannot progress | @@ -2441,6 +2442,7 @@ joined into a single file: next as a held frame-0 guide (see *H3: holding a clip with `guides`*, *Chaining with a guide*). - `guide_frames` — with `guide`, how many frames the guide carries: 22 (default) or 39. +- `guide_backoff` — with `guide`, a multiple of 17 (default 0): carry the frames that many before each segment's end and drop that tail at assembly. Not with `match_audio`. - `anchor` — `none` (default) or `segment_1` (MiniMax-H3 `t2va`/`fl2va`, with `last_frame` or `guide` continuity): every segment after the first is rescaled to segment 1's per-channel latent statistics before decode, holding the chain's colour diff --git a/dw/guides.py b/dw/guides.py index c0c67f707..8439c5ab8 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -62,6 +62,7 @@ anchor_strength_problem, RENDER_GRID, chain_anchor_problem, + guide_backoff_problems, guide_chain_problems, guide_end_problem, guide_frame_problem, @@ -265,6 +266,13 @@ def guide_chain_errors(workflow_definition, source_indices=None): } ) continue + for key, message in guide_backoff_problems(chain): + errors.append( + { + "path": render_path(base + ("chain", key)), + "message": message, + } + ) if continuity != GUIDE_CONTINUITY: continue problem = _takes_guides_problem(pipeline, pipeline.get("arguments")) diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index 8cbb1abf6..240f89371 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -69,6 +69,7 @@ LATENT_ANCHOR_STRENGTH_INPUT, anchor_strength_problem, chain_anchor_problem, + guide_backoff_problems, guide_chain_problems, ) @@ -164,7 +165,17 @@ def __init__(self, config): def extract(self, artifact): count = self.config.guide_frames - frames = frames_as_pil_list(artifact)[-count:] + # With backoff the held span ends one block early; the segment's tail + # past it is dropped at assembly, so the join lands inside the copy + back = self.config.guide_backoff + all_frames = frames_as_pil_list(artifact) + end = len(all_frames) - back + if end < count: + raise ValueError( + f"A guide chain with guide_backoff {back} needs segments of at " + f"least {count + back} frames, and this one made {len(all_frames)}" + ) + frames = all_frames[end - count : end] if not self.config.carry_audio: return _SegmentCarry(frames, None, None) @@ -180,7 +191,10 @@ def extract(self, artifact): "on the chain or a 'frame_rate' pipeline argument" ) samples = frames_to_samples(count, self.config.fps, sample_rate) - return _SegmentCarry(frames, audio[:, -samples:], sample_rate) + stop = audio.shape[1] - frames_to_samples(back, self.config.fps, sample_rate) + return _SegmentCarry( + frames, audio[:, max(stop - samples, 0) : stop], sample_rate + ) def inject(self, arguments, carry, segment_argument): if carry.audio is not None: @@ -314,6 +328,17 @@ def run_chain(pipeline, chain_definition, arguments): segment_audio, len(segment_frames), config.fps, segment_rate ) + if config.tail_trim and segment.index < len(config.plan) - 1: + # The carry is already taken; the tail it backed off from is + # replaced by the next segment's fresh take (#1189) + segment_frames = segment_frames[: -config.tail_trim] + if segment_audio is not None: + segment_audio = segment_audio[ + :, + : segment_audio.shape[1] + - frames_to_samples(config.tail_trim, config.fps, segment_rate), + ] + kept_frames = segment_frames[segment.head_trim :] start_sample = audio.shape[1] if audio is not None else 0 shots.append( @@ -548,6 +573,12 @@ def __init__( # held soundtrack has nothing to crossfade self.guide_frames = None self.guide_holds_audio = False + # Frames dropped from the tail of every segment but the last (#1189) + self.guide_backoff = 0 + self.tail_trim = 0 + problems = guide_backoff_problems(chain_definition) + if problems: + raise ValueError("; ".join(f"Chain {message}" for _, message in problems)) head_trim = self.trim_frames if self.continuity == GUIDE_CONTINUITY: problems = guide_chain_problems(chain_definition) @@ -559,6 +590,8 @@ def __init__( "guide_frames", GUIDE_CHAIN_DEFAULT ) head_trim = self.guide_frames + self.guide_backoff = int(chain_definition.get("guide_backoff", 0)) + self.tail_trim = self.guide_backoff if self.carry_audio: self.guide_holds_audio = True self.crossfade_ms = 0.0 @@ -586,10 +619,18 @@ def __init__( num_frames = arguments.get("num_frames", None) if num_frames is not None: validate_frame_snap(int(num_frames), self.frame_snap) - if self.guide_frames is not None and int(num_frames) <= head_trim: + if ( + self.guide_frames is not None + and int(num_frames) <= head_trim + self.tail_trim + ): raise ValueError( f"Segments of {num_frames} frames cannot progress past " f"a {head_trim}-frame guide" + + ( + f" and a {self.tail_trim}-frame back-off" + if self.tail_trim + else "" + ) ) self.plan = [ Segment( diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index f67c36a94..812cb0f26 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -549,6 +549,8 @@ def guide_frame_problem(frame): GUIDE_CONTINUITY = "guide" GUIDE_CHAIN_FRAMES = (22, 39) GUIDE_CHAIN_DEFAULT = 22 +# guide_backoff (#1189) moves the carry back whole 17-frame blocks +GUIDE_BACKOFF_BLOCK = 17 # Where a guide chain runs is where `guides` runs (dw/guides.py); the run and # validate lead their refusal with this rule GUIDE_CHAIN_RULE = ( @@ -671,6 +673,52 @@ def guide_chain_problems(chain): return problems +def guide_backoff_problems(chain): + """[(key, message)] for a chain block's `guide_backoff` (#1189): a whole + number of 17-frame blocks, on a guide-continuity chain, planned from a + segment count - a `match_audio` plan counts every segment's new frames from + its head trim alone. An unresolved reference is left to the run.""" + if "guide_backoff" not in chain: + return [] + backoff = chain["guide_backoff"] + if isinstance(backoff, str) and references.is_ref(references.UNRESOLVED, backoff): + return [] + block = GUIDE_BACKOFF_BLOCK + if ( + isinstance(backoff, bool) + or not isinstance(backoff, int) + or backoff < 0 + or backoff % block + ): + return [ + ( + "guide_backoff", + f"guide_backoff must be a multiple of {block} frames (whole " + f"latent blocks), got {backoff!r}", + ) + ] + if backoff == 0: + return [] + if chain.get("continuity", "last_frame") != GUIDE_CONTINUITY: + return [ + ( + "guide_backoff", + "guide_backoff is a guide continuity setting - set continuity " + "'guide' or drop guide_backoff", + ) + ] + if chain.get("match_audio"): + return [ + ( + "guide_backoff", + "guide_backoff does not combine with match_audio: the plan " + "covers the track from each segment's head trim alone, and " + "backing off drops more frames per segment - use 'segments'", + ) + ] + return [] + + def guide_end_problem(frame, length, num_frames): """Why a guide of `length` frames at `frame` runs past a `num_frames` render, or None. Ending exactly at `num_frames` fits.""" diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index 02e3f17e0..b93081da6 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -340,6 +340,14 @@ ], "default": 22 }, + "guide_backoff": { + "description": "'guide' only - frames, a multiple of 17, to back the carry off from the end of each segment: the next segment is guided by the frames that many before the end, and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Not with 'match_audio'. Accepts a 'variable:' reference.", + "type": [ + "integer", + "string" + ], + "default": 0 + }, "anchor": { "description": "Latent statistics anchor (MiniMax-H3 t2va and fl2va only, with continuity 'guide' or 'last_frame'). 'segment_1' passes segment 1's decoded frames to every later segment as its 'latent_anchor': each later segment's video latent is rescaled, per VAE latent channel, to segment 1's latent mean and std before decode, so colour and contrast stop drifting from segment to segment. Segment 1 itself is not corrected, and audio is never touched. 'none' (the default) anchors nothing; any other value, of any type, is refused naming the two. Accepts a 'variable:' reference.", "default": "none" diff --git a/tests/test_chain.py b/tests/test_chain.py index 4755a408a..be64604ac 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -921,6 +921,102 @@ def test_segments_mode_lays_the_previous_tail_in_as_a_guide(self, guide_frames): assert result.frames[124].getpixel((0, 0)) == (50, guide_frames, 7) assert result.frames[-1].getpixel((0, 0)) == (100, 123, 7) + def test_backoff_carries_the_frames_one_block_before_the_end(self): + pipeline = FakePipeline(self.output) + + run_chain( + pipeline, + self.chain(guide_frames=22, guide_backoff=17), + self.arguments(), + ) + + guide = pipeline.calls[1]["guides"][0] + got = [frame.getpixel((0, 0)) for frame in guide["video"].frames] + assert got == [(0, f, 7) for f in range(85, 107)] + samples = frames_to_samples(22, self.FPS, self.RATE) + assert guide["video"].audio.shape[-1] == samples + + def test_backoff_trims_every_segment_but_the_last(self): + pipeline = FakePipeline(self.output) + + result = run_chain( + pipeline, + self.chain(guide_frames=22, guide_backoff=17), + self.arguments(), + ) + + assert len(result.frames) == 124 + 2 * (124 - 22) - 2 * 17 + # segment 1 ends 17 frames short, the last segment keeps its tail + assert result.frames[106].getpixel((0, 0)) == (0, 106, 7) + assert result.frames[107].getpixel((0, 0)) == (50, 22, 7) + assert result.frames[-1].getpixel((0, 0)) == (100, 123, 7) + assert [s["num_frames"] for s in result.shots] == [107, 85, 102] + assert [s["start_frame"] for s in result.shots] == [0, 107, 192] + + def test_backoff_slices_the_held_audio_to_the_same_span(self): + pipeline = FakePipeline(self.output) + + result = run_chain( + pipeline, + self.chain(guide_frames=22, guide_backoff=17), + self.arguments(), + ) + + per_frame = frames_to_samples(1, self.FPS, self.RATE) + # the held span is 22 frames of segment 1's audio (value 1.0), taken + # before the tail, so it is all segment 1's + held = pipeline.calls[1]["guides"][0]["video"].audio + assert held.shape[-1] == frames_to_samples(22, self.FPS, self.RATE) + assert bool((held == 1.0).all()) + # picture and track stay the same length + total = result.audio.shape[-1] + assert abs( + total - frames_to_samples(len(result.frames), self.FPS, self.RATE) + ) <= (3 * per_frame) + assert [s["num_samples"] for s in result.shots][-1] > 0 + + def test_backoff_zero_is_todays_chain(self): + plain = run_chain(FakePipeline(self.output), self.chain(), self.arguments()) + zero = run_chain( + FakePipeline(self.output), self.chain(guide_backoff=0), self.arguments() + ) + + assert [f.getpixel((0, 0)) for f in zero.frames] == [ + f.getpixel((0, 0)) for f in plain.frames + ] + assert zero.shots == plain.shots + + def test_backoff_must_be_a_multiple_of_17(self): + with pytest.raises(ValueError, match="17"): + ChainConfig(self.chain(guide_backoff=10), self.arguments()) + + def test_backoff_needs_guide_continuity(self): + chain = self.chain(guide_backoff=17, continuity="last_frame") + with pytest.raises(ValueError, match="guide_backoff"): + ChainConfig(chain, self.arguments()) + + def test_backoff_is_refused_with_match_audio(self): + audio = torch.zeros(1, 2, int(256 / self.FPS * self.RATE)) + chain = { + "match_audio": True, + "continuity": "guide", + "guide_backoff": 17, + "fps": self.FPS, + "frame_snap": MINIMAX_SNAP, + } + arguments = { + "num_frames": 124, + "references": [FakeAudioReference(audio, self.RATE)], + } + with pytest.raises(ValueError, match="match_audio"): + ChainConfig(chain, arguments) + + def test_backoff_tail_trim_is_on_the_config(self): + config = ChainConfig(self.chain(guide_backoff=17), self.arguments()) + + assert config.tail_trim == 17 + assert ChainConfig(self.chain(), self.arguments()).tail_trim == 0 + def test_the_guide_is_appended_to_the_segments_own_guides(self): pipeline = FakePipeline(self.output) own = {"video": "clip.mp4", "frame": 17} diff --git a/tests/test_guide_chain_validation.py b/tests/test_guide_chain_validation.py index c090c649c..b538fd3f2 100644 --- a/tests/test_guide_chain_validation.py +++ b/tests/test_guide_chain_validation.py @@ -101,6 +101,37 @@ def test_a_guide_length_other_than_22_or_39(self, frames): assert "22" in error["message"] assert "39" in error["message"] + @pytest.mark.parametrize("backoff", [10, -17, 1.5, True]) + def test_a_backoff_that_is_not_a_multiple_of_17(self, backoff): + chain = {"segments": 2, "continuity": "guide", "guide_backoff": backoff} + + error = one(definition(chain=chain)) + + assert error["path"] == "steps[0].pipeline.chain.guide_backoff" + assert "17" in error["message"] + + @pytest.mark.parametrize("backoff", [0, 17, 34]) + def test_a_backoff_of_whole_blocks(self, backoff): + chain = {"segments": 2, "continuity": "guide", "guide_backoff": backoff} + + assert guide_chain_errors(definition(chain=chain)) == [] + + def test_a_backoff_off_guide_continuity(self): + chain = {"segments": 2, "continuity": "last_frame", "guide_backoff": 17} + + error = one(definition("t2va", chain=chain)) + + assert error["path"] == "steps[0].pipeline.chain.guide_backoff" + assert "guide" in error["message"] + + def test_a_backoff_with_match_audio(self): + chain = {"match_audio": True, "continuity": "guide", "guide_backoff": 17} + + error = one(definition(chain=chain)) + + assert error["path"] == "steps[0].pipeline.chain.guide_backoff" + assert "match_audio" in error["message"] + def test_carry_frames(self): chain = {"segments": 2, "continuity": "guide", "carry_frames": 10} From e8c2dac97ad2f5c6becbfdeef3a503d0f94c035c Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:36:40 -0600 Subject: [PATCH 017/405] feat(chain): #1187 - keystone: segment 1's frame 0 as every later keyframe Co-Authored-By: Claude Opus 5.5 --- docs/ATTRIBUTIONS.md | 7 ++- docs/WORKFLOW_GUIDE.md | 4 ++ dw/guides.py | 9 ++++ dw/pipeline_processors/chain.py | 38 +++++++++++++--- dw/pipeline_processors/h3_rules.py | 35 +++++++++++++++ dw/workflow_schema.json | 4 ++ tests/test_chain.py | 71 ++++++++++++++++++++++++++++++ 7 files changed, 162 insertions(+), 6 deletions(-) diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md index f4eb337b0..d9ef2226e 100644 --- a/docs/ATTRIBUTIONS.md +++ b/docs/ATTRIBUTIONS.md @@ -142,4 +142,9 @@ concept was taken, together with seitanism's context hold. It inspired ([dw/pipeline_processors/chain.py](../dw/pipeline_processors/chain.py); [#1189](https://github.com/dkackman/diffusers-workflow/issues/1189)): handing the next segment the frames one block before the end and dropping the previous -segment's last block at assembly, so the join lands inside the held copy. +segment's last block at assembly, so the join lands inside the held copy. The +same suite's idea of a first-segment keyframe, and NerdNotes' (no licence, so +concepts only), inspired `keystone` +([#1187](https://github.com/dkackman/diffusers-workflow/issues/1187)): segment 1's +frame 0 as every later segment's keyframe, pulling the chain back toward its +clean start. diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 337bdb7be..4d404c360 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -1859,6 +1859,7 @@ head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 | where | MiniMax-H3 `t2va` or `fl2va`; refused on `ref2va`, with `references`, or on a non-H3 step - guides stay off ref2va | | `carry_frames` | refused - the carry is always the last `guide_frames` frames | | `guide_backoff` | 0 (default) or a multiple of 17: the carry is the `guide_frames` frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs `num_frames` above P + back-off. Refused off `guide` continuity and with `match_audio` | +| `keystone` | `none` (default) or `segment_1`: on `fl2va`, every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain. The guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Refused on `last_frame` (there the keyframe is the only carry), and any other value is refused naming the two | | `trim_frames` | ignored (the trim is P), with a note in the job log | | `crossfade_ms` | ignored when `carry_audio` is true: the guide held the audio, so the seam is joined with only a declick | | `num_frames` | must exceed P, or the chain cannot progress | @@ -2447,6 +2448,9 @@ joined into a single file: `last_frame` or `guide` continuity): every segment after the first is rescaled to segment 1's per-channel latent statistics before decode, holding the chain's colour and contrast (see *Anchoring a chain's colour*). +- `keystone` — `none` (default) or `segment_1` (with `guide` continuity only): every later + segment's keyframe is segment 1's frame 0 instead of the guide's first frame (see + *Chaining with a guide*). - `anchor_strength` — with `anchor: "segment_1"`, how far that match goes: 0 to 1, default 1 (the full match). - `carry_frames` — with `last_segment`, bound the carry to the last N frames of the diff --git a/dw/guides.py b/dw/guides.py index 8439c5ab8..7efee4324 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -62,6 +62,7 @@ anchor_strength_problem, RENDER_GRID, chain_anchor_problem, + chain_keystone_problem, guide_backoff_problems, guide_chain_problems, guide_end_problem, @@ -395,6 +396,14 @@ def anchor_errors(workflow_definition, source_indices=None): "message": f"{problem}{where}", } ) + problem = chain_keystone_problem(chain) + if problem: + errors.append( + { + "path": render_path(base + ("chain", "keystone")), + "message": f"{problem}{where}", + } + ) path = render_path(base + ("chain", "anchor")) problem = chain_anchor_problem(chain) if problem: diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index 240f89371..eea2686a5 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -68,7 +68,10 @@ LATENT_ANCHOR_INPUT, LATENT_ANCHOR_STRENGTH_INPUT, anchor_strength_problem, + KEYSTONE_NONE, + KEYSTONE_SEGMENT_1, chain_anchor_problem, + chain_keystone_problem, guide_backoff_problems, guide_chain_problems, ) @@ -196,7 +199,7 @@ def extract(self, artifact): frames, audio[:, max(stop - samples, 0) : stop], sample_rate ) - def inject(self, arguments, carry, segment_argument): + def inject(self, arguments, carry, segment_argument, keystone=None): if carry.audio is not None: video = AudioVideo( carry.frames, @@ -211,7 +214,10 @@ def inject(self, arguments, carry, segment_argument): arguments[GUIDES_INPUT] = list(arguments.get(GUIDES_INPUT) or []) + [guide] # fl2va's keyframe - t2va passes none, and gets none if arguments.get(segment_argument) is not None: - arguments[segment_argument] = carry.frames[0] + # A keystone (#1187) is segment 1's frame 0 in place of the guide's + arguments[segment_argument] = ( + keystone if keystone is not None else carry.frames[0] + ) # The names are h3_rules.CHAIN_CONTINUITY_MODES, which validation reads too; @@ -285,6 +291,9 @@ def run_chain(pipeline, chain_definition, arguments): # under `anchor: "segment_1"` (#1166) - the same frame objects the output # holds, so keeping them costs no copy on an unspilled chain anchor_frames = None + # Segment 1's frame 0 alone, the keyframe of every later segment under + # `keystone: "segment_1"` (#1187) + keystone_frame = None # One shot per segment, measured as the picture and track grow (#378); # the frames are counted rather than read off `frames`, which a spilled # chain never fills @@ -294,7 +303,13 @@ def run_chain(pipeline, chain_definition, arguments): try: for segment in config.plan: segment_arguments = _segment_arguments( - arguments, config, segment, continuity, carry, anchor_frames + arguments, + config, + segment, + continuity, + carry, + anchor_frames, + keystone_frame, ) logger.info( @@ -316,6 +331,8 @@ def run_chain(pipeline, chain_definition, arguments): segment_frames = frames_as_pil_list(artifact) if config.anchor == ANCHOR_SEGMENT_1 and segment.index == 0: anchor_frames = segment_frames + if config.keystone == KEYSTONE_SEGMENT_1 and segment.index == 0: + keystone_frame = segment_frames[0] segment_audio, segment_rate = _generated_audio(artifact) # A segment's own generated audio can run a codec-padding sliver # short of the frames it was asked for - the same gap #197 fixed for @@ -418,7 +435,7 @@ def _stitched(config, spill, frames, audio, audio_rate, shots): def _segment_arguments( - arguments, config, segment, continuity, carry, anchor_frames=None + arguments, config, segment, continuity, carry, anchor_frames=None, keystone=None ): """One segment's arguments: the step's, with this segment's prompt, its slice of a match_audio track, the previous segment's carry, and - under @@ -441,7 +458,12 @@ def _segment_arguments( ) if segment.index > 0: - continuity.inject(segment_arguments, carry, config.segment_argument) + if keystone is not None: + continuity.inject( + segment_arguments, carry, config.segment_argument, keystone=keystone + ) + else: + continuity.inject(segment_arguments, carry, config.segment_argument) if anchor_frames is not None: segment_arguments[LATENT_ANCHOR_INPUT] = anchor_frames if config.anchor_strength is not None: @@ -558,6 +580,12 @@ def __init__( if self.anchor_strength is not None: self.anchor_strength = float(self.anchor_strength) + # Segment 1's frame 0 as every later keyframe (#1187) + problem = chain_keystone_problem(chain_definition) + if problem: + raise ValueError(f"Chain {problem}") + self.keystone = chain_definition.get("keystone", KEYSTONE_NONE) + self.segment_argument = chain_definition.get("segment_argument", "image") self.carry_frames = chain_definition.get("carry_frames", None) if self.carry_frames is not None: diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index 812cb0f26..bd6fcdd24 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -577,6 +577,13 @@ def guide_frame_problem(frame): ANCHOR_CONTINUITIES = ("last_frame", GUIDE_CONTINUITY) +# The keystone (#1187): segment 1's frame 0 as every later segment's fl2va +# keyframe, instead of the guide's first frame. Guide continuity only +KEYSTONE_NONE = "none" +KEYSTONE_SEGMENT_1 = "segment_1" +CHAIN_KEYSTONE_MODES = (KEYSTONE_NONE, KEYSTONE_SEGMENT_1) + + def anchor_strength_problem(value): """Why `value` cannot be a `latent_anchor_strength` / chain `anchor_strength`, naming the range, or None. An unresolved reference is left to the run.""" @@ -595,6 +602,34 @@ def anchor_strength_problem(value): return None +def chain_keystone_problem(chain): + """The reason a chain block's `keystone` is refused, or None: an unknown + value (naming the values), or `segment_1` without `continuity: "guide"`. + Unresolved references are left to the run.""" + keystone = chain.get("keystone", KEYSTONE_NONE) + if isinstance(keystone, str) and references.is_ref(references.UNRESOLVED, keystone): + return None + if keystone not in CHAIN_KEYSTONE_MODES: + return ( + f"unknown chain keystone {keystone!r} - expected one of " + f"{', '.join(CHAIN_KEYSTONE_MODES)}" + ) + if keystone == KEYSTONE_NONE: + return None + continuity = chain.get("continuity", "last_frame") + if isinstance(continuity, str) and references.is_ref( + references.UNRESOLVED, continuity + ): + return None + if continuity != GUIDE_CONTINUITY: + return ( + f"keystone 'segment_1' needs continuity 'guide' - the keystone " + f"replaces the keyframe, and on {continuity!r} the keyframe is the " + f"only carry; got {continuity!r}" + ) + return None + + def chain_anchor_problem(chain): """The reason a chain block's `anchor` is refused for its own settings, or None: an unknown value (naming the values), or `segment_1` with a continuity diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index b93081da6..a72948423 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -355,6 +355,10 @@ "anchor_strength": { "description": "How far each later segment's latent statistics match goes, with anchor 'segment_1' (refused without it): a number from 0 to 1, default 1. 1 is the full per-channel mean/std match; 0 leaves the latent alone; 0.5 moves each channel halfway to segment 1's statistics, holding the look without flattening a deliberate change in light. Anything outside [0, 1], or not a number, is refused naming the range. Accepts a 'variable:' reference." }, + "keystone": { + "description": "Keyframe source for a guide chain on MiniMax-H3 fl2va. 'segment_1' sets every later segment's keyframe (the 'segment_argument' image) to segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain; the guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Needs continuity 'guide': on 'last_frame' the keyframe is the only carry and is refused. 'none' (the default) leaves the keyframe as it is; any other value, of any type, is refused naming the two. Accepts a 'variable:' reference.", + "default": "none" + }, "segment_argument": { "description": "The pipeline argument the carry-over lands in - e.g. 'image' for image-to-video pipelines, or 'references' for reference-conditioned modular pipelines, where the carry-over is appended as an image or video reference.", "type": "string", diff --git a/tests/test_chain.py b/tests/test_chain.py index be64604ac..52caa7435 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -975,6 +975,77 @@ def test_backoff_slices_the_held_audio_to_the_same_span(self): ) <= (3 * per_frame) assert [s["num_samples"] for s in result.shots][-1] > 0 + def test_keystone_sets_every_later_keyframe_to_segment_1_frame_0(self): + pipeline = FakePipeline(self.output) + keyframe = solid_frame((1, 2, 3)) + + run_chain( + pipeline, + self.chain(keystone="segment_1"), + self.arguments(image=keyframe), + ) + + assert pipeline.calls[0]["image"] is keyframe + assert pipeline.calls[1]["image"].getpixel((0, 0)) == (0, 0, 7) + assert pipeline.calls[2]["image"].getpixel((0, 0)) == (0, 0, 7) + + def test_keystone_leaves_the_guide_as_the_carried_tail(self): + pipeline = FakePipeline(self.output) + + result = run_chain( + pipeline, + self.chain(keystone="segment_1"), + self.arguments(image=solid_frame((1, 2, 3))), + ) + + guide = pipeline.calls[1]["guides"][0] + got = [frame.getpixel((0, 0)) for frame in guide["video"].frames] + assert got == [(0, f, 7) for f in range(102, 124)] + assert len(result.frames) == 124 + 2 * (124 - 22) + # segment 1's frame 0 appears once, at the head of the output + assert result.frames[0].getpixel((0, 0)) == (0, 0, 7) + assert result.frames[124].getpixel((0, 0)) == (50, 22, 7) + + def test_keystone_is_refused_without_guide_continuity(self): + chain = self.chain(keystone="segment_1", continuity="last_frame") + with pytest.raises(ValueError, match="only carry"): + ChainConfig(chain, self.arguments()) + + def test_keystone_refuses_an_unknown_value_naming_both(self): + with pytest.raises(ValueError, match="none, segment_1"): + ChainConfig(self.chain(keystone="segment_2"), self.arguments()) + + def test_keystone_none_is_todays_chain(self): + key = solid_frame((1, 2, 3)) + plain = run_chain( + FakePipeline(self.output), self.chain(), self.arguments(image=key) + ) + pipeline = FakePipeline(self.output) + none = run_chain( + pipeline, self.chain(keystone="none"), self.arguments(image=key) + ) + + assert [f.getpixel((0, 0)) for f in none.frames] == [ + f.getpixel((0, 0)) for f in plain.frames + ] + assert pipeline.calls[1]["image"].getpixel((0, 0)) == (0, 102, 7) + + def test_keystone_validation_wires_the_rule(self): + from dw.guides import anchor_errors + + definition = { + "steps": [ + { + "name": "s", + "pipeline": { + "chain": {"segments": 2, "keystone": "segment_1"}, + "arguments": {}, + }, + } + ] + } + assert any("keystone" in e["message"] for e in anchor_errors(definition)) + def test_backoff_zero_is_todays_chain(self): plain = run_chain(FakePipeline(self.output), self.chain(), self.arguments()) zero = run_chain( From e6a91c470229d1729836bdd3d7e2bdb0adae52df Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:40:33 -0600 Subject: [PATCH 018/405] feat(chain): #1032 - carry_pick sharpest of the last N frames Co-Authored-By: Claude Opus 5.5 --- docs/ATTRIBUTIONS.md | 6 ++ docs/WORKFLOW_GUIDE.md | 1 + dw/guides.py | 9 +++ dw/pipeline_processors/chain.py | 33 +++++++++-- dw/pipeline_processors/chain_pick.py | 53 +++++++++++++++++ dw/pipeline_processors/h3_rules.py | 45 ++++++++++++++ dw/workflow_schema.json | 13 ++++ tests/test_chain.py | 88 ++++++++++++++++++++++++++++ tests/test_chain_pick.py | 61 +++++++++++++++++++ 9 files changed, 305 insertions(+), 4 deletions(-) create mode 100644 dw/pipeline_processors/chain_pick.py create mode 100644 tests/test_chain_pick.py diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md index d9ef2226e..a15958cf6 100644 --- a/docs/ATTRIBUTIONS.md +++ b/docs/ATTRIBUTIONS.md @@ -148,3 +148,9 @@ concepts only), inspired `keystone` ([#1187](https://github.com/dkackman/diffusers-workflow/issues/1187)): segment 1's frame 0 as every later segment's keyframe, pulling the chain back toward its clean start. + +harlixay7/Kinoloom (MIT; concept only, no code taken) inspired `carry_pick` +([dw/pipeline_processors/chain_pick.py](../dw/pipeline_processors/chain_pick.py); +[#1032](https://github.com/dkackman/diffusers-workflow/issues/1032)): picking the +sharpest of a segment's last frames as the next keyframe rather than always the +last, and ending the segment there. diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 4d404c360..5e2e16501 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -1860,6 +1860,7 @@ head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 | `carry_frames` | refused - the carry is always the last `guide_frames` frames | | `guide_backoff` | 0 (default) or a multiple of 17: the carry is the `guide_frames` frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs `num_frames` above P + back-off. Refused off `guide` continuity and with `match_audio` | | `keystone` | `none` (default) or `segment_1`: on `fl2va`, every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain. The guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Refused on `last_frame` (there the keyframe is the only carry), and any other value is refused naming the two | +| `carry_pick` | `{"sharpest": N}`, N from 1 to 8, with `last_frame` continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones | | `trim_frames` | ignored (the trim is P), with a note in the job log | | `crossfade_ms` | ignored when `carry_audio` is true: the guide held the audio, so the seam is joined with only a declick | | `num_frames` | must exceed P, or the chain cannot progress | diff --git a/dw/guides.py b/dw/guides.py index 7efee4324..18c1c0e79 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -62,6 +62,7 @@ anchor_strength_problem, RENDER_GRID, chain_anchor_problem, + chain_carry_pick_problem, chain_keystone_problem, guide_backoff_problems, guide_chain_problems, @@ -396,6 +397,14 @@ def anchor_errors(workflow_definition, source_indices=None): "message": f"{problem}{where}", } ) + problem = chain_carry_pick_problem(chain) + if problem: + errors.append( + { + "path": render_path(base + ("chain", "carry_pick")), + "message": f"{problem}{where}", + } + ) problem = chain_keystone_problem(chain) if problem: errors.append( diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index eea2686a5..6fb26c587 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -53,6 +53,7 @@ SegmentSpill, spilled_frames, ) +from .chain_pick import sharpest_index from .h3_anchor import anchor_refusal from .h3_guides import guides_refusal from .h3_hold import hold_audio_reference @@ -71,6 +72,7 @@ KEYSTONE_NONE, KEYSTONE_SEGMENT_1, chain_anchor_problem, + chain_carry_pick_problem, chain_keystone_problem, guide_backoff_problems, guide_chain_problems, @@ -90,6 +92,13 @@ def __init__(self, config): self.config = config def extract(self, artifact): + # With carry_pick, the sharpest of the last N frames; `picked_back` + # is how many frames after it the segment drops (0 = the last frame) + self.picked_back = 0 + if self.config.carry_pick: + frames = frames_as_pil_list(artifact) + self.picked_back = sharpest_index(frames, self.config.carry_pick) + return frames[len(frames) - 1 - self.picked_back] return extract_frame(artifact, -1) def inject(self, arguments, carry, segment_argument): @@ -345,15 +354,24 @@ def run_chain(pipeline, chain_definition, arguments): segment_audio, len(segment_frames), config.fps, segment_rate ) - if config.tail_trim and segment.index < len(config.plan) - 1: + tail_trim = config.tail_trim + getattr(continuity, "picked_back", 0) + if config.carry_pick and segment.index < len(config.plan) - 1: + emit_log( + f"Chain seam {segment.index + 1}/{len(config.plan) - 1}: " + f"carry_pick took frame {len(segment_frames) - tail_trim} of " + f"{len(segment_frames)} (sharpest of the last " + f"{config.carry_pick}), dropping {tail_trim} after it" + ) + if tail_trim and segment.index < len(config.plan) - 1: # The carry is already taken; the tail it backed off from is - # replaced by the next segment's fresh take (#1189) - segment_frames = segment_frames[: -config.tail_trim] + # replaced by the next segment's fresh take (#1189), and a + # picked frame ends its segment (#1032) + segment_frames = segment_frames[:-tail_trim] if segment_audio is not None: segment_audio = segment_audio[ :, : segment_audio.shape[1] - - frames_to_samples(config.tail_trim, config.fps, segment_rate), + - frames_to_samples(tail_trim, config.fps, segment_rate), ] kept_frames = segment_frames[segment.head_trim :] @@ -586,6 +604,13 @@ def __init__( raise ValueError(f"Chain {problem}") self.keystone = chain_definition.get("keystone", KEYSTONE_NONE) + # The sharpest of the last N frames as the keyframe (#1032) + problem = chain_carry_pick_problem(chain_definition) + if problem: + raise ValueError(f"Chain {problem}") + pick = chain_definition.get("carry_pick") + self.carry_pick = int(pick["sharpest"]) if pick else 0 + self.segment_argument = chain_definition.get("segment_argument", "image") self.carry_frames = chain_definition.get("carry_frames", None) if self.carry_frames is not None: diff --git a/dw/pipeline_processors/chain_pick.py b/dw/pipeline_processors/chain_pick.py new file mode 100644 index 000000000..d6557d73e --- /dev/null +++ b/dw/pipeline_processors/chain_pick.py @@ -0,0 +1,53 @@ +"""Picking the sharpest of a segment's last frames as the next keyframe (#1032). + +A motion-blurred final frame passes its softness to the next segment as that +segment's keyframe. `carry_pick: {"sharpest": N}` looks at the last N frames +instead. Numpy only: the sharpness is the variance of a 3x3 Laplacian on luma. + +The idea comes from harlixay7/Kinoloom (MIT; concept only, no code). +""" + +import numpy + + +def _luma(frame): + pixels = numpy.asarray(frame, dtype=numpy.float64) + if pixels.ndim == 3 and pixels.shape[-1] >= 3: + return 0.299 * pixels[..., 0] + 0.587 * pixels[..., 1] + 0.114 * pixels[..., 2] + if pixels.ndim == 3: + return pixels[..., 0] + return pixels + + +def sharpness(frame): + """The variance of a 3x3 Laplacian on the frame's luma - higher is sharper. + + Accepts a PIL image or an array. A frame under 3 pixels either way has no + interior to measure and scores 0. + """ + luma = _luma(frame) + if luma.shape[0] < 3 or luma.shape[1] < 3: + return 0.0 + laplacian = ( + luma[:-2, 1:-1] + + luma[2:, 1:-1] + + luma[1:-1, :-2] + + luma[1:-1, 2:] + - 4.0 * luma[1:-1, 1:-1] + ) + return float(laplacian.var()) + + +def sharpest_index(frames, window): + """Index from the end of the sharpest of the last `window` frames. + + 0 is the last frame. Ties go to the latest frame; a window longer than the + clip uses the whole clip. + """ + count = min(int(window), len(frames)) + best_back, best_score = 0, None + for back in range(count): + score = sharpness(frames[len(frames) - 1 - back]) + if best_score is None or score > best_score: + best_back, best_score = back, score + return best_back diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index bd6fcdd24..53382a7d0 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -630,6 +630,51 @@ def chain_keystone_problem(chain): return None +# The carry pick (#1032): the sharpest of the last N frames as the next +# keyframe, the segment trimmed after it. Family-independent, `last_frame` only +CARRY_PICK_MIN = 1 +CARRY_PICK_MAX = 8 + + +def chain_carry_pick_problem(chain): + """The reason a chain block's `carry_pick` is refused, or None: a shape + other than {"sharpest": N}, N outside 1..8, or a continuity other than + `last_frame`. Unresolved references are left to the run.""" + if "carry_pick" not in chain: + return None + pick = chain["carry_pick"] + if isinstance(pick, str) and references.is_ref(references.UNRESOLVED, pick): + return None + expected = ( + f"expected {{'sharpest': N}} with N from {CARRY_PICK_MIN} to {CARRY_PICK_MAX}" + ) + if not isinstance(pick, dict) or set(pick) != {"sharpest"}: + return f"invalid carry_pick {pick!r} - {expected}" + window = pick["sharpest"] + if isinstance(window, str) and references.is_ref(references.UNRESOLVED, window): + return None + if ( + isinstance(window, bool) + or not isinstance(window, int) + or not CARRY_PICK_MIN <= window <= CARRY_PICK_MAX + ): + return ( + f"carry_pick sharpest must be a whole number between {CARRY_PICK_MIN} " + f"and {CARRY_PICK_MAX}, got {window!r}" + ) + continuity = chain.get("continuity", "last_frame") + if isinstance(continuity, str) and references.is_ref( + references.UNRESOLVED, continuity + ): + return None + if continuity != "last_frame": + return ( + f"carry_pick needs continuity 'last_frame' - the other modes carry " + f"more than one frame; got {continuity!r}" + ) + return None + + def chain_anchor_problem(chain): """The reason a chain block's `anchor` is refused for its own settings, or None: an unknown value (naming the values), or `segment_1` with a continuity diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index a72948423..230ff75bb 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -359,6 +359,19 @@ "description": "Keyframe source for a guide chain on MiniMax-H3 fl2va. 'segment_1' sets every later segment's keyframe (the 'segment_argument' image) to segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain; the guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Needs continuity 'guide': on 'last_frame' the keyframe is the only carry and is refused. 'none' (the default) leaves the keyframe as it is; any other value, of any type, is refused naming the two. Accepts a 'variable:' reference.", "default": "none" }, + "carry_pick": { + "description": "Continuity 'last_frame' only, any model family: pick the next segment's keyframe as the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma; ties go to the latest) instead of always the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame, so motion continues from it; the last segment keeps its tail, and the job log names the picked frame per seam. {'sharpest': N} with N a whole number from 1 to 8 (1 is the last frame); anything else, or any continuity but 'last_frame', is refused naming the allowed values.", + "type": "object", + "properties": { + "sharpest": { + "type": ["integer", "string"], + "minimum": 1, + "maximum": 8 + } + }, + "required": ["sharpest"], + "additionalProperties": false + }, "segment_argument": { "description": "The pipeline argument the carry-over lands in - e.g. 'image' for image-to-video pipelines, or 'references' for reference-conditioned modular pipelines, where the carry-over is appended as an image or video reference.", "type": "string", diff --git a/tests/test_chain.py b/tests/test_chain.py index 52caa7435..0951c1c27 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -1350,3 +1350,91 @@ def test_run_refuses_string_prompts(self): action = SimpleNamespace(pipeline_definition={"chain": {"prompts": "abc"}}) with pytest.raises(ValueError, match="one prompt per segment"): resolve_chain_prompts(action, {}) + + +class TestCarryPick: + """carry_pick (#1032): the sharpest of the last N frames is the keyframe, + and the segment is trimmed after it.""" + + def output(self, arguments, index): + # frame 6 of 8 is the sharp one; the rest are flat + frames = [] + for f in range(8): + if f == 6: + image = Image.new("RGB", (16, 16), (0, 0, 0)) + for x in range(0, 16, 2): + for y in range(0, 16, 2): + image.putpixel((x, y), (255, 255, 255)) + else: + image = Image.new("RGB", (16, 16), (100, f, index)) + frames.append(image) + return SimpleNamespace(frames=[frames]) + + def chain(self, **extra): + return {"segments": 2, "trim_frames": 0, "fps": 4, **extra} + + def test_a_blurred_last_frame_is_replaced_by_the_sharp_one_before_it(self): + pipeline = FakePipeline(self.output) + run_chain(pipeline, self.chain(carry_pick={"sharpest": 3}), {"num_frames": 8}) + assert pipeline.calls[1]["image"].getpixel((0, 0)) == (255, 255, 255) + + def test_window_one_is_the_last_frame(self): + pipeline = FakePipeline(self.output) + result = run_chain( + pipeline, self.chain(carry_pick={"sharpest": 1}), {"num_frames": 8} + ) + assert pipeline.calls[1]["image"].getpixel((0, 0)) == (100, 7, 0) + assert len(result.frames) == 16 + + def test_the_segment_is_trimmed_after_the_picked_frame(self): + pipeline = FakePipeline(self.output) + result = run_chain( + pipeline, self.chain(carry_pick={"sharpest": 3}), {"num_frames": 8} + ) + # frame 6 is one back from the end: segment 1 keeps frames 0..6 + assert [s["num_frames"] for s in result.shots] == [7, 8] + assert len(result.frames) == 15 + assert result.frames[6].getpixel((0, 0)) == (255, 255, 255) + + def test_the_pick_is_logged_per_seam(self): + pipeline = FakePipeline(self.output) + with patch("dw.pipeline_processors.chain.emit_log") as emit_log: + run_chain( + pipeline, self.chain(carry_pick={"sharpest": 3}), {"num_frames": 8} + ) + notes = [call.args[0] for call in emit_log.call_args_list] + assert any("carry_pick" in n and "frame 7" in n for n in notes) + + def test_the_last_segment_keeps_its_tail(self): + pipeline = FakePipeline(self.output) + result = run_chain( + pipeline, + {"segments": 3, "trim_frames": 0, "fps": 4, "carry_pick": {"sharpest": 3}}, + {"num_frames": 8}, + ) + assert [s["num_frames"] for s in result.shots] == [7, 7, 8] + + def test_it_is_refused_off_last_frame_naming_the_allowed_values(self): + chain = self.chain(carry_pick={"sharpest": 3}, continuity="guide") + with pytest.raises(ValueError, match="last_frame"): + ChainConfig(chain, {"num_frames": 8}) + + def test_it_is_refused_out_of_range_naming_the_range(self): + with pytest.raises(ValueError, match="1 and 8"): + ChainConfig(self.chain(carry_pick={"sharpest": 9}), {"num_frames": 8}) + + def test_validation_wires_the_rule(self): + from dw.guides import anchor_errors + + definition = { + "steps": [ + { + "name": "s", + "pipeline": { + "chain": {"segments": 2, "carry_pick": {"sharpest": 99}}, + "arguments": {}, + }, + } + ] + } + assert any("carry_pick" in e["message"] for e in anchor_errors(definition)) diff --git a/tests/test_chain_pick.py b/tests/test_chain_pick.py new file mode 100644 index 000000000..592f9dad5 --- /dev/null +++ b/tests/test_chain_pick.py @@ -0,0 +1,61 @@ +"""carry_pick (#1032): the sharpest of the last N frames as the next keyframe.""" + +import numpy +from PIL import Image + +from dw.pipeline_processors.chain_pick import sharpest_index, sharpness + + +def flat(value=128): + return numpy.full((16, 16, 3), value, dtype=numpy.uint8) + + +def checker(): + frame = numpy.zeros((16, 16, 3), dtype=numpy.uint8) + frame[::2, ::2] = 255 + frame[1::2, 1::2] = 255 + return frame + + +def test_a_blurred_last_frame_loses_to_a_sharp_one_before_it(): + frames = [flat(), checker(), flat()] + assert sharpest_index(frames, 3) == 1 + + +def test_window_one_is_always_the_last_frame(): + assert sharpest_index([checker(), flat()], 1) == 0 + + +def test_ties_go_to_the_latest_frame(): + assert sharpest_index([flat(), flat(), flat()], 3) == 0 + + +def test_the_window_bounds_the_search(): + frames = [checker(), flat(), flat()] + assert sharpest_index(frames, 2) == 0 + + +def test_a_window_larger_than_the_clip_uses_the_whole_clip(): + assert sharpest_index([checker(), flat()], 8) == 1 + + +def test_sharpness_accepts_pil_and_orders_detail_over_flat(): + assert sharpness(Image.fromarray(checker())) > sharpness(Image.fromarray(flat())) + assert sharpness(flat()) == 0.0 + + +def test_a_tiny_frame_has_no_sharpness_to_measure(): + assert sharpness(numpy.zeros((2, 2, 3), dtype=numpy.uint8)) == 0.0 + + +def test_the_rule_names_the_allowed_values(): + from dw.pipeline_processors.h3_rules import chain_carry_pick_problem + + assert chain_carry_pick_problem({}) is None + assert chain_carry_pick_problem({"carry_pick": {"sharpest": 4}}) is None + for bad in ({"sharpest": 0}, {"sharpest": 9}, {"best": 3}, 3, {}): + assert "1" in chain_carry_pick_problem({"carry_pick": bad}) + problem = chain_carry_pick_problem( + {"carry_pick": {"sharpest": 3}, "continuity": "guide"} + ) + assert "last_frame" in problem From 3ebe9b73083be5f3fd1675bf14b8ac39f9eb9fdf Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:43:43 -0600 Subject: [PATCH 019/405] fix(chain): #1032 - carry_pick refused with match_audio; window must fit the segment Co-Authored-By: Claude Opus 5.5 --- docs/WORKFLOW_GUIDE.md | 2 +- dw/guides.py | 7 ++++++- dw/pipeline_processors/chain.py | 5 ++++- dw/pipeline_processors/h3_rules.py | 29 ++++++++++++++++++++++++++--- tests/test_chain.py | 10 ++++++++++ tests/test_chain_pick.py | 30 ++++++++++++++++++++++++++++++ 6 files changed, 77 insertions(+), 6 deletions(-) diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 5e2e16501..25ae42a31 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -1860,7 +1860,7 @@ head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 | `carry_frames` | refused - the carry is always the last `guide_frames` frames | | `guide_backoff` | 0 (default) or a multiple of 17: the carry is the `guide_frames` frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs `num_frames` above P + back-off. Refused off `guide` continuity and with `match_audio` | | `keystone` | `none` (default) or `segment_1`: on `fl2va`, every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain. The guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Refused on `last_frame` (there the keyframe is the only carry), and any other value is refused naming the two | -| `carry_pick` | `{"sharpest": N}`, N from 1 to 8, with `last_frame` continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones | +| `carry_pick` | `{"sharpest": N}`, N from 1 to 8, with `last_frame` continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones; `match_audio` is refused (the plan covers the track up front and the pick drops frames per seam, so the video would fall short - use `segments`); and when `num_frames` is known N must be less than `num_frames` minus `trim_frames`, or the pick could empty a segment | | `trim_frames` | ignored (the trim is P), with a note in the job log | | `crossfade_ms` | ignored when `carry_audio` is true: the guide held the audio, so the seam is joined with only a declick | | `num_frames` | must exceed P, or the chain cannot progress | diff --git a/dw/guides.py b/dw/guides.py index 18c1c0e79..b7817e848 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -397,7 +397,12 @@ def anchor_errors(workflow_definition, source_indices=None): "message": f"{problem}{where}", } ) - problem = chain_carry_pick_problem(chain) + problem = chain_carry_pick_problem( + chain, + pipeline.get("arguments", {}).get("num_frames") + if isinstance(pipeline.get("arguments"), dict) + else None, + ) if problem: errors.append( { diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index 6fb26c587..e42bc80b1 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -605,7 +605,10 @@ def __init__( self.keystone = chain_definition.get("keystone", KEYSTONE_NONE) # The sharpest of the last N frames as the keyframe (#1032) - problem = chain_carry_pick_problem(chain_definition) + problem = chain_carry_pick_problem( + chain_definition, + arguments.get("num_frames") if isinstance(arguments, dict) else None, + ) if problem: raise ValueError(f"Chain {problem}") pick = chain_definition.get("carry_pick") diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index 53382a7d0..a3918a887 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -636,10 +636,13 @@ def chain_keystone_problem(chain): CARRY_PICK_MAX = 8 -def chain_carry_pick_problem(chain): +def chain_carry_pick_problem(chain, num_frames=None): """The reason a chain block's `carry_pick` is refused, or None: a shape - other than {"sharpest": N}, N outside 1..8, or a continuity other than - `last_frame`. Unresolved references are left to the run.""" + other than {"sharpest": N}, N outside 1..8, a continuity other than + `last_frame`, `match_audio` (the plan is fixed from the track's length and + the pick drops frames per seam), or - when the step's `num_frames` is known + - an N that could empty a segment. Unresolved references are left to the + run.""" if "carry_pick" not in chain: return None pick = chain["carry_pick"] @@ -672,6 +675,26 @@ def chain_carry_pick_problem(chain): f"carry_pick needs continuity 'last_frame' - the other modes carry " f"more than one frame; got {continuity!r}" ) + if chain.get("match_audio"): + return ( + "carry_pick does not combine with match_audio: the plan covers the " + "track up front and the pick drops up to " + f"{CARRY_PICK_MAX - 1} more frames per seam, leaving the video " + "shorter than the track - use 'segments'" + ) + trim = chain.get("trim_frames", 1) + if ( + isinstance(num_frames, int) + and not isinstance(num_frames, bool) + and isinstance(trim, int) + and not isinstance(trim, bool) + and window >= num_frames - trim + ): + return ( + f"carry_pick sharpest {window} must be less than num_frames " + f"{num_frames} minus trim_frames {trim} ({num_frames - trim}), or " + "the pick could empty a segment" + ) return None diff --git a/tests/test_chain.py b/tests/test_chain.py index 0951c1c27..7bee2be9e 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -1423,6 +1423,16 @@ def test_it_is_refused_out_of_range_naming_the_range(self): with pytest.raises(ValueError, match="1 and 8"): ChainConfig(self.chain(carry_pick={"sharpest": 9}), {"num_frames": 8}) + def test_it_is_refused_with_match_audio(self): + chain = self.chain(carry_pick={"sharpest": 3}, match_audio=True) + with pytest.raises(ValueError, match="match_audio"): + ChainConfig(chain, {"num_frames": 8}) + + def test_it_is_refused_when_the_window_could_empty_a_segment(self): + with pytest.raises(ValueError, match=r"num_frames 8.*trim_frames 0"): + ChainConfig(self.chain(carry_pick={"sharpest": 8}), {"num_frames": 8}) + ChainConfig(self.chain(carry_pick={"sharpest": 7}), {"num_frames": 8}) + def test_validation_wires_the_rule(self): from dw.guides import anchor_errors diff --git a/tests/test_chain_pick.py b/tests/test_chain_pick.py index 592f9dad5..b184cc4ca 100644 --- a/tests/test_chain_pick.py +++ b/tests/test_chain_pick.py @@ -59,3 +59,33 @@ def test_the_rule_names_the_allowed_values(): {"carry_pick": {"sharpest": 3}, "continuity": "guide"} ) assert "last_frame" in problem + + +def test_the_rule_refuses_match_audio_and_a_window_that_fills_the_segment(): + from dw.pipeline_processors.h3_rules import chain_carry_pick_problem + + pick = {"sharpest": 4} + assert "match_audio" in chain_carry_pick_problem( + {"carry_pick": pick, "match_audio": True} + ) + problem = chain_carry_pick_problem({"carry_pick": pick, "trim_frames": 1}, 5) + assert "5" in problem and "trim_frames 1" in problem + assert chain_carry_pick_problem({"carry_pick": pick, "trim_frames": 1}, 6) is None + assert chain_carry_pick_problem({"carry_pick": pick}) is None + + +def test_validation_refuses_match_audio(): + from dw.guides import anchor_errors + + definition = { + "steps": [ + { + "name": "s", + "pipeline": { + "chain": {"match_audio": True, "carry_pick": {"sharpest": 3}}, + "arguments": {"num_frames": 49}, + }, + } + ] + } + assert any("match_audio" in e["message"] for e in anchor_errors(definition)) From 283967c9a22fc09d11b0336b082a26920aaf51b1 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:50:16 -0600 Subject: [PATCH 020/405] feat(chain): #1191 - look_hold: pickup grade, creep and texture ease-out, precompensated carry Co-Authored-By: Claude Opus 5.5 --- docs/ATTRIBUTIONS.md | 8 + docs/WORKFLOW_GUIDE.md | 29 +++ dw/guides.py | 9 + dw/pipeline_processors/chain.py | 31 +++ dw/pipeline_processors/chain_look.py | 301 +++++++++++++++++++++++++++ dw/pipeline_processors/h3_rules.py | 61 ++++++ dw/workflow_schema.json | 11 + tests/test_chain.py | 75 +++++++ tests/test_chain_look.py | 113 ++++++++++ 9 files changed, 638 insertions(+) create mode 100644 dw/pipeline_processors/chain_look.py create mode 100644 tests/test_chain_look.py diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md index a15958cf6..99b728085 100644 --- a/docs/ATTRIBUTIONS.md +++ b/docs/ATTRIBUTIONS.md @@ -154,3 +154,11 @@ harlixay7/Kinoloom (MIT; concept only, no code taken) inspired `carry_pick` [#1032](https://github.com/dkackman/diffusers-workflow/issues/1032)): picking the sharpest of a segment's last frames as the next keyframe rather than always the last, and ending the segment there. + +Smite79/MiniMax-H3-LongVideos (a custom no-redistribution licence; concepts only, +no code taken) inspired `look_hold` +([dw/pipeline_processors/chain_look.py](../dw/pipeline_processors/chain_look.py); +[#1191](https://github.com/dkackman/diffusers-workflow/issues/1191)): a quantile +grade of a segment's opening to the carried frame, an ease-out of contrast and +saturation creep, an ease-out of faint texture, and a keyframe pre-compensated for +the measured boost. diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 25ae42a31..2dd36d2ec 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -1861,11 +1861,40 @@ head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 | `guide_backoff` | 0 (default) or a multiple of 17: the carry is the `guide_frames` frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs `num_frames` above P + back-off. Refused off `guide` continuity and with `match_audio` | | `keystone` | `none` (default) or `segment_1`: on `fl2va`, every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain. The guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Refused on `last_frame` (there the keyframe is the only carry), and any other value is refused naming the two | | `carry_pick` | `{"sharpest": N}`, N from 1 to 8, with `last_frame` continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones; `match_audio` is refused (the plan covers the track up front and the pick drops frames per seam, so the video would fall short - use `segments`); and when `num_frames` is known N must be less than `num_frames` minus `trim_frames`, or the pick could empty a segment | +| `look_hold` | `true` or `{"pickup", "creep", "texture", "precompensate"}` booleans, any model family: pixel corrections on each later segment's frames before its carry is taken (below). On a guide chain `pickup` and `precompensate` do not apply (the carried guide is a whole clip, left alone); an explicit `true` for either there is refused, and `true` alone runs `creep` and `texture` only | | `trim_frames` | ignored (the trim is P), with a note in the job log | | `crossfade_ms` | ignored when `carry_audio` is true: the guide held the audio, so the seam is joined with only a declick | | `num_frames` | must exceed P, or the chain cannot progress | | `guide_noise` | rides on the step's arguments to every segment, so segments 2+ noise the carried guide (above) | +#### Holding a chain's look in pixels: `look_hold` + +`latent_anchor` works inside the H3 latent; `look_hold` works on the decoded frames, so it suits any +family. Each segment after the first is corrected before its carry is taken, so the +carried keyframe and the stitched video are the corrected frames (applied once, not +again to the stitched output). Segment 1 is the reference and is left as rendered. +`"look_hold": true` turns all four on; an object turns on the ones it names: + +- `pickup`: matches the segment's opening (frame 1; frame 0 is the keyframe copy) to + the carried frame by a per-channel quantile map on YCbCr, carries the same map to the + last frame and blends it per frame. Skipped on a flat frame, and on a jump too large + to be drift (a cut). +- `creep`: when the last frame's contrast or saturation exceeds segment 1's by more + than 0.03, scales both down by a factor running from 1 at the first frame to the + needed gain at the last. Gains are at most 1: a darker or paler shot is untouched. +- `texture`: when either end's faint texture (the flat-area detail, not edges) + exceeds the roughest of segment 1's first, middle and last frames by more than 0.03, + subtracts up to half of it, easing in across the shot. +- `precompensate`: divides the carried keyframe's contrast and saturation by the + boost measured so far (the running mean of the rendered opening's look over the fed + keyframe's, each clamped to 1..1.25), so a model that boosts its own keyframe lands + back on the look. + +`pickup` and `precompensate` act on one carried keyframe, so they need continuity +`last_frame`; the carried frame of a guide chain is a whole guide clip and is left +alone. Absent or `false`, the chain is unchanged bit for bit. Concepts from +Smite79/MiniMax-H3-LongVideos ([ATTRIBUTIONS](ATTRIBUTIONS.md)). + #### Anchoring a chain's colour: `latent_anchor` and `anchor: "segment_1"` A long H3 chain drifts in colour and contrast: each segment's latent statistics wander diff --git a/dw/guides.py b/dw/guides.py index b7817e848..1b744d867 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -63,6 +63,7 @@ RENDER_GRID, chain_anchor_problem, chain_carry_pick_problem, + chain_look_hold_problem, chain_keystone_problem, guide_backoff_problems, guide_chain_problems, @@ -410,6 +411,14 @@ def anchor_errors(workflow_definition, source_indices=None): "message": f"{problem}{where}", } ) + problem = chain_look_hold_problem(chain) + if problem: + errors.append( + { + "path": render_path(base + ("chain", "look_hold")), + "message": f"{problem}{where}", + } + ) problem = chain_keystone_problem(chain) if problem: errors.append( diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index e42bc80b1..005f26279 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -29,6 +29,7 @@ previous_result fan-out: three keyframes in, three chained videos out. """ +import copy import gc import logging import math @@ -53,6 +54,7 @@ SegmentSpill, spilled_frames, ) +from .chain_look import LookHold from .chain_pick import sharpest_index from .h3_anchor import anchor_refusal from .h3_guides import guides_refusal @@ -74,6 +76,8 @@ chain_anchor_problem, chain_carry_pick_problem, chain_keystone_problem, + chain_look_hold_flags, + chain_look_hold_problem, guide_backoff_problems, guide_chain_problems, ) @@ -296,6 +300,10 @@ def run_chain(pipeline, chain_definition, arguments): audio = None # joined generated audio, (channels, samples) float32 audio_rate = None carry = None + # The corrected frames' look-hold state, and the last carry before any + # pre-compensation (what a later segment's opening is graded to) + hold = LookHold(config.look_hold) if config.look_hold else None + raw_carry = None # Segment 1's decoded frames, the latent anchor of every later segment # under `anchor: "segment_1"` (#1166) - the same frame objects the output # holds, so keeping them costs no copy on an unspilled chain @@ -336,7 +344,11 @@ def run_chain(pipeline, chain_definition, arguments): output = pipeline._run_once(segment_arguments) artifact = _single_artifact(output) + if hold is not None: + artifact = _look_held(artifact, hold, raw_carry, segment.index == 0) carry = continuity.extract(artifact) + if hold is not None and config.continuity == "last_frame": + raw_carry, carry = carry, hold.feed(carry) segment_frames = frames_as_pil_list(artifact) if config.anchor == ANCHOR_SEGMENT_1 and segment.index == 0: anchor_frames = segment_frames @@ -427,6 +439,19 @@ def run_chain(pipeline, chain_definition, arguments): return _stitched(config, spill, frames, audio, audio_rate, shots) +def _look_held(artifact, hold, raw_carry, first): + """The segment with its frames look-corrected (#1191), audio untouched - + the corrected frames are what the carry is taken from and what the + stitched video is made of.""" + frames = hold.correct(frames_as_pil_list(artifact), raw_carry, first) + if isinstance(artifact, AudioVideo): + # A shallow copy with the new frames: audio, rate and shots as they were + held = copy.copy(artifact) + held.frames = frames + return held + return frames + + def _stitched(config, spill, frames, audio, audio_rate, shots): """The chain's one AudioVideo: the frames (replayed from disk when spilled), and either the joined generated audio, the original match_audio @@ -614,6 +639,12 @@ def __init__( pick = chain_definition.get("carry_pick") self.carry_pick = int(pick["sharpest"]) if pick else 0 + # Pixel corrections on each segment before its carry (#1191) + problem = chain_look_hold_problem(chain_definition) + if problem: + raise ValueError(f"Chain {problem}") + self.look_hold = chain_look_hold_flags(chain_definition) + self.segment_argument = chain_definition.get("segment_argument", "image") self.carry_frames = chain_definition.get("carry_frames", None) if self.carry_frames is not None: diff --git a/dw/pipeline_processors/chain_look.py b/dw/pipeline_processors/chain_look.py new file mode 100644 index 000000000..00405ae2a --- /dev/null +++ b/dw/pipeline_processors/chain_look.py @@ -0,0 +1,301 @@ +"""Pixel-domain look holding for a long chain (#1191, #1033, #1067). + +Each finished segment's frames can be corrected before its carry is taken, so a +long chain keeps the look of its first segment: a grade at the pickup, an +ease-out of saturation/contrast creep, an ease-out of faint texture, and a +keyframe pre-compensated for the boost a model measurably adds. Arrays are +(n, h, w, 3) uint8 in and out; the work is float32 inside, clipped on the way +out. Numpy and torch only. + +The ideas come from Smite79/MiniMax-H3-LongVideos (a no-redistribution +licence; concepts only, no code taken). +""" + +import numpy +import torch +from PIL import Image + +POOL = 64 # tone and look are read off a frame pooled to POOL x POOL by area +QUANTILES = numpy.linspace(0.0, 1.0, 17) +FLAT_SPREAD = 0.02 # a frame whose luma spans less than this has no tone to match +CUT_JUMP = ( + 0.25 # per-channel mean quantile gap past which an opening is a cut, not drift +) +LUT_SIZE = 1024 +TEXTURE_AMPLITUDE = 0.12 # detail stronger than this (in luma) is structure +TEXTURE_GRADIENT = 0.03 # a blurred-luma step stronger than this is an edge +BOOST_MAX = 1.25 # precompensate clamps each gain to [1, BOOST_MAX] +_EPS = 1e-4 + + +def _float(frames): + return numpy.asarray(frames, dtype=numpy.float32) / 255.0 + + +def _uint8(pixels): + return numpy.clip(numpy.rint(pixels * 255.0), 0, 255).astype(numpy.uint8) + + +def _to_ycc(rgb): + r, g, b = rgb[..., 0], rgb[..., 1], rgb[..., 2] + y = 0.299 * r + 0.587 * g + 0.114 * b + return numpy.stack( + [y, 0.5 + (b - y) * 0.564, 0.5 + (r - y) * 0.713], axis=-1 + ).astype(numpy.float32) + + +def _from_ycc(ycc): + y, cb, cr = ycc[..., 0], ycc[..., 1] - 0.5, ycc[..., 2] - 0.5 + r = y + cr / 0.713 + b = y + cb / 0.564 + g = (y - 0.299 * r - 0.114 * b) / 0.587 + return numpy.stack([r, g, b], axis=-1).astype(numpy.float32) + + +def _pooled(frame): + """A frame (uint8 h, w, 3) pooled to POOL x POOL by area, as float YCbCr.""" + pixels = torch.from_numpy(_float(frame)).permute(2, 0, 1)[None] + pooled = torch.nn.functional.adaptive_avg_pool2d(pixels, POOL) + return _to_ycc(pooled[0].permute(1, 2, 0).numpy()) + + +def tone(frame): + """A frame's (3, 17) YCbCr quantiles - 0, 1/16, ..., 1 per channel.""" + ycc = _pooled(frame).reshape(-1, 3) + return numpy.quantile(ycc, QUANTILES, axis=0).T.astype(numpy.float32) + + +def look(frame): + """(luma spread between the 5th and 95th percentile, mean chroma deviation + from 0.5) of a frame.""" + ycc = _pooled(frame) + low, high = numpy.quantile(ycc[..., 0], [0.05, 0.95]) + chroma = float(numpy.abs(ycc[..., 1:] - 0.5).mean()) + return float(high - low), chroma + + +def _lut(source, target): + """A monotone map from `source` quantiles onto `target` quantiles, sampled + on LUT_SIZE points of [0, 1]; the ends shift by the end offsets.""" + grid = numpy.linspace(0.0, 1.0, LUT_SIZE, dtype=numpy.float32) + xs = numpy.concatenate([[0.0], source, [1.0]]) + fs = numpy.concatenate( + [ + [numpy.clip(target[0] - source[0], 0, 1)], + target, + [numpy.clip(1.0 + target[-1] - source[-1], 0, 1)], + ] + ) + xs = numpy.maximum.accumulate(xs) + out = numpy.interp(grid, xs, numpy.maximum.accumulate(fs)) + return numpy.maximum.accumulate(out).astype(numpy.float32) + + +def _apply_lut(channel, lut): + return numpy.interp(channel, numpy.linspace(0.0, 1.0, LUT_SIZE), lut) + + +def pickup_grade(frames, carried): + """Match frame 1 to the carried frame by quantile LUT, and carry the same + mapping to the last frame, interpolating the LUT per frame between. + + Frame 0 is the keyframe copy and stays as it is. Returned unchanged when + there are under two frames after it is set aside, when a frame is flat, or + when the opening differs from the carried frame by more than CUT_JUMP (a + cut, not drift). + """ + frames = numpy.asarray(frames) + count = len(frames) + if count < 2: + return frames + carried_tone = tone(numpy.asarray(carried)) + first_tone, last_tone = tone(frames[1]), tone(frames[-1]) + for each in (carried_tone, first_tone, last_tone): + if each[0, -1] - each[0, 0] < FLAT_SPREAD: + return frames + if float(numpy.abs(first_tone - carried_tone).mean(axis=1).max()) > CUT_JUMP: + return frames + + first_luts = [_lut(first_tone[c], carried_tone[c]) for c in range(3)] + last_luts = [ + _lut( + last_tone[c], + _apply_lut(last_tone[c], first_luts[c]), + ) + for c in range(3) + ] + out = frames.copy() + for index in range(1, count): + weight = (index - 1) / (count - 2) if count > 2 else 0.0 + ycc = _to_ycc(_float(frames[index])) + for c in range(3): + lut = (1 - weight) * first_luts[c] + weight * last_luts[c] + ycc[..., c] = _apply_lut(ycc[..., c], lut) + out[index] = _uint8(_from_ycc(ycc)) + return out + + +def ease_creep(frames, scene_look, tolerance=0.03): + """Scale contrast (about the frame mean) and chroma (about 0.5) down by a + factor running linearly from 1 at the first frame to the needed gain at the + last, when the last frame's look exceeds the scene's by more than the + tolerance. Gains are at most 1: a darker or paler shot is untouched.""" + frames = numpy.asarray(frames) + count = len(frames) + if count < 2: + return frames + spread, chroma = look(frames[-1]) + scene_spread, scene_chroma = scene_look + contrast_gain = chroma_gain = 1.0 + if spread - scene_spread > tolerance and spread > _EPS: + contrast_gain = min(1.0, scene_spread / spread) + if chroma - scene_chroma > tolerance and chroma > _EPS: + chroma_gain = min(1.0, scene_chroma / chroma) + if contrast_gain == 1.0 and chroma_gain == 1.0: + return frames + out = frames.copy() + for index in range(count): + ramp = index / (count - 1) + out[index] = _scaled( + frames[index], + 1.0 + (contrast_gain - 1.0) * ramp, + 1.0 + (chroma_gain - 1.0) * ramp, + ) + return out + + +def _scaled(frame, contrast, chroma): + """Contrast about the frame's mean luma and chroma about 0.5, as factors.""" + ycc = _to_ycc(_float(frame)) + mean = ycc[..., 0].mean() + ycc[..., 0] = mean + (ycc[..., 0] - mean) * contrast + ycc[..., 1:] = 0.5 + (ycc[..., 1:] - 0.5) * chroma + return _uint8(_from_ycc(ycc)) + + +def _detail(luma): + """(luma - its box blur, mask of faint detail in flat areas). The blur + radius is about height / 64; edges and strong detail fall outside the + mask.""" + radius = max(1, luma.shape[0] // 64) + tensor = torch.from_numpy(luma)[None, None] + padded = torch.nn.functional.pad(tensor, (radius,) * 4, mode="replicate") + blurred = torch.nn.functional.avg_pool2d(padded, 2 * radius + 1, stride=1)[0, 0] + blurred = blurred.numpy() + gy, gx = numpy.gradient(blurred) + flat = numpy.hypot(gx, gy) < TEXTURE_GRADIENT + detail = luma - blurred + return detail, flat & (numpy.abs(detail) < TEXTURE_AMPLITUDE) + + +def texture(frame): + """The mean of the faint detail |luma - blur(luma)| over the flat, faint + area of a frame; 0 when there is none.""" + detail, mask = _detail(_to_ycc(_float(frame))[..., 0]) + return float(numpy.abs(detail)[mask].mean()) if mask.any() else 0.0 + + +def scene_texture(frames): + """The roughest of a segment's first, middle and last frames.""" + frames = numpy.asarray(frames) + picks = {0, len(frames) // 2, len(frames) - 1} + return max(texture(frames[i]) for i in picks) + + +def _eased(frame, cut): + ycc = _to_ycc(_float(frame)) + detail, mask = _detail(ycc[..., 0]) + ycc[..., 0] -= cut * mask * detail + return _uint8(_from_ycc(ycc)) + + +def _cut_for(frame, limit, max_cut): + """The smallest cut in [0, max_cut] that brings the frame's texture to + `limit`, by bisection; max_cut when even that falls short.""" + if texture(frame) <= limit: + return 0.0 + if texture(_eased(frame, max_cut)) > limit: + return max_cut + low, high = 0.0, max_cut + for _ in range(10): + middle = (low + high) / 2 + if texture(_eased(frame, middle)) > limit: + low = middle + else: + high = middle + return high + + +def ease_texture(frames, scene_texture, tolerance=0.03, max_cut=0.5): + """Subtract a fraction (at most max_cut) of the masked faint detail at + whichever end exceeds the scene's texture by more than the tolerance, + interpolating the fraction linearly across the shot.""" + frames = numpy.asarray(frames) + count = len(frames) + if count < 2: + return frames + limit = scene_texture + tolerance + ends = (_cut_for(frames[0], limit, max_cut), _cut_for(frames[-1], limit, max_cut)) + if not any(ends): + return frames + out = frames.copy() + for index in range(count): + cut = ends[0] + (ends[1] - ends[0]) * index / (count - 1) + if cut > 0: + out[index] = _eased(frames[index], cut) + return out + + +def precompensate(frame, boost): + """Divide contrast and chroma by the measured boost (contrast_gain, + chroma_gain), each clamped to [1, BOOST_MAX]; (1, 1) is the identity.""" + contrast, chroma = (min(max(float(g), 1.0), BOOST_MAX) for g in boost) + if contrast == 1.0 and chroma == 1.0: + return frame + return _scaled(numpy.asarray(frame), 1.0 / contrast, 1.0 / chroma) + + +class LookHold: + """What a chain keeps between segments: segment 1's look and texture, the + running boost, and the keyframe the next segment was fed.""" + + def __init__(self, flags): + self.flags = flags + self.scene_look = None + self.scene_texture = None + self._ratios = [] + self._fed = None + + def boost(self): + if not self._ratios: + return (1.0, 1.0) + return tuple(float(v) for v in numpy.mean(self._ratios, axis=0)) + + def correct(self, frames, carried, first): + """One segment's PIL frames, corrected. Segment 1 only sets the scene; + later ones also learn the boost from their rendered opening.""" + stack = numpy.stack([numpy.asarray(f.convert("RGB")) for f in frames]) + opening = stack[min(1, len(stack) - 1)] + if first: + self.scene_look, self.scene_texture = look(opening), scene_texture(stack) + return frames + if self._fed is not None: + rendered = look(opening) + if min(self._fed) > _EPS: + self._ratios.append([r / f for r, f in zip(rendered, self._fed)]) + if self.flags["pickup"] and carried is not None: + stack = pickup_grade(stack, numpy.asarray(carried.convert("RGB"))) + if self.flags["creep"]: + stack = ease_creep(stack, self.scene_look) + if self.flags["texture"]: + stack = ease_texture(stack, self.scene_texture) + return [Image.fromarray(each) for each in stack] + + def feed(self, carry): + """The keyframe for the next segment: the carry, pre-compensated by the + boost so far. Records its look to measure the next opening against.""" + if self.flags["precompensate"]: + pixels = numpy.asarray(carry.convert("RGB")) + carry = Image.fromarray(precompensate(pixels, self.boost())) + self._fed = look(numpy.asarray(carry.convert("RGB"))) + return carry diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index a3918a887..3c7b2b081 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -961,3 +961,64 @@ def media_problems(name, shape, width, height, num_frames=None): f"pad it to {on_grid}" ) return problems + + +# The look hold (#1191): pixel corrections applied to each finished segment +# before its carry is taken. Family-independent +LOOK_HOLD_PARTS = ("pickup", "creep", "texture", "precompensate") +# Parts that act on a single carried keyframe, so need `last_frame` +LOOK_HOLD_KEYFRAME_PARTS = ("pickup", "precompensate") + + +def chain_look_hold_flags(chain): + """The four look-hold switches as {part: bool}, or None when off. `true` + turns on all four; an object names the ones it wants.""" + hold = chain.get("look_hold") + if hold is True: + return dict.fromkeys(LOOK_HOLD_PARTS, True) + if isinstance(hold, dict) and any(hold.get(part) is True for part in hold): + return {part: hold.get(part) is True for part in LOOK_HOLD_PARTS} + return None + + +def chain_look_hold_problem(chain): + """The reason a chain block's `look_hold` is refused, or None: a value that + is neither a boolean nor an object of the four boolean switches, or an + explicit `pickup` / `precompensate` on a continuity other than `last_frame` + (a guide or last_segment carry is a whole clip, not one keyframe - `true` + alone quietly skips those two there). Unresolved references are left to the + run.""" + if "look_hold" not in chain: + return None + hold = chain["look_hold"] + if isinstance(hold, str) and references.is_ref(references.UNRESOLVED, hold): + return None + expected = ( + "expected true, false or an object of " + + ", ".join(LOOK_HOLD_PARTS) + + " booleans" + ) + if isinstance(hold, bool): + return None + if not isinstance(hold, dict): + return f"invalid look_hold {hold!r} - {expected}" + unknown = sorted(set(hold) - set(LOOK_HOLD_PARTS)) + if unknown: + return f"look_hold has unknown key(s) {unknown} - {expected}" + for part, value in hold.items(): + if not isinstance(value, bool): + return f"look_hold {part} must be true or false, got {value!r}" + continuity = chain.get("continuity", "last_frame") + if isinstance(continuity, str) and references.is_ref( + references.UNRESOLVED, continuity + ): + return None + if continuity != "last_frame": + wrong = [p for p in LOOK_HOLD_KEYFRAME_PARTS if hold.get(p)] + if wrong: + return ( + f"look_hold {' and '.join(wrong)} need continuity 'last_frame' - " + f"the {continuity!r} carry is a whole clip, not one keyframe; " + "creep and texture work on any continuity" + ) + return None diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index 230ff75bb..96d06f559 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -372,6 +372,17 @@ "required": ["sharpest"], "additionalProperties": false }, + "look_hold": { + "description": "Hold a long chain's look to its first segment with pixel-domain corrections applied to each later segment's decoded frames before its carry is taken, so the carried keyframe and the stitched video are the corrected frames (any model family; segment 1 is the reference and is left as rendered). 'pickup' grades the segment's opening to the carried frame by quantile LUT and carries the mapping to its last frame; 'creep' eases a rise in contrast or saturation back to segment 1's by the last frame, never brightening or touching a darker or paler shot; 'texture' eases faint texture (not edges) back to segment 1's, by at most half; 'precompensate' divides the carried keyframe's contrast and saturation by the boost measured so far (the running mean of rendered opening over fed keyframe, clamped to 1..1.25). true turns all four on; an object picks. 'pickup' and 'precompensate' work on one carried keyframe, so need continuity 'last_frame' (true alone skips them on a guide or last_segment chain, an explicit true there is refused). Absent or false = off, and the chain is unchanged.", + "type": ["boolean", "object", "string"], + "properties": { + "pickup": {"type": "boolean"}, + "creep": {"type": "boolean"}, + "texture": {"type": "boolean"}, + "precompensate": {"type": "boolean"} + }, + "additionalProperties": false + }, "segment_argument": { "description": "The pipeline argument the carry-over lands in - e.g. 'image' for image-to-video pipelines, or 'references' for reference-conditioned modular pipelines, where the carry-over is appended as an image or video reference.", "type": "string", diff --git a/tests/test_chain.py b/tests/test_chain.py index 7bee2be9e..cb994c801 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -12,6 +12,7 @@ from types import SimpleNamespace from unittest.mock import patch +import numpy import pytest import torch from PIL import Image @@ -1448,3 +1449,77 @@ def test_validation_wires_the_rule(self): ] } assert any("carry_pick" in e["message"] for e in anchor_errors(definition)) + + +class TestLookHold: + """look_hold (#1191): off is today's chain; on, the carry is the corrected + frame and the stitched video is made of corrected frames.""" + + def output(self, arguments, index): + # each segment renders one shade brighter than the last, flat but not + # flat enough to skip the grade + ramp = [[(40 + 12 * x + 20 * index,) * 3 for x in range(16)]] * 16 + image = Image.new("RGB", (16, 16)) + image.putdata([p for row in ramp for p in row]) + return SimpleNamespace(frames=[[image.copy() for _ in range(4)]]) + + def chain(self, **extra): + return {"segments": 3, "trim_frames": 0, "fps": 4, **extra} + + def test_look_hold_off_is_todays_chain(self): + plain = FakePipeline(self.output) + off = FakePipeline(self.output) + a = run_chain(plain, self.chain(), {}) + b = run_chain(off, self.chain(look_hold=False), {}) + assert [numpy.asarray(f).tolist() for f in a.frames] == [ + numpy.asarray(f).tolist() for f in b.frames + ] + + def test_look_hold_pulls_a_later_segment_onto_the_carried_frame(self): + pipeline = FakePipeline(self.output) + out = run_chain(pipeline, self.chain(look_hold={"pickup": True}), {}) + first_last = numpy.asarray(out.frames[3]).astype(int) + carried = numpy.asarray(pipeline.calls[1]["image"]).astype(int) + # segment 2's opening (frame 1 of it) now sits on the carried tone + opening = numpy.asarray(out.frames[5]).astype(int) + assert abs(opening.mean() - carried.mean()) < abs( + numpy.asarray(self.output({}, 1).frames[0][1]).astype(int).mean() + - carried.mean() + ) + assert first_last.shape == opening.shape + + def test_segment_one_is_left_as_rendered(self): + out = run_chain(FakePipeline(self.output), self.chain(look_hold=True), {}) + raw = numpy.asarray(self.output({}, 0).frames[0][0]) + assert (numpy.asarray(out.frames[0]) == raw).all() + + def test_validation_refuses_bad_shapes_and_guide_keyframe_parts(self): + from dw.pipeline_processors.h3_rules import chain_look_hold_problem + + assert chain_look_hold_problem({}) is None + assert chain_look_hold_problem({"look_hold": True}) is None + assert chain_look_hold_problem({"look_hold": {"creep": True}}) is None + assert "unknown" in chain_look_hold_problem({"look_hold": {"gloss": True}}) + assert chain_look_hold_problem({"look_hold": "yes"}) + assert chain_look_hold_problem({"look_hold": {"creep": 1}}) + guide = {"continuity": "guide"} + assert chain_look_hold_problem({**guide, "look_hold": True}) is None + assert chain_look_hold_problem({**guide, "look_hold": {"pickup": True}}) + with pytest.raises(ValueError, match="look_hold"): + ChainConfig(self.chain(look_hold=3), {}) + + def test_validate_names_a_bad_look_hold(self): + from dw.guides import anchor_errors + + definition = { + "steps": [ + { + "name": "s", + "pipeline": { + "chain": {"segments": 2, "look_hold": "nope"}, + "arguments": {}, + }, + } + ] + } + assert any("look_hold" in e["message"] for e in anchor_errors(definition)) diff --git a/tests/test_chain_look.py b/tests/test_chain_look.py new file mode 100644 index 000000000..cd3f8f776 --- /dev/null +++ b/tests/test_chain_look.py @@ -0,0 +1,113 @@ +"""look_hold (#1191): pixel corrections on a chain segment's frames.""" + +import numpy + +from dw.pipeline_processors.chain_look import ( + ease_creep, + ease_texture, + look, + pickup_grade, + precompensate, + texture, + tone, +) + +H = W = 64 + + +def gradient(scale=1.0, shift=(0, 0, 0)): + ramp = numpy.linspace(40, 200, W, dtype=numpy.float32) + base = numpy.tile(ramp, (H, 1)) + frame = numpy.stack([base, base * 0.9 + 10, base * 0.8 + 20], axis=-1) + mean = frame.mean() + frame = mean + (frame - mean) * scale + numpy.array(shift, dtype=numpy.float32) + return numpy.clip(frame, 0, 255).astype(numpy.uint8) + + +def clip_of(frame, count=5): + return numpy.stack([frame] * count) + + +def tinted(frame, cb): + from dw.pipeline_processors.chain_look import _float, _from_ycc, _to_ycc, _uint8 + + ycc = _to_ycc(_float(frame)) + ycc[..., 1] += cb / 255.0 + return _uint8(_from_ycc(ycc)) + + +def test_pickup_grade_matches_the_opening_to_the_carried_frame(): + carried = gradient() + frames = clip_of(tinted(carried, 10)) + graded = pickup_grade(frames, carried) + assert numpy.abs(tone(graded[1]) - tone(carried)).max() * 255 <= 1.0 + assert numpy.abs(tone(graded[-1]) - tone(carried)).max() * 255 <= 1.0 + assert (graded[0] == frames[0]).all() # the keyframe copy is left alone + + +def test_pickup_grade_skips_a_cut_and_a_flat_frame(): + carried = gradient() + cut = clip_of((carried * 0.3).astype(numpy.uint8)) + assert (pickup_grade(cut, carried) == cut).all() + flat = clip_of(numpy.full((H, W, 3), 90, numpy.uint8)) + assert (pickup_grade(flat, carried) == flat).all() + + +def test_creep_brings_the_last_frame_to_the_scene_look_within_tolerance(): + scene = look(gradient()) + frames = numpy.stack([gradient(1.0), gradient(1.2), gradient(1.4)]) + assert look(frames[-1])[0] - scene[0] > 0.03 + eased = ease_creep(frames, scene) + assert look(eased[-1])[0] - scene[0] <= 0.03 + assert (eased[0] == frames[0]).all() + + +def test_creep_never_brightens_and_never_touches_a_darker_shot(): + scene = look(gradient()) + duller = clip_of(gradient(0.6)) + assert (ease_creep(duller, scene) == duller).all() + frames = numpy.stack([gradient(1.0), gradient(1.5)]) + eased = ease_creep(frames, scene) + assert look(eased[-1])[0] <= look(frames[-1])[0] + assert abs(float(eased[-1].mean()) - float(frames[-1].mean())) < 3 + + +def test_texture_ease_leaves_edges_alone(): + rng = numpy.random.default_rng(0) + field = numpy.full((H, W, 3), 128.0) + field[:, W // 2 :] = 200.0 # a sharp step edge + noisy = field.copy() + noisy[:, : W // 2 - 4] += rng.normal(0, 25, (H, W // 2 - 4, 1)) # faint noise + noisy[:, W // 2 + 4 :] += rng.normal(0, 25, (H, W // 2 - 4, 1)) + frame = numpy.clip(noisy, 0, 255).astype(numpy.uint8) + frames = clip_of(frame) + before = texture(frame) + assert before > 0.03 + eased = ease_texture(frames, 0.0) + after = texture(eased[-1]) + assert after < before + assert after >= before * 0.5 - 1e-3 # at most half + # the edge columns survive + assert ( + eased[-1][:, W // 2 - 2 : W // 2 + 2] == frame[:, W // 2 - 2 : W // 2 + 2] + ).all() + + +def test_texture_ease_leaves_a_smooth_shot_alone(): + frames = clip_of(gradient()) + assert (ease_texture(frames, texture(gradient())) == frames).all() + + +def test_precompensate_is_identity_at_boost_one(): + frame = gradient(1.3) + assert (precompensate(frame, (1.0, 1.0)) == frame).all() + assert (precompensate(frame, (0.8, 0.5)) == frame).all() # clamped up to 1 + + +def test_precompensate_divides_and_clamps(): + frame = gradient(1.3) + out = precompensate(frame, (1.1, 1.0)) + assert look(out)[0] < look(frame)[0] + big = precompensate(frame, (9.0, 9.0)) + capped = precompensate(frame, (1.25, 1.25)) + assert (big == capped).all() From 1f5b3e3673d79dd96293c619a540a2443889b101 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 18:57:21 -0600 Subject: [PATCH 021/405] fix(chain): #1191 - look_hold texture edges hold at any resolution; precompensate tested in a chain Co-Authored-By: Claude Opus 5.5 --- dw/pipeline_processors/chain_look.py | 38 ++++++++++------ tests/test_chain.py | 67 +++++++++++++++++++++++++--- tests/test_chain_look.py | 46 +++++++++++++++++++ 3 files changed, 131 insertions(+), 20 deletions(-) diff --git a/dw/pipeline_processors/chain_look.py b/dw/pipeline_processors/chain_look.py index 00405ae2a..649f2b757 100644 --- a/dw/pipeline_processors/chain_look.py +++ b/dw/pipeline_processors/chain_look.py @@ -23,7 +23,7 @@ ) LUT_SIZE = 1024 TEXTURE_AMPLITUDE = 0.12 # detail stronger than this (in luma) is structure -TEXTURE_GRADIENT = 0.03 # a blurred-luma step stronger than this is an edge +TEXTURE_GRADIENT = 0.03 # edge threshold on blurred luma, as at a 3 px box BOOST_MAX = 1.25 # precompensate clamps each gain to [1, BOOST_MAX] _EPS = 1e-4 @@ -95,14 +95,20 @@ def _apply_lut(channel, lut): return numpy.interp(channel, numpy.linspace(0.0, 1.0, LUT_SIZE), lut) +def is_cut(opening_tone, reference_tone): + """True when an opening's tone is past CUT_JUMP from the reference's.""" + gap = numpy.abs(opening_tone - reference_tone).mean(axis=1).max() + return float(gap) > CUT_JUMP + + def pickup_grade(frames, carried): """Match frame 1 to the carried frame by quantile LUT, and carry the same mapping to the last frame, interpolating the LUT per frame between. Frame 0 is the keyframe copy and stays as it is. Returned unchanged when - there are under two frames after it is set aside, when a frame is flat, or - when the opening differs from the carried frame by more than CUT_JUMP (a - cut, not drift). + there are under two frames (frame 0 plus one more), when a frame is flat, + or when the opening differs from the carried frame by more than CUT_JUMP + (a cut, not drift). """ frames = numpy.asarray(frames) count = len(frames) @@ -113,7 +119,7 @@ def pickup_grade(frames, carried): for each in (carried_tone, first_tone, last_tone): if each[0, -1] - each[0, 0] < FLAT_SPREAD: return frames - if float(numpy.abs(first_tone - carried_tone).mean(axis=1).max()) > CUT_JUMP: + if is_cut(first_tone, carried_tone): return frames first_luts = [_lut(first_tone[c], carried_tone[c]) for c in range(3)] @@ -176,14 +182,16 @@ def _scaled(frame, contrast, chroma): def _detail(luma): """(luma - its box blur, mask of faint detail in flat areas). The blur radius is about height / 64; edges and strong detail fall outside the - mask.""" + mask. A blurred step's gradient falls as 1 / (2 radius + 1), so the edge + test scales it back to the 3 px box TEXTURE_GRADIENT was set at: the mask + reads the same at any resolution.""" radius = max(1, luma.shape[0] // 64) tensor = torch.from_numpy(luma)[None, None] padded = torch.nn.functional.pad(tensor, (radius,) * 4, mode="replicate") blurred = torch.nn.functional.avg_pool2d(padded, 2 * radius + 1, stride=1)[0, 0] blurred = blurred.numpy() gy, gx = numpy.gradient(blurred) - flat = numpy.hypot(gx, gy) < TEXTURE_GRADIENT + flat = numpy.hypot(gx, gy) * (2 * radius + 1) < TEXTURE_GRADIENT * 3 detail = luma - blurred return detail, flat & (numpy.abs(detail) < TEXTURE_AMPLITUDE) @@ -229,18 +237,20 @@ def _cut_for(frame, limit, max_cut): def ease_texture(frames, scene_texture, tolerance=0.03, max_cut=0.5): """Subtract a fraction (at most max_cut) of the masked faint detail at whichever end exceeds the scene's texture by more than the tolerance, - interpolating the fraction linearly across the shot.""" + interpolating the fraction linearly across the shot. Frame 0 is the keyframe + copy and stays as it is; the ends are frame 1 and the last.""" frames = numpy.asarray(frames) count = len(frames) if count < 2: return frames limit = scene_texture + tolerance - ends = (_cut_for(frames[0], limit, max_cut), _cut_for(frames[-1], limit, max_cut)) + ends = (_cut_for(frames[1], limit, max_cut), _cut_for(frames[-1], limit, max_cut)) if not any(ends): return frames out = frames.copy() - for index in range(count): - cut = ends[0] + (ends[1] - ends[0]) * index / (count - 1) + for index in range(1, count): + weight = (index - 1) / (count - 2) if count > 2 else 0.0 + cut = ends[0] + (ends[1] - ends[0]) * weight if cut > 0: out[index] = _eased(frames[index], cut) return out @@ -265,6 +275,7 @@ def __init__(self, flags): self.scene_texture = None self._ratios = [] self._fed = None + self._fed_tone = None def boost(self): if not self._ratios: @@ -279,7 +290,7 @@ def correct(self, frames, carried, first): if first: self.scene_look, self.scene_texture = look(opening), scene_texture(stack) return frames - if self._fed is not None: + if self._fed is not None and not is_cut(tone(opening), self._fed_tone): rendered = look(opening) if min(self._fed) > _EPS: self._ratios.append([r / f for r, f in zip(rendered, self._fed)]) @@ -297,5 +308,6 @@ def feed(self, carry): if self.flags["precompensate"]: pixels = numpy.asarray(carry.convert("RGB")) carry = Image.fromarray(precompensate(pixels, self.boost())) - self._fed = look(numpy.asarray(carry.convert("RGB"))) + pixels = numpy.asarray(carry.convert("RGB")) + self._fed, self._fed_tone = look(pixels), tone(pixels) return carry diff --git a/tests/test_chain.py b/tests/test_chain.py index cb994c801..7a6234cc2 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -1467,13 +1467,66 @@ def chain(self, **extra): return {"segments": 3, "trim_frames": 0, "fps": 4, **extra} def test_look_hold_off_is_todays_chain(self): - plain = FakePipeline(self.output) - off = FakePipeline(self.output) - a = run_chain(plain, self.chain(), {}) - b = run_chain(off, self.chain(look_hold=False), {}) - assert [numpy.asarray(f).tolist() for f in a.frames] == [ - numpy.asarray(f).tolist() for f in b.frames - ] + off_flags = { + "pickup": False, + "creep": False, + "texture": False, + "precompensate": False, + } + runs = [] + for extra in ({}, {"look_hold": False}, {"look_hold": off_flags}): + pipeline = FakePipeline(self.audio_output) + out = run_chain(pipeline, self.chain(**extra), {}) + runs.append((pipeline, out)) + base_pipeline, base = runs[0] + for pipeline, out in runs[1:]: + assert [numpy.asarray(f).tolist() for f in out.frames] == [ + numpy.asarray(f).tolist() for f in base.frames + ] + assert torch.equal(torch.as_tensor(out.audio), torch.as_tensor(base.audio)) + fed = [numpy.asarray(c["image"]).tolist() for c in pipeline.calls[1:]] + assert fed == [ + numpy.asarray(c["image"]).tolist() for c in base_pipeline.calls[1:] + ] + + def audio_output(self, arguments, index): + frames = self.output(arguments, index).frames[0] + audio = torch.full((1, 2, 100), float(index + 1)) + return {"videos": [frames], "audio": audio, "sampling_rate": 100} + + def boosted_output(self, arguments, index): + """Every segment opens brighter and more contrasty than its keyframe.""" + fed = arguments.get("image") + base = numpy.asarray(fed).astype(float) if fed is not None else None + if base is None: + ramp = numpy.tile(numpy.linspace(90, 170, 16), (16, 1)) + base = numpy.stack([ramp, ramp * 0.8 + 10, ramp * 0.6 + 20], axis=-1) + mean = base.mean() + boosted = numpy.clip(mean + 8 + (base - mean) * 1.2, 0, 255).astype(numpy.uint8) + image = Image.fromarray(boosted) + frames = [image.copy() for _ in range(4)] + if fed is not None: + frames[0] = fed + return SimpleNamespace(frames=[frames]) + + def test_precompensate_feeds_a_flatter_keyframe_and_pickup_keeps_the_raw_carry( + self, + ): + from dw.pipeline_processors.chain_look import look + + pipeline = FakePipeline(self.boosted_output) + out = run_chain(pipeline, self.chain(segments=4, look_hold=True), {}) + fed = [c.get("image") for c in pipeline.calls] + # segment 2 is fed segment 1's carry untouched: no boost measured yet + assert (numpy.asarray(fed[1]) == numpy.asarray(out.frames[3])).all() + # segment 3 is fed a keyframe flatter than the raw carry it was cut from + raw_carry_2 = numpy.asarray(out.frames[7]) # segment 2's last frame + assert look(numpy.asarray(fed[2]))[0] < look(raw_carry_2)[0] + # pickup grades segment 3's opening toward that raw carry, not the fed one + opening = numpy.asarray(out.frames[9]) # segment 3, frame 1 + to_raw = abs(look(opening)[0] - look(raw_carry_2)[0]) + to_fed = abs(look(opening)[0] - look(numpy.asarray(fed[2]))[0]) + assert to_raw < to_fed def test_look_hold_pulls_a_later_segment_onto_the_carried_frame(self): pipeline = FakePipeline(self.output) diff --git a/tests/test_chain_look.py b/tests/test_chain_look.py index cd3f8f776..d84ad2dc2 100644 --- a/tests/test_chain_look.py +++ b/tests/test_chain_look.py @@ -1,8 +1,10 @@ """look_hold (#1191): pixel corrections on a chain segment's frames.""" import numpy +import pytest from dw.pipeline_processors.chain_look import ( + LookHold, ease_creep, ease_texture, look, @@ -93,6 +95,32 @@ def test_texture_ease_leaves_edges_alone(): ).all() +@pytest.mark.parametrize("height,width", [(512, 512), (1088, 1920)]) +def test_texture_ease_leaves_edges_alone_at_full_resolution(height, width): + rng = numpy.random.default_rng(0) + mid = width // 2 + field = numpy.full((height, width, 3), 128.0) + field[:, mid:] = 200.0 + noisy = field.copy() + noisy[:, : mid - 4] += rng.normal(0, 25, (height, mid - 4, 1)) + noisy[:, mid + 4 :] += rng.normal(0, 25, (height, width - mid - 4, 1)) + frame = numpy.clip(noisy, 0, 255).astype(numpy.uint8) + eased = ease_texture(clip_of(frame, 3), 0.0) + band = slice(mid - 4, mid + 4) + change = numpy.abs(eased[-1][:, band].astype(int) - frame[:, band].astype(int)) + assert change.max() <= 1 + assert texture(eased[-1]) < texture(frame) # the flat-field noise still drops + + +def test_texture_ease_leaves_the_keyframe_copy_alone(): + rng = numpy.random.default_rng(1) + frame = numpy.clip(128 + rng.normal(0, 25, (H, W, 3)), 0, 255).astype(numpy.uint8) + frames = clip_of(frame, 4) + eased = ease_texture(frames, 0.0) + assert (eased[0] == frames[0]).all() + assert (eased[-1] != frames[-1]).any() + + def test_texture_ease_leaves_a_smooth_shot_alone(): frames = clip_of(gradient()) assert (ease_texture(frames, texture(gradient())) == frames).all() @@ -111,3 +139,21 @@ def test_precompensate_divides_and_clamps(): big = precompensate(frame, (9.0, 9.0)) capped = precompensate(frame, (1.25, 1.25)) assert (big == capped).all() + + +def test_a_cut_opening_does_not_move_the_boost(): + from PIL import Image + + hold = LookHold( + dict.fromkeys(("pickup", "creep", "texture", "precompensate"), True) + ) + carry = Image.fromarray(gradient()) + hold.correct([carry] * 3, None, True) + hold.feed(carry) + cut = Image.fromarray((gradient() * 0.3).astype(numpy.uint8)) + hold.correct([carry, cut, cut], carry, False) + assert hold.boost() == (1.0, 1.0) + drift = Image.fromarray(gradient(1.2)) + hold.feed(carry) + hold.correct([carry, drift, drift], carry, False) + assert hold.boost()[0] > 1.1 From 0f075b35cce40af3e241479e20a5b1df07b91e44 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 19:03:20 -0600 Subject: [PATCH 022/405] feat(chain): duration_seconds - ask for a chain by its length Co-Authored-By: Claude Opus 5.5 --- docs/WORKFLOW_GUIDE.md | 18 +- dw/cost_scaling.py | 62 +++++- dw/pipeline_processors/chain.py | 112 +++++++++- dw/pipeline_processors/chain_spill.py | 4 +- dw/pipeline_processors/h3_rules.py | 19 +- dw/workflow_schema.json | 36 ++- tests/test_chain_duration.py | 205 ++++++++++++++++++ .../templates/ltx2/chained-segments.json | 6 +- .../templates/minimax/chained-segments.json | 5 +- 9 files changed, 434 insertions(+), 33 deletions(-) create mode 100644 tests/test_chain_duration.py diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 2dd36d2ec..f316306ef 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -1858,9 +1858,9 @@ head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 | `guide_frames` | 22 (default) or 39 - a whole-latent guide length; anything else is refused | | where | MiniMax-H3 `t2va` or `fl2va`; refused on `ref2va`, with `references`, or on a non-H3 step - guides stay off ref2va | | `carry_frames` | refused - the carry is always the last `guide_frames` frames | -| `guide_backoff` | 0 (default) or a multiple of 17: the carry is the `guide_frames` frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs `num_frames` above P + back-off. Refused off `guide` continuity and with `match_audio` | +| `guide_backoff` | 0 (default) or a multiple of 17: the carry is the `guide_frames` frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs `num_frames` above P + back-off. Refused off `guide` continuity and with `match_audio` or `duration_seconds` | | `keystone` | `none` (default) or `segment_1`: on `fl2va`, every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain. The guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Refused on `last_frame` (there the keyframe is the only carry), and any other value is refused naming the two | -| `carry_pick` | `{"sharpest": N}`, N from 1 to 8, with `last_frame` continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones; `match_audio` is refused (the plan covers the track up front and the pick drops frames per seam, so the video would fall short - use `segments`); and when `num_frames` is known N must be less than `num_frames` minus `trim_frames`, or the pick could empty a segment | +| `carry_pick` | `{"sharpest": N}`, N from 1 to 8, with `last_frame` continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones; `match_audio` and `duration_seconds` are refused (the plan covers the length up front and the pick drops frames per seam, so the video would fall short - use `segments`); and when `num_frames` is known N must be less than `num_frames` minus `trim_frames`, or the pick could empty a segment | | `look_hold` | `true` or `{"pickup", "creep", "texture", "precompensate"}` booleans, any model family: pixel corrections on each later segment's frames before its carry is taken (below). On a guide chain `pickup` and `precompensate` do not apply (the carried guide is a whole clip, left alone); an explicit `true` for either there is refused, and `true` alone runs `creep` and `texture` only | | `trim_frames` | ignored (the trim is P), with a note in the job log | | `crossfade_ms` | ignored when `carry_audio` is true: the guide held the audio, so the seam is joined with only a declick | @@ -2461,8 +2461,18 @@ joined into a single file: held track's slice is that segment's `hold_audio`, a reference's replaces the reference - and the final video is muxed with the **original, unsliced track** - so the soundtrack has no seams at all. Requires - `num_frames` (the per-segment length) and a frame rate. Exactly one of `segments` - or `match_audio` must be given. + `num_frames` (the per-segment length) and a frame rate. Exactly one of `segments`, + `match_audio` or `duration_seconds` must be given. +- `duration_seconds` — instead of a count, ask for the chain by its length: the plan + covers `round(duration_seconds * fps)` frames (the same planner `match_audio` uses, + so the last segment shrinks to what remains, snapped by `frame_snap`), and the + stitched video and its generated audio are both trimmed to exactly that length, shot + records included. Needs `num_frames` in the step's arguments and a frame rate (chain + `fps` or a `frame_rate` argument). Refused with `guide_backoff` and `carry_pick`, + which drop frames the up-front plan cannot count. A cost estimate scales by the + segment count it implies. In the chained templates it is a variable defaulting to + null (a null variable drops the key): set `duration_seconds` and set `segments` to + null; with both set the chain is refused. - `continuity` — how continuity carries across segments. `last_frame` (the default) extracts each segment's last frame and passes it to the next segment - single-frame conditioning, which carries pose and colour. `last_segment` carries the previous diff --git a/dw/cost_scaling.py b/dw/cost_scaling.py index 75badf167..fbce45975 100644 --- a/dw/cost_scaling.py +++ b/dw/cost_scaling.py @@ -19,6 +19,8 @@ import json import math +from . import references + # The basis an estimate quotes its minutes on UNKNOWN = "unknown" CATALOG = "catalog" @@ -216,6 +218,45 @@ def _list_frames_scaling(name, default_entries, effective_entries, list_entries) return factor, f"frames {_shown(now)} vs {_shown(was)} over {name}" +# The declared drivers that set how many segments a chain runs; the count +# itself is the chain's own to tell (`chain_segment_count`), so a figure +# measured per segment scales by it +CHAIN_LENGTH_DRIVERS = ("segments", "duration_seconds") + + +def _resolved(value, variables): + """`value` with a `variable:` reference read from `variables`.""" + name = references.ref_name(references.VARIABLE, value) + return variables.get(name) if name is not None else value + + +def _chain_blocks(node): + """Every (chain block, arguments) pair under `node`: a dict holding a + `chain` beside its `arguments`.""" + if isinstance(node, dict): + if isinstance(node.get("chain"), dict): + yield node["chain"], node.get("arguments") or {} + for value in node.values(): + yield from _chain_blocks(value) + elif isinstance(node, list): + for value in node: + yield from _chain_blocks(value) + + +def _chain_segments(definition, variables): + """The segments the workflow's chain runs at `variables`, or None when + there is no single chain or its count cannot be told.""" + from .pipeline_processors.chain import chain_segment_count + + blocks = list(_chain_blocks(definition.get("steps"))) + if len(blocks) != 1: + return None + chain, arguments = blocks[0] + chain = {key: _resolved(value, variables) for key, value in chain.items()} + arguments = {key: _resolved(value, variables) for key, value in arguments.items()} + return chain_segment_count(chain, arguments) + + def _driver_scaling(definition, expanded, list_entries): """How far the declared `cost_drivers` moved the run away from the defaults the curated `cost` was measured at, as (factor, axes, unscaled), @@ -235,8 +276,12 @@ def _driver_scaling(definition, expanded, list_entries): at_default = {} running = {} unscaled = [] + chain_drivers = [] for name in _declared_drivers(definition): default_value = defaults.get(name) + if name in CHAIN_LENGTH_DRIVERS: + chain_drivers.append(name) + continue if isinstance(default_value, list): frames = _list_frames_scaling( name, default_value, effective.get(name), list_entries @@ -259,7 +304,22 @@ def _driver_scaling(definition, expanded, list_entries): scaled = scale_curated(factor, at_default, running) if scaled is None: return None - return scaled[0], scaled[1] + axes, unscaled + factor, scaled_axes = scaled + scaled_axes = scaled_axes + axes + if chain_drivers: + was = _chain_segments(definition, defaults) + now = _chain_segments(definition, effective) + if was and now: + if was != now: + factor *= now / was + scaled_axes.append(f"segments {now} vs {was}") + elif any( + _driver_comparable(effective.get(name)) + != _driver_comparable(defaults.get(name)) + for name in chain_drivers + ): + unscaled.extend(chain_drivers) + return factor, scaled_axes, unscaled def _scaled(price, scaling): diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index 005f26279..04b3f392f 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -16,8 +16,10 @@ and trimmed off it again, so the next segment continues the motion itself. MiniMax-H3 t2va and fl2va only: guides stay off ref2va -Two ways to specify the length: +Three ways to specify the length: - segments: N - run the pipeline N times as configured +- duration_seconds: S - plan the segments that cover S seconds at the chain's + frame rate, then trim the picture and the generated audio to exactly S - match_audio: true - derive the total frame count from the step's `hold_audio` track, or else its one audio reference, slice that audio into frame-aligned per-segment chunks, and mux the final video with the original, @@ -472,7 +474,19 @@ def _stitched(config, spill, frames, audio, audio_rate, shots): shots=_trimmed_shots(shots, config.total_frames), ) - if audio is None: + if config.duration_seconds is not None: + # The plan overshoots by design; the picture and the generated audio + # are cut to the asked length together + if spill is None: + frames = frames[: config.total_frames] + if audio is not None: + audio = audio[ + :, : frames_to_samples(config.total_frames, config.fps, audio_rate) + ] + shots = _trimmed_shots( + shots, config.total_frames, audio.shape[1] if audio is not None else None + ) + elif audio is None: shots = without_samples(shots) return AudioVideo(frames, audio, audio_rate, fps=config.fps, shots=shots) @@ -572,19 +586,24 @@ def _check_guide_chain(pipeline, arguments, chain_definition, config): ) -def _trimmed_shots(shots, total_frames): - """A match_audio chain's shots, cut where its overshooting picture is. +def _trimmed_shots(shots, total_frames, total_samples=None): + """A length-planned chain's shots, cut where its overshooting picture is. - The soundtrack is the caller's own, laid under whole rather than built - segment by segment, so no shot has a stretch of it to measure: the sample - side is cleared, not derived. + A match_audio soundtrack is the caller's own, laid under whole rather than + built segment by segment, so no shot has a stretch of it to measure: the + sample side is cleared, not derived. A duration_seconds chain's generated + audio is cut to `total_samples`, and each shot's stretch of it with it. """ trimmed = [] - for shot in without_samples(shots): + for shot in shots if total_samples is not None else without_samples(shots): if shot["start_frame"] >= total_frames: break end = min(shot["start_frame"] + shot["num_frames"], total_frames) - trimmed.append({**shot, "num_frames": end - shot["start_frame"]}) + cut = {**shot, "num_frames": end - shot["start_frame"]} + if total_samples is not None: + start = min(shot["start_sample"], total_samples) + cut["num_samples"] = min(shot["num_samples"], total_samples - start) + trimmed.append(cut) return trimmed @@ -596,8 +615,12 @@ def __init__( ): segments = chain_definition.get("segments", None) match_audio = bool(chain_definition.get("match_audio", False)) - if (segments is not None) == match_audio: - raise ValueError("A chain needs exactly one of 'segments' or 'match_audio'") + duration = chain_definition.get("duration_seconds", None) + if (segments is not None) + match_audio + (duration is not None) != 1: + raise ValueError( + "A chain needs exactly one of 'segments', 'match_audio' or " + "'duration_seconds'" + ) self.continuity = chain_definition.get("continuity", "last_frame") if self.continuity not in CONTINUITY_MODES: @@ -694,9 +717,16 @@ def __init__( self.audio_reference = None self.holds_audio = False self.total_frames = None + self.duration_seconds = float(duration) if duration is not None else None if match_audio: self._plan_from_audio(arguments, base_dir) + elif duration is not None: + self.total_frames, self.plan = _duration_plan(chain_definition, arguments) + logger.info( + f"Chaining to {float(duration):g}s: {self.total_frames} frames " + f"across {len(self.plan)} segments" + ) else: segments = int(segments) if not 1 <= segments <= MAX_SEGMENTS: @@ -771,6 +801,66 @@ def _plan_from_audio(self, arguments, base_dir=None): ) +def _head_trim(chain_definition): + """Head frames each segment after the first drops: the guide a `guide` + chain laid in, else `trim_frames`.""" + if chain_definition.get("continuity", "last_frame") == GUIDE_CONTINUITY: + return int(chain_definition.get("guide_frames", GUIDE_CHAIN_DEFAULT)) + return int(chain_definition.get("trim_frames", 1)) + + +def _duration_plan(chain_definition, arguments): + """(total_frames, plan) for a `duration_seconds` chain - the same plan a + `match_audio` chain makes from a track, from the asked length instead.""" + duration = float(chain_definition["duration_seconds"]) + if not math.isfinite(duration) or duration <= 0: + raise ValueError( + f"Chain 'duration_seconds' must be greater than 0, got {duration:g}" + ) + fps = _resolve_fps(chain_definition, arguments) + if fps is None: + raise ValueError( + "A duration_seconds chain needs the frame rate - set 'fps' on the " + "chain or a 'frame_rate' pipeline argument" + ) + num_frames = arguments.get("num_frames", None) + if num_frames is None: + raise ValueError( + "A duration_seconds chain needs 'num_frames' in the step's arguments " + "as the per-segment length" + ) + total_frames = max(1, round(duration * fps)) + plan = plan_segments( + total_frames, + int(num_frames), + _head_trim(chain_definition), + chain_definition.get("frame_snap", None), + ) + return total_frames, plan + + +def chain_segment_count(chain_definition, arguments): + """How many segments a chain block runs, or None when it cannot be told + from the block and the step's arguments alone (`match_audio`, an + unresolved reference, a missing frame rate or `num_frames`, or both + `segments` and `duration_seconds` given). Pure: the run's `ChainConfig` + and the cost estimate both ask it.""" + segments = chain_definition.get("segments", None) + duration = chain_definition.get("duration_seconds", None) + if chain_definition.get("match_audio") or (segments is None) == (duration is None): + return None + try: + if segments is not None: + return int(segments) + frame_snap = chain_definition.get("frame_snap", None) + definition = {**chain_definition} + if not isinstance(frame_snap, dict): + definition.pop("frame_snap", None) + return len(_duration_plan(definition, arguments)[1]) + except (TypeError, ValueError): + return None + + def plan_segments(total_frames, segment_frames, trim_frames, frame_snap=None): """Plan the segments that cover a total frame count. diff --git a/dw/pipeline_processors/chain_spill.py b/dw/pipeline_processors/chain_spill.py index 141d23a2a..cd245bb4e 100644 --- a/dw/pipeline_processors/chain_spill.py +++ b/dw/pipeline_processors/chain_spill.py @@ -205,11 +205,11 @@ def _decode_segment(path): def spilled_frames(config, spill): """A spilled chain's frames, replayed lazily from its segment files. - match_audio overshoots by design - the tail trim happens as the lazy + match_audio and duration_seconds overshoot by design - the tail trim happens as the lazy frames replay, so the files themselves stay whole.""" return SegmentedFrames( spill.paths, - config.total_frames if config.source_audio is not None else None, + config.total_frames, config.keep_segments, spill.frame_count, ) diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index 3c7b2b081..e03982f8b 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -675,12 +675,12 @@ def chain_carry_pick_problem(chain, num_frames=None): f"carry_pick needs continuity 'last_frame' - the other modes carry " f"more than one frame; got {continuity!r}" ) - if chain.get("match_audio"): + if chain.get("match_audio") or chain.get("duration_seconds") is not None: return ( - "carry_pick does not combine with match_audio: the plan covers the " - "track up front and the pick drops up to " + "carry_pick does not combine with match_audio or duration_seconds: " + "the plan covers the length up front and the pick drops up to " f"{CARRY_PICK_MAX - 1} more frames per seam, leaving the video " - "shorter than the track - use 'segments'" + "shorter than asked - use 'segments'" ) trim = chain.get("trim_frames", 1) if ( @@ -779,7 +779,7 @@ def guide_chain_problems(chain): def guide_backoff_problems(chain): """[(key, message)] for a chain block's `guide_backoff` (#1189): a whole number of 17-frame blocks, on a guide-continuity chain, planned from a - segment count - a `match_audio` plan counts every segment's new frames from + segment count - a `match_audio` or `duration_seconds` plan counts every segment's new frames from its head trim alone. An unresolved reference is left to the run.""" if "guide_backoff" not in chain: return [] @@ -810,13 +810,14 @@ def guide_backoff_problems(chain): "'guide' or drop guide_backoff", ) ] - if chain.get("match_audio"): + if chain.get("match_audio") or chain.get("duration_seconds") is not None: return [ ( "guide_backoff", - "guide_backoff does not combine with match_audio: the plan " - "covers the track from each segment's head trim alone, and " - "backing off drops more frames per segment - use 'segments'", + "guide_backoff does not combine with match_audio or " + "duration_seconds: the plan covers the length from each " + "segment's head trim alone, and backing off drops more frames " + "per segment - use 'segments'", ) ] return [] diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index 96d06f559..0574c160e 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -299,7 +299,7 @@ ] }, "chain": { - "description": "Run this pipeline repeatedly, carrying visual continuity from each segment into the next, and stitch the segments into one long video. Specify the length with exactly one of 'segments' or 'match_audio'.", + "description": "Run this pipeline repeatedly, carrying visual continuity from each segment into the next, and stitch the segments into one long video. Specify the length with exactly one of 'segments', 'match_audio' or 'duration_seconds'.", "type": "object", "properties": { "segments": { @@ -310,6 +310,15 @@ ], "minimum": 1 }, + "duration_seconds": { + "description": "Ask for the chain by its length: the segment count and the last segment's frame snap are derived from this many seconds at the chain's frame rate, and the stitched video and its generated audio are trimmed to exactly this length. Needs 'num_frames' in the step's arguments and a frame rate ('fps' on the chain or a 'frame_rate' argument). Not with 'guide_backoff' or 'carry_pick', which drop frames the up-front plan cannot count. Given beside 'segments', exactly one of the two must resolve to a value: a variable left null counts as absent, so a template can offer both. Accepts a 'variable:' reference.", + "type": [ + "number", + "string", + "null" + ], + "exclusiveMinimum": 0 + }, "match_audio": { "description": "Derive the segment count from the duration of this step's hold_audio track, or else its one audio reference, slicing it into frame-aligned per-segment chunks (each segment's hold_audio, or its audio reference); the final video is muxed with the original, unsliced audio track.", "type": "boolean" @@ -341,7 +350,7 @@ "default": 22 }, "guide_backoff": { - "description": "'guide' only - frames, a multiple of 17, to back the carry off from the end of each segment: the next segment is guided by the frames that many before the end, and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Not with 'match_audio'. Accepts a 'variable:' reference.", + "description": "'guide' only - frames, a multiple of 17, to back the carry off from the end of each segment: the next segment is guided by the frames that many before the end, and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Not with 'match_audio' or 'duration_seconds'. Accepts a 'variable:' reference.", "type": [ "integer", "string" @@ -485,7 +494,7 @@ "default": false } }, - "oneOf": [ + "anyOf": [ { "required": [ "segments" @@ -495,8 +504,29 @@ "required": [ "match_audio" ] + }, + { + "required": [ + "duration_seconds" + ] } ], + "not": { + "anyOf": [ + { + "required": [ + "segments", + "match_audio" + ] + }, + { + "required": [ + "duration_seconds", + "match_audio" + ] + } + ] + }, "additionalProperties": false }, "pipeline": { diff --git a/tests/test_chain_duration.py b/tests/test_chain_duration.py new file mode 100644 index 000000000..c3b345bba --- /dev/null +++ b/tests/test_chain_duration.py @@ -0,0 +1,205 @@ +"""A chain asked for by its length in seconds (`duration_seconds`).""" + +import pytest + +from dw.cost_scaling import _driver_scaling +from dw.pipeline_processors.chain import ( + ChainConfig, + chain_segment_count, + run_chain, +) +from dw.task_domains import frames_to_samples + +from tests.test_chain import MINIMAX_SNAP, FakePipeline, modular_output + + +def chain(**overrides): + return {"duration_seconds": 4.5, "fps": 4, "trim_frames": 2} | overrides + + +def test_duration_plans_whole_segments_and_trims_to_the_asked_length(): + # 40 s at 24 fps with 124-frame guide segments + definition = { + "duration_seconds": 40, + "fps": 24, + "continuity": "guide", + "guide_frames": 22, + "frame_snap": MINIMAX_SNAP, + } + config = ChainConfig(definition, {"num_frames": 124}) + assert config.total_frames == 960 + produced = sum(s.num_frames - s.head_trim for s in config.plan) + assert produced >= 960 + + # And the run: 4.5 s at 4 fps is 18 frames, from 8 + 6 + 6 generated + pipeline = FakePipeline(modular_output) + result = run_chain(pipeline, chain(), {"num_frames": 8}) + assert len(pipeline.calls) == 3 + assert len(result.frames) == 18 + assert result.audio.shape[1] == frames_to_samples(18, 4, 100) == 450 + assert sum(s["num_frames"] for s in result.shots) == 18 + + +def test_exactly_one_of_segments_match_audio_duration(): + with pytest.raises(ValueError, match="exactly one of"): + ChainConfig({"fps": 4}, {"num_frames": 8}) + with pytest.raises(ValueError, match="exactly one of"): + ChainConfig(chain(segments=2), {"num_frames": 8}) + with pytest.raises(ValueError, match="exactly one of"): + ChainConfig(chain(match_audio=True), {"num_frames": 8}) + # A null duration is absent + assert len(ChainConfig(chain(duration_seconds=None, segments=2), {}).plan) == 2 + + +def test_duration_needs_a_frame_rate_and_num_frames_and_a_positive_length(): + with pytest.raises(ValueError, match="frame rate"): + ChainConfig({"duration_seconds": 4}, {"num_frames": 8}) + with pytest.raises(ValueError, match="num_frames"): + ChainConfig(chain(), {}) + with pytest.raises(ValueError, match="greater than 0"): + ChainConfig(chain(duration_seconds=0), {"num_frames": 8}) + + +def test_the_frame_rate_may_come_from_the_pipeline_argument(): + config = ChainConfig({"duration_seconds": 4.5}, {"num_frames": 8, "frame_rate": 4}) + assert config.total_frames == 18 + + +def test_duration_with_frame_snap_snaps_the_last_segment_up(): + definition = {"duration_seconds": 10, "fps": 24, "trim_frames": 2} + definition["frame_snap"] = MINIMAX_SNAP + config = ChainConfig(definition, {"num_frames": 124}) + # 240 frames: 124, then 122 more would need 120 + 2 = 122 -> snapped to 124 + assert [s.num_frames for s in config.plan] == [124, 124] + assert all((s.num_frames - 5) % 17 == 0 for s in config.plan) + + +@pytest.mark.parametrize( + "extra, match", + [ + ({"continuity": "guide", "guide_backoff": 17}, "match_audio"), + ({"carry_pick": {"sharpest": 3}}, "duration_seconds"), + ], +) +def test_duration_refuses_the_trims_an_up_front_plan_cannot_count(extra, match): + with pytest.raises(ValueError, match=match): + ChainConfig(chain(**extra), {"num_frames": 124}) + + +def test_chain_segment_count_is_pure_and_shared(): + assert chain_segment_count({"segments": 5}, {}) == 5 + assert chain_segment_count(chain(), {"num_frames": 8}) == 3 + assert chain_segment_count(chain(), {}) is None + assert chain_segment_count({"match_audio": True}, {}) is None + assert chain_segment_count({"segments": "variable:x"}, {}) is None + assert chain_segment_count(chain(duration_seconds=None, segments=2), {}) == 2 + + +def _definition(segments, duration): + return { + "cost_drivers": ["segments", "duration_seconds"], + "variables": { + "segments": segments, + "duration_seconds": duration, + "num_frames": 8, + }, + "steps": [ + { + "pipeline": { + "chain": { + "segments": "variable:segments", + "duration_seconds": "variable:duration_seconds", + "fps": 4, + "trim_frames": 2, + }, + "arguments": {"num_frames": "variable:num_frames"}, + } + } + ], + } + + +def test_the_estimate_scales_by_the_segment_count_duration_implies(): + definition = _definition(3, None) + # 4.5 s -> 3 segments, the default: unchanged and nothing flagged + same = { + "variables": { + **definition["variables"], + "segments": None, + "duration_seconds": 4.5, + } + } + factor, axes, unscaled = _driver_scaling(definition, same, {}) + assert factor == pytest.approx(1.0) and unscaled == [] + # 12.5 s at 4 fps = 50 frames -> 8, then 6 each: 1 + ceil(42/6) = 8 segments + longer = { + "variables": { + **definition["variables"], + "segments": None, + "duration_seconds": 12.5, + } + } + factor, axes, unscaled = _driver_scaling(definition, longer, {}) + assert factor == pytest.approx(8 / 3) + assert unscaled == [] and any("segments 8 vs 3" in a for a in axes) + # And a plain segment count scales the same way + five = {"variables": {**definition["variables"], "segments": 6}} + factor, _, _ = _driver_scaling(definition, five, {}) + assert factor == pytest.approx(2.0) + + +TEMPLATES = ("minimax", "ltx2") + + +def _template(family): + import json + from pathlib import Path + + path = ( + Path(__file__).parent.parent + / f"workflows/templates/{family}/chained-segments.json" + ) + return json.loads(path.read_text()) + + +def _resolved(definition, **overrides): + from dw.variables import replace_variables + + variables = {**definition["variables"], **overrides} + step = replace_variables(definition["steps"][0], variables) + chain = step["pipeline"]["chain"] + if isinstance(chain.get("frame_snap"), str): + # The engine resolves 'constraint:' before the chain runs + chain["frame_snap"] = MINIMAX_SNAP + return chain, step["pipeline"]["arguments"] + + +@pytest.mark.parametrize("family", TEMPLATES) +def test_a_template_with_duration_null_keeps_its_segments(family): + chain, arguments = _resolved(_template(family)) + assert "duration_seconds" not in chain + assert len(ChainConfig(chain, arguments).plan) == 3 + + +@pytest.mark.parametrize("family", TEMPLATES) +def test_a_template_asked_for_seconds_plans_by_length(family): + chain, arguments = _resolved(_template(family), segments=None, duration_seconds=20) + config = ChainConfig(chain, arguments) + assert config.total_frames == 480 + assert len(config.plan) == chain_segment_count(chain, arguments) >= 4 + + +@pytest.mark.parametrize("family", TEMPLATES) +def test_a_template_estimate_scales_by_the_implied_segment_count(family): + definition = _template(family) + ask = { + "variables": { + **definition["variables"], + "segments": None, + "duration_seconds": 20, + } + } + factor, axes, _ = _driver_scaling(definition, ask, {}) + chain, arguments = _resolved(definition, segments=None, duration_seconds=20) + assert factor == pytest.approx(chain_segment_count(chain, arguments) / 3) + assert any(axis.startswith("segments") for axis in axes) diff --git a/workflows/templates/ltx2/chained-segments.json b/workflows/templates/ltx2/chained-segments.json index 1fffb725c..5ff7a14ae 100644 --- a/workflows/templates/ltx2/chained-segments.json +++ b/workflows/templates/ltx2/chained-segments.json @@ -1,8 +1,8 @@ { "id": "LTX2I2VChained", - "description": "A long clip built from short ones. The chain re-runs the pipeline per segment, handing each the last frame of the one before, and stitches the frames and the generated audio back into a single video. `prompt` is used for every segment, so any line of dialogue in it is spoken again in each one; to give each segment its own action or line, pass `prompts`, a list with one prompt per segment (segment i uses prompts[min(i, len - 1)]).", + "description": "A long clip built from short ones. The chain re-runs the pipeline per segment, handing each the last frame of the one before, and stitches the frames and the generated audio back into a single video. `prompt` is used for every segment, so any line of dialogue in it is spoken again in each one; to give each segment its own action or line, pass `prompts`, a list with one prompt per segment (segment i uses prompts[min(i, len - 1)]). To ask for a length instead of a count, set `duration_seconds` and set `segments` to null: the plan derives the segments and the video and audio are trimmed to exactly that length.", "summary": "A long LTX-2.5 clip with audio, chained from short segments that each start on the previous one's last frame.", - "cost_drivers": ["segments", "num_frames", "width", "height"], + "cost_drivers": ["segments", "duration_seconds", "num_frames", "width", "height"], "variables": { "transformer_weights_dtype": "{uint4}", "text_encoder_weights_dtype": "{int8}", @@ -14,6 +14,7 @@ "frame_rate": 24.0, "num_frames": 121, "segments": 3, + "duration_seconds": null, "image": { "location": "https://pbs.twimg.com/media/HPxFgsIXIAAjtUu?format=jpg&name=medium" }, @@ -136,6 +137,7 @@ }, "chain": { "segments": "variable:segments", + "duration_seconds": "variable:duration_seconds", "prompts": "variable:prompts", "continuity": "last_frame", "segment_argument": "image", diff --git a/workflows/templates/minimax/chained-segments.json b/workflows/templates/minimax/chained-segments.json index eb480646e..92d53b030 100644 --- a/workflows/templates/minimax/chained-segments.json +++ b/workflows/templates/minimax/chained-segments.json @@ -1,6 +1,6 @@ { "id": "MiniMaxH3I2VChained", - "description": "Getting past the model's fixed clip length by running one pipeline repeatedly and stitching the output. The simplest form of 'chain': a fixed segment count with 'last_frame' continuity, where each segment's final frame becomes the next segment's keyframe. Also shows the bookkeeping a stitched video needs - 'frame_snap' to keep every segment a length the model accepts, 'trim_frames' to drop the duplicated keyframe at each seam, and a crossfade over the generated audio so the joins are not audible. `prompt` is used for every segment, so a spoken line in it repeats in each one; pass `prompts`, one prompt per segment, to give each its own. Set `continuity` to 'guide' to carry motion and voice rather than a still: each segment's last `guide_frames` frames (22 or 39) and their audio are laid into the next at frame 0 as an H3 guide, the next keyframe is the guide's first frame, and the guide is trimmed off again - 'trim_frames' and the crossfade then do not apply. Guides are t2va/fl2va only, so the ref2va chains have no 'guide' mode.", + "description": "Getting past the model's fixed clip length by running one pipeline repeatedly and stitching the output. The simplest form of 'chain': a fixed segment count with 'last_frame' continuity, where each segment's final frame becomes the next segment's keyframe. Also shows the bookkeeping a stitched video needs - 'frame_snap' to keep every segment a length the model accepts, 'trim_frames' to drop the duplicated keyframe at each seam, and a crossfade over the generated audio so the joins are not audible. `prompt` is used for every segment, so a spoken line in it repeats in each one; pass `prompts`, one prompt per segment, to give each its own. To ask for a length instead of a count, set `duration_seconds` and set `segments` to null: the plan derives the segments and the video and audio are trimmed to exactly that length (not with a guide back-off or carry_pick). Set `continuity` to 'guide' to carry motion and voice rather than a still: each segment's last `guide_frames` frames (22 or 39) and their audio are laid into the next at frame 0 as an H3 guide, the next keyframe is the guide's first frame, and the guide is trimmed off again - 'trim_frames' and the crossfade then do not apply. Guides are t2va/fl2va only, so the ref2va chains have no 'guide' mode.", "summary": "A long H3 clip from a supplied still, chained by last frame past the model's fixed clip length.", "cost": [ { @@ -23,6 +23,7 @@ }, "cost_drivers": [ "segments", + "duration_seconds", "num_frames", "num_inference_steps", "width", @@ -40,6 +41,7 @@ }, "num_frames": 124, "segments": 3, + "duration_seconds": null, "continuity": "last_frame", "guide_frames": 22, "width": 960, @@ -198,6 +200,7 @@ "chain": { "prompts": "variable:prompts", "segments": "variable:segments", + "duration_seconds": "variable:duration_seconds", "continuity": "variable:continuity", "guide_frames": "variable:guide_frames", "segment_argument": "image", From 315bfaf6d0c0ea20b9c0a24ca49637683818e2d6 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 19:06:57 -0600 Subject: [PATCH 023/405] fix(chain): duration_seconds overrides segments, and says so Co-Authored-By: Claude Opus 5.5 --- docs/WORKFLOW_GUIDE.md | 9 ++-- dw/pipeline_processors/chain.py | 18 +++++-- dw/pipeline_processors/chain_spill.py | 4 +- dw/workflow_schema.json | 4 +- tests/test_chain_duration.py | 47 ++++++++++++++++++- .../templates/ltx2/chained-segments.json | 2 +- .../templates/minimax/chained-segments.json | 2 +- 7 files changed, 69 insertions(+), 17 deletions(-) diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index f316306ef..be5401c35 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -2461,7 +2461,7 @@ joined into a single file: held track's slice is that segment's `hold_audio`, a reference's replaces the reference - and the final video is muxed with the **original, unsliced track** - so the soundtrack has no seams at all. Requires - `num_frames` (the per-segment length) and a frame rate. Exactly one of `segments`, + `num_frames` (the per-segment length) and a frame rate. One of `segments`, `match_audio` or `duration_seconds` must be given. - `duration_seconds` — instead of a count, ask for the chain by its length: the plan covers `round(duration_seconds * fps)` frames (the same planner `match_audio` uses, @@ -2470,9 +2470,10 @@ joined into a single file: records included. Needs `num_frames` in the step's arguments and a frame rate (chain `fps` or a `frame_rate` argument). Refused with `guide_backoff` and `carry_pick`, which drop frames the up-front plan cannot count. A cost estimate scales by the - segment count it implies. In the chained templates it is a variable defaulting to - null (a null variable drops the key): set `duration_seconds` and set `segments` to - null; with both set the chain is refused. + segment count it implies. It overrides `segments` when both resolve (the run logs + that `segments` was ignored), so the chained templates, whose `duration_seconds` + variable defaults to null (a null variable drops the key), take just + `duration_seconds=20`. It is refused with `match_audio`. - `continuity` — how continuity carries across segments. `last_frame` (the default) extracts each segment's last frame and passes it to the next segment - single-frame conditioning, which carries pose and colour. `last_segment` carries the previous diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index 04b3f392f..931565a79 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -616,11 +616,19 @@ def __init__( segments = chain_definition.get("segments", None) match_audio = bool(chain_definition.get("match_audio", False)) duration = chain_definition.get("duration_seconds", None) - if (segments is not None) + match_audio + (duration is not None) != 1: + # A resolved duration wins over a segment count: a template carries a + # default `segments`, and asking for seconds must not need it nulled + counted = segments is not None or duration is not None + if counted == match_audio: raise ValueError( "A chain needs exactly one of 'segments', 'match_audio' or " "'duration_seconds'" ) + if segments is not None and duration is not None: + emit_log( + f"Chain 'segments' ({segments}) ignored: 'duration_seconds' " + f"({float(duration):g}) sets the segment count" + ) self.continuity = chain_definition.get("continuity", "last_frame") if self.continuity not in CONTINUITY_MODES: @@ -842,15 +850,15 @@ def _duration_plan(chain_definition, arguments): def chain_segment_count(chain_definition, arguments): """How many segments a chain block runs, or None when it cannot be told from the block and the step's arguments alone (`match_audio`, an - unresolved reference, a missing frame rate or `num_frames`, or both - `segments` and `duration_seconds` given). Pure: the run's `ChainConfig` + unresolved reference, or a missing frame rate or `num_frames`). A + `duration_seconds` wins over `segments`. Pure: the run's `ChainConfig` and the cost estimate both ask it.""" segments = chain_definition.get("segments", None) duration = chain_definition.get("duration_seconds", None) - if chain_definition.get("match_audio") or (segments is None) == (duration is None): + if chain_definition.get("match_audio") or (segments is None and duration is None): return None try: - if segments is not None: + if duration is None: return int(segments) frame_snap = chain_definition.get("frame_snap", None) definition = {**chain_definition} diff --git a/dw/pipeline_processors/chain_spill.py b/dw/pipeline_processors/chain_spill.py index cd245bb4e..a3e7d215c 100644 --- a/dw/pipeline_processors/chain_spill.py +++ b/dw/pipeline_processors/chain_spill.py @@ -205,8 +205,8 @@ def _decode_segment(path): def spilled_frames(config, spill): """A spilled chain's frames, replayed lazily from its segment files. - match_audio and duration_seconds overshoot by design - the tail trim happens as the lazy - frames replay, so the files themselves stay whole.""" + match_audio and duration_seconds overshoot by design - the tail trim happens + as the lazy frames replay, so the files themselves stay whole.""" return SegmentedFrames( spill.paths, config.total_frames, diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index 0574c160e..de0dc6272 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -299,7 +299,7 @@ ] }, "chain": { - "description": "Run this pipeline repeatedly, carrying visual continuity from each segment into the next, and stitch the segments into one long video. Specify the length with exactly one of 'segments', 'match_audio' or 'duration_seconds'.", + "description": "Run this pipeline repeatedly, carrying visual continuity from each segment into the next, and stitch the segments into one long video. Specify the length with one of 'segments', 'match_audio' or 'duration_seconds' ('duration_seconds' overrides 'segments' when both are given).", "type": "object", "properties": { "segments": { @@ -311,7 +311,7 @@ "minimum": 1 }, "duration_seconds": { - "description": "Ask for the chain by its length: the segment count and the last segment's frame snap are derived from this many seconds at the chain's frame rate, and the stitched video and its generated audio are trimmed to exactly this length. Needs 'num_frames' in the step's arguments and a frame rate ('fps' on the chain or a 'frame_rate' argument). Not with 'guide_backoff' or 'carry_pick', which drop frames the up-front plan cannot count. Given beside 'segments', exactly one of the two must resolve to a value: a variable left null counts as absent, so a template can offer both. Accepts a 'variable:' reference.", + "description": "Ask for the chain by its length: the segment count and the last segment's frame snap are derived from this many seconds at the chain's frame rate, and the stitched video and its generated audio are trimmed to exactly this length. Needs 'num_frames' in the step's arguments and a frame rate ('fps' on the chain or a 'frame_rate' argument). Not with 'guide_backoff' or 'carry_pick', which drop frames the up-front plan cannot count. Given beside 'segments', the duration wins and 'segments' is ignored (the run logs it), so a template can offer both with a default segment count. Not with 'match_audio'. Accepts a 'variable:' reference.", "type": [ "number", "string", diff --git a/tests/test_chain_duration.py b/tests/test_chain_duration.py index c3b345bba..151af4083 100644 --- a/tests/test_chain_duration.py +++ b/tests/test_chain_duration.py @@ -43,10 +43,12 @@ def test_duration_plans_whole_segments_and_trims_to_the_asked_length(): def test_exactly_one_of_segments_match_audio_duration(): with pytest.raises(ValueError, match="exactly one of"): ChainConfig({"fps": 4}, {"num_frames": 8}) - with pytest.raises(ValueError, match="exactly one of"): - ChainConfig(chain(segments=2), {"num_frames": 8}) + # The duration wins over a segment count (R18); the run says so + assert len(ChainConfig(chain(segments=2), {"num_frames": 8}).plan) == 3 with pytest.raises(ValueError, match="exactly one of"): ChainConfig(chain(match_audio=True), {"num_frames": 8}) + with pytest.raises(ValueError, match="exactly one of"): + ChainConfig(chain(match_audio=True, segments=2), {"num_frames": 8}) # A null duration is absent assert len(ChainConfig(chain(duration_seconds=None, segments=2), {}).plan) == 2 @@ -93,6 +95,8 @@ def test_chain_segment_count_is_pure_and_shared(): assert chain_segment_count({"match_audio": True}, {}) is None assert chain_segment_count({"segments": "variable:x"}, {}) is None assert chain_segment_count(chain(duration_seconds=None, segments=2), {}) == 2 + # The duration wins when both resolve + assert chain_segment_count(chain(segments=2), {"num_frames": 8}) == 3 def _definition(segments, duration): @@ -203,3 +207,42 @@ def test_a_template_estimate_scales_by_the_implied_segment_count(family): chain, arguments = _resolved(definition, segments=None, duration_seconds=20) assert factor == pytest.approx(chain_segment_count(chain, arguments) / 3) assert any(axis.startswith("segments") for axis in axes) + + +def test_duration_overrides_the_templates_default_segments(monkeypatch): + from dw.pipeline_processors import chain as chain_module + + logged = [] + monkeypatch.setattr( + chain_module, "emit_log", lambda message, **data: logged.append(message) + ) + definition = _template("minimax") + chain_block, arguments = _resolved(definition, duration_seconds=20) + # Only the duration was passed; segments is still the template's 3 + assert chain_block["segments"] == 3 + + config = ChainConfig(chain_block, arguments) + assert config.total_frames == 480 + assert len(config.plan) == chain_segment_count(chain_block, arguments) > 3 + produced = sum(s.num_frames - s.head_trim for s in config.plan) + assert produced >= 480 + assert any("segments" in line and "ignored" in line for line in logged) + + ask = {"variables": {**definition["variables"], "duration_seconds": 20}} + factor, axes, _ = _driver_scaling(definition, ask, {}) + assert factor == pytest.approx(len(config.plan) / 3) + assert any(axis.startswith("segments") for axis in axes) + + +def test_a_spilled_duration_chain_trims_frames_and_audio(tmp_path): + # A rate the segment files' AAC track can be written at + pipeline = FakePipeline( + lambda arguments, index: modular_output(arguments, index, sample_rate=8000), + output_dir=str(tmp_path), + file_prefix="wf-step", + ) + result = run_chain(pipeline, chain(save_segments=True), {"num_frames": 8}) + + assert sum(len(chunk) for chunk in result.frames) == 18 + assert result.audio.shape[1] == frames_to_samples(18, 4, 8000) == 36000 + result.frames.cleanup() diff --git a/workflows/templates/ltx2/chained-segments.json b/workflows/templates/ltx2/chained-segments.json index 5ff7a14ae..b3598d9d6 100644 --- a/workflows/templates/ltx2/chained-segments.json +++ b/workflows/templates/ltx2/chained-segments.json @@ -1,6 +1,6 @@ { "id": "LTX2I2VChained", - "description": "A long clip built from short ones. The chain re-runs the pipeline per segment, handing each the last frame of the one before, and stitches the frames and the generated audio back into a single video. `prompt` is used for every segment, so any line of dialogue in it is spoken again in each one; to give each segment its own action or line, pass `prompts`, a list with one prompt per segment (segment i uses prompts[min(i, len - 1)]). To ask for a length instead of a count, set `duration_seconds` and set `segments` to null: the plan derives the segments and the video and audio are trimmed to exactly that length.", + "description": "A long clip built from short ones. The chain re-runs the pipeline per segment, handing each the last frame of the one before, and stitches the frames and the generated audio back into a single video. `prompt` is used for every segment, so any line of dialogue in it is spoken again in each one; to give each segment its own action or line, pass `prompts`, a list with one prompt per segment (segment i uses prompts[min(i, len - 1)]). To ask for a length instead of a count, set `duration_seconds`; it overrides `segments` (the run logs that `segments` was ignored), and the plan derives the segments and the video and audio are trimmed to exactly that length.", "summary": "A long LTX-2.5 clip with audio, chained from short segments that each start on the previous one's last frame.", "cost_drivers": ["segments", "duration_seconds", "num_frames", "width", "height"], "variables": { diff --git a/workflows/templates/minimax/chained-segments.json b/workflows/templates/minimax/chained-segments.json index 92d53b030..42bbd46d3 100644 --- a/workflows/templates/minimax/chained-segments.json +++ b/workflows/templates/minimax/chained-segments.json @@ -1,6 +1,6 @@ { "id": "MiniMaxH3I2VChained", - "description": "Getting past the model's fixed clip length by running one pipeline repeatedly and stitching the output. The simplest form of 'chain': a fixed segment count with 'last_frame' continuity, where each segment's final frame becomes the next segment's keyframe. Also shows the bookkeeping a stitched video needs - 'frame_snap' to keep every segment a length the model accepts, 'trim_frames' to drop the duplicated keyframe at each seam, and a crossfade over the generated audio so the joins are not audible. `prompt` is used for every segment, so a spoken line in it repeats in each one; pass `prompts`, one prompt per segment, to give each its own. To ask for a length instead of a count, set `duration_seconds` and set `segments` to null: the plan derives the segments and the video and audio are trimmed to exactly that length (not with a guide back-off or carry_pick). Set `continuity` to 'guide' to carry motion and voice rather than a still: each segment's last `guide_frames` frames (22 or 39) and their audio are laid into the next at frame 0 as an H3 guide, the next keyframe is the guide's first frame, and the guide is trimmed off again - 'trim_frames' and the crossfade then do not apply. Guides are t2va/fl2va only, so the ref2va chains have no 'guide' mode.", + "description": "Getting past the model's fixed clip length by running one pipeline repeatedly and stitching the output. The simplest form of 'chain': a fixed segment count with 'last_frame' continuity, where each segment's final frame becomes the next segment's keyframe. Also shows the bookkeeping a stitched video needs - 'frame_snap' to keep every segment a length the model accepts, 'trim_frames' to drop the duplicated keyframe at each seam, and a crossfade over the generated audio so the joins are not audible. `prompt` is used for every segment, so a spoken line in it repeats in each one; pass `prompts`, one prompt per segment, to give each its own. To ask for a length instead of a count, set `duration_seconds`; it overrides `segments` (the run logs that `segments` was ignored), and the plan derives the segments and the video and audio are trimmed to exactly that length (not with a guide back-off or carry_pick). Set `continuity` to 'guide' to carry motion and voice rather than a still: each segment's last `guide_frames` frames (22 or 39) and their audio are laid into the next at frame 0 as an H3 guide, the next keyframe is the guide's first frame, and the guide is trimmed off again - 'trim_frames' and the crossfade then do not apply. Guides are t2va/fl2va only, so the ref2va chains have no 'guide' mode.", "summary": "A long H3 clip from a supplied still, chained by last frame past the model's fixed clip length.", "cost": [ { From 845ec3bb1fafe3b4499719e6ac775a1e92682deb Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 19:12:25 -0600 Subject: [PATCH 024/405] feat(workflow): presets - one variable selects a bundle of others Co-Authored-By: Claude Opus 5.5 --- docs/ARCHITECTURE.md | 1 + docs/WORKFLOW_GUIDE.md | 24 ++++++++ dw/for_each.py | 2 + dw/presets.py | 110 ++++++++++++++++++++++++++++++++++++ dw/schema.py | 1 + dw/validation.py | 6 ++ dw/workflow.py | 2 + dw/workflow_schema.json | 11 ++++ tests/test_plan.py | 1 + tests/test_presets.py | 119 +++++++++++++++++++++++++++++++++++++++ tests/test_validation.py | 1 + 11 files changed, 278 insertions(+) create mode 100644 dw/presets.py create mode 100644 tests/test_presets.py diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index f3b84b325..2b3308ad9 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -97,6 +97,7 @@ one to open. | Validation registry | `dw/validation.py`: `ERROR_CHECKS`, `WARNING_CHECKS`, `run_checks` | A check is one `Check` in a registry, and a check that raises becomes one `internal` finding while the rest still run. | `tests/test_validation.py::TestExceptionPolicy::test_a_raising_check_is_one_internal_error_and_the_rest_still_run` | | Step-value checks | `dw/step_value_checks.py`: `fps_errors`, `cache_errors`, `null_media_errors`, `select_errors`, `chain_prompts_errors` | The only caller of these checks is the error registry. | `dw/validation.py`: `ERROR_CHECKS` | | Admission | `dw/server/admission.py`: `admit` | Validate, submit, rerun and enhance load and check a request once, and `JobManager.submit` does not check it again. | `tests/test_admission.py::test_a_submit_expands_once` | +| Presets | `dw/presets.py`: `apply_presets`, `preset_errors` | A `presets` bundle sets the variables the caller did not pass, through `set_variables` right before the caller's own arguments fold, so an explicit argument wins and a bundle value is validated like a caller's. | `tests/test_presets.py::test_a_caller_value_beats_the_bundle` | | Variable constraints | `dw/variable_constraints.py`: `apply_constraints`; the grid arithmetic `aligned` / `aligned_down` | A model's rule about a value is declared in the workflow's `variable_constraints`, and it is checked at validation and again before anything loads. `plan_cuts` (and its static check) rounds render lengths through `aligned` / `aligned_down`, never its own copy, so a planned `num_frames` is one the template's constraint accepts; `crop_face_track` pads its crops to the step's `modulus`/`remainder` grid through `aligned` too (`padding_to_grid`). | `tests/test_variable_constraints.py::TestTheGrid::test_without_snap_an_off_grid_value_is_refused_not_rounded`, `tests/test_variable_constraints.py::TestTheGrid::test_plan_cuts_rounds_through_the_constraint_owner`, `tests/test_face_track.py::TestPadding::test_padding_to_grid_rounds_through_the_constraint_owner`, `tests/test_variable_constraints.py::TestTheRunTimePass::test_an_illegal_value_is_refused_before_anything_loads` | | Bound acknowledgement | `dw/plan.py`: `fingerprint`, `dw/server/admission.py`: `check_bound_acknowledgement` | A bound `acknowledged_cost` is refused with 409 when the plan's fingerprint or its required downloads changed. The fingerprint is keyed only by what can change the output: `DOCUMENTATION_KEYS` (`cost`, `cost_drivers`, `vram_estimate`, descriptions, catalog shape/traits), a constraint's `reason` and a result's role, subfolder and file name are left out, of the workflow and of each composed child; every schema key is sorted into left out or kept. A number digests by value, not spelling (`12.0` as `12`), and the seed stays out (#85). | `tests/test_plan.py::TestFingerprintKeys::test_the_exclusion_lists_are_pinned`, `tests/test_plan.py::TestTheCatalogH3Template::test_a_client_round_trip_keeps_it`, `tests/test_plan.py::TestFingerprintKeys::test_every_top_level_schema_key_is_sorted`, `tests/test_server.py::TestBoundRerun::test_a_rerun_bound_to_a_stale_plan_is_refused`, `tests/test_server.py::TestBoundAcknowledgement::test_a_matching_fingerprint_queues` | | Per-repo gating | `dw/plan_downloads.py`: `_model_info`, `_probe_gate_blocked`, assembled by `dw/plan.py`: `build_plan` | Each gated repo, including a `loras` entry's repo, is probed and reported on its own. | `tests/test_plan.py::TestDownloadsRequired::test_a_gated_repo_this_token_lacks_access_to_is_blocked` | diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index be5401c35..ca732b244 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -356,6 +356,30 @@ packages `dw/trust.py` allows (`diffusers`, `torch`, `transformers`, `sdnq`, class defined there, or to a `torch.dtype` under a `dtype` key. Anything else is refused at load. +### Presets + +`presets` lets one variable select a bundle of others - a template's quality +dial. The shape is `{selector: {value: {variable: value}}}`: + +```json +"variables": {"tier": "standard", "width": 1024, "num_inference_steps": 30}, +"presets": { + "tier": { + "draft": {"width": 512, "num_inference_steps": 4}, + "standard": {"width": 768, "num_inference_steps": 12} + } +} +``` + +The chosen bundle (the caller's value for the selector, else its default) sets +every variable the caller did not pass; an explicit argument always wins, so +`tier=draft num_inference_steps=30` runs 30 steps on the draft canvas. The +estimate, `variable_constraints` and substitution all see the bundle's values. +Validation refuses a bundle key that is no declared variable, a selector whose +default is no bundle name, and a caller value for the selector outside the +bundle names (at `arguments.`, naming the allowed ones). Bundle values +are validated like caller values. + ### What a variable is allowed to be A model's own rule about a value belongs in the workflow, not in engine code. diff --git a/dw/for_each.py b/dw/for_each.py index cbd942f4c..9299e6165 100644 --- a/dw/for_each.py +++ b/dw/for_each.py @@ -37,6 +37,7 @@ from .references import FROM_PREVIOUS_RESULT_KEY from .security import InvalidInputError, validate_variable_name from .step_cache import reference_resolves_to +from .presets import apply_presets from .variables import argument_errors, set_variables FOR_EACH_KEY = "for_each" @@ -450,6 +451,7 @@ def entry_field_warnings(definition, arguments=None): variables = copy.deepcopy(variables) supplied = set() if arguments and not argument_errors(definition, arguments): + apply_presets(definition, arguments, variables) set_variables(arguments, variables) supplied = set(arguments) warnings = [] diff --git a/dw/presets.py b/dw/presets.py new file mode 100644 index 000000000..ad0d5e865 --- /dev/null +++ b/dw/presets.py @@ -0,0 +1,110 @@ +"""`presets`: one variable selects a bundle of other variables' values. + +A template exposes one quality dial - `tier`, say, with `draft`, `standard` +and `best` - that moves canvas, weights and steps together, while every +variable stays individually settable. The shape is +`{"": {"": {"": , ...}, ...}}`. + +The bundle sets defaults and the caller's arguments land after it +(`apply_presets`, then `set_variables`), so an explicit argument always wins. +A bundle value goes through `set_variables` exactly as a caller's would - +name and string validation, type coercion - so a bundle is no way around +those checks. Both places a run's arguments are folded (`Workflow._fold`, +which the estimate and the realized copy read too, and the argument fold in +`dw/validation.py`) call `apply_presets` first. +""" + +from .variables import set_variables + +PRESETS_KEY = "presets" + + +def _presets(definition): + presets = definition.get(PRESETS_KEY) if isinstance(definition, dict) else None + return presets if isinstance(presets, dict) else {} + + +def _chosen(selector, arguments, variables): + """The bundle name a run selects: the caller's value, else the default.""" + if arguments and selector in arguments: + return arguments[selector] + return variables.get(selector) + + +def apply_presets(definition, arguments, variables): + """Set, on `variables`, the chosen bundle's values for every variable the + caller did not pass. A selector naming no bundle sets nothing here; + `preset_errors` is what refuses it.""" + arguments = arguments if isinstance(arguments, dict) else {} + if not isinstance(variables, dict): + return + for selector, bundles in _presets(definition).items(): + if not isinstance(bundles, dict): + continue + chosen = _chosen(selector, arguments, variables) + bundle = bundles.get(chosen) if isinstance(chosen, str) else None + if not isinstance(bundle, dict): + continue + values = { + name: value + for name, value in bundle.items() + if name not in arguments and name != selector + } + set_variables(values, variables) + + +def preset_errors(definition, arguments=None): + """What is wrong with the `presets` block, as [{path, message}]: a bundle + key that is no declared variable, a selector that is not a declared + variable or whose default is no bundle name, and a caller value for the + selector outside the bundle names (at `arguments.`).""" + presets = _presets(definition) + if not presets: + return [] + declared = definition.get("variables") + declared = declared if isinstance(declared, dict) else {} + names = ", ".join(sorted(declared)) or "" + errors = [] + for selector, bundles in presets.items(): + if not isinstance(bundles, dict): + continue + allowed = sorted(bundles) + listed = ", ".join(allowed) + if selector not in declared: + errors.append( + { + "path": f"presets.{selector}", + "message": f"The selector '{selector}' is not a declared " + f"variable; declared: {names}", + } + ) + elif declared[selector] not in bundles: + errors.append( + { + "path": f"variables.{selector}", + "message": f"The default of the preset selector " + f"'{selector}' must be one of its bundles: {listed}", + } + ) + if isinstance(arguments, dict) and selector in arguments: + if arguments[selector] not in bundles: + errors.append( + { + "path": f"arguments.{selector}", + "message": f"'{arguments[selector]}' is not a " + f"'{selector}' value; allowed: {listed}", + } + ) + for value, bundle in bundles.items(): + if not isinstance(bundle, dict): + continue + for name in bundle: + if name not in declared: + errors.append( + { + "path": f"presets.{selector}.{value}.{name}", + "message": f"'{name}' is not a declared " + f"variable; declared: {names}", + } + ) + return errors diff --git a/dw/schema.py b/dw/schema.py index bca09e909..9a85803f2 100644 --- a/dw/schema.py +++ b/dw/schema.py @@ -176,6 +176,7 @@ def load_schema(schema_name): "cost", "cost_drivers", "variables", + "presets", "variable_constraints", "seed", ], diff --git a/dw/validation.py b/dw/validation.py index ebeaa8f29..15678c1be 100644 --- a/dw/validation.py +++ b/dw/validation.py @@ -40,6 +40,7 @@ from .content_types import content_type_errors from .dissolve_frame_errors import dissolve_frame_errors from .for_each import ForEachError, entry_field_warnings +from .presets import apply_presets from .guides import ( anchor_errors, guide_chain_errors, @@ -79,6 +80,7 @@ component_type_errors, unbraced_type_warnings, ) +from .presets import preset_errors from .variable_constraints import ( ConstraintReferenceError, constraint_errors, @@ -533,6 +535,9 @@ def _task_errors(context): "constraints", lambda c: constraint_errors(c.definition, c.arguments, supplied=c.supplied), ), + # A `presets` block naming nothing declared, or a selector value that + # is no bundle (dw/presets.py) + Check("presets", lambda c: preset_errors(c.definition, c.arguments)), # An 'attn_processor_type' whose Hub kernel this machine has no build # variant for (dw/kernel_availability.py, #178) Check( @@ -898,6 +903,7 @@ def undeclared_variable_errors(workflow, arguments=None): supplied = set() if isinstance(variables, dict) and arguments: if not argument_errors(definition, arguments): + apply_presets(definition, arguments, variables) set_variables(arguments, variables) supplied = set(arguments) declared = sorted(variables or {}) diff --git a/dw/workflow.py b/dw/workflow.py index e23980d1b..bcbed8fc7 100644 --- a/dw/workflow.py +++ b/dw/workflow.py @@ -20,6 +20,7 @@ from .runs import activate_output_root, deactivate_output_root from .schema import format_validation_errors from .for_each import expand_for_each +from .presets import apply_presets from .variables import ( ConstantError, argument_errors, @@ -353,6 +354,7 @@ def _fold(self, definition, arguments, *, fold_arguments, constrain): if fold_arguments: # set variable values from the arguments passed to the workflow; # these may come from the command line or from a parent workflow + apply_presets(definition, arguments, variables) set_variables(arguments, variables) # an entry of a list-valued variable may name another variable; # resolve those before anything inside it is realized, so a diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index de0dc6272..c17d0ea93 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -65,6 +65,17 @@ "items": {"type": "string"}, "uniqueItems": true }, + "presets": { + "description": "One variable that selects a bundle of other variables' values - a template's quality dial. {selector: {value: {variable: value}}}: the selector is a declared variable whose default names one of its bundles, and every key of a bundle is a declared variable. The chosen bundle sets the variables the caller did not pass; an explicit argument always wins. Bundle values are validated exactly as caller values are.", + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": {"$ref": "#/$defs/arguments/additionalProperties"} + } + } + }, "variables": { "$ref": "#/$defs/arguments" }, diff --git a/tests/test_plan.py b/tests/test_plan.py index 5f78b8f4c..33d4dd38f 100644 --- a/tests/test_plan.py +++ b/tests/test_plan.py @@ -241,6 +241,7 @@ def test_a_step_added_removed_or_renamed(self, plan): "id", # names every file the run saves, and the step cache keys on it "variables", # the folded values are the work "variable_constraints", # a rule rounds or refuses a value + "presets", # a bundle sets the variables the caller did not pass "argument_template", # read as arguments when not run as a child "seed", # removed separately: a seed is not the work, but it is the output "steps", diff --git a/tests/test_presets.py b/tests/test_presets.py new file mode 100644 index 000000000..67da8f4ee --- /dev/null +++ b/tests/test_presets.py @@ -0,0 +1,119 @@ +"""`presets`: one variable selects a bundle of other variables' values. + +A template exposes one quality dial (`tier`) that moves canvas, steps and so +on, while every variable stays individually settable: a caller's explicit +argument always wins over the bundle. +""" + +import copy + +from dw.presets import apply_presets, preset_errors +from dw.workflow import workflow_from_definition + +PRESETS = { + "tier": { + "draft": {"width": 512, "num_inference_steps": 4}, + "standard": {"width": 768, "num_inference_steps": 12}, + } +} + + +def definition(presets=PRESETS, **overrides): + variables = { + "tier": "standard", + "width": 1024, + "num_inference_steps": 30, + "prompt": "a cat", + } + variables.update(overrides) + return {"id": "dial", "variables": variables, "presets": copy.deepcopy(presets)} + + +def test_the_bundle_sets_the_variables_the_caller_did_not_pass(): + d = definition() + variables = copy.deepcopy(d["variables"]) + + apply_presets(d, {"tier": "draft"}, variables) + + assert variables["width"] == 512 + assert variables["num_inference_steps"] == 4 + assert variables["prompt"] == "a cat" + + +def test_a_caller_value_beats_the_bundle(): + d = definition() + arguments = {"tier": "draft", "num_inference_steps": 30} + variables = copy.deepcopy(d["variables"]) + + apply_presets(d, arguments, variables) + from dw.variables import set_variables + + set_variables(arguments, variables) + + assert variables["num_inference_steps"] == 30 + assert variables["width"] == 512 + + +def test_the_default_bundle_applies_with_no_arguments(): + d = definition() + variables = copy.deepcopy(d["variables"]) + + apply_presets(d, {}, variables) + + assert variables["width"] == 768 + assert variables["num_inference_steps"] == 12 + + +def test_an_unknown_selector_value_is_refused_naming_the_values(): + (problem,) = preset_errors(definition(), {"tier": "ultra"}) + + assert problem["path"] == "arguments.tier" + assert "draft" in problem["message"] and "standard" in problem["message"] + + +def test_a_bundle_naming_an_undeclared_variable_is_a_validation_error(): + d = definition({"tier": {"standard": {"nope": 1}}}) + + (problem,) = preset_errors(d, None) + + assert problem["path"] == "presets.tier.standard.nope" + assert "declared" in problem["message"] + + +def test_a_selector_whose_default_is_not_a_bundle_is_an_error(): + problems = preset_errors(definition(tier="gold"), None) + + assert any(p["path"] == "variables.tier" for p in problems) + + +def test_a_workflow_without_presets_is_unchanged(): + d = definition() + del d["presets"] + variables = copy.deepcopy(d["variables"]) + + apply_presets(d, {"tier": "draft"}, variables) + + assert variables == d["variables"] + assert preset_errors(d, {"tier": "draft"}) == [] + + +def test_a_bundle_value_is_validated_like_a_caller_value(): + import pytest + + d = definition({"tier": {"standard": {"prompt": "bad\x00value"}}}) + variables = copy.deepcopy(d["variables"]) + + with pytest.raises(Exception): + apply_presets(d, {}, variables) + + +def test_the_workflow_folds_the_bundle_for_a_run_and_for_validation(tmp_path): + d = definition() + d["steps"] = [{"name": "a", "task": {"command": "no_op", "arguments": {}}}] + workflow = workflow_from_definition(d, str(tmp_path)) + + variables = workflow.folded_variables({"tier": "draft"}) + assert variables["width"] == 512 + + errors = workflow.validation_errors({"tier": "ultra"}) + assert any(e["path"] == "arguments.tier" for e in errors), errors diff --git a/tests/test_validation.py b/tests/test_validation.py index a223c15a0..b44afd13f 100644 --- a/tests/test_validation.py +++ b/tests/test_validation.py @@ -208,6 +208,7 @@ def test_validation_does_not_import_the_workflow_module(): "component_types", "component_names", "constraints", + "presets", "kernel_availability", "sub_workflows", ] From 420fef50bd77a6227f4d0b0fb67dff3be2e9f76d Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 19:26:02 -0600 Subject: [PATCH 025/405] fix(workflow): presets seen by validate, the estimate and every merge site The default bundle now folds with no or bad arguments, so {} and the selector's default share a fingerprint. effective_variables in dw/presets.py owns a written definition's variables for some arguments; the constraint checks, argument warnings, host-RAM key, VRAM inheritance, observed-cost buckets, estimate defaults and the jobs fallback read it. Every bundle is coerced and checked against variable_constraints at presets...; a non-string selector default is refused. Co-Authored-By: Claude Opus 5.5 --- docs/ARCHITECTURE.md | 2 +- docs/WORKFLOW_GUIDE.md | 29 ++++- dw/argument_warnings.py | 4 +- dw/cost_scaling.py | 9 +- dw/for_each.py | 7 +- dw/plan.py | 29 +++-- dw/presets.py | 141 +++++++++++++++++++-- dw/server/deps.py | 15 ++- dw/server/host_ram_need.py | 6 +- dw/server/observed_cost.py | 17 ++- dw/server/routes/jobs.py | 5 +- dw/validation.py | 9 +- dw/variable_constraints.py | 38 +++++- dw/vram_inheritance.py | 18 +-- dw/workflow.py | 21 +++- tests/test_presets.py | 248 ++++++++++++++++++++++++++++++++++++- tests/test_server.py | 18 +++ 17 files changed, 540 insertions(+), 76 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2b3308ad9..f7cf463fe 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -97,7 +97,7 @@ one to open. | Validation registry | `dw/validation.py`: `ERROR_CHECKS`, `WARNING_CHECKS`, `run_checks` | A check is one `Check` in a registry, and a check that raises becomes one `internal` finding while the rest still run. | `tests/test_validation.py::TestExceptionPolicy::test_a_raising_check_is_one_internal_error_and_the_rest_still_run` | | Step-value checks | `dw/step_value_checks.py`: `fps_errors`, `cache_errors`, `null_media_errors`, `select_errors`, `chain_prompts_errors` | The only caller of these checks is the error registry. | `dw/validation.py`: `ERROR_CHECKS` | | Admission | `dw/server/admission.py`: `admit` | Validate, submit, rerun and enhance load and check a request once, and `JobManager.submit` does not check it again. | `tests/test_admission.py::test_a_submit_expands_once` | -| Presets | `dw/presets.py`: `apply_presets`, `preset_errors` | A `presets` bundle sets the variables the caller did not pass, through `set_variables` right before the caller's own arguments fold, so an explicit argument wins and a bundle value is validated like a caller's. | `tests/test_presets.py::test_a_caller_value_beats_the_bundle` | +| Presets | `dw/presets.py`: `apply_presets`, `effective_variables`, `preset_errors` | A `presets` bundle sets the variables the caller did not pass, through `set_variables` right before the caller's own arguments fold, so an explicit argument wins and a bundle value is validated like a caller's. The fold applies the default bundle with no usable arguments. Every reader of a written definition's variables for some arguments (constraint checks, argument warnings, host-RAM key, VRAM inheritance, observed-cost buckets, the estimate's defaults) goes through `effective_variables`; a reader of an expanded definition reads its folded `variables`. | `tests/test_presets.py::test_a_caller_value_beats_the_bundle`, `tests/test_presets.py::test_no_arguments_and_the_default_selector_have_one_fingerprint`, `tests/test_presets.py::test_a_bundle_value_breaking_a_rule_is_an_error_at_the_bundle` | | Variable constraints | `dw/variable_constraints.py`: `apply_constraints`; the grid arithmetic `aligned` / `aligned_down` | A model's rule about a value is declared in the workflow's `variable_constraints`, and it is checked at validation and again before anything loads. `plan_cuts` (and its static check) rounds render lengths through `aligned` / `aligned_down`, never its own copy, so a planned `num_frames` is one the template's constraint accepts; `crop_face_track` pads its crops to the step's `modulus`/`remainder` grid through `aligned` too (`padding_to_grid`). | `tests/test_variable_constraints.py::TestTheGrid::test_without_snap_an_off_grid_value_is_refused_not_rounded`, `tests/test_variable_constraints.py::TestTheGrid::test_plan_cuts_rounds_through_the_constraint_owner`, `tests/test_face_track.py::TestPadding::test_padding_to_grid_rounds_through_the_constraint_owner`, `tests/test_variable_constraints.py::TestTheRunTimePass::test_an_illegal_value_is_refused_before_anything_loads` | | Bound acknowledgement | `dw/plan.py`: `fingerprint`, `dw/server/admission.py`: `check_bound_acknowledgement` | A bound `acknowledged_cost` is refused with 409 when the plan's fingerprint or its required downloads changed. The fingerprint is keyed only by what can change the output: `DOCUMENTATION_KEYS` (`cost`, `cost_drivers`, `vram_estimate`, descriptions, catalog shape/traits), a constraint's `reason` and a result's role, subfolder and file name are left out, of the workflow and of each composed child; every schema key is sorted into left out or kept. A number digests by value, not spelling (`12.0` as `12`), and the seed stays out (#85). | `tests/test_plan.py::TestFingerprintKeys::test_the_exclusion_lists_are_pinned`, `tests/test_plan.py::TestTheCatalogH3Template::test_a_client_round_trip_keeps_it`, `tests/test_plan.py::TestFingerprintKeys::test_every_top_level_schema_key_is_sorted`, `tests/test_server.py::TestBoundRerun::test_a_rerun_bound_to_a_stale_plan_is_refused`, `tests/test_server.py::TestBoundAcknowledgement::test_a_matching_fingerprint_queues` | | Per-repo gating | `dw/plan_downloads.py`: `_model_info`, `_probe_gate_blocked`, assembled by `dw/plan.py`: `build_plan` | Each gated repo, including a `loras` entry's repo, is probed and reported on its own. | `tests/test_plan.py::TestDownloadsRequired::test_a_gated_repo_this_token_lacks_access_to_is_blocked` | diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index ca732b244..c1a728cd7 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -373,12 +373,29 @@ dial. The shape is `{selector: {value: {variable: value}}}`: The chosen bundle (the caller's value for the selector, else its default) sets every variable the caller did not pass; an explicit argument always wins, so -`tier=draft num_inference_steps=30` runs 30 steps on the draft canvas. The -estimate, `variable_constraints` and substitution all see the bundle's values. -Validation refuses a bundle key that is no declared variable, a selector whose -default is no bundle name, and a caller value for the selector outside the -bundle names (at `arguments.`, naming the allowed ones). Bundle values -are validated like caller values. +`tier=draft num_inference_steps=30` runs 30 steps on the draft canvas. With no +arguments - or when a bad argument leaves them all unfolded - the default +bundle applies, so a run with no arguments and one passing the selector's +default are the same run, with the same fingerprint. + +Everything that judges or prices a run sees the bundle's values: validation, +substitution, the estimate (the curated `cost` counts as measured at the +default bundle), the fingerprint, the VRAM projection, the host-RAM +projection, the observed-cost buckets, `variable_constraints` and the +argument warnings. A bundle value is rounded by a +`snap: "up"` rule, with the same warning a caller's value draws. + +Validation checks every bundle, not only the chosen one, and reports each +problem where it was written: + +- a bundle key that is no declared variable, a bundle value that will not + coerce to its default's type or fails string validation, or one outside a + `variable_constraints` rule, at `presets...`; +- a selector that is not a declared variable, at `presets.`; +- a selector whose default is not a string (bundle names are JSON keys, so a + number can never match one) or names no bundle, at `variables.`; +- a caller value for the selector outside the bundle names, at + `arguments.`, naming the allowed ones. ### What a variable is allowed to be diff --git a/dw/argument_warnings.py b/dw/argument_warnings.py index 225f98ecf..9803a6224 100644 --- a/dw/argument_warnings.py +++ b/dw/argument_warnings.py @@ -8,6 +8,7 @@ from . import references from .introspection import CLASS_NAME_PATTERN, unknown_call_arguments +from .presets import effective_variables from .variables import undeclared_variable_references @@ -164,7 +165,8 @@ def workflow_argument_warnings(workflow_definition, arguments=None): same values `constraint_warnings` checks a constraint against. """ warnings = [] - values = {**(workflow_definition.get("variables") or {}), **(arguments or {})} + # the chosen preset bundle included, as the run folds it + values = effective_variables(workflow_definition, arguments) declared = sorted(workflow_definition.get("variables") or {}) for path, name in undeclared_variable_references(workflow_definition): hint = ( diff --git a/dw/cost_scaling.py b/dw/cost_scaling.py index fbce45975..641e684e0 100644 --- a/dw/cost_scaling.py +++ b/dw/cost_scaling.py @@ -20,6 +20,7 @@ import math from . import references +from .presets import effective_variables # The basis an estimate quotes its minutes on UNKNOWN = "unknown" @@ -269,7 +270,9 @@ def _driver_scaling(definition, expanded, list_entries): for every template: its name goes in `unscaled`, and the figure is kept but flagged (#866, Q4). """ - defaults = definition.get("variables") or {} + # the defaults the curated cost was measured at, the default preset + # bundle among them + defaults = effective_variables(definition) effective = expanded.get("variables") or {} factor = 1.0 axes = [] @@ -351,7 +354,9 @@ def _scaled(price, scaling): def _entry_cost_field_shifted(definition, expanded): """Whether any declared list driver carries a per-entry cost field outside the values the default entries had.""" - defaults = definition.get("variables") or {} + # the defaults the curated cost was measured at, the default preset + # bundle among them + defaults = effective_variables(definition) effective = expanded.get("variables") or {} return any( isinstance(defaults.get(name), list) diff --git a/dw/for_each.py b/dw/for_each.py index 9299e6165..58494ac8e 100644 --- a/dw/for_each.py +++ b/dw/for_each.py @@ -450,8 +450,11 @@ def entry_field_warnings(definition, arguments=None): return [] variables = copy.deepcopy(variables) supplied = set() - if arguments and not argument_errors(definition, arguments): - apply_presets(definition, arguments, variables) + good = bool(arguments) and not argument_errors(definition, arguments) + # the chosen bundle - the default one when no argument folds - as + # Workflow._fold applies it + apply_presets(definition, arguments if good else {}, variables, strict=False) + if good: set_variables(arguments, variables) supplied = set(arguments) warnings = [] diff --git a/dw/plan.py b/dw/plan.py index 055649231..2375c1e30 100644 --- a/dw/plan.py +++ b/dw/plan.py @@ -34,6 +34,7 @@ from .elision import elide_definition from . import references from .plan_downloads import _sub_workflow_paths, downloads_required +from .presets import effective_variables # Re-exported: dw/server/routes/jobs.py and the tests read the plan's gate # warnings from here, beside build_plan @@ -136,7 +137,11 @@ def build_plan( # cost is the cost of the work that happens elided = elide_definition(expanded, definition) entries = list_entries(definition, realized) - measured_entries = list_entries(definition, definition) + # The defaults a curated cost was measured at - the default preset + # bundle among them + measured_entries = list_entries( + definition, {"variables": effective_variables(definition)} + ) step_count = len(expanded.get("steps") or []) cache_hits = cached_steps(definition, realized, arguments, cache_probe) # The work includes the text a stored prompt holds now, not only its @@ -207,8 +212,9 @@ def list_entries(definition, realized): read from `realized`'s folded variables - a literal list is not an argument and is not listed. - Passing `definition` as both arguments answers the lengths the - workflow's *stored defaults* carry, which is the list a catalog `cost` + Passing `{"variables": effective_variables(definition)}` as `realized` + answers the lengths the workflow's *stored defaults* carry (its default + preset bundle included), which is the list a catalog `cost` figure was measured against (`estimate`). """ variables = realized.get("variables") or {} @@ -449,14 +455,17 @@ def _child_catalog_price(child_definition, child_cost, step_arguments, device): child_measured_entries = {} child_expanded = {"variables": {}} if child_definition is not None: - child_measured_entries = _list_entries(child_definition, child_definition) + child_measured_entries = _list_entries( + child_definition, {"variables": effective_variables(child_definition)} + ) # The composing step's own `arguments` are what the child - # actually runs with - folded over its declared defaults the - # same way a caller's arguments are, since `expanded` has - # already substituted them to concrete values (#341) - effective_variables = dict(child_definition.get("variables") or {}) - effective_variables.update(step_arguments) - child_expanded = {"variables": effective_variables} + # actually runs with - folded over its declared defaults (and the + # preset bundle they choose) the same way a caller's arguments are, + # since `expanded` has already substituted them to concrete values + # (#341) + child_expanded = { + "variables": effective_variables(child_definition, step_arguments) + } child_list_entries = _list_entries(child_definition, child_expanded) child = _price(child_cost, device, child_list_entries, child_measured_entries) if ( diff --git a/dw/presets.py b/dw/presets.py index ad0d5e865..6dd4d4312 100644 --- a/dw/presets.py +++ b/dw/presets.py @@ -9,11 +9,28 @@ (`apply_presets`, then `set_variables`), so an explicit argument always wins. A bundle value goes through `set_variables` exactly as a caller's would - name and string validation, type coercion - so a bundle is no way around -those checks. Both places a run's arguments are folded (`Workflow._fold`, -which the estimate and the realized copy read too, and the argument fold in -`dw/validation.py`) call `apply_presets` first. +those checks. + +Validate checks what the run will use (`dw/workflow_run.py`), so every +reader of a run's variables sees the bundle, the default one included when +no arguments are passed (or when a bad argument leaves them all unfolded): + +- the fold (`Workflow._fold`), which the run, validation, the estimate, the + fingerprint and the VRAM projection all read - the projection reads the + expanded definition's already-folded variables, so it never re-applies; +- `effective_variables`, the one owner of "a written definition's variables + for these arguments", for the readers that take the definition as written + rather than folded: the constraint checks, the argument warnings, the + host-RAM memory key, the VRAM-inheritance comparisons and the observed-cost + buckets; +- `preset_errors`, the static check over *every* bundle, chosen or not, and + the constraint checks (`dw/variable_constraints.py`), which report a bundle + value outside a declared rule at `presets...`. """ +import copy + +from .security import SecurityError from .variables import set_variables PRESETS_KEY = "presets" @@ -31,10 +48,15 @@ def _chosen(selector, arguments, variables): return variables.get(selector) -def apply_presets(definition, arguments, variables): +def apply_presets(definition, arguments, variables, *, strict=True): """Set, on `variables`, the chosen bundle's values for every variable the caller did not pass. A selector naming no bundle sets nothing here; - `preset_errors` is what refuses it.""" + `preset_errors` is what refuses it. + + `strict` (the run) raises what `set_variables` raises for a bundle value + that will not coerce; otherwise (validation, the readers of + `effective_variables`) that one value is left at its declared default, + since `preset_errors` reports it at its own path.""" arguments = arguments if isinstance(arguments, dict) else {} if not isinstance(variables, dict): return @@ -50,14 +72,90 @@ def apply_presets(definition, arguments, variables): for name, value in bundle.items() if name not in arguments and name != selector } - set_variables(values, variables) + if strict: + set_variables(values, variables) + continue + for name, value in values.items(): + try: + set_variables({name: value}, variables) + except (ValueError, TypeError, SecurityError): + continue + + +def effective_variables(definition, arguments=None): + """The variables a run of the written `definition` uses for `arguments`, + as a new dict: the declared defaults, the chosen bundle's values over + them, and the caller's `arguments` over those. + + Never raises and never mutates `definition`. The arguments are laid over + as given, uncoerced - the shallow merge every reader made before presets + existed, so a bad argument is still there for the reader that reports + it. For a definition that is already folded (an expanded one), read its + `variables` instead: applying the bundle again would undo a rule's + rounding.""" + declared = definition.get("variables") if isinstance(definition, dict) else None + variables = dict(declared) if isinstance(declared, dict) else {} + arguments = arguments if isinstance(arguments, dict) else {} + apply_presets(definition, arguments, variables, strict=False) + variables.update(arguments) + return variables + + +def bundle_sources(definition, arguments=None): + """{variable: "presets.."} for each variable the chosen + bundle sets on a run with `arguments` - where a finding about that value + belongs, since the author wrote it there.""" + arguments = arguments if isinstance(arguments, dict) else {} + declared = definition.get("variables") if isinstance(definition, dict) else None + declared = declared if isinstance(declared, dict) else {} + sources = {} + for selector, bundles in _presets(definition).items(): + if not isinstance(bundles, dict): + continue + chosen = _chosen(selector, arguments, declared) + bundle = bundles.get(chosen) if isinstance(chosen, str) else None + if not isinstance(bundle, dict): + continue + for name in bundle: + if name not in arguments and name != selector and name in declared: + sources[name] = f"{PRESETS_KEY}.{selector}.{chosen}" + return sources + + +def coerced_bundles(definition): + """(selector, value, {variable: coerced value}) for every bundle, each + value coerced against the declared default as the run coerces it. A value + that will not coerce, or names no declared variable, is left out - + `preset_errors` reports those.""" + declared = definition.get("variables") if isinstance(definition, dict) else None + declared = declared if isinstance(declared, dict) else {} + for selector, bundles in _presets(definition).items(): + if not isinstance(bundles, dict): + continue + for value, bundle in bundles.items(): + if not isinstance(bundle, dict): + continue + coerced = {} + for name, item in bundle.items(): + if name not in declared or name == selector: + continue + scratch = {name: declared[name]} + try: + set_variables({name: item}, scratch) + except (ValueError, TypeError, SecurityError): + continue + coerced[name] = scratch[name] + yield selector, value, coerced def preset_errors(definition, arguments=None): """What is wrong with the `presets` block, as [{path, message}]: a bundle - key that is no declared variable, a selector that is not a declared - variable or whose default is no bundle name, and a caller value for the - selector outside the bundle names (at `arguments.`).""" + key that is no declared variable, a bundle value that will not coerce to + its declared default's type or fails string validation, a selector that + is not a declared variable or whose default is not a string naming one of + its bundles, and a caller value for the selector outside the bundle names + (at `arguments.`). Every bundle is checked, not only the chosen + one; a value outside a declared rule is `constraint_errors`' to report.""" presets = _presets(definition) if not presets: return [] @@ -78,6 +176,17 @@ def preset_errors(definition, arguments=None): f"variable; declared: {names}", } ) + elif not isinstance(declared[selector], str): + default = declared[selector] + errors.append( + { + "path": f"variables.{selector}", + "message": f"The preset selector '{selector}' defaults to " + f"{type(default).__name__} {default!r}, but bundle names " + "are JSON object keys and so always strings - it can never " + f"select one. Make its default a string, one of: {listed}", + } + ) elif declared[selector] not in bundles: errors.append( { @@ -98,13 +207,23 @@ def preset_errors(definition, arguments=None): for value, bundle in bundles.items(): if not isinstance(bundle, dict): continue - for name in bundle: + for name, item in bundle.items(): + path = f"{PRESETS_KEY}.{selector}.{value}.{name}" if name not in declared: errors.append( { - "path": f"presets.{selector}.{value}.{name}", + "path": path, "message": f"'{name}' is not a declared " f"variable; declared: {names}", } ) + continue + # Coerced against a copy, one value at a time, the way + # argument_errors checks a caller's - so every bundle is + # checked, chosen or not, and a bad one is a finding rather + # than the ValueError the run's set_variables would raise + try: + set_variables({name: item}, copy.deepcopy(declared)) + except (ValueError, TypeError, SecurityError) as e: + errors.append({"path": path, "message": str(e)}) return errors diff --git a/dw/server/deps.py b/dw/server/deps.py index 3afd6bd51..10d70ef89 100644 --- a/dw/server/deps.py +++ b/dw/server/deps.py @@ -15,6 +15,7 @@ from .. import references from ..adapter_compatibility import catalog_index from ..lora_catalog import entry_errors +from ..presets import PRESETS_KEY, effective_variables from ..realize import strings_with_prefix from ..security import SecurityError, validate_asset_reference from ..vram_inheritance import ( @@ -295,8 +296,10 @@ def _scale_past_template(definition, template, drivers, arguments): catalog figure and this one quote a request alike - and the reason names each axis that moved. (1, None) when the workflow is the template's size; (None, reason) when its size cannot be compared.""" - # The template as the bucket's drivers set it - bucket = {**template, "variables": {**(template.get("variables") or {}), **drivers}} + # The template as the bucket's drivers set it - its preset bundle + # folded in under them, and the block dropped so it is not applied twice + bucket = {key: value for key, value in template.items() if key != PRESETS_KEY} + bucket["variables"] = effective_variables(template, drivers) # Counted with for_each expanded, so a list longer than the one the # bucket ran (a fallback to the template's default bucket) re-prices by # its member count @@ -381,8 +384,8 @@ def inherited_observed(state, ws, definition, arguments=None, *, card=None): if identity is None: return None arguments = arguments or {} - variables = definition.get("variables") - variables = variables if isinstance(variables, dict) else {} + # the inline run's own values, its chosen preset bundle included + variables = effective_variables(definition, arguments) best = None withheld = None for name, template, writable in _catalog_listing(state, ws): @@ -392,9 +395,9 @@ def inherited_observed(state, ws, definition, arguments=None, *, card=None): # workflow's own value where it has that variable, else the template's # default - so the bucket is the one the inline run would fall in drivers = { - driver: arguments.get(driver, variables[driver]) + driver: variables[driver] for driver in declared_drivers(template) - if driver in arguments or driver in variables + if driver in variables } block = observed_for_name( state, diff --git a/dw/server/host_ram_need.py b/dw/server/host_ram_need.py index 6aef27098..98c1ffdb8 100644 --- a/dw/server/host_ram_need.py +++ b/dw/server/host_ram_need.py @@ -32,6 +32,7 @@ import logging from ..host_memory_projection import projected_mb, releases_between_iterations +from ..presets import effective_variables from ..variables import VariableNotFoundError, replace_variables from ..vram_estimate import pipeline_identity from ..vram_inheritance import _found @@ -61,9 +62,8 @@ def memory_key(definition, arguments=None): references a variable it doesn't declare.""" if not isinstance(definition, dict): return None - variables = definition.get("variables") - variables = dict(variables) if isinstance(variables, dict) else {} - variables.update(arguments or {}) + # the chosen preset bundle included, as the run folds it + variables = effective_variables(definition, arguments) # Deep substitution, not `_found`'s: `weights_dtype` sits inside a # quantization config rather than being one try: diff --git a/dw/server/observed_cost.py b/dw/server/observed_cost.py index 4e85341af..458e25254 100644 --- a/dw/server/observed_cost.py +++ b/dw/server/observed_cost.py @@ -41,6 +41,8 @@ import logging import time +from ..presets import effective_variables + logger = logging.getLogger("dw") COST_DRIVERS_KEY = "cost_drivers" @@ -84,10 +86,10 @@ def _bucket_key(definition, arguments): variables = definition.get("variables") or {} drivers = declared_drivers(definition) if drivers: - return tuple( - (name, _comparable(arguments.get(name, variables.get(name)))) - for name in sorted(drivers) - ) + # the chosen preset bundle included: a driver a bundle moves buckets + # on the value the run used + values = effective_variables(definition, arguments) + return tuple((name, _comparable(values.get(name))) for name in sorted(drivers)) if _is_task_only(definition): # No pipeline loads, so a task's inputs (an audio file, a list of # lines) are data rather than a size; with nothing declared to @@ -244,7 +246,6 @@ def observed_for(definition, rows, device=None, device_name=None, arguments=None runs = len(cold) + len(warm) + unclassified if not runs: return None - variables = definition.get("variables") or {} drivers = declared_drivers(definition) observed = { "device": device, @@ -261,10 +262,8 @@ def observed_for(definition, rows, device=None, device_name=None, arguments=None # The *effective* driver values - the caller's where they supplied # one, else the stored default - so the block says which run it is a # figure for rather than always describing the defaults - effective = { - name: (arguments or {}).get(name, variables.get(name)) - for name in sorted(drivers) - } + values = effective_variables(definition, arguments) + effective = {name: values.get(name) for name in sorted(drivers)} observed["drivers"] = { name: len(value) if isinstance(value, list) else value for name, value in effective.items() diff --git a/dw/server/routes/jobs.py b/dw/server/routes/jobs.py index 5f7dc0ec2..7acb5af65 100644 --- a/dw/server/routes/jobs.py +++ b/dw/server/routes/jobs.py @@ -32,6 +32,7 @@ from ...events import select_kinds from ...host_memory_projection import CEILING_FRACTION, host_memory_warnings from ...plan import build_plan, gate_warnings +from ...presets import apply_presets from ...schema import format_validation_errors from ...security import SecurityError, validate_path from ...variables import set_variables @@ -368,8 +369,10 @@ def get_job_workflow(request: Request, job_id: str): name: arguments[name] for name in arguments if name in variables } # Fold through the owner, so a value is validated and coerced to - # the default's type exactly as it was when the run started + # the default's type exactly as it was when the run started - + # the chosen preset bundle first, as Workflow._fold applies it try: + apply_presets(definition, declared, variables) set_variables(declared, variables) folded = list(declared) except (SecurityError, TypeError, ValueError) as exc: diff --git a/dw/validation.py b/dw/validation.py index 15678c1be..1123c316a 100644 --- a/dw/validation.py +++ b/dw/validation.py @@ -901,9 +901,12 @@ def undeclared_variable_errors(workflow, arguments=None): definition = copy.deepcopy(workflow.workflow_definition) variables = definition.get("variables") supplied = set() - if isinstance(variables, dict) and arguments: - if not argument_errors(definition, arguments): - apply_presets(definition, arguments, variables) + if isinstance(variables, dict): + good = bool(arguments) and not argument_errors(definition, arguments) + # the chosen bundle - the default one when no argument folds - as + # Workflow._fold applies it + apply_presets(definition, arguments if good else {}, variables, strict=False) + if good: set_variables(arguments, variables) supplied = set(arguments) declared = sorted(variables or {}) diff --git a/dw/variable_constraints.py b/dw/variable_constraints.py index 7de29e1b1..2bafdc95d 100644 --- a/dw/variable_constraints.py +++ b/dw/variable_constraints.py @@ -48,6 +48,7 @@ import numbers from . import references +from .presets import PRESETS_KEY, bundle_sources, coerced_bundles, effective_variables logger = logging.getLogger("dw") @@ -292,20 +293,26 @@ def constraint_errors(definition, arguments=None, supplied=()): """Every declared constraint a value breaks, as [{path, message}]. A value the caller supplied is reported at `arguments.`, where they - wrote it; a stored default at `variables.`. A constraint that + wrote it; a stored default at `variables.`; a preset bundle's value + at `presets...` - every bundle, chosen or not. A constraint that declares `snap: "up"` and can reach a legal value is a warning rather than an error - the run will round it, and saying so is the point. """ constraints = declared_constraints(definition) if not constraints: return [] - variables = definition.get("variables") or {} - values = {**variables, **(arguments or {})} + # What the run uses: the chosen preset bundle under the arguments. A + # value the chosen bundle set is left to the every-bundle pass below, + # so it is reported once, where its author wrote it + values = effective_variables(definition, arguments) + from_bundle = bundle_sources(definition, arguments) errors = [] for name in sorted(constraints): constraint = constraints[name] if not isinstance(constraint, dict) or name not in values: continue + if name in from_bundle: + continue message = refusal(name, values[name], constraint) if message is None: # Either legal, or legal once rounded - a value the workflow @@ -314,6 +321,8 @@ def constraint_errors(definition, arguments=None, supplied=()): where = "arguments" if name in (supplied or ()) else "variables" errors.append({"path": f"{where}.{name}", "message": message}) for variable, index, name, entry in entry_targets(definition, values): + if variable in from_bundle: + continue message = refusal(name, entry[name], constraints[name]) if message is None: continue @@ -321,6 +330,26 @@ def constraint_errors(definition, arguments=None, supplied=()): errors.append( {"path": f"{where}.{variable}[{index}].{name}", "message": message} ) + # Every bundle, chosen or not: a bundle is a default the author wrote, + # so one outside a rule is as wrong as a default outside it would be + for selector, label, bundle in coerced_bundles(definition): + where = f"{PRESETS_KEY}.{selector}.{label}" + for name in sorted(bundle): + constraint = constraints.get(name) + if not isinstance(constraint, dict): + continue + message = refusal(name, bundle[name], constraint) + if message is not None: + errors.append({"path": f"{where}.{name}", "message": message}) + for variable, index, name, entry in entry_targets(definition, bundle): + message = refusal(name, entry[name], constraints[name]) + if message is not None: + errors.append( + { + "path": f"{where}.{variable}[{index}].{name}", + "message": message, + } + ) return errors @@ -335,7 +364,8 @@ def constraint_warnings(definition, arguments=None): constraints = declared_constraints(definition) if not constraints: return [] - values = {**(definition.get("variables") or {}), **(arguments or {})} + # the chosen preset bundle's values round as a caller's would + values = effective_variables(definition, arguments) notices = [] for name in sorted(constraints): constraint = constraints[name] diff --git a/dw/vram_inheritance.py b/dw/vram_inheritance.py index ff7549116..aaeaf6a49 100644 --- a/dw/vram_inheritance.py +++ b/dw/vram_inheritance.py @@ -34,6 +34,7 @@ import json from .for_each import ForEachError, expand_for_each +from .presets import effective_variables from .vram_estimate import KEY as ESTIMATE_KEY from .vram_estimate import reference_count from .variables import ( @@ -159,8 +160,7 @@ def reference_counts(definition, arguments=None): `variable:` references resolved and `for_each` expanded, so each member step counts its own entry's `references` (#915). A step's count is `vram_estimate.reference_count`'s, the one the VRAM projection uses.""" - variables = dict(definition.get("variables") or {}) - variables.update(arguments or {}) + variables = effective_variables(definition, arguments) expanded = _expanded_steps(definition, variables) if expanded is None: return set() @@ -197,8 +197,7 @@ def _compared_variables(definition, arguments=None): a variable whose default is null (`lora_alpha`) then drops that key as the template it was copied from does, instead of comparing the reference text against nothing (#866).""" - variables = dict(definition.get("variables") or {}) - variables.update(arguments or {}) + variables = effective_variables(definition, arguments) for _, name in undeclared_variable_references(definition): variables.setdefault(name, None) return variables @@ -219,8 +218,7 @@ def pipeline_steps(definition, arguments=None): six-entry list is six steps, each with its entry's `item:` values - the count and values an inherited price is scaled by (#851). The authored steps when the expansion fails.""" - variables = dict(definition.get("variables") or {}) - variables.update(arguments or {}) + variables = effective_variables(definition, arguments) steps = _expanded_steps(definition, variables) if steps is None: steps = definition.get("steps") or [] @@ -237,8 +235,7 @@ def pipeline_values(definition, key, arguments=None): sets it twice), None for a step that sets none or a non-numeric one - for scaling an inherited price along it (#851, #866). A `for_each` step is read per member, as `pipeline_steps` expands it.""" - variables = dict(definition.get("variables") or {}) - variables.update(arguments or {}) + variables = effective_variables(definition, arguments) found = [] for step in pipeline_steps(definition, arguments): numbers = [ @@ -368,10 +365,7 @@ def inherited_vram_warnings( identities = [pipeline_identity(step, variables) for step in steps] written_definition = written if isinstance(written, dict) else definition written_variables = written_definition.get("variables") - written_variables = { - **(written_variables if isinstance(written_variables, dict) else {}), - **(arguments or {}), - } + written_variables = effective_variables(written_definition, arguments) written_steps = written_definition.get("steps") written_steps = written_steps if isinstance(written_steps, list) else [] warnings = [] diff --git a/dw/workflow.py b/dw/workflow.py index bcbed8fc7..c19c0efd3 100644 --- a/dw/workflow.py +++ b/dw/workflow.py @@ -318,7 +318,9 @@ def step_output_dir(self, step_definition): os.makedirs(target, exist_ok=True) return target - def _fold(self, definition, arguments, *, fold_arguments, constrain): + def _fold( + self, definition, arguments, *, fold_arguments, constrain, strict_presets=True + ): """Stage one of preparing a definition, shared by validation, the run and the record: constants realized, the caller's `arguments` folded in when `fold_arguments`, list entries' own references @@ -330,6 +332,13 @@ def _fold(self, definition, arguments, *, fold_arguments, constrain): Raises ConstantError for a 'constant:' variable default that fails to resolve, naming the variable. + + The chosen preset bundle (dw/presets.py) is applied whatever + `fold_arguments` is - with no usable arguments, the default bundle - + since a bundle is part of what the declared defaults mean. + `strict_presets` false (validation) leaves a bundle value that will + not coerce at its default rather than raising; `preset_errors` + reports it. """ variables = definition.get("variables") if not isinstance(variables, dict): @@ -351,10 +360,17 @@ def _fold(self, definition, arguments, *, fold_arguments, constrain): realize_constants(value) except (ValueError, InvalidInputError, UntrustedWorkflowError) as e: raise ConstantError(f"variables.{name}", str(e)) from e + # the chosen bundle first, so the caller's own values land over it; + # with no usable arguments that is the selector's default bundle + apply_presets( + definition, + arguments if fold_arguments else {}, + variables, + strict=strict_presets, + ) if fold_arguments: # set variable values from the arguments passed to the workflow; # these may come from the command line or from a parent workflow - apply_presets(definition, arguments, variables) set_variables(arguments, variables) # an entry of a list-valued variable may name another variable; # resolve those before anything inside it is realized, so a @@ -407,6 +423,7 @@ def _folded_and_expanded(self, arguments): fold_arguments=bool(arguments) and not argument_errors(definition, arguments), constrain=snap_constraints, + strict_presets=False, ) source_indices = [] expanded = self._expand(definition, variables, source_indices) diff --git a/tests/test_presets.py b/tests/test_presets.py index 67da8f4ee..c07835276 100644 --- a/tests/test_presets.py +++ b/tests/test_presets.py @@ -7,7 +7,13 @@ import copy +import pytest + +from dw.plan import build_plan from dw.presets import apply_presets, preset_errors +from dw.security import SecurityError +from dw.validation import workflow_errors +from dw.variable_constraints import constraint_errors, constraint_warnings from dw.workflow import workflow_from_definition PRESETS = { @@ -98,13 +104,13 @@ def test_a_workflow_without_presets_is_unchanged(): def test_a_bundle_value_is_validated_like_a_caller_value(): - import pytest - d = definition({"tier": {"standard": {"prompt": "bad\x00value"}}}) variables = copy.deepcopy(d["variables"]) - with pytest.raises(Exception): + with pytest.raises(SecurityError): apply_presets(d, {}, variables) + (problem,) = preset_errors(d, None) + assert problem["path"] == "presets.tier.standard.prompt" def test_the_workflow_folds_the_bundle_for_a_run_and_for_validation(tmp_path): @@ -117,3 +123,239 @@ def test_the_workflow_folds_the_bundle_for_a_run_and_for_validation(tmp_path): errors = workflow.validation_errors({"tier": "ultra"}) assert any(e["path"] == "arguments.tier" for e in errors), errors + + +# --- every reader sees the bundle (validate checks what the run will use) --- + + +def resizing(presets=PRESETS, **overrides): + """The definition with a step that reads every variable.""" + d = definition(presets, **overrides) + d["steps"] = [ + { + "name": "a", + "task": { + "command": "no_op", + "arguments": { + "width": "variable:width", + "steps": "variable:num_inference_steps", + }, + }, + } + ] + return d + + +@pytest.mark.parametrize("arguments", [None, {}]) +def test_the_default_bundle_folds_with_no_arguments(tmp_path, arguments): + workflow = workflow_from_definition(resizing(), str(tmp_path)) + + assert workflow.folded_variables(arguments)["width"] == 768 + expanded = workflow.expanded_definition(arguments) + assert expanded["steps"][0]["task"]["arguments"] == {"width": 768, "steps": 12} + + +def test_a_bad_argument_still_folds_the_default_bundle(tmp_path): + """A bad argument leaves every argument unfolded - the declared + defaults, of which the default bundle is one.""" + workflow = workflow_from_definition(resizing(), str(tmp_path)) + + folded = workflow.folded_variables({"tier": "draft", "width": "wide"}) + + assert folded["width"] == 768 + assert folded["num_inference_steps"] == 12 + + +def test_no_arguments_and_the_default_selector_have_one_fingerprint(tmp_path): + def plan(arguments): + d = resizing() + d["steps"][0]["result"] = {"content_type": "image/png"} + candidate = workflow_from_definition(d, str(tmp_path)) + return build_plan( + candidate, + arguments, + device="cuda", + prompt_dir=str(tmp_path), + lookup_sizes=False, + )["fingerprint"] + + assert plan({}) == plan({"tier": "standard"}) + assert plan({}) != plan({"tier": "draft"}) + + +def framed(draft_frames=100, standard_frames=97, snap=False): + rule = {"modulus": 8, "remainder": 1, "reason": "VAE grid"} + if snap: + rule["snap"] = "up" + return { + "id": "framed", + "variables": {"tier": "standard", "num_frames": 121}, + "variable_constraints": {"num_frames": rule}, + "presets": { + "tier": { + "draft": {"num_frames": draft_frames}, + "standard": {"num_frames": standard_frames}, + } + }, + } + + +@pytest.mark.parametrize("arguments", [{}, {"tier": "draft"}, {"tier": "standard"}]) +def test_a_bundle_value_breaking_a_rule_is_an_error_at_the_bundle(arguments): + """Every bundle is checked, the chosen one or not, and only once.""" + errors = constraint_errors(framed(), arguments, supplied=set(arguments)) + + assert [e["path"] for e in errors] == ["presets.tier.draft.num_frames"] + assert "8 * n + 1" in errors[0]["message"] + + +def test_the_default_bundle_breaking_a_rule_is_an_error_with_no_arguments(): + errors = constraint_errors(framed(draft_frames=97, standard_frames=100), {}) + + assert [e["path"] for e in errors] == ["presets.tier.standard.num_frames"] + + +def test_a_caller_value_over_a_bundle_is_reported_where_the_caller_wrote_it(): + arguments = {"tier": "draft", "num_frames": 100} + errors = constraint_errors(framed(draft_frames=97), arguments, set(arguments)) + + assert [e["path"] for e in errors] == ["arguments.num_frames"] + + +def test_a_snapped_bundle_value_warns_as_a_caller_value_would(): + d = framed(snap=True) + + assert constraint_errors(d, {"tier": "draft"}, {"tier"}) == [] + (notice,) = constraint_warnings(d, {"tier": "draft"}) + assert "100" in notice and "105" in notice + assert constraint_warnings(d, {}) == [] + + +def test_validation_reports_a_bundle_value_breaking_a_rule(tmp_path): + d = framed() + d["steps"] = [ + { + "name": "a", + "task": {"command": "no_op", "arguments": {"n": "variable:num_frames"}}, + } + ] + for arguments in ({}, {"tier": "draft"}): + workflow = workflow_from_definition(copy.deepcopy(d), str(tmp_path)) + paths = [e["path"] for e in workflow_errors(workflow, arguments)] + assert "presets.tier.draft.num_frames" in paths, (arguments, paths) + + +@pytest.mark.parametrize("arguments", [{}, {"tier": "draft"}]) +def test_a_bundle_value_that_will_not_coerce_is_a_finding_not_a_raise( + tmp_path, arguments +): + d = resizing({"tier": {"draft": {"width": "abc"}, "standard": {"width": 768}}}) + workflow = workflow_from_definition(d, str(tmp_path)) + + errors = workflow_errors(workflow, arguments) + + (problem,) = [e for e in errors if e["path"].startswith("presets.")] + assert problem["path"] == "presets.tier.draft.width" + assert "int" in problem["message"] + + +def test_a_selector_whose_default_is_not_a_string_is_refused(): + d = definition({"1": {}}, tier=1) + d["presets"] = {"tier": {"1": {"width": 512}}} + + (problem,) = preset_errors(d, None) + + assert problem["path"] == "variables.tier" + assert "string" in problem["message"] + + +def test_effective_variables_carries_the_bundle_under_the_arguments(): + from dw.presets import effective_variables + + d = definition() + + assert effective_variables(d)["width"] == 768 + values = effective_variables(d, {"tier": "draft", "width": 640}) + assert values["width"] == 640 + assert values["num_inference_steps"] == 4 + # the declared block itself is never touched + assert d["variables"]["width"] == 1024 + + +def test_the_memory_key_sees_the_bundle(): + from dw.server.host_ram_need import memory_key + + d = definition({"tier": {"draft": {"offload": "sequential"}, "standard": {}}}) + d["variables"]["offload"] = "none" + d["steps"] = [ + { + "name": "gen", + "pipeline": { + "configuration": { + "component_type": "{Fake}", + "offload": "variable:offload", + }, + "from_pretrained_arguments": {"model_name": "m"}, + "arguments": {"prompt": "variable:prompt"}, + }, + } + ] + + assert memory_key(d, {}) == memory_key(d, {"tier": "standard"}) + assert memory_key(d, {"tier": "draft"}) != memory_key(d, {}) + assert memory_key(d, {"tier": "draft"}) == memory_key(d, {"offload": "sequential"}) + + +def test_argument_warnings_resolve_a_bundle_value(): + from dw.argument_warnings import workflow_argument_warnings + + d = { + "id": "cut", + "variables": {"tier": "smooth", "seam_fade_ms": 0, "audio_bleed_ms": 40}, + "presets": {"tier": {"smooth": {"seam_fade_ms": 120}, "hard": {}}}, + "steps": [ + { + "name": "cut", + "task": { + "command": "concat_videos", + "arguments": { + "seam_fade_ms": "variable:seam_fade_ms", + "audio_bleed_ms": "variable:audio_bleed_ms", + }, + }, + } + ], + } + + assert any("seam_fade_ms" in w for w in workflow_argument_warnings(d, {})) + assert not any( + "seam_fade_ms" in w for w in workflow_argument_warnings(d, {"tier": "hard"}) + ) + + +def test_the_estimate_prices_a_default_run_at_the_default_bundle(tmp_path): + """The curated cost was measured at the defaults, the default bundle + included - a run with no arguments is not scaled away from it.""" + d = framed(draft_frames=49) + d["cost"] = [{"device": "cuda", "name": "card", "vram_gb": 8, "minutes": 10}] + d["cost_drivers"] = ["num_frames"] + d["steps"] = [ + { + "name": "a", + "task": {"command": "no_op", "arguments": {"n": "variable:num_frames"}}, + "result": {"content_type": "image/png"}, + } + ] + + def estimate(arguments): + candidate = workflow_from_definition(copy.deepcopy(d), str(tmp_path)) + return build_plan( + candidate, + arguments, + device="cuda", + prompt_dir=str(tmp_path), + lookup_sizes=False, + )["estimate"] + + assert estimate({}) == estimate({"tier": "standard"}) + assert estimate({})["minutes"] == 10 diff --git a/tests/test_server.py b/tests/test_server.py index 0f97a5280..3e3ebf3b8 100644 --- a/tests/test_server.py +++ b/tests/test_server.py @@ -6757,3 +6757,21 @@ def raise_boom(self): assert response.status_code == 200 records = [r for r in caplog.records if r.levelno >= logging.ERROR] assert len(records) == 1, [r.getMessage() for r in records] + + +def test_unrealized_job_workflow_folds_the_chosen_preset(server): + """The fallback fold applies the bundle the recorded selector chose, as + the run did (presets).""" + workflow = valid_workflow() + workflow["variables"].update({"tier": "standard", "steps": 20}) + workflow["presets"] = {"tier": {"draft": {"steps": 4}, "standard": {"steps": 12}}} + with server(success_script) as client: + submitted = client.post( + "/api/jobs", json={"workflow": workflow, "arguments": {"tier": "draft"}} + ).json() + wait_for_status(client, submitted["id"], TERMINAL_STATES) + client.app.state.job_manager.jobs.pop(submitted["id"]) + body = client.get(f"/api/jobs/{submitted['id']}/workflow").json() + + assert body["realized"] is False + assert body["definition"]["variables"]["steps"] == 4 From 4bdf6d385681152ccbaa0f914c45f5e5ca3b2dbe Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 19:47:55 -0600 Subject: [PATCH 026/405] feat(chain): per-segment check with a bounded re-roll After each chain segment after the first renders, chain_check measures its kept frames against the previous segment (the seam jump and look step, on analyze_seams' own arithmetic) and segment 1 (fine-band texture drift). A segment that fails is re-rendered from the same carry at the step's seed plus attempt x 1000003, up to `retries`; when every attempt fails the chain keeps the closest one, records check_failed on its shot, emits one chain_check_failed warning, and finishes. Nothing of an attempt reaches the timeline until it is kept. Off by default: absent `check` is no check and absent `retries` is 0, so existing workflows are unchanged. The default ceilings are provisional until the bench noise floor sets them. The estimate quotes the worst case, 1 + (N - 1)(1 + retries) renders for N segments. Ratchet: modules +1 for dw/pipeline_processors/chain_check.py, the module the plan names. Co-Authored-By: Claude Opus 5.5 --- docs/ARCHITECTURE.md | 1 + docs/WORKFLOW_GUIDE.md | 43 +++- dw/cost_scaling.py | 43 +++- dw/guides.py | 10 + dw/pipeline_processors/chain.py | 345 +++++++++++++++++++++----- dw/pipeline_processors/chain_check.py | 183 ++++++++++++++ dw/pipeline_processors/h3_rules.py | 68 +++++ dw/tasks/assess.py | 21 +- dw/workflow_schema.json | 18 ++ tests/test_chain.py | 148 +++++++++++ tests/test_chain_check.py | 241 ++++++++++++++++++ 11 files changed, 1060 insertions(+), 61 deletions(-) create mode 100644 dw/pipeline_processors/chain_check.py create mode 100644 tests/test_chain_check.py diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index f7cf463fe..0b1175ca3 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -80,6 +80,7 @@ one to open. | JSON records beside media | `dw/media_types.py`: `JsonRecord`, `dw/result.py`: `Result.save_artifact` | A dict marked `JsonRecord` is saved whole as one `.json`, whatever content type the step declares, and is never split into one file per key. | `tests/test_face_track.py::test_json_record_is_saved_whole_under_a_video_content_type`, `tests/test_face_track.py::test_the_task_result_saves_crops_as_video_and_track_as_one_json` | | Reading a JSON record back | `dw/locations.py`: `load_json_record` | A task reads a saved JSON record (`fit_to_model`'s, `crop_face_track`'s) only through `load_json_record`: the path is confined like any media path, then the extension must be `.json`, then the size is checked against `MAX_JSON_SIZE`, and only then is it parsed. `fit._read_fit` and `face_track._read_track` call it, never `open`. | `tests/test_locations.py::TestLoadJsonRecord::test_traversal_out_of_the_roots_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_a_non_json_extension_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_an_oversized_file_is_refused_before_it_is_read`, `tests/test_fit_to_model.py::TestRestore::test_fit_path_with_traversal_is_refused` | | Shots in a joined video | `dw/shots.py` | Every `AudioVideo` constructor decides what to do with `shots`, and the sample spans are measured from the joined waveform. | `tests/test_shots.py::test_every_audio_video_constructor_site_is_decided`, `tests/test_shots.py::TestConcatVideosShots::test_an_overrun_track_is_measured_not_derived` | +| Chain per-segment check and re-roll | `dw/pipeline_processors/chain_check.py`: `check_segment`, `SegmentCheck`, `best_attempt`, `reroll_seed`; the keys' rule is `dw/pipeline_processors/h3_rules.py`: `chain_check_problem`; the attempt loop is `dw/pipeline_processors/chain.py`: `_checked_attempt`, `_render_attempt` | A checked segment is measured on the arithmetic `analyze_seams` uses (`dw/tasks/assess.py`: `seam_measures`, `frame_measures`) and `dw/texture_bands.py`, never a copy of it. Every attempt renders from the same `_ChainState`; nothing of an attempt (frames, audio join, shot, spill, look-hold state, segment 1's anchor or keystone) reaches the timeline until it is kept, and a re-roll runs on its own generator at `reroll_seed`, leaving the step's generator where attempt 0 left it. Segment 1 and a hard-cut seam are never checked. Absent `check` is off and absent `retries` is 0. The worst case is quoted by `dw/cost_scaling.py`: `_worst_case_retries`, 1 + (N - 1)(1 + retries) renders for N segments. | `tests/test_chain.py::TestCheckAndReroll::test_a_failed_segment_is_rerolled_with_a_new_seed_from_the_same_carry`, `tests/test_chain.py::TestCheckAndReroll::test_budget_exhausted_keeps_the_best_attempt_and_warns`, `tests/test_chain.py::TestCheckAndReroll::test_absent_check_is_todays_chain`, `tests/test_chain_check.py::test_the_estimate_quotes_the_worst_case_retries` | | Overlap windows of a long video | `dw/tasks/windows.py`: `window_video`, `window_span` | Window `i` covers source frames `[i*stride - overlap, i*stride + stride)`, its synthetic frames repeat the first or last frame, and its audio is cut on the source's own `frames_to_samples` boundaries, so adjacent windows' strides tile the track with no sample lost or repeated. | `tests/test_window_video.py::TestAudio::test_strides_tile_the_source_track_exactly`, `tests/test_window_video.py::TestRealPath::test_a_path_argument_keeps_its_audio`, `tests/test_window_video.py::TestFileMatchesInMemory::test_every_window_equals_the_in_memory_window`, `tests/test_read_frame_range.py` | | Keep a span of a video's frames and audio | `dw/tasks/trim.py`: `trim_video`; `dw/shots.py`: `trimmed_shots` | Frames `[start_frame, start_frame + num_frames)` are kept and the audio is cut on `frames_to_samples` of each end at the track's own rate, so consecutive trims tile the track. A span past the clip's end is refused, never shortened. Shots are clipped to the span, sample side cleared. | `tests/test_trim_video.py::TestFramesAndAudio::test_consecutive_trims_tile_the_track`, `tests/test_trim_video.py::TestShots::test_shots_are_clipped_across_the_boundary` | | Join processed windows back to the source's length | `dw/tasks/windows.py`: `join_windows`; `dw/task_domains.py`: `window_count`, `window_count_problem`, `join_windows_errors` | Output is exactly the source's frames, each once: later windows' first `overlap` frames blend over the previous window's last on the open ramp `dissolve_videos` uses. The window count is `window_count`'s `ceil(source_frames / stride)` and nothing else (it is also `window_video`'s last index plus one). Audio is the source's own track; shots are one per window with `overlap_frames` on each seam, their sample side laid over that track by `dw/shots.py`'s `remeasured_shots` (the same rounding as `frames_to_samples`, so the spans are cumulative). | `tests/test_join_windows.py::TestRoundTrip::test_unprocessed_windows_rejoin_to_the_source`, `tests/test_join_windows.py::TestShots::test_spans_partition_the_frames_and_tile_the_audio`, `tests/test_join_windows.py::TestUint8Output::test_matches_the_float32_accumulator_exactly` | diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index c1a728cd7..1a42ded48 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -2559,6 +2559,11 @@ joined into a single file: instead, so the numbers live in one place (*What a variable is allowed to be*). - `prompts` — optional per-segment prompt list for narrative progression; segment `i` uses `prompts[min(i, len - 1)]`. +- `check` — check each segment after the first before it is kept (any model family): + `true` for the default ceilings, an object to set any of them, absent or `false` for + no check (the chain is unchanged). See *Checking each segment* below. +- `retries` — with `check` on, how many times a segment that fails is re-rendered, + 0 to 3 (default 0). Above 0 without `check` is refused. - `save_segments` — write each completed segment to the output directory as a playable mp4 and free its frames, bounding memory to roughly one segment regardless of chain length. The final video is streamed from the segment files @@ -2572,7 +2577,7 @@ The chain runs inside one iteration of the step, so it composes with `previous_result` fan-out (three keyframes in, three chained videos out), and a `pipeline_reference` step can carry its own `chain`. Seeds behave like a normal run: the step's generator advances across segments, so one seed reproduces the whole -chain. Expect some visual drift across many segments with `last_frame` continuity - +chain (a `check` re-roll runs on a generator of its own, below). Expect some visual drift across many segments with `last_frame` continuity - it is single-frame conditioning; `last_segment` continuity exists for exactly that, where the pipeline can take a video reference. @@ -2581,6 +2586,42 @@ See [workflows/templates/ltx2/chained-segments.json](../workflows/templates/ltx2 [workflows/templates/minimax/chain-matched-to-audio.json](../workflows/templates/minimax/chain-matched-to-audio.json) (audio-matched lip-sync of arbitrary length). +#### Checking each segment: `check` and `retries` + +With `check` on, each segment after the first is measured before anything of it reaches +the stitched video, its audio, its shot record, a `save_segments` file or the look-hold +state: + +| Measure | Against | Default ceiling | Fails when | +|---|---|---|---| +| `jump_ratio` | the previous segment: the largest frame change across the seam over the larger of the two segments' own typical change, floored at 2 grey levels (`analyze_seams`'s measure) | `jump_ratio_max` 25 | above the ceiling | +| `look_ratio` | the previous segment: the largest saturation, lightness or contrast step across the seam over that measure's own rate inside either segment (`analyze_seams`'s measure) | `look_ratio_max` 12 | above the ceiling | +| `texture_ratio` | segment 1: the mean fine-band (2-8 px) luma texture of 4 evenly spaced frames, whole frame | `texture_ratio_max` 2 | above it or below its inverse (doubled or halved); `null` skips it | + +`"check": true` uses those ceilings; `"check": {"jump_ratio_max": 20}` overrides the ones +it names. `face` is reserved and must be `null`. The ceilings are provisional, set to +catch gross failures only until the bench's noise floor sets them. A static segment's own +motion is ~0, so the jump is divided by the floor rather than by ~0. Segment 1 has no seam +and is never checked, nor is a seam whose incoming shot is a hard cut (a chain's own +segments never are). + +A segment that fails is rendered again from the same carry and the same arguments, at a +new seed: attempt `n` (0 is the first) runs at the step's seed plus `n` x 1000003, on a +fresh generator, so the step's own generator - and every later segment - runs as it +would have. A workflow that names no seed draws one before the run and records it, so a +re-roll is repeatable from the manifest. The first attempt that passes is kept. When all +`1 + retries` fail, the chain keeps the one with the fewest failed measures, then the +lowest `jump_ratio`, and finishes - it never restructures the run. The kept attempt's +frames and audio are the ones stitched, and the seam is crossfaded against them. + +Each shot of a checked chain records `attempts`; a re-rolled one records the `seed` its +kept attempt ran at, and one that failed every attempt records `check_failed` (the +measures its kept attempt failed). The job gets one `warning` (`kind: +"chain_check_failed"`) per such segment, naming the measures, their values and the kept +attempt, and the job log names each re-roll. A cost estimate quotes the worst case: +segment 1 is never re-rolled, so N segments cost up to 1 + (N - 1)(1 + `retries`) +segment renders where the curated figure, measured without re-rolls, paid N. + ## Schedulers Override the default scheduler: diff --git a/dw/cost_scaling.py b/dw/cost_scaling.py index 641e684e0..334e5e389 100644 --- a/dw/cost_scaling.py +++ b/dw/cost_scaling.py @@ -258,6 +258,46 @@ def _chain_segments(definition, variables): return chain_segment_count(chain, arguments) +def _chain_retries(definition, variables): + """The re-rolls a checked segment of the workflow's one chain may take + at `variables`: its `retries` with `check` on, else 0.""" + blocks = list(_chain_blocks(definition.get("steps"))) + if len(blocks) != 1: + return 0 + chain = blocks[0][0] + check = _resolved(chain.get("check"), variables) + retries = _resolved(chain.get("retries", 0), variables) + if not (check is True or isinstance(check, dict)): + return 0 + if isinstance(retries, bool) or not isinstance(retries, int): + return 0 + return max(0, retries) + + +def _worst_case_retries(definition, variables): + """(factor, axes, unscaled) for the worst case of a checked chain's + re-rolls, as `chain_check` runs them: segment 1 is never checked, and + each later segment may render 1 + retries times, so N segments cost up + to 1 + (N - 1)(1 + retries) renders where the curated figure, measured + without re-rolls, paid N. `retries` is flagged unscaled when the segment + count cannot be told; with no re-rolls nothing moves.""" + retries = _chain_retries(definition, variables) + segments = _chain_segments(definition, variables) if retries else None + if not retries: + return 1.0, [], [] + if not segments: + return 1.0, [], ["retries"] + renders = 1 + (segments - 1) * (1 + retries) + return ( + renders / segments, + [ + f"worst case {retries} re-roll(s) of each of {segments - 1} checked " + f"segment(s): {renders} renders for {segments}" + ], + [], + ) + + def _driver_scaling(definition, expanded, list_entries): """How far the declared `cost_drivers` moved the run away from the defaults the curated `cost` was measured at, as (factor, axes, unscaled), @@ -322,7 +362,8 @@ def _driver_scaling(definition, expanded, list_entries): for name in chain_drivers ): unscaled.extend(chain_drivers) - return factor, scaled_axes, unscaled + worst, worst_axes, worst_unscaled = _worst_case_retries(definition, effective) + return factor * worst, scaled_axes + worst_axes, unscaled + worst_unscaled def _scaled(price, scaling): diff --git a/dw/guides.py b/dw/guides.py index 1b744d867..fba8290d7 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -63,6 +63,7 @@ RENDER_GRID, chain_anchor_problem, chain_carry_pick_problem, + chain_check_problem, chain_look_hold_problem, chain_keystone_problem, guide_backoff_problems, @@ -427,6 +428,15 @@ def anchor_errors(workflow_definition, source_indices=None): "message": f"{problem}{where}", } ) + problem = chain_check_problem(chain) + if problem: + key = "retries" if problem.startswith("retries") else "check" + errors.append( + { + "path": render_path(base + ("chain", key)), + "message": f"{problem}{where}", + } + ) path = render_path(base + ("chain", "anchor")) problem = chain_anchor_problem(chain) if problem: diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index 931565a79..b05db28e7 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -44,7 +44,7 @@ from .. import empty_device_cache from ..media_types import AudioVideo, fit_codec_padding from ..output_extraction import get_artifact_list -from ..events import emit_log +from ..events import emit_log, emit_warning from ..shots import shot_record, without_samples from ..dsp import as_channels_samples, slice_samples from ..task_domains import frames_to_samples @@ -56,6 +56,13 @@ SegmentSpill, spilled_frames, ) +from .chain_check import ( + best_attempt, + check_segment, + reroll_seed, + segment_check, + texture_samples, +) from .chain_look import LookHold from .chain_pick import sharpest_index from .h3_anchor import anchor_refusal @@ -77,6 +84,7 @@ KEYSTONE_SEGMENT_1, chain_anchor_problem, chain_carry_pick_problem, + chain_check_problem, chain_keystone_problem, chain_look_hold_flags, chain_look_hold_problem, @@ -319,76 +327,63 @@ def run_chain(pipeline, chain_definition, arguments): shots = [] frame_count = 0 + # The step's seed, which a re-roll offsets (chain_check.reroll_seed); the + # previous segment's kept frames, which a check measures the seam + # against, and segment 1's texture samples, which it measures drift by + seed = _step_seed(arguments) + previous_kept = None + segment_1_texture = None + try: for segment in config.plan: - segment_arguments = _segment_arguments( - arguments, - config, - segment, - continuity, - carry, - anchor_frames, - keystone_frame, - ) - logger.info( f"Chain segment {segment.index + 1}/{len(config.plan)}" + (f": {segment.num_frames} frames" if segment.num_frames else "") ) - - # The denoise counter restarts for every segment - without this the - # bar rewinds to zero with nothing saying why - pipeline.segment_label = f"segment {segment.index + 1}/{len(config.plan)}" - # Names the anchor in the job log when this segment carries one - pipeline.latent_anchor_source = ( - "segment 1" if segment.index > 0 and anchor_frames is not None else None + state = _ChainState( + carry, raw_carry, hold, anchor_frames, keystone_frame, seed ) - output = pipeline._run_once(segment_arguments) - artifact = _single_artifact(output) - - if hold is not None: - artifact = _look_held(artifact, hold, raw_carry, segment.index == 0) - carry = continuity.extract(artifact) - if hold is not None and config.continuity == "last_frame": - raw_carry, carry = carry, hold.feed(carry) - segment_frames = frames_as_pil_list(artifact) - if config.anchor == ANCHOR_SEGMENT_1 and segment.index == 0: - anchor_frames = segment_frames - if config.keystone == KEYSTONE_SEGMENT_1 and segment.index == 0: - keystone_frame = segment_frames[0] - segment_audio, segment_rate = _generated_audio(artifact) - # A segment's own generated audio can run a codec-padding sliver - # short of the frames it was asked for - the same gap #197 fixed for - # a decoded file (_decode_audio_video) and for an in-memory - # previous_result shot (Result.save). A chained segment goes through - # neither of those, so the shortfall was surviving here uncorrected - # and compounding once per segment (#408). - if segment_audio is not None and segment_rate and config.fps: - segment_audio = fit_codec_padding( - segment_audio, len(segment_frames), config.fps, segment_rate + if config.check is not None and segment.index > 0: + attempt, checked = _checked_attempt( + pipeline, + arguments, + config, + segment, + continuity, + state, + previous_kept, + segment_1_texture, + ) + else: + attempt = _render_attempt( + pipeline, arguments, config, segment, continuity, state ) + checked = None - tail_trim = config.tail_trim + getattr(continuity, "picked_back", 0) + # Only the kept attempt reaches the timeline: its carry, its + # look-hold state, its frames and its audio + carry, raw_carry = attempt.carry, attempt.raw_carry + hold = attempt.hold + if config.anchor == ANCHOR_SEGMENT_1 and segment.index == 0: + anchor_frames = attempt.rendered + if config.keystone == KEYSTONE_SEGMENT_1 and segment.index == 0: + keystone_frame = attempt.rendered[0] + if config.check is not None: + previous_kept = attempt.kept + if segment.index == 0: + segment_1_texture = texture_samples(attempt.kept) + segment_audio, segment_rate = attempt.audio, attempt.rate + + tail_trim = attempt.tail_trim if config.carry_pick and segment.index < len(config.plan) - 1: emit_log( f"Chain seam {segment.index + 1}/{len(config.plan) - 1}: " - f"carry_pick took frame {len(segment_frames) - tail_trim} of " - f"{len(segment_frames)} (sharpest of the last " + f"carry_pick took frame {len(attempt.rendered) - tail_trim} of " + f"{len(attempt.rendered)} (sharpest of the last " f"{config.carry_pick}), dropping {tail_trim} after it" ) - if tail_trim and segment.index < len(config.plan) - 1: - # The carry is already taken; the tail it backed off from is - # replaced by the next segment's fresh take (#1189), and a - # picked frame ends its segment (#1032) - segment_frames = segment_frames[:-tail_trim] - if segment_audio is not None: - segment_audio = segment_audio[ - :, - : segment_audio.shape[1] - - frames_to_samples(tail_trim, config.fps, segment_rate), - ] - - kept_frames = segment_frames[segment.head_trim :] + + kept_frames = attempt.kept start_sample = audio.shape[1] if audio is not None else 0 shots.append( shot_record( @@ -398,6 +393,8 @@ def run_chain(pipeline, chain_definition, arguments): start_sample, ) ) + if config.check is not None: + _record_check(shots[-1], segment, config, checked) frame_count += len(kept_frames) if spill is not None: spill.write( @@ -429,7 +426,7 @@ def run_chain(pipeline, chain_definition, arguments): # The segment's raw output is finished with - the frames live on # (in RAM or on disk) and the carry frame is extracted. Free it # before the next segment needs the accelerator. - del output, artifact, segment_frames, segment_audio, kept_frames + del attempt, segment_audio, kept_frames gc.collect() empty_device_cache() finally: @@ -441,6 +438,231 @@ def run_chain(pipeline, chain_definition, arguments): return _stitched(config, spill, frames, audio, audio_rate, shots) +@dataclass +class _ChainState: + """What a segment is rendered from: the previous segment's carry (and, + under look_hold, the carry before pre-compensation and the hold's state), + segment 1's anchor frames and keystone, and the step's seed. A re-roll + renders from exactly this again.""" + + carry: object + raw_carry: object + hold: object # LookHold or None + anchor_frames: list + keystone_frame: object + seed: int # None when the step passes neither a generator nor a seed + + +@dataclass +class _Attempt: + """One render of a segment and everything derived from it, before any of + it is committed to the chain's timeline.""" + + rendered: list # its frames as rendered (look-held), before any trim + kept: list # the frames it adds to the timeline: tail and head trimmed + audio: object # (channels, samples) numpy, tail trimmed, or None + rate: int + carry: object + raw_carry: object + hold: object # the look-hold state after this segment, or None + tail_trim: int + seed: int = None + + +def _render_attempt( + pipeline, arguments, config, segment, continuity, state, number=0, attempts=1 +): + """Render one attempt of a segment from `state`. Attempt 0 runs on the + step's own generator; attempt n > 0 on `chain_check.reroll_seed`'s seed. + The look-hold state is forked when more than one attempt may run, so a + discarded attempt leaves nothing behind in it.""" + segment_arguments = _segment_arguments( + arguments, + config, + segment, + continuity, + state.carry, + state.anchor_frames, + state.keystone_frame, + ) + seed = _reseeded(segment_arguments, state.seed, number) if number else None + + # The denoise counter restarts for every segment - without this the + # bar rewinds to zero with nothing saying why + pipeline.segment_label = f"segment {segment.index + 1}/{len(config.plan)}" + ( + f", attempt {number + 1}/{attempts}" if number else "" + ) + # Names the anchor in the job log when this segment carries one + pipeline.latent_anchor_source = ( + "segment 1" if segment.index > 0 and state.anchor_frames is not None else None + ) + output = pipeline._run_once(segment_arguments) + artifact = _single_artifact(output) + del output + + hold, raw_carry = state.hold, state.raw_carry + if hold is not None: + if attempts > 1: + hold = copy.deepcopy(hold) + artifact = _look_held(artifact, hold, raw_carry, segment.index == 0) + carry = continuity.extract(artifact) + if hold is not None and config.continuity == "last_frame": + raw_carry, carry = carry, hold.feed(carry) + rendered = frames_as_pil_list(artifact) + segment_audio, segment_rate = _generated_audio(artifact) + # A segment's own generated audio can run a codec-padding sliver + # short of the frames it was asked for - the same gap #197 fixed for + # a decoded file (_decode_audio_video) and for an in-memory + # previous_result shot (Result.save). A chained segment goes through + # neither of those, so the shortfall was surviving here uncorrected + # and compounding once per segment (#408). + if segment_audio is not None and segment_rate and config.fps: + segment_audio = fit_codec_padding( + segment_audio, len(rendered), config.fps, segment_rate + ) + + segment_frames = rendered + tail_trim = config.tail_trim + getattr(continuity, "picked_back", 0) + if tail_trim and segment.index < len(config.plan) - 1: + # The carry is already taken; the tail it backed off from is + # replaced by the next segment's fresh take (#1189), and a + # picked frame ends its segment (#1032) + segment_frames = segment_frames[:-tail_trim] + if segment_audio is not None: + segment_audio = segment_audio[ + :, + : segment_audio.shape[1] + - frames_to_samples(tail_trim, config.fps, segment_rate), + ] + return _Attempt( + rendered, + segment_frames[segment.head_trim :], + segment_audio, + segment_rate, + carry, + raw_carry, + hold, + tail_trim, + seed, + ) + + +def _checked_attempt( + pipeline, + arguments, + config, + segment, + continuity, + state, + previous_kept, + segment_1_texture, +): + """Render a segment, check it against the one before and segment 1, and + re-roll it from the same state up to `config.retries` times while it + fails. Returns (the kept attempt, its record): the attempt with the + fewest failed measures, then the lowest jump ratio (`best_attempt`), and + {"attempts", "result", "kept"}. Only the best attempt so far is held.""" + attempts = 1 + config.retries + results = [] + best, best_index = None, None + for number in range(attempts): + if number: + emit_log( + f"Chain segment {segment.index + 1}/{len(config.plan)}: check " + f"failed ({_failures(results[-1])}) - re-rolling, attempt " + f"{number + 1} of {attempts}" + + ( + f" at seed {reroll_seed(state.seed, number)}" + if state.seed is not None + else " (the step passes no seed to vary)" + ) + ) + gc.collect() + empty_device_cache() + attempt = _render_attempt( + pipeline, arguments, config, segment, continuity, state, number, attempts + ) + results.append( + check_segment(previous_kept, attempt.kept, segment_1_texture, config.check) + ) + kept_index = best_attempt(results) + if kept_index == number: + best, best_index = attempt, number + del attempt + if results[-1].passed: + break + return best, { + "attempts": len(results), + "result": results[best_index], + "kept": best_index, + "seed": best.seed, + } + + +def _failures(result): + return ", ".join(f"{name} {result.measures.get(name)}" for name in result.failed) + + +def _record_check(shot, segment, config, checked): + """Put a checked segment's attempts on its shot - and, when every attempt + failed, the measures its kept attempt failed, with one job warning that + reaches the caller. Segment 1 is never checked: one attempt.""" + if checked is None: + shot["attempts"] = 1 + return + shot["attempts"] = checked["attempts"] + result, kept = checked["result"], checked["kept"] + if kept: + # The seed the kept re-roll ran at, so the run can be repeated + shot["seed"] = checked.get("seed") + if result.passed: + return + shot["check_failed"] = list(result.failed) + emit_warning( + f"Chain segment {segment.index + 1}/{len(config.plan)} failed its check " + f"on every attempt ({_failures(result)}); kept attempt {kept + 1} of " + f"{checked['attempts']}, the closest", + kind="chain_check_failed", + segment=segment.index + 1, + measures=list(result.failed), + values={name: result.measures.get(name) for name in result.failed}, + kept_attempt=kept + 1, + attempts=checked["attempts"], + ) + + +def _step_seed(arguments): + """The seed a chain's segments run at: the step's generator's (a + workflow naming none drew one before the run, `workflow_run`, and seeded + the generator with it), else a `seed` argument, else None.""" + generator = arguments.get("generator") + if isinstance(generator, torch.Generator): + return generator.initial_seed() + seed = arguments.get("seed") + if isinstance(seed, int) and not isinstance(seed, bool): + return seed + return None + + +def _reseeded(segment_arguments, seed, number): + """Point a re-roll's arguments at attempt `number`'s seed - a fresh + generator on the step generator's device, or the `seed` argument - and + return that seed; None, with the arguments as they were, without one. + The step's own generator is left where attempt 0 left it, so the + segments after a re-roll run as they would have.""" + if seed is None: + return None + rerolled = reroll_seed(seed, number) + generator = segment_arguments.get("generator") + if isinstance(generator, torch.Generator): + segment_arguments["generator"] = torch.Generator( + device=generator.device + ).manual_seed(rerolled) + else: + segment_arguments["seed"] = rerolled + return rerolled + + def _look_held(artifact, hold, raw_carry, first): """The segment with its frames look-corrected (#1191), audio untouched - the corrected frames are what the carry is taken from and what the @@ -676,6 +898,13 @@ def __init__( raise ValueError(f"Chain {problem}") self.look_hold = chain_look_hold_flags(chain_definition) + # The per-segment check and its re-roll budget; absent is off and 0 + problem = chain_check_problem(chain_definition) + if problem: + raise ValueError(f"Chain {problem}") + self.check = segment_check(chain_definition.get("check")) + self.retries = int(chain_definition.get("retries", 0)) if self.check else 0 + self.segment_argument = chain_definition.get("segment_argument", "image") self.carry_frames = chain_definition.get("carry_frames", None) if self.carry_frames is not None: diff --git a/dw/pipeline_processors/chain_check.py b/dw/pipeline_processors/chain_check.py new file mode 100644 index 000000000..3e032ed7e --- /dev/null +++ b/dw/pipeline_processors/chain_check.py @@ -0,0 +1,183 @@ +"""The per-segment check a chain runs on each segment before keeping it, and +the policy of its bounded re-roll. + +After segment k renders (k > 1), its kept frames are measured against +segment k - 1's - the seam's picture jump and look step, by the arithmetic +`analyze_seams` uses (`dw/tasks/assess.py` `seam_measures`) - and against +segment 1's, for fine-band texture drift (`dw/texture_bands.py`). A segment +over any ceiling fails, naming the measures; `run_chain` re-renders it from +the same carry with `reroll_seed`'s seed up to the chain's `retries`, and +keeps `best_attempt` of what it rendered. Segment 1 has no seam before it +and is never checked; nor is a seam whose incoming shot is a hard cut. + +Measurement and choice only: nothing here renders, and nothing here touches +the timeline. The chain keys are validated by `h3_rules.chain_check_problem`. +""" + +import math +from dataclasses import dataclass, field + +import numpy + +# PROVISIONAL: the default ceilings, to be set from the 99th percentile of +# each measure across the noise-floor runs of S1-S3, plus margin, once +# bench/records/noise-floor.md has them. Until then they are set to catch +# gross failures only: +# - jump_ratio: `seam_frame_jump`'s own threshold (dw/assessment_rules.py), +# above every intended cut the stage D runs measured (9.9-20.3; a dissolve +# reads about 1.8) - a chain seam over it is a cut nobody asked for +# - look_ratio: no assessment rule reads it yet; its calibration +# (docs/proposals/complete/seam-look-measures-complete.md) read 0.64 for a +# clip joined to itself, 19.2 for a saturation-1.6 grade (the size of +# join Kinoloom reported) and 47.5 for an exposure step - 12 is ~19x the +# control and under the visible grade +# - texture_ratio: the segment's fine-band luma against segment 1's, either +# way - halved (blur) or doubled (crust) +JUMP_RATIO_MAX = 25.0 +LOOK_RATIO_MAX = 12.0 +TEXTURE_RATIO_MAX = 2.0 + +# The measures a check can fail, in the order a failure names them +MEASURES = ("jump_ratio", "look_ratio", "texture_ratio") + +# Frames of a segment, evenly spaced, its fine-band texture is the mean of +TEXTURE_SAMPLES = 4 + +# Attempt n of a segment (0 is the first render) runs at the step's seed +# plus n times this prime, so a re-roll is reproducible from the step's seed +REROLL_PRIME = 1000003 + + +@dataclass(frozen=True) +class SegmentCheck: + """The ceilings one segment is held to. `texture_ratio_max` bounds the + fine-band ratio both ways (None skips it); `face` is reserved.""" + + jump_ratio_max: float = JUMP_RATIO_MAX + look_ratio_max: float = LOOK_RATIO_MAX + texture_ratio_max: float | None = TEXTURE_RATIO_MAX + face: None = None + + +@dataclass +class CheckResult: + passed: bool + measures: dict + failed: list = field(default_factory=list) + + +def segment_check(value): + """A chain block's `check` as a SegmentCheck, or None when off: true is + the defaults, an object overrides the fields it names. The value has + passed `h3_rules.chain_check_problem`.""" + if value is None or value is False: + return None + if value is True: + return SegmentCheck() + fields = { + name: float(value[name]) + for name in ("jump_ratio_max", "look_ratio_max") + if name in value + } + if "texture_ratio_max" in value: + texture = value["texture_ratio_max"] + fields["texture_ratio_max"] = None if texture is None else float(texture) + return SegmentCheck(**fields) + + +def texture_samples(frames): + """The TEXTURE_SAMPLES evenly spaced frames the texture measure reads - + all a chain needs to keep of segment 1.""" + if len(frames) <= TEXTURE_SAMPLES: + return list(frames) + picks = numpy.linspace(0, len(frames) - 1, TEXTURE_SAMPLES).round().astype(int) + return [frames[i] for i in picks] + + +def check_segment(previous_frames, frames, segment_1_frames, check, hard_cut=False): + """Measure a segment's kept frames against the segment before it and + segment 1, as a CheckResult. `hard_cut` marks a seam meant as a cut, + which is never checked.""" + if hard_cut: + return CheckResult(True, {"skipped": "hard_cut"}, []) + from ..tasks.assess import frame_measures, seam_measures + + before_thumbs, before_looks = frame_measures(previous_frames) + after_thumbs, after_looks = frame_measures(frames) + if before_thumbs is None or after_thumbs is None: + return CheckResult(True, {"skipped": "no frames"}, []) + seam = seam_measures( + numpy.concatenate([before_thumbs, after_thumbs]), + numpy.concatenate([before_looks, after_looks]), + {"start_frame": 0, "num_frames": len(before_thumbs)}, + {"start_frame": len(before_thumbs), "num_frames": len(after_thumbs)}, + ) + measures = { + "jump_ratio": seam["jump_ratio"], + "frame_delta": seam["frame_delta"], + "typical_delta": seam["typical_delta"], + "look_ratio": seam["look_ratio"], + "look_ratio_measure": seam["look_ratio_measure"], + "texture_ratio": _texture_ratio(frames, segment_1_frames), + } + failed = [] + if _over(measures["jump_ratio"], check.jump_ratio_max): + failed.append("jump_ratio") + if _over(measures["look_ratio"], check.look_ratio_max): + failed.append("look_ratio") + texture = measures["texture_ratio"] + if ( + check.texture_ratio_max is not None + and texture is not None + and max(texture, 1 / texture if texture else math.inf) > check.texture_ratio_max + ): + failed.append("texture_ratio") + return CheckResult(not failed, measures, failed) + + +def best_attempt(results): + """The index of the attempt to keep: fewest failed measures, then the + lowest jump ratio, then the earliest.""" + + def rank(index): + result = results[index] + jump = result.measures.get("jump_ratio") + return (len(result.failed), math.inf if jump is None else jump, index) + + return min(range(len(results)), key=rank) + + +def reroll_seed(seed, attempt): + """The seed attempt `attempt` of a segment runs at (0 is the first).""" + return seed + attempt * REROLL_PRIME + + +def _over(value, ceiling): + return value is not None and value > ceiling + + +def _fine_luma(frames): + """The mean fine-band luma of `frames`' texture samples, or None when a + frame is too small to hold the band.""" + from ..texture_bands import MIN_SIDE, texture_bands + + values = [] + for frame in texture_samples(frames): + # A chain's frames are PIL images; an array is taken as RGB already + rgb = numpy.asarray( + frame.convert("RGB") if hasattr(frame, "convert") else frame + ) + if min(rgb.shape[:2]) < MIN_SIDE: + return None + values.append(texture_bands(rgb)["luma"]["fine"]) + return float(numpy.mean(values)) if values else None + + +def _texture_ratio(frames, segment_1_frames): + """The segment's fine-band luma over segment 1's, or None where either + has none to measure.""" + base = _fine_luma(segment_1_frames) + value = _fine_luma(frames) + if not base or value is None: + return None + return round(value / base, 4) diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index e03982f8b..7cdeb3ab1 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -1023,3 +1023,71 @@ def chain_look_hold_problem(chain): "creep and texture work on any continuity" ) return None + + +# The per-segment check and its bounded re-roll (chain_check.py): the fields a +# `check` object may set (`face` is reserved, null only) and the most re-rolls +# a checked segment may take. Family-independent +CHAIN_CHECK_FIELDS = ("jump_ratio_max", "look_ratio_max", "texture_ratio_max", "face") +CHAIN_RETRIES_MAX = 3 + + +def _number_over(value, floor): + """Whether `value` is a finite number above `floor` (never a bool), or an + unresolved reference, which is left to the run.""" + if isinstance(value, str): + return references.is_ref(references.UNRESOLVED, value) + return ( + not isinstance(value, bool) + and isinstance(value, (int, float)) + and math.isfinite(value) + and value > floor + ) + + +def chain_check_problem(chain): + """The reason a chain block's `check` or `retries` is refused, or None. + `check` is true (the default ceilings), false, or an object of + `CHAIN_CHECK_FIELDS`: the ratio ceilings above 0, `texture_ratio_max` + above 1 (it bounds the ratio both ways) or null, `face` null. `retries` + is a whole number from 0 to `CHAIN_RETRIES_MAX`; above 0 it needs the + check on, as with no check nothing fails. Unresolved references are left + to the run, whose `ChainConfig` asks this same rule.""" + check = chain.get("check") + expected = "expected true, false or an object of " + ", ".join(CHAIN_CHECK_FIELDS) + if isinstance(check, dict): + unknown = sorted(set(check) - set(CHAIN_CHECK_FIELDS)) + if unknown: + return f"check has unknown key(s) {unknown} - {expected}" + for name in ("jump_ratio_max", "look_ratio_max"): + if name in check and not _number_over(check[name], 0): + return f"check {name} must be a number above 0, got {check[name]!r}" + texture = check.get("texture_ratio_max") + if texture is not None and not _number_over(texture, 1): + return ( + "check texture_ratio_max bounds the fine-band ratio both ways, " + f"so must be a number above 1 or null, got {texture!r}" + ) + if check.get("face") is not None: + return f"check face is reserved and must be null, got {check['face']!r}" + elif not ( + check is None + or isinstance(check, bool) + or (isinstance(check, str) and _number_over(check, 0)) + ): + return f"invalid check {check!r} - {expected}" + retries = chain.get("retries", 0) + if isinstance(retries, str) and _number_over(retries, 0): + return None + whole = isinstance(retries, int) and not isinstance(retries, bool) + if not whole or not 0 <= retries <= CHAIN_RETRIES_MAX: + return ( + f"retries must be a whole number between 0 and {CHAIN_RETRIES_MAX}, " + f"got {retries!r}" + ) + if retries and (check is None or check is False): + return ( + f"retries {retries} needs check on - a re-roll replaces a segment " + "its check failed, and with no check no segment fails" + ) + return None diff --git a/dw/tasks/assess.py b/dw/tasks/assess.py index 208dbc518..52d7673f9 100644 --- a/dw/tasks/assess.py +++ b/dw/tasks/assess.py @@ -224,7 +224,7 @@ def _in_memory_thumbs(frames): """(thumbnails, looks) of an AudioVideo's frames, one frame at a time: a list, an (N, H, W, C) array, or a SegmentedFrames replaying (N, H, W, 3) chunks. Both None for no frames.""" - from ..pipeline_processors.chain import SegmentedFrames + from ..pipeline_processors.chain_spill import SegmentedFrames thumbs = [] looks = [] @@ -246,6 +246,25 @@ def add(frame): return numpy.stack(thumbs), numpy.stack(looks) +def frame_measures(frames): + """(thumbnails, looks) of in-memory frames - the public name the chain's + per-segment check (dw/pipeline_processors/chain_check.py) reads them by; + `_in_memory_thumbs` is the one reduction.""" + return _in_memory_thumbs(frames) + + +def seam_measures(thumbs, looks, previous, shot, fade=0): + """The picture and look measurements `analyze_seams` takes at the seam + `shot` opens, over thumbnails and looks already reduced - the public name + the chain's per-segment check measures a seam by, before the segment is + on any timeline. `_seam_video` and `_seam_look` are the arithmetic.""" + media = Media(thumbs, None, None, None, looks=looks) + return { + **_seam_video(media, previous, shot, fade), + **_seam_look(media, previous, shot, fade), + } + + def _file_path(video): """The validated file a probe's 'video' names, or None for an in-memory clip: a path, or the VideoFileReference an asset:/output: reference or a diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index c17d0ea93..c7fa11b57 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -403,6 +403,24 @@ }, "additionalProperties": false }, + "check": { + "description": "Check each segment after the first before keeping it, any model family: its seam against the previous segment (the picture jump and the look step, measured as analyze_seams measures them) and its fine-band texture against segment 1's. true uses the default ceilings; an object overrides any of 'jump_ratio_max', 'look_ratio_max' and 'texture_ratio_max' (a number above 1 bounding the ratio both ways, or null to skip it); 'face' is reserved and must be null. A failed segment is re-rendered up to 'retries' times. Segment 1 and a seam opened by a hard cut are never checked. Absent or false = off, and the chain is unchanged. Accepts a 'variable:' reference.", + "type": ["boolean", "object", "string"], + "properties": { + "jump_ratio_max": {"type": ["number", "string"], "exclusiveMinimum": 0}, + "look_ratio_max": {"type": ["number", "string"], "exclusiveMinimum": 0}, + "texture_ratio_max": {"type": ["number", "string", "null"], "exclusiveMinimum": 1}, + "face": {"type": "null"} + }, + "additionalProperties": false + }, + "retries": { + "description": "With 'check' on, how many times a segment that fails its check is re-rendered from the same carry, each at a new seed (the step's seed plus the attempt number times 1000003): 0 to 3, default 0. When every attempt fails, the chain keeps the attempt with the fewest failed measures (then the lowest jump), records 'check_failed' on its shot, warns and finishes. A cost estimate quotes the worst case. Above 0 without 'check' is refused. Accepts a 'variable:' reference.", + "type": ["integer", "string"], + "minimum": 0, + "maximum": 3, + "default": 0 + }, "segment_argument": { "description": "The pipeline argument the carry-over lands in - e.g. 'image' for image-to-video pipelines, or 'references' for reference-conditioned modular pipelines, where the carry-over is appended as an image or video reference.", "type": "string", diff --git a/tests/test_chain.py b/tests/test_chain.py index 7a6234cc2..e7f0f8f67 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -1576,3 +1576,151 @@ def test_validate_names_a_bad_look_hold(self): ] } assert any("look_hold" in e["message"] for e in anchor_errors(definition)) + + +class TestCheckAndReroll: + """The per-segment check (chain_check.py) and its bounded re-roll: a + failed segment is rendered again from the same carry with a new seed, + and only the kept attempt reaches the timeline.""" + + SEED = 42 + PRIME = 1000003 + + def chain(self, **extra): + return {"segments": 2, "trim_frames": 1, "fps": 4, **extra} + + def arguments(self): + return {"generator": torch.Generator().manual_seed(self.SEED)} + + @staticmethod + def pattern(): + x = numpy.arange(32) + row = 128 + 100 * numpy.sin(2 * numpy.pi * x / 32) + plane = numpy.tile(row, (32, 1)) + return numpy.stack([plane] * 3, axis=-1).astype(numpy.uint8) + + def output_for(self, shifts): + """Segment 1 is a sinusoid; a later segment is its carry rolled + sideways by the shift its seed maps to (0 = a seamless take, and + the look is the same at any shift), and its audio is the call + number, so the stitched track says which attempt it came from.""" + + def output(arguments, index): + seed = arguments["generator"].initial_seed() + carry = arguments.get("image") + if carry is None: + pixels = self.pattern() + else: + pixels = numpy.roll(numpy.asarray(carry), shifts[seed], axis=1) + frames = [Image.fromarray(pixels) for _ in range(8)] + audio = torch.full((1, 2, 200), float(index + 1)) + return {"videos": [frames], "audio": audio, "sampling_rate": 100} + + return output + + def seeds(self, *shifts): + return {self.SEED + n * self.PRIME: k for n, k in enumerate(shifts)} + + def test_absent_check_is_todays_chain(self): + pipeline = FakePipeline(self.output_for(self.seeds(16))) + out = run_chain(pipeline, self.chain(), self.arguments()) + assert len(pipeline.calls) == 2 + assert all("attempts" not in shot for shot in out.shots) + + def test_a_failed_segment_is_rerolled_with_a_new_seed_from_the_same_carry(self): + pipeline = FakePipeline(self.output_for(self.seeds(16, 0))) + out = run_chain(pipeline, self.chain(check=True, retries=1), self.arguments()) + + assert len(pipeline.calls) == 3 + first, reroll = pipeline.calls[1], pipeline.calls[2] + assert first["generator"].initial_seed() == self.SEED + assert reroll["generator"].initial_seed() == self.SEED + self.PRIME + assert numpy.array_equal( + numpy.asarray(reroll["image"]), numpy.asarray(first["image"]) + ) + assert out.shots[1]["attempts"] == 2 + assert out.shots[1]["seed"] == self.SEED + self.PRIME + assert "check_failed" not in out.shots[1] + assert out.shots[0]["attempts"] == 1 + + def test_the_kept_attempts_audio_replaces_the_failed_one(self): + pipeline = FakePipeline(self.output_for(self.seeds(16, 0))) + out = run_chain(pipeline, self.chain(check=True, retries=1), self.arguments()) + plain = run_chain( + FakePipeline(self.output_for(self.seeds(0))), + self.chain(), + self.arguments(), + ) + + audio = numpy.asarray(out.audio) + # 200 samples, then 200 less the 25-sample trimmed head + assert audio.shape[-1] == plain.audio.shape[-1] == 375 + # Call 3 (the re-roll) is what follows the seam; call 2 is gone + assert set(numpy.unique(audio[..., 200:])) == {3.0} + assert 2.0 not in numpy.unique(audio) + assert out.shots[1]["crossfade_ms"] == plain.shots[1]["crossfade_ms"] + assert out.shots[1]["num_samples"] == plain.shots[1]["num_samples"] + # And the kept attempt's frames are the ones stitched + assert len(out.frames) == 15 + assert numpy.array_equal( + numpy.asarray(out.frames[-1]), numpy.asarray(out.frames[0]) + ) + + def test_budget_exhausted_keeps_the_best_attempt_and_warns(self): + pipeline = FakePipeline(self.output_for(self.seeds(16, 8, 12))) + with patch("dw.pipeline_processors.chain.emit_warning") as warn: + out = run_chain( + pipeline, self.chain(check=True, retries=2), self.arguments() + ) + + assert len(pipeline.calls) == 4 + shot = out.shots[1] + assert shot["attempts"] == 3 + assert "jump_ratio" in shot["check_failed"] + assert shot["seed"] == self.SEED + self.PRIME + # The shift of 8 (call 3) is kept: the lowest jump of three failures + assert set(numpy.unique(numpy.asarray(out.audio)[..., 200:])) == {3.0} + expected = numpy.roll(self.pattern(), 8, axis=1) + assert numpy.array_equal(numpy.asarray(out.frames[8]), expected) + assert warn.call_count == 1 + message = warn.call_args.args[0] + assert "segment 2" in message and "jump_ratio" in message + assert "attempt 2 of 3" in message + assert warn.call_args.kwargs["kind"] == "chain_check_failed" + assert warn.call_args.kwargs["kept_attempt"] == 2 + + def test_retries_zero_never_rerolls(self): + pipeline = FakePipeline(self.output_for(self.seeds(16))) + with patch("dw.pipeline_processors.chain.emit_warning") as warn: + out = run_chain( + pipeline, self.chain(check=True, retries=0), self.arguments() + ) + assert len(pipeline.calls) == 2 + assert out.shots[1]["attempts"] == 1 + assert "jump_ratio" in out.shots[1]["check_failed"] + assert "seed" not in out.shots[1] + assert warn.call_count == 1 + + def test_a_passing_segment_is_not_rerolled(self): + pipeline = FakePipeline(self.output_for(self.seeds(0))) + out = run_chain( + pipeline, self.chain(segments=3, check=True, retries=3), self.arguments() + ) + assert len(pipeline.calls) == 3 + assert [shot["attempts"] for shot in out.shots] == [1, 1, 1] + + def test_a_reroll_leaves_the_look_hold_as_a_single_render_would(self): + # The failed attempts' look-hold state is dropped with them: a run + # that re-rolls into seamless takes stitches what one that rendered + # them first time does + chain = self.chain(segments=3, check=True, retries=1, look_hold=True) + rerolled = run_chain( + FakePipeline(self.output_for(self.seeds(16, 0))), chain, self.arguments() + ) + direct = run_chain( + FakePipeline(self.output_for(self.seeds(0))), chain, self.arguments() + ) + assert [shot["attempts"] for shot in rerolled.shots] == [1, 2, 2] + assert [numpy.asarray(f).tolist() for f in rerolled.frames] == [ + numpy.asarray(f).tolist() for f in direct.frames + ] diff --git a/tests/test_chain_check.py b/tests/test_chain_check.py new file mode 100644 index 000000000..f0fa848ba --- /dev/null +++ b/tests/test_chain_check.py @@ -0,0 +1,241 @@ +"""The per-segment check a chain runs before it keeps a segment: the seam jump +and look against the previous segment, and texture drift against segment 1 +(dw/pipeline_processors/chain_check.py), and the worst-case minutes its +re-rolls can cost (dw/cost_scaling.py).""" + +import numpy +import pytest +from PIL import Image + +from dw.pipeline_processors.chain_check import ( + JUMP_RATIO_MAX, + LOOK_RATIO_MAX, + TEXTURE_RATIO_MAX, + SegmentCheck, + check_segment, + segment_check, +) +from dw.pipeline_processors.h3_rules import chain_check_problem + + +def noise(seed, size=32): + rng = numpy.random.default_rng(seed) + return rng.integers(0, 256, (size, size, 3), dtype=numpy.uint8) + + +def frames_of(array, count=8): + return [Image.fromarray(array) for _ in range(count)] + + +def grey(level, size=32, count=8): + return frames_of(numpy.full((size, size, 3), level, dtype=numpy.uint8), count) + + +def ramp(reverse=False, size=32): + row = numpy.linspace(0, 255, size) + plane = numpy.tile(row[::-1] if reverse else row, (size, 1)) + return numpy.stack([plane] * 3, axis=-1).astype(numpy.uint8) + + +def test_a_jump_over_the_threshold_fails_and_names_the_measure(): + # The picture flips across the seam, and its look does not move + before = frames_of(ramp()) + after = frames_of(ramp(reverse=True)) + + result = check_segment(before, after, before, SegmentCheck()) + + assert not result.passed + assert result.failed == ["jump_ratio"] + assert result.measures["jump_ratio"] > JUMP_RATIO_MAX + + +def test_a_seamless_segment_passes(): + before = frames_of(noise(1)) + + result = check_segment(before, frames_of(noise(1)), before, SegmentCheck()) + + assert result.passed and result.failed == [] + assert result.measures["jump_ratio"] == 0 + + +def test_a_static_shot_with_near_zero_motion_does_not_fail_on_jump(): + # Both shots static: their own typical delta is ~0, under the assess + # floor, so the seam's change is divided by the floor and not by ~0 + result = check_segment(grey(100), grey(130), grey(100), SegmentCheck()) + + assert "jump_ratio" not in result.failed + assert result.measures["typical_delta"] == 2.0 + assert result.measures["jump_ratio"] == pytest.approx(15.0) + + +def test_hard_cut_seams_are_never_checked(): + before = frames_of(noise(1)) + after = frames_of(noise(2)) + + result = check_segment(before, after, before, SegmentCheck(), hard_cut=True) + + assert result.passed and result.failed == [] + assert result.measures == {"skipped": "hard_cut"} + + +def test_a_look_step_over_the_threshold_fails_on_look(): + result = check_segment(grey(100), grey(160), grey(100), SegmentCheck()) + + assert "look_ratio" in result.failed + assert result.measures["look_ratio_measure"] == "lightness" + + +def test_texture_drift_from_segment_1_fails_both_ways(): + textured = frames_of(noise(1)) + smooth = grey(128) + check = SegmentCheck(jump_ratio_max=1e9, look_ratio_max=1e9) + + blurred = check_segment(textured, smooth, textured, check) + crusted = check_segment(smooth, textured, smooth + smooth, check) + + assert blurred.failed == ["texture_ratio"] + assert blurred.measures["texture_ratio"] < 1 / TEXTURE_RATIO_MAX + # A flat segment 1 has no fine band to measure a ratio against + assert crusted.measures["texture_ratio"] is None + assert "texture_ratio" not in crusted.failed + + +def test_texture_is_not_checked_when_its_threshold_is_none_or_frames_are_small(): + textured = frames_of(noise(1)) + off = SegmentCheck(1e9, 1e9, None) + assert check_segment(textured, grey(128), textured, off).passed + tiny = frames_of(noise(1, size=8)) + small = check_segment(tiny, tiny, tiny, SegmentCheck()) + assert small.measures["texture_ratio"] is None + + +def test_segment_check_reads_the_chain_value(): + assert segment_check(None) is None + assert segment_check(False) is None + assert segment_check(True) == SegmentCheck( + JUMP_RATIO_MAX, LOOK_RATIO_MAX, TEXTURE_RATIO_MAX + ) + custom = segment_check({"jump_ratio_max": 10, "texture_ratio_max": None}) + assert custom == SegmentCheck(10.0, LOOK_RATIO_MAX, None) + + +class TestRule: + def test_absent_and_off_are_accepted(self): + assert chain_check_problem({}) is None + assert chain_check_problem({"check": False}) is None + assert chain_check_problem({"check": True, "retries": 3}) is None + assert chain_check_problem({"check": {"look_ratio_max": 8}}) is None + assert chain_check_problem({"check": True, "retries": "variable:r"}) is None + assert chain_check_problem({"check": "variable:c", "retries": 2}) is None + + @pytest.mark.parametrize( + "chain, words", + [ + ({"check": "yes"}, "check"), + ({"check": {"gloss": 1}}, "unknown"), + ({"check": {"jump_ratio_max": 0}}, "jump_ratio_max"), + ({"check": {"jump_ratio_max": True}}, "jump_ratio_max"), + ({"check": {"texture_ratio_max": 0.5}}, "texture_ratio_max"), + ({"check": {"face": 1}}, "face"), + ({"check": True, "retries": 4}, "0 and 3"), + ({"check": True, "retries": -1}, "0 and 3"), + ({"check": True, "retries": 1.5}, "0 and 3"), + ({"check": True, "retries": True}, "0 and 3"), + ({"retries": 1}, "needs check"), + ({"check": False, "retries": 2}, "needs check"), + ], + ) + def test_bad_values_are_refused_naming_the_key(self, chain, words): + assert words in chain_check_problem(chain) + + def test_chain_config_asks_the_rule(self): + from dw.pipeline_processors.chain import ChainConfig + + with pytest.raises(ValueError, match="0 and 3"): + ChainConfig({"segments": 2, "check": True, "retries": 5}, {}) + config = ChainConfig({"segments": 2}, {}) + assert config.check is None and config.retries == 0 + config = ChainConfig({"segments": 2, "check": True, "retries": 2}, {}) + assert config.check == SegmentCheck() and config.retries == 2 + + def test_validation_wires_the_rule(self): + from dw.guides import anchor_errors + + definition = { + "steps": [ + { + "name": "s", + "pipeline": { + "chain": {"segments": 2, "check": True, "retries": 9}, + "arguments": {}, + }, + } + ] + } + errors = anchor_errors(definition) + assert any( + "retries" in e["message"] and e["path"].endswith("chain.retries") + for e in errors + ), errors + + def test_the_schema_takes_both_keys(self): + import json + from pathlib import Path + + import jsonschema + + schema = json.loads( + (Path(__file__).parent.parent / "dw/workflow_schema.json").read_text() + ) + chain = {"$defs": schema["$defs"], **schema["$defs"]["chain"]} + jsonschema.validate({"segments": 2, "check": True, "retries": 1}, chain) + jsonschema.validate( + {"segments": 2, "check": {"jump_ratio_max": 20, "face": None}}, chain + ) + jsonschema.validate({"segments": 2, "retries": "variable:retries"}, chain) + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate({"segments": 2, "retries": 4}, chain) + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate({"segments": 2, "check": {"gloss": 1}}, chain) + + +def _priced(segments, check, retries): + """A one-chain workflow measured at 3 segments in 30 minutes.""" + chain = {"segments": "variable:segments", "fps": 4} + if check is not None: + chain["check"] = check + if retries is not None: + chain["retries"] = "variable:retries" + variables = {"segments": 3, "retries": 0} + definition = { + "cost": [{"device": "cuda", "name": "card", "minutes": 30}], + "cost_drivers": ["segments"], + "variables": variables, + "steps": [{"pipeline": {"chain": chain, "arguments": {}}}], + } + expanded = {"variables": {**variables, "segments": segments, "retries": retries}} + return definition, expanded + + +def _minutes(segments, check, retries): + from dw.plan import estimate + + definition, expanded = _priced(segments, check, retries) + return estimate(definition, expanded, {}, "cuda", None, None) + + +def test_the_estimate_quotes_the_worst_case_retries(): + per_segment = 10.0 # 30 minutes over the 3 segments it was measured at + # Segment 1 is never checked, so never re-rolled: of 5 segments, 4 can + # each run 1 + retries times + for segments, retries in ((3, 1), (5, 2), (5, 3)): + quoted = _minutes(segments, True, retries) + worst = per_segment * (1 + (segments - 1) * (1 + retries)) + assert quoted["minutes"] == pytest.approx(worst), (segments, retries) + assert "re-roll" in quoted["scaled"] + + +def test_the_estimate_is_unchanged_with_no_check_or_no_retries(): + assert _minutes(3, None, None)["minutes"] == pytest.approx(30) + assert _minutes(3, True, 0)["minutes"] == pytest.approx(30) + assert _minutes(5, True, 0)["minutes"] == pytest.approx(50) From 1e77425cf19e4371fae2054f5f4e859ac674b50f Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 19:57:21 -0600 Subject: [PATCH 027/405] feat(tasks): export_timeline - a joined piece as an editable OTIO bundle Co-Authored-By: Claude Opus 5.5 --- bench/records/for-don.md | 5 + docs/ATTRIBUTIONS.md | 7 + docs/TASKS.md | 34 +++++ dw/arguments.py | 1 + dw/content_types.py | 7 + dw/tasks/timeline_export.py | 278 ++++++++++++++++++++++++++++++++++ dw/tasks/video_handlers.py | 23 +++ tests/test_timeline_export.py | 219 ++++++++++++++++++++++++++ 8 files changed, 574 insertions(+) create mode 100644 bench/records/for-don.md create mode 100644 dw/tasks/timeline_export.py create mode 100644 tests/test_timeline_export.py diff --git a/bench/records/for-don.md b/bench/records/for-don.md new file mode 100644 index 000000000..5b35a495e --- /dev/null +++ b/bench/records/for-don.md @@ -0,0 +1,5 @@ +# For Don + +Things only you can check. + +- [ ] Import one exported `.otioz` (`export_timeline`, #1197) in DaVinci Resolve 18.5+ once a bench cut exists: File > Import > Timeline, then check one clip per shot, the rate (23.976 as 24000/1001) and that each clip plays its span. Nothing here can run Resolve. diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md index 99b728085..f23674cfa 100644 --- a/docs/ATTRIBUTIONS.md +++ b/docs/ATTRIBUTIONS.md @@ -162,3 +162,10 @@ no code taken) inspired `look_hold` grade of a segment's opening to the carried frame, an ease-out of contrast and saturation creep, an ease-out of faint texture, and a keyframe pre-compensated for the measured boost. + +leonardofrito/Directors-Cut (MIT; concept and schema shape, no code taken) +inspired `export_timeline` +([dw/tasks/timeline_export.py](../dw/tasks/timeline_export.py); +[#1197](https://github.com/dkackman/diffusers-workflow/issues/1197)): a joined +piece written as an OpenTimelineIO `.otioz` bundle with one clip per shot, so an +editor opens it as a timeline. diff --git a/docs/TASKS.md b/docs/TASKS.md index 19a0091f6..0901626a6 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -719,6 +719,40 @@ to `join_into_song`). if it reaches a frame. The shot records are re-measured against the file as written, so `media.shots` stays accurate. +### export_timeline + +A joined piece as an editable timeline: an OpenTimelineIO `.otioz` bundle that +DaVinci Resolve (18.5+) imports with one clip per shot. The bundle is a zip of +`version.txt` (`1.0.0`), `content.otio` and a `media/` folder; each clip's +`target_url` is `media/`. The OTIO JSON is written directly, so there is +no `opentimelineio` dependency. Rates are the exact rational the engine writes +(23.976 is 24000/1001, stored by OTIO as a float) with integral frame values. + +```json +{ + "task": { + "command": "export_timeline", + "arguments": { + "video": "output:/latest/final/cut.mp4", + "name": "episode-1" + } + }, + "result": { "content_type": "application/x-otioz" } +} +``` + +| Argument | Meaning | +| -------- | ------- | +| `video` | The joined video, a stored file (`asset:`, `output:` or a path). A clip still in memory is refused: the bundle carries the file | +| `sources` | One file per shot, in order. Each clip then points at its own file over that file's whole range. Default: every clip points into `video` over its shot's span, so the bundle holds one media file | +| `shots` | Shot records; else the video's own from the run manifest, else the whole file is one clip | +| `name` | The timeline's name | + +By default there is one video track and, when the file has a soundtrack, one +audio track. Where the shots leave frames uncovered the track carries a gap, so +clips and gaps add up to the video's length. The concept and schema come from +leonardofrito/Directors-Cut (MIT); see [ATTRIBUTIONS.md](ATTRIBUTIONS.md). + ### dissolve_videos Join videos with a cross-dissolve at every seam, and fade the whole piece in diff --git a/dw/arguments.py b/dw/arguments.py index 67d31640e..f4e9d4cae 100644 --- a/dw/arguments.py +++ b/dw/arguments.py @@ -926,6 +926,7 @@ def resolve_relative_path(path, base_dir): "analyze_seams", "analyze_sync_drift", "analyze_face_drift", + "export_timeline", "window_video", "trim_video", "generate_sound_effects", diff --git a/dw/content_types.py b/dw/content_types.py index b01f425a6..016cdf33b 100644 --- a/dw/content_types.py +++ b/dw/content_types.py @@ -38,6 +38,10 @@ CONTENT_TYPE_KEY = "content_type" +# The OpenTimelineIO bundle export_timeline writes: a zip no MIME database +# names, so its extension is stated here +OTIOZ_CONTENT_TYPE = "application/x-otioz" + # Audio content types soundfile can write, mapped to their file extension and to any # write arguments the extension alone does not imply. Opus has no extension of its own # in libsndfile - it is a subtype of the ogg container. @@ -198,6 +202,9 @@ def guess_extension(content_type): if content_type in AUDIO_FORMATS: return AUDIO_FORMATS[content_type][0] + if content_type == OTIOZ_CONTENT_TYPE: + return ".otioz" + ext = mimetypes.guess_extension(content_type) if ext is not None: return ext diff --git a/dw/tasks/timeline_export.py b/dw/tasks/timeline_export.py new file mode 100644 index 000000000..7b33e7327 --- /dev/null +++ b/dw/tasks/timeline_export.py @@ -0,0 +1,278 @@ +"""export_timeline: a joined piece as an editable OpenTimelineIO bundle (#1197). + +A joined video is one file to an editor, and a recut of it means finding every +seam again. The shots a join records say where each one starts, so this writes +them as the timeline they already are: an `.otioz` bundle that DaVinci Resolve +(18.5+) imports as a timeline with one clip per shot. + +The bundle is a zip holding `version.txt` (`1.0.0`), `content.otio` (OTIO's +JSON, written directly - no `opentimelineio` dependency) and a `media/` folder +with the files the clips point at, each clip's `target_url` being the relative +path `media/`. By default there is one video track and, when the file +has a soundtrack, one audio track; each shot is a clip whose media is the +joined video itself with `source_range` set to the shot's span, so the bundle +holds one media file and every clip points into it. With `sources` (one file +per shot, in order) each clip points at its own file over that file's full +range, the shot lengths of the join giving way to the files' own. + +Frame values are integers on a track whose rate is the exact rational the +engine writes (23.976 is 24000/1001); OTIO stores the rate as a float. + +The concept and the schema shape come from leonardofrito/Directors-Cut (MIT); +see docs/ATTRIBUTIONS.md. +""" + +import json +import logging +import os +import zipfile +from types import SimpleNamespace + +from ..locations import validate_media_path +from ..security import ( + InvalidInputError, + validate_output_path, + validate_string_input, +) +from ..writers import exact_frame_rate +from .assess import resolve_shots + +logger = logging.getLogger("dw") + +COMMAND = "export_timeline" +OTIO_VERSION = "1.0.0" +MEDIA_FOLDER = "media" +MAX_NAME_LENGTH = 200 + + +def rational_time(frames, rate): + return {"OTIO_SCHEMA": "RationalTime.1", "rate": rate, "value": int(frames)} + + +def time_range(start, duration, rate): + return { + "OTIO_SCHEMA": "TimeRange.1", + "duration": rational_time(duration, rate), + "start_time": rational_time(start, rate), + } + + +def _gap(frames, rate): + return { + "OTIO_SCHEMA": "Gap.1", + "name": "", + "metadata": {}, + "source_range": time_range(0, frames, rate), + "effects": [], + "markers": [], + "enabled": True, + } + + +def _clip(name, start, duration, target_url, available_frames, rate): + return { + "OTIO_SCHEMA": "Clip.1", + "name": name, + "metadata": {}, + "source_range": time_range(start, duration, rate), + "effects": [], + "markers": [], + "enabled": True, + "media_reference": { + "OTIO_SCHEMA": "ExternalReference.1", + "name": os.path.basename(target_url), + "metadata": {}, + "available_range": time_range(0, available_frames, rate), + "target_url": target_url, + }, + } + + +def _track(kind, children): + return { + "OTIO_SCHEMA": "Track.1", + "name": kind, + "metadata": {}, + "source_range": None, + "effects": [], + "markers": [], + "enabled": True, + "children": children, + "kind": kind, + } + + +def _timeline(name, tracks): + return { + "OTIO_SCHEMA": "Timeline.1", + "name": name, + "metadata": {}, + "global_start_time": None, + "tracks": { + "OTIO_SCHEMA": "Stack.1", + "name": "tracks", + "metadata": {}, + "source_range": None, + "effects": [], + "markers": [], + "enabled": True, + "children": tracks, + }, + } + + +class TimelineBundle: + """The timeline and the files it points at; `save` writes the `.otioz`. + + It has a `save`, so the result writer takes it for any content type it + has no other writer for - `application/x-otioz` names the extension. + """ + + def __init__(self, document, media): + self.document = document + # {arcname: absolute source path} + self.media = media + + def save(self, path): + """Write the zip at `path`. The member names are fixed here, never + taken from a path, so no source file can place itself outside + `media/`.""" + directory = os.path.dirname(os.path.abspath(path)) + path = validate_output_path(path, directory) + with zipfile.ZipFile(path, "w") as bundle: + bundle.writestr("version.txt", OTIO_VERSION, zipfile.ZIP_DEFLATED) + bundle.writestr( + "content.otio", + json.dumps(self.document, indent=4), + zipfile.ZIP_DEFLATED, + ) + for arcname, source in self.media.items(): + bundle.write(source, arcname, zipfile.ZIP_STORED) + return path + + +def _media_name(source, taken): + """`media/` for a source, unique among the bundle's members.""" + base = os.path.basename(source) or "media" + arcname = f"{MEDIA_FOLDER}/{base}" + stem, extension = os.path.splitext(base) + counter = 1 + while arcname in taken and taken[arcname] != source: + counter += 1 + arcname = f"{MEDIA_FOLDER}/{stem}-{counter}{extension}" + return arcname + + +def _stored_file(value, what): + """The confined, existing file a video argument names.""" + from .video_utils import VideoFileReference + + if isinstance(value, VideoFileReference): + value = value.path + if isinstance(value, dict) and "location" in value: + value = value["location"] + if not isinstance(value, str): + raise InvalidInputError( + f"{COMMAND}: {what} must be a stored video (asset:, output: or a " + f"path) - the bundle carries the file itself, so a clip still in " + f"memory has to be saved first, not {type(value).__name__}" + ) + return validate_media_path(value, None, what) + + +def _facts(path): + from ..media import probe_metadata + + info = probe_metadata(path) + if not info or not info.get("fps") or not info.get("frame_count"): + raise InvalidInputError(f"{COMMAND}: '{os.path.basename(path)}' is not a video") + return info + + +def export_timeline(video, sources=None, shots=None, name="timeline"): + """Task command: the shots of a joined video as an OpenTimelineIO bundle. + + Args: + video: The joined video - a stored file (asset:, output: or a path) + sources: One file per shot, in order, for each clip to point at its + own file over that file's whole range. Default: every clip points + into `video` over its shot's span + shots: Shot records, else the video's own from the run manifest beside + the file, else the whole file is one clip + name: The timeline's name + + Returns: + A bundle the result writes as `.otioz` - content type + `application/x-otioz` + """ + name = validate_string_input(str(name), max_length=MAX_NAME_LENGTH) + path = _stored_file(video, "the video") + info = _facts(path) + rate = float(exact_frame_rate(info["fps"])) + total = int(info["frame_count"]) + + records, _source = resolve_shots(video, SimpleNamespace(shots=None), shots) + if not records: + records = [{"name": "whole", "start_frame": 0, "num_frames": total}] + records = sorted(records, key=lambda shot: int(shot["start_frame"])) + + media = {} + video_arc = _media_name(path, media) + media[video_arc] = path + + if sources: + if len(sources) != len(records): + raise InvalidInputError( + f"{COMMAND}: sources names {len(sources)} files for " + f"{len(records)} shots - give one per shot, in order" + ) + children = [] + for index, (shot, source) in enumerate(zip(records, sources)): + source_path = _stored_file(source, f"sources[{index}]") + source_info = _facts(source_path) + arc = _media_name(source_path, media) + media[arc] = source_path + frames = int(source_info["frame_count"]) + children.append( + _clip( + str(shot.get("name", f"shot {index + 1}")), + 0, + frames, + arc, + frames, + rate, + ) + ) + has_audio = bool(info.get("sample_rate")) + tracks = [_track("Video", children)] + if has_audio: + tracks.append(_track("Audio", json.loads(json.dumps(children)))) + return TimelineBundle(_timeline(name, tracks), media) + + def lay_out(): + children = [] + cursor = 0 + for index, shot in enumerate(records): + start = int(shot["start_frame"]) + frames = int(shot["num_frames"]) + if start > cursor: + children.append(_gap(start - cursor, rate)) + children.append( + _clip( + str(shot.get("name", f"shot {index + 1}")), + start, + frames, + video_arc, + total, + rate, + ) + ) + cursor = start + frames + if cursor < total: + children.append(_gap(total - cursor, rate)) + return children + + tracks = [_track("Video", lay_out())] + if info.get("sample_rate"): + tracks.append(_track("Audio", lay_out())) + return TimelineBundle(_timeline(name, tracks), media) diff --git a/dw/tasks/video_handlers.py b/dw/tasks/video_handlers.py index 6bb157fa8..ddb7aaab8 100644 --- a/dw/tasks/video_handlers.py +++ b/dw/tasks/video_handlers.py @@ -83,6 +83,29 @@ def _handle_dissolve_videos(task, arguments, previous_pipelines): return dissolve_videos(**arguments) +@register_command( + "export_timeline", + implementation="dw.tasks.timeline_export.export_timeline", + summary=( + "Write a joined video's shots as an editable OpenTimelineIO .otioz " + "bundle (DaVinci Resolve 18.5+), one clip per shot." + ), + parameter_descriptions={ + "video": "The joined video - an asset:/output: reference or a path to a stored file.", + "sources": "One file per shot, in order, for each clip to point at its own file; default the shots' spans of video.", + "shots": "Shot records; else the video's own from the run manifest, else the whole file is one clip.", + "name": "The timeline's name.", + }, + media_arguments=("sources",), +) +def _handle_export_timeline(task, arguments, previous_pipelines): + """Write a joined video's shots as an OpenTimelineIO bundle""" + logger.debug("Exporting a timeline") + from .timeline_export import export_timeline + + return export_timeline(**arguments) + + @register_command( "join_into_song", implementation="dw.tasks.join_into_song.join_into_song", diff --git a/tests/test_timeline_export.py b/tests/test_timeline_export.py new file mode 100644 index 000000000..edc942c70 --- /dev/null +++ b/tests/test_timeline_export.py @@ -0,0 +1,219 @@ +"""Tests for export_timeline (dw/tasks/timeline_export.py, #1197). + +Header probing is replaced by a table of facts per file, so no video is +decoded: the files are placeholder bytes, which is all the zip needs. +""" + +import json +import os +import zipfile +from fractions import Fraction + +import pytest + +import dw.media +import dw.tasks.timeline_export as te +from dw.locations import location_errors +from dw.security import SecurityError +from dw.shots import shot_record +from dw.tasks.task import task_command_info +from dw.trust import TRUST_WORKFLOWS_ENV_VAR + +NTSC = 24000 / 1001 + + +@pytest.fixture +def facts(monkeypatch): + table = {} + monkeypatch.setattr(dw.media, "probe_metadata", lambda path: table.get(path)) + return table + + +def a_file(directory, name, facts, frames, fps=24.0, sample_rate=48000): + path = os.path.join(str(directory), name) + with open(path, "wb") as file: + file.write(b"mp4:" + name.encode()) + facts[path] = { + "fps": fps, + "frame_count": frames, + "sample_rate": sample_rate, + } + return path + + +def shots(*spans): + return [ + shot_record(f"shot {i}", start, count) for i, (start, count) in enumerate(spans) + ] + + +def bundle_of(video, tmp_path, **kwargs): + out = tmp_path / "out.otioz" + te.export_timeline(video, **kwargs).save(str(out)) + archive = zipfile.ZipFile(out) + return archive, json.loads(archive.read("content.otio")) + + +def frames_of(item): + return item["source_range"]["duration"]["value"] + + +def test_the_bundle_has_its_three_members(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 72) + archive, content = bundle_of( + video, tmp_path, shots=shots((0, 24), (24, 48)), name="cut" + ) + names = archive.namelist() + assert archive.read("version.txt").decode().strip() == "1.0.0" + assert "content.otio" in names + assert names.count("media/joined.mp4") == 1 + assert len([n for n in names if n.startswith("media/")]) == 1 + assert content["OTIO_SCHEMA"] == "Timeline.1" + assert content["name"] == "cut" + + +def test_one_clip_per_shot_on_a_video_and_an_audio_track(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 72) + _, content = bundle_of(video, tmp_path, shots=shots((0, 24), (24, 48))) + stack = content["tracks"] + assert stack["OTIO_SCHEMA"] == "Stack.1" + assert [t["kind"] for t in stack["children"]] == ["Video", "Audio"] + for track in stack["children"]: + assert track["OTIO_SCHEMA"] == "Track.1" + clips = track["children"] + assert [c["OTIO_SCHEMA"] for c in clips] == ["Clip.1", "Clip.1"] + assert [c["name"] for c in clips] == ["shot 0", "shot 1"] + assert [c["source_range"]["start_time"]["value"] for c in clips] == [0, 24] + reference = clips[0]["media_reference"] + assert reference["OTIO_SCHEMA"] == "ExternalReference.1" + assert reference["target_url"] == "media/joined.mp4" + assert reference["available_range"]["duration"]["value"] == 72 + + +def test_a_silent_video_has_no_audio_track(tmp_path, facts): + video = a_file(tmp_path, "mute.mp4", facts, 24, sample_rate=None) + _, content = bundle_of(video, tmp_path) + assert [t["kind"] for t in content["tracks"]["children"]] == ["Video"] + + +def test_clips_and_gaps_sum_to_the_videos_duration(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 100) + _, content = bundle_of(video, tmp_path, shots=shots((10, 20), (50, 30))) + for track in content["tracks"]["children"]: + kinds = [c["OTIO_SCHEMA"] for c in track["children"]] + assert kinds == ["Gap.1", "Clip.1", "Gap.1", "Clip.1", "Gap.1"] + assert sum(frames_of(c) for c in track["children"]) == 100 + + +def test_a_video_with_no_shots_is_one_clip(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 50) + _, content = bundle_of(video, tmp_path) + clips = content["tracks"]["children"][0]["children"] + assert [frames_of(c) for c in clips] == [50] + + +def test_a_fractional_rate_is_the_exact_rational(tmp_path, facts): + video = a_file(tmp_path, "ntsc.mp4", facts, 48, fps=23.976) + _, content = bundle_of(video, tmp_path, shots=shots((0, 24), (24, 24))) + for item in content["tracks"]["children"][0]["children"]: + for part in ( + item["source_range"]["start_time"], + item["source_range"]["duration"], + ): + assert part["rate"] == NTSC + assert Fraction(part["rate"]).limit_denominator(1001) == Fraction( + 24000, 1001 + ) + assert isinstance(part["value"], int) + + +def test_sources_give_each_clip_its_own_file_over_its_full_range(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 60) + a = a_file(tmp_path, "a.mp4", facts, 24) + b = a_file(tmp_path, "b.mp4", facts, 36) + archive, content = bundle_of( + video, tmp_path, shots=shots((0, 24), (24, 36)), sources=[a, b] + ) + clips = content["tracks"]["children"][0]["children"] + assert [c["media_reference"]["target_url"] for c in clips] == [ + "media/a.mp4", + "media/b.mp4", + ] + assert [frames_of(c) for c in clips] == [24, 36] + assert [c["source_range"]["start_time"]["value"] for c in clips] == [0, 0] + assert {"media/a.mp4", "media/b.mp4"} <= set(archive.namelist()) + + +def test_sources_must_match_the_shots(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 60) + a = a_file(tmp_path, "a.mp4", facts, 24) + with pytest.raises(Exception, match="one per shot"): + te.export_timeline(video, shots=shots((0, 24), (24, 36)), sources=[a]) + + +def test_same_named_sources_stay_apart(tmp_path, facts): + (tmp_path / "x").mkdir() + (tmp_path / "y").mkdir() + video = a_file(tmp_path, "joined.mp4", facts, 48) + a = a_file(tmp_path / "x", "s.mp4", facts, 24) + b = a_file(tmp_path / "y", "s.mp4", facts, 24) + archive, content = bundle_of( + video, tmp_path, shots=shots((0, 24), (24, 24)), sources=[a, b] + ) + urls = [ + c["media_reference"]["target_url"] + for c in content["tracks"]["children"][0]["children"] + ] + assert len(set(urls)) == 2 + assert set(urls) <= set(archive.namelist()) + + +def test_the_bundle_path_is_confined_by_validate_output_path(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 24) + bundle = te.export_timeline(video) + with pytest.raises(SecurityError): + bundle.save(str(tmp_path / "a" / ".." / ".." / "escape.otioz")) + assert not (tmp_path.parent / "escape.otioz").exists() + + +def test_member_names_never_come_from_a_path(tmp_path, facts): + video = a_file(tmp_path, "joined.mp4", facts, 24) + bundle = te.export_timeline(video) + for arcname in bundle.media: + assert arcname.startswith("media/") and ".." not in arcname + + +def test_an_unreadable_video_is_refused(monkeypatch, tmp_path, facts): + monkeypatch.setenv(TRUST_WORKFLOWS_ENV_VAR, "0") + with pytest.raises(SecurityError): + te.export_timeline("/etc/hostname") + + +def test_a_clip_in_memory_is_refused(facts): + with pytest.raises(Exception, match="stored video"): + te.export_timeline(object()) + + +def test_sources_is_a_media_argument(monkeypatch, tmp_path): + monkeypatch.setenv(TRUST_WORKFLOWS_ENV_VAR, "0") + definition = { + "steps": [ + { + "name": "t", + "task": { + "command": "export_timeline", + "arguments": {"video": "asset:a.mp4", "sources": ["/etc/hostname"]}, + }, + } + ] + } + errors = location_errors(definition, base_dir=str(tmp_path)) + assert [e["path"] for e in errors] == ["steps[0].task.arguments.sources[0]"] + + +def test_the_command_is_registered_with_an_otioz_content_type(): + info = task_command_info("export_timeline") + assert info is not None + from dw.content_types import guess_extension + + assert guess_extension(te.OTIOZ_CONTENT_TYPE) == ".otioz" From 75423d474df8b49c0cca8a7b6a5eb15f92210ccb Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 19:57:28 -0600 Subject: [PATCH 028/405] test(tasks): export_timeline content type check Co-Authored-By: Claude Opus 5.5 --- tests/test_timeline_export.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_timeline_export.py b/tests/test_timeline_export.py index edc942c70..d9a9e8275 100644 --- a/tests/test_timeline_export.py +++ b/tests/test_timeline_export.py @@ -216,4 +216,4 @@ def test_the_command_is_registered_with_an_otioz_content_type(): assert info is not None from dw.content_types import guess_extension - assert guess_extension(te.OTIOZ_CONTENT_TYPE) == ".otioz" + assert guess_extension("application/x-otioz") == ".otioz" From 3fedddbfa5ad24320c50a9d3071115eae579d267 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 20:08:00 -0600 Subject: [PATCH 029/405] feat(tasks): split_takes and the long narration and music templates Co-Authored-By: Claude Opus 5.5 --- docs/TASKS.md | 32 ++++ dw/tasks/task.py | 1 + dw/tasks/text_takes.py | 156 ++++++++++++++++++++ tests/test_catalog_structure.py | 9 +- tests/test_step_cache.py | 6 + tests/test_text_takes.py | 84 +++++++++++ workflows/templates/minimax/music-long.json | 124 ++++++++++++++++ workflows/templates/narration-long.json | 100 +++++++++++++ workflows/templates/split-takes.json | 27 ++++ 9 files changed, 538 insertions(+), 1 deletion(-) create mode 100644 dw/tasks/text_takes.py create mode 100644 tests/test_text_takes.py create mode 100644 workflows/templates/minimax/music-long.json create mode 100644 workflows/templates/narration-long.json create mode 100644 workflows/templates/split-takes.json diff --git a/docs/TASKS.md b/docs/TASKS.md index 0901626a6..3aafccc8c 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -3041,6 +3041,38 @@ number (the reference resolved to something other than text); and a list left with nothing once empty parts are dropped. The parts are positional; there is no named form. +## Splitting Text into Takes + +`split_takes` cuts a long text into takes a speech model can say in one go +(`dw/tasks/text_takes.py`). Audio8-TTS holds about 95 s of reference, text and +speech together, so a narration longer than that is spoken take by take and +joined (`templates/narration-long`). + +```json +{ + "name": "split", + "task": { + "command": "split_takes", + "arguments": { "text": "variable:text", "max_seconds": 60, "words_per_second": 2.5 } + }, + "result": { "content_type": "application/json" } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `text` | Yes | The text to speak | +| `max_seconds` | No | The longest a take may run (default 60: the ~95 s voice-clone context less the reference clip) | +| `words_per_second` | No | The rate that turns the limit into a word count (default 2.5; written dialogue measured 2.5 to 4 words a second as filling a clip) | + +It answers `{takes, count, words}`. A take ends at a sentence boundary; a +sentence that alone is over the limit is cut at its last comma that fits, else +at a word, and the run warns. The takes rejoin to the text, modulo whitespace. +The engine expands `for_each` before a run, so the list cannot feed a +`for_each` in the same workflow: run `templates/split-takes`, then pass its +`takes` to `templates/narration-long`. Refused: empty text, a limit that is not +above zero. + ## Extracting Sections Reduce generated text to a known set of labelled sections, dropping anything else: diff --git a/dw/tasks/task.py b/dw/tasks/task.py index 972bf2ca3..3e11590f2 100644 --- a/dw/tasks/task.py +++ b/dw/tasks/task.py @@ -23,6 +23,7 @@ from . import phrases # noqa: F401 - registers analyze_phrases from . import cuts # noqa: F401 - registers plan_cuts from . import trim # noqa: F401 - registers trim_video +from . import text_takes # noqa: F401 - registers split_takes # The handlers by family, split out of this module when it outgrew the size # ceiling (#790). Each registers its commands on import, so these lines are diff --git a/dw/tasks/text_takes.py b/dw/tasks/text_takes.py new file mode 100644 index 000000000..ac955b0c9 --- /dev/null +++ b/dw/tasks/text_takes.py @@ -0,0 +1,156 @@ +"""split_takes: cut a long text into takes a speech model can say in one go. + +A TTS model has a context, not a stamina: Audio8-TTS holds about 95 s of +reference, text and speech together (`templates/voice-clone`), and a line past +that is refused. A narration longer than one take is therefore spoken take by +take under one voice and seed and joined (`templates/narration-long`). + +Where a text is cut decides whether the join can be heard. A cut between two +sentences lands on a pause the speaker would have made anyway; a cut inside one +puts a breath where there is none. So takes end at sentence boundaries, and only +a sentence that alone is over the limit is cut inside - at its last comma that +fits, else at a word. + +The limit is a duration turned into words with `words_per_second`, an estimate +(written dialogue measured 2.5-4 words/s as filling a clip) - the first take of +a run is the check on it, not this function. +""" + +import logging +import math +import re + +logger = logging.getLogger("dw") + +# A sentence ends at . ! ? or an ellipsis, optionally closed by quotes or +# brackets, and is followed by whitespace +_SENTENCE_END = re.compile(r"(?<=[.!?…])[\"'”’)\]]*\s+") + + +def _words_allowed(max_seconds, words_per_second): + for name, value in ( + ("max_seconds", max_seconds), + ("words_per_second", words_per_second), + ): + if ( + isinstance(value, bool) + or not isinstance(value, (int, float)) + or not math.isfinite(value) + or value <= 0 + ): + raise ValueError(f"split_takes: {name} must be above zero, got {value!r}") + return max(1, int(max_seconds * words_per_second)) + + +def _sentences(text): + """The sentences of the text, each with its whitespace collapsed.""" + parts, start = [], 0 + for match in _SENTENCE_END.finditer(text): + # Keep the closing quote with the sentence it closes + parts.append(text[start : match.start() + len(match.group().rstrip())]) + start = match.end() + parts.append(text[start:]) + return [" ".join(p.split()) for p in parts if p.strip()] + + +def _cut_long(sentence, limit): + """A sentence over the limit as pieces within it: cut after the last comma + that fits, else at the limit.""" + pieces, words = [], sentence.split() + while len(words) > limit: + cut = limit + for index in range(limit - 1, 0, -1): + if words[index].endswith((",", ";", ":", "—")): + cut = index + 1 + break + pieces.append(" ".join(words[:cut])) + words = words[cut:] + pieces.append(" ".join(words)) + return pieces + + +def split_takes(text, max_seconds=60.0, words_per_second=2.5): + """Task command: split a text into takes of at most `max_seconds` of speech. + + Args: + text: The text to speak + max_seconds: The longest a take may run. For `voice-clone` this is the + ~95 s shared context less the reference clip, so 60 leaves room + words_per_second: The speaking rate used to turn the limit into words + + Returns: + A list of takes, in order. Joined with a space they read as the text, + less its original whitespace + + Raises: + ValueError: If the text is empty or a limit is not above zero + """ + limit = _words_allowed(max_seconds, words_per_second) + if not isinstance(text, str) or not text.strip(): + raise ValueError("split_takes needs some text to split") + + units = [] + for sentence in _sentences(text): + if len(sentence.split()) > limit: + logger.warning( + f"split_takes: a sentence of {len(sentence.split())} words is over " + f"the {limit}-word limit and is cut inside" + ) + units.extend(_cut_long(sentence, limit)) + else: + units.append(sentence) + + takes, current, used = [], [], 0 + for unit in units: + size = len(unit.split()) + if current and used + size > limit: + takes.append(" ".join(current)) + current, used = [], 0 + current.append(unit) + used += size + if current: + takes.append(" ".join(current)) + return takes + + +def split_takes_task(text, max_seconds=60.0, words_per_second=2.5): + """Task command: split a text into takes, answered as a JSON document. + + The takes are a `for_each` list (`templates/narration-long`): the engine + expands `for_each` before a run, so run this first - `templates/split-takes` + - and pass its `takes` in. + + Args: + text: The text to speak + max_seconds: The longest a take may run + words_per_second: The speaking rate used to turn the limit into words + + Returns: + {takes, count, words}: the takes in order, how many, and each one's + word count + """ + takes = split_takes(text, max_seconds, words_per_second) + return { + "takes": takes, + "count": len(takes), + "words": [len(t.split()) for t in takes], + } + + +def _register(): + from .registry import register_command + from ..task_domains import POSITIVE + + @register_command( + "split_takes", + implementation="dw.tasks.text_takes.split_takes_task", + returns="json", + domains={"max_seconds": POSITIVE, "words_per_second": POSITIVE}, + ) + def _handle_split_takes(task, arguments, previous_pipelines): + """Split a long text into takes a speech model can say in one go""" + logger.debug("Splitting takes") + return split_takes_task(**arguments) + + +_register() diff --git a/tests/test_catalog_structure.py b/tests/test_catalog_structure.py index 61ea03128..9675ff137 100644 --- a/tests/test_catalog_structure.py +++ b/tests/test_catalog_structure.py @@ -132,6 +132,7 @@ def load(path): "workflows/templates/attribute-lines.json", "workflows/templates/audio-trim-fade.json", "workflows/templates/check-script.json", + "workflows/templates/split-takes.json", "workflows/templates/face-drift-check.json", "workflows/templates/image-processors.json", "workflows/templates/minimax/music-video-cuts.json", @@ -323,6 +324,8 @@ def test_per_entry_problems_reports_a_mismatched_entries_count(): "workflows/templates/minimax/last-frame-only.json": {"image"}, # a 'result' field the modular pipeline needs declared "workflows/templates/minimax/music.json": {"sample_rate"}, + # fields of a 'sections' list entry, not this workflow's own variables + "workflows/templates/minimax/music-long.json": {"audio_duration", "lyrics"}, # pair_audio's 'fit' argument and its value 'video', and the per-entry # 'num_frames' of a shot "workflows/templates/minimax/music-video.json": {"fit", "num_frames", "video"}, @@ -552,7 +555,11 @@ def test_no_stale_entry_in_the_allowlist(): # since the template loads no model and only encodes on the CPU, so neither # entry carries a cost block now. Re-measured at 12_780 after that and # face-drift-check's `samples_per_shot`, so the budget stands. -COMPACT_BUDGET = 12_800 +# Then to 13_150 for `narration-long`, `split-takes` and `minimax/music-long` +# (long-form media, task 22, 2026-11-03), measured at 13_112: three new +# entries, the catalog's only long narration, its text splitter and its +# multi-section music, with no cost blocks yet. +COMPACT_BUDGET = 13_150 FILTERED_BUDGET = 1_500 diff --git a/tests/test_step_cache.py b/tests/test_step_cache.py index 33424a48b..7e4ab7530 100644 --- a/tests/test_step_cache.py +++ b/tests/test_step_cache.py @@ -683,6 +683,8 @@ def test_normalized_downstream_false_when_nothing_references_it(): "text_to_image": "af1014c287ace2f22907a6a91d05107d5874f22f6aeb066c58c639c0243d6a66" }, "generate-speech.json": {}, + "narration-long.json": {}, + "split-takes.json": {}, "image-edit.json": { "edit": "241e4cc77f00d837873d01e3ea9ec7105513af8362908d8229c740cbca997737" }, @@ -849,6 +851,10 @@ def test_normalized_downstream_false_when_nothing_references_it(): "minimax/music.json": { "generate_music": "1abd10e3c899fb5bbc490baa64467ed46d5005f78fa8f94bc777f3ddcaf81fb5" }, + "minimax/music-long.json": { + "generate_section@intro": "1abd10e3c899fb5bbc490baa64467ed46d5005f78fa8f94bc777f3ddcaf81fb5", + "generate_section@chorus": "1abd10e3c899fb5bbc490baa64467ed46d5005f78fa8f94bc777f3ddcaf81fb5", + }, "minimax/reference-to-video.json": { "reference_to_video_audio": "ba9915580e6a8017d8c68efbddea776985d107222546f84cca18e319298243bf" }, diff --git a/tests/test_text_takes.py b/tests/test_text_takes.py new file mode 100644 index 000000000..ea43f55fa --- /dev/null +++ b/tests/test_text_takes.py @@ -0,0 +1,84 @@ +"""split_takes: a long text into takes a speech model can say in one go.""" + +import re + +import pytest + +from dw.tasks.text_takes import split_takes + +SENTENCES = [ + "Somebody finished the pistachio and nobody is saying anything.", + "The fridge hums.", + "Nobody looks up from their phone, and nobody admits to the crime.", + "It was a quiet kind of guilt.", + "Eventually someone laughs.", +] +TEXT = " ".join(SENTENCES * 4) + + +def words(text): + return len(text.split()) + + +def test_no_take_is_over_the_limit(): + takes = split_takes(TEXT, max_seconds=10, words_per_second=2.5) + assert len(takes) > 1 + assert all(words(t) <= 25 for t in takes) + + +def test_takes_end_on_sentence_boundaries(): + for take in split_takes(TEXT, max_seconds=10, words_per_second=2.5): + assert re.search(r"[.!?]$", take) + + +def test_takes_rejoin_to_the_text(): + takes = split_takes(TEXT, max_seconds=10, words_per_second=2.5) + assert " ".join(takes).split() == TEXT.split() + + +def test_short_text_is_one_take(): + assert split_takes("Hello there.", 60, 2.5) == ["Hello there."] + + +def test_a_long_sentence_splits_at_its_last_comma_before_the_limit(): + sentence = "one two three, four five six, seven eight nine ten eleven twelve." + takes = split_takes(sentence, max_seconds=4, words_per_second=2.5) # 10 words + assert takes[0] == "one two three, four five six," + assert " ".join(takes).split() == sentence.split() + assert all(words(t) <= 10 for t in takes) + + +def test_a_long_sentence_without_commas_splits_at_a_word(): + sentence = " ".join(f"w{i}" for i in range(25)) + "." + takes = split_takes(sentence, max_seconds=4, words_per_second=2.5) + assert all(words(t) <= 10 for t in takes) + assert " ".join(takes).split() == sentence.split() + + +def test_paragraph_breaks_are_whitespace(): + takes = split_takes("First one.\n\nSecond one.", 60, 2.5) + assert " ".join(takes).split() == ["First", "one.", "Second", "one."] + + +@pytest.mark.parametrize("bad", [0, -1, float("nan")]) +def test_bad_limits_are_refused(bad): + with pytest.raises(ValueError): + split_takes("Hi.", max_seconds=bad, words_per_second=2.5) + with pytest.raises(ValueError): + split_takes("Hi.", max_seconds=10, words_per_second=bad) + + +def test_empty_text_is_refused(): + with pytest.raises(ValueError): + split_takes(" ", 60, 2.5) + + +def test_the_registered_task_answers_the_takes(): + import dw.tasks.task # noqa: F401 + from dw.tasks.registry import _COMMAND_INFO + + assert _COMMAND_INFO["split_takes"]["returns"] == "json" + from dw.tasks.text_takes import split_takes_task + + out = split_takes_task(TEXT, max_seconds=10) + assert out["count"] == len(out["takes"]) > 1 diff --git a/workflows/templates/minimax/music-long.json b/workflows/templates/minimax/music-long.json new file mode 100644 index 000000000..f1d06501f --- /dev/null +++ b/workflows/templates/minimax/music-long.json @@ -0,0 +1,124 @@ +{ + "id": "MiniMaxMusicLong", + "description": "A piece longer than one Music 3 track, written as sections and joined. Music 3 is practical to about 300 s (stay at or under it); a longer piece is generated section by section, each section trimmed to its length, faded out a little, and crossfaded into the next. 'sections' is a list of {name, caption, lyrics, seconds, audio_duration}: 'caption' is that section's Music 3 prompt, 'lyrics' its tagged lyrics (empty for an instrumental), 'seconds' the length kept, and 'audio_duration' the generation ceiling - 1.5 times 'seconds', per the Music 3 skill, because the ceiling is not a target and a ceiling set at the wanted length cuts the end off and stretches the caption; the engine does no arithmetic, so write both. The loaded model is reused across sections. The sections are generated independently: section-to-section style continuity is NOT guaranteed - key, tempo, instrumentation and mix can all change at a seam, so repeat the same style tags in every caption, keep each section's caption describing the same band, and expect to listen at the joins. 'crossfade_ms' (1500 by default) overlaps the end of one section with the start of the next, which hides a change of sound better than a short fade but cannot hide a change of tempo; 'section_fade_out_ms' (400) rounds each section's trimmed end. A beat-aligned join through analyze_beats is not attempted. The joined piece is peak-normalized to -3 dBFS (Music 3 lands at or over full scale; see 'minimax/music'). Cost grows with the sum of the sections' 'audio_duration'.", + "cost_drivers": [ + "sections" + ], + "variables": { + "sections": [ + { + "name": "intro", + "caption": "warm acoustic folk, fingerpicked guitar, soft brushed drums, intimate male vocal, 90 BPM", + "lyrics": "[verse]\nMorning comes in slow across the field\nEverything I carried finally healed", + "seconds": 30, + "audio_duration": 45 + }, + { + "name": "chorus", + "caption": "warm acoustic folk, fingerpicked guitar, soft brushed drums, intimate male vocal, 90 BPM, a lift into the chorus", + "lyrics": "[chorus]\nLet it rise, let it rise\nGold along the skyline", + "seconds": 30, + "audio_duration": 45 + } + ], + "crossfade_ms": 1500, + "section_fade_out_ms": 400, + "seed": null + }, + "seed": "variable:seed", + "steps": [ + { + "name": "generate_section", + "for_each": "variable:sections", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "load_components": { + "dtype": "torch.bfloat16" + }, + "components_manager": { + "enable_auto_cpu_offload": true + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-Music3" + }, + "arguments": { + "prompt": "item:caption", + "lyrics": "item:lyrics", + "audio_duration": "item:audio_duration", + "output": "audios" + } + }, + "result": { + "content_type": "audio/mp3", + "sample_rate": 44100, + "save": false + } + }, + { + "name": "trim", + "for_each": "variable:sections", + "task": { + "command": "slice_audio", + "arguments": { + "audio": "previous_result:generate_section", + "start_seconds": 0, + "duration_seconds": "item:seconds" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "fade", + "for_each": "variable:sections", + "task": { + "command": "fade_audio", + "arguments": { + "audio": "previous_result:trim", + "fade_in_ms": 0, + "fade_out_ms": "variable:section_fade_out_ms" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "joined", + "task": { + "command": "crossfade_audio", + "arguments": { + "audios": "gather:fade", + "crossfade_ms": "variable:crossfade_ms", + "sample_rate": 44100 + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "balanced", + "task": { + "command": "normalize_audio", + "arguments": { + "audio": "previous_result:joined", + "peak_dbfs": -3.0, + "sample_rate": 44100 + } + }, + "result": { + "content_type": "audio/mp3", + "sample_rate": 44100, + "subfolder": "final" + } + } + ], + "summary": "A piece longer than one Music 3 track, written as sections and crossfaded together." +} diff --git a/workflows/templates/narration-long.json b/workflows/templates/narration-long.json new file mode 100644 index 000000000..6f52c741b --- /dev/null +++ b/workflows/templates/narration-long.json @@ -0,0 +1,100 @@ +{ + "id": "NarrationLong", + "description": "A narration longer than one take, spoken in one cloned voice and joined without audible seams. Audio8-TTS holds about 95 s of reference, text and speech together (see 'voice-clone'), so a long text is cut into takes at sentence boundaries, each take is spoken with 'voice-clone's own generate_speech arguments under one 'reference_audio', 'reference_text' and 'seed', and the takes are joined in order. 'takes' is a list of strings, and it is a variable rather than a step because the engine expands 'for_each' before a run starts, so the cut has to exist first: run 'templates/split-takes' (the 'split_takes' task) on the text - 60 s a take by default, 2.5 words a second, so a take is about 150 words - and pass its 'takes' here. The default is three short takes to show the shape. Each take is followed by 'gap_seconds' of silence (0.3 by default, a first guess to be tuned by ear rather than a measured pause) cut to the take's own sample rate, and 'crossfade_ms' (30 by default) is the overlap at every join: the crossfade also overlaps each take into its gap, so the audible pause is about gap_seconds less one crossfade; the last take keeps its trailing gap, and 'slice_audio' trims it if it matters. The same seed does not make takes sound alike by itself - a different text is a different draw - so listen for a voice that drifts between takes and split into shorter takes if one does; a seam is also where a level step shows, so check the joined file's loudness before relying on it. Consent: clone only a voice you have the right to use - your own, or one whose owner has agreed - and never to impersonate or deceive. There is no default voice: 'reference_audio' (an asset: reference, at most 20 s) and 'reference_text' (exactly what is said in it) are required together, as in 'voice-clone'.", + "variables": { + "takes": [ + "Somebody finished the pistachio and nobody is saying anything.", + "The fridge hums, and nobody looks up from their phone.", + "It was a quiet kind of guilt." + ], + "reference_audio": "asset:voice.wav", + "reference_text": "The words spoken in the reference clip, exactly as said.", + "seed": null, + "gap_seconds": 0.3, + "crossfade_ms": 30 + }, + "seed": "variable:seed", + "steps": [ + { + "name": "speak", + "for_each": "variable:takes", + "task": { + "command": "generate_speech", + "arguments": { + "text": "item:", + "model_name": "Edge0/Audio8-TTS-Realistic-0.6b", + "reference_audio": "variable:reference_audio", + "reference_text": "variable:reference_text" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "silence_slice", + "for_each": "variable:takes", + "task": { + "command": "slice_audio", + "arguments": { + "audio": "previous_result:speak", + "start_seconds": 0, + "duration_seconds": "variable:gap_seconds" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "silence", + "for_each": "variable:takes", + "task": { + "command": "gain_audio", + "arguments": { + "audio": "previous_result:silence_slice", + "gain_db": -120 + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "gapped", + "for_each": "variable:takes", + "task": { + "command": "crossfade_audio", + "arguments": { + "audios": [ + "previous_result:speak", + "previous_result:silence" + ], + "crossfade_ms": "variable:crossfade_ms" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "narration", + "task": { + "command": "crossfade_audio", + "arguments": { + "audios": "gather:gapped", + "crossfade_ms": "variable:crossfade_ms" + } + }, + "result": { + "content_type": "audio/wav", + "subfolder": "final" + } + } + ], + "summary": "A narration longer than one take, spoken in one cloned voice and joined take by take." +} diff --git a/workflows/templates/split-takes.json b/workflows/templates/split-takes.json new file mode 100644 index 000000000..0b62efa26 --- /dev/null +++ b/workflows/templates/split-takes.json @@ -0,0 +1,27 @@ +{ + "id": "SplitTakes", + "description": "Cuts a long text into takes a speech model can say in one go, at sentence boundaries, and answers them as JSON: {takes, count, words}. Run it before 'narration-long', whose list of takes is the list it answers - the engine expands a 'for_each' before a run, so the list cannot come from a step of the same workflow. A take ends at a sentence boundary unless one sentence alone is over the limit, which is cut at its last comma that fits, else at a word. 'max_seconds' is the longest a take may run (60 keeps a voice-clone take, its reference and its speech inside Audio8-TTS's ~95 s context); 'words_per_second' turns it into words (2.5 default; written dialogue measured 2.5 to 4 words a second as filling a clip, so a text that is slow to say wants a lower figure).", + "variables": { + "text": "Somebody finished the pistachio and nobody is saying anything. The fridge hums. Nobody looks up from their phone, and nobody admits to the crime.", + "max_seconds": 60, + "words_per_second": 2.5 + }, + "steps": [ + { + "name": "split", + "task": { + "command": "split_takes", + "arguments": { + "text": "variable:text", + "max_seconds": "variable:max_seconds", + "words_per_second": "variable:words_per_second" + } + }, + "result": { + "content_type": "application/json", + "subfolder": "final" + } + } + ], + "summary": "Cuts a long text into speakable takes at sentence boundaries, answered as JSON." +} From 8e69a2342de1db4ee5cebc1e49a80e6038cacb04 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 20:08:26 -0600 Subject: [PATCH 030/405] bench: for-don notes (listing budget, module count) Co-Authored-By: Claude Opus 5.5 --- bench/records/for-don.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/bench/records/for-don.md b/bench/records/for-don.md index 5b35a495e..6eff79d4c 100644 --- a/bench/records/for-don.md +++ b/bench/records/for-don.md @@ -3,3 +3,5 @@ Things only you can check. - [ ] Import one exported `.otioz` (`export_timeline`, #1197) in DaVinci Resolve 18.5+ once a bench cut exists: File > Import > Timeline, then check one clip per shot, the rate (23.976 as 24000/1001) and that each clip plays its span. Nothing here can run Resolve. +- Compact catalog listing budget raised to 13_150 (tests) for the new split-takes, narration-long and music-long templates. Recommendation: keep (discovery). +- Arch ratchet module count grows on feat/long-form-media (presets, chain_pick, chain_look, chain_check, timeline_export, text_takes, ...). Re-baseline at merge needs your ok. Recommendation: approve. From 5fe98c7816908eacf4f6927f3d3558c7735c53cf Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 20:14:25 -0600 Subject: [PATCH 031/405] fix(tasks): one-call long narration from text; abbreviations are not sentence ends narration-long takes plain text and runs narration-from-takes as a child workflow; abbreviations and initials no longer end a sentence; music-long refuses a section audio_duration over 300. Co-Authored-By: Claude Opus 5.5 --- bench/records/for-don.md | 2 +- docs/TASKS.md | 5 +- dw/tasks/text_takes.py | 28 ++++- tests/test_catalog_structure.py | 6 +- tests/test_step_cache.py | 1 + tests/test_text_takes.py | 114 ++++++++++++++++++ workflows/templates/minimax/music-long.json | 8 +- workflows/templates/narration-from-takes.json | 100 +++++++++++++++ workflows/templates/narration-long.json | 87 +++---------- workflows/templates/split-takes.json | 2 +- 10 files changed, 275 insertions(+), 78 deletions(-) create mode 100644 workflows/templates/narration-from-takes.json diff --git a/bench/records/for-don.md b/bench/records/for-don.md index 6eff79d4c..ad4ebc517 100644 --- a/bench/records/for-don.md +++ b/bench/records/for-don.md @@ -3,5 +3,5 @@ Things only you can check. - [ ] Import one exported `.otioz` (`export_timeline`, #1197) in DaVinci Resolve 18.5+ once a bench cut exists: File > Import > Timeline, then check one clip per shot, the rate (23.976 as 24000/1001) and that each clip plays its span. Nothing here can run Resolve. -- Compact catalog listing budget raised to 13_150 (tests) for the new split-takes, narration-long and music-long templates. Recommendation: keep (discovery). +- Compact catalog listing budget raised to 13_250 (tests) for the new split-takes, narration-long, narration-from-takes and music-long templates. Recommendation: keep (discovery). - Arch ratchet module count grows on feat/long-form-media (presets, chain_pick, chain_look, chain_check, timeline_export, text_takes, ...). Re-baseline at merge needs your ok. Recommendation: approve. diff --git a/docs/TASKS.md b/docs/TASKS.md index 3aafccc8c..5813e1ae4 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -3069,8 +3069,9 @@ It answers `{takes, count, words}`. A take ends at a sentence boundary; a sentence that alone is over the limit is cut at its last comma that fits, else at a word, and the run warns. The takes rejoin to the text, modulo whitespace. The engine expands `for_each` before a run, so the list cannot feed a -`for_each` in the same workflow: run `templates/split-takes`, then pass its -`takes` to `templates/narration-long`. Refused: empty text, a limit that is not +`for_each` in the same workflow. `templates/narration-long` takes the plain +text, runs this task, and hands the takes to `templates/narration-from-takes` +as a child workflow, which expands its own `for_each`. Refused: empty text, a limit that is not above zero. ## Extracting Sections diff --git a/dw/tasks/text_takes.py b/dw/tasks/text_takes.py index ac955b0c9..34ce7dde7 100644 --- a/dw/tasks/text_takes.py +++ b/dw/tasks/text_takes.py @@ -27,6 +27,25 @@ _SENTENCE_END = re.compile(r"(?<=[.!?…])[\"'”’)\]]*\s+") +# A period after one of these, or after a single capital letter (an initial), +# is not the end of a sentence +_ABBREVIATIONS = frozenset( + "mr mrs ms dr prof st sr jr vs etc e.g i.e mt no fig".split() +) + + +def _is_abbreviation(before): + """Whether the text before a boundary ends in an abbreviation or initial.""" + tokens = before.rstrip().rsplit(None, 1) + token = tokens[-1] if tokens else "" + if not token.endswith("."): + return False + word = token[:-1].lstrip("\"'“‘([").lower() + return word in _ABBREVIATIONS or ( + len(word) == 1 and word.isalpha() and token[-2].isupper() + ) + + def _words_allowed(max_seconds, words_per_second): for name, value in ( ("max_seconds", max_seconds), @@ -46,6 +65,8 @@ def _sentences(text): """The sentences of the text, each with its whitespace collapsed.""" parts, start = [], 0 for match in _SENTENCE_END.finditer(text): + if _is_abbreviation(text[start : match.start()]): + continue # Keep the closing quote with the sentence it closes parts.append(text[start : match.start() + len(match.group().rstrip())]) start = match.end() @@ -116,9 +137,10 @@ def split_takes(text, max_seconds=60.0, words_per_second=2.5): def split_takes_task(text, max_seconds=60.0, words_per_second=2.5): """Task command: split a text into takes, answered as a JSON document. - The takes are a `for_each` list (`templates/narration-long`): the engine - expands `for_each` before a run, so run this first - `templates/split-takes` - - and pass its `takes` in. + The takes are a `for_each` list (`templates/narration-from-takes`): a + `for_each` list cannot come from a step of the same workflow, so + `templates/narration-long` runs this and hands the takes to a child + workflow, which expands its own `for_each`. Args: text: The text to speak diff --git a/tests/test_catalog_structure.py b/tests/test_catalog_structure.py index 9675ff137..b6bc33bd0 100644 --- a/tests/test_catalog_structure.py +++ b/tests/test_catalog_structure.py @@ -558,8 +558,10 @@ def test_no_stale_entry_in_the_allowlist(): # Then to 13_150 for `narration-long`, `split-takes` and `minimax/music-long` # (long-form media, task 22, 2026-11-03), measured at 13_112: three new # entries, the catalog's only long narration, its text splitter and its -# multi-section music, with no cost blocks yet. -COMPACT_BUDGET = 13_150 +# multi-section music, with no cost blocks yet. Then to 13_250 (measured +# 13_236) for `narration-from-takes`: `narration-long` now takes plain text and +# the take-list template keeps its own entry. +COMPACT_BUDGET = 13_250 FILTERED_BUDGET = 1_500 diff --git a/tests/test_step_cache.py b/tests/test_step_cache.py index 7e4ab7530..0dad01a5e 100644 --- a/tests/test_step_cache.py +++ b/tests/test_step_cache.py @@ -683,6 +683,7 @@ def test_normalized_downstream_false_when_nothing_references_it(): "text_to_image": "af1014c287ace2f22907a6a91d05107d5874f22f6aeb066c58c639c0243d6a66" }, "generate-speech.json": {}, + "narration-from-takes.json": {}, "narration-long.json": {}, "split-takes.json": {}, "image-edit.json": { diff --git a/tests/test_text_takes.py b/tests/test_text_takes.py index ea43f55fa..7ad3383cf 100644 --- a/tests/test_text_takes.py +++ b/tests/test_text_takes.py @@ -82,3 +82,117 @@ def test_the_registered_task_answers_the_takes(): out = split_takes_task(TEXT, max_seconds=10) assert out["count"] == len(out["takes"]) > 1 + + +def test_abbreviations_are_not_sentence_ends(): + takes = split_takes("Hi there now. Dr. Smith went home. Mr. Lee left.", 3, 1) + assert all(not t.endswith(("Dr.", "Mr.")) for t in takes) + takes = split_takes("see e.g. this one. Next.", 3, 1) + assert all(not t.endswith("e.g.") for t in takes) + + +def test_an_initial_is_not_a_sentence_end(): + from dw.tasks.text_takes import _sentences + + assert _sentences("J. R. Smith spoke.") == ["J. R. Smith spoke."] + assert _sentences("It ended. Then J. Smith spoke.") == [ + "It ended.", + "Then J. Smith spoke.", + ] + + +def test_narration_long_validates_and_names_its_child(): + import json + from pathlib import Path + + from dw.workflow import Workflow + + root = Path(__file__).resolve().parent.parent / "workflows" / "templates" + path = root / "narration-long.json" + definition = json.loads(path.read_text()) + assert Workflow(definition, str(root), str(path)).validation_errors() == [] + child = definition["steps"][1]["workflow"] + assert child["path"] == "./narration-from-takes.json" + assert child["arguments"]["takes"] == "previous_result:split.takes" + assert (root / "narration-from-takes.json").exists() + + +def test_a_child_receives_the_parents_takes(tmp_path): + import json + + from dw.workflow import Workflow + + child = { + "id": "child", + "variables": {"takes": ["x"]}, + "steps": [ + { + "name": "each", + "for_each": "variable:takes", + "task": { + "command": "split_takes", + "arguments": {"text": "item:"}, + }, + "result": {"content_type": "application/json", "save": False}, + } + ], + } + (tmp_path / "child.json").write_text(json.dumps(child)) + parent = { + "id": "parent", + "variables": {"text": "One short. Two short. Three short."}, + "steps": [ + { + "name": "split", + "task": { + "command": "split_takes", + "arguments": { + "text": "variable:text", + "max_seconds": 2, + "words_per_second": 1, + }, + }, + "result": {"content_type": "application/json", "save": False}, + }, + { + "name": "run", + "workflow": { + "path": "./child.json", + "arguments": {"takes": "previous_result:split.takes"}, + }, + "result": {"content_type": "application/json", "save": False}, + }, + ], + } + workflow = Workflow(parent, str(tmp_path), str(tmp_path / "parent.json")) + assert workflow.validation_errors() == [] + seen = [] + from dw.workflow import Workflow as W + + original = W.run + + def spy(self, arguments, *a, **k): + seen.append(arguments) + return original(self, arguments, *a, **k) + + from unittest.mock import patch + + with patch.object(W, "run", spy): + workflow.run({}) + child_args = [s for s in seen if isinstance(s, dict) and "takes" in s] + assert child_args[0]["takes"] == ["One short.", "Two short.", "Three short."] + + +def test_music_long_refuses_a_section_over_300_seconds(): + import json + from pathlib import Path + + from dw.workflow import Workflow + + root = Path(__file__).resolve().parent.parent / "workflows" / "templates" + path = root / "minimax" / "music-long.json" + definition = json.loads(path.read_text()) + assert Workflow(definition, str(path.parent), str(path)).validation_errors() == [] + definition["variables"]["sections"][0]["audio_duration"] = 301 + errors = Workflow(definition, str(path.parent), str(path)).validation_errors() + assert any("300" in json.dumps(e) for e in errors), errors diff --git a/workflows/templates/minimax/music-long.json b/workflows/templates/minimax/music-long.json index f1d06501f..3e2491eca 100644 --- a/workflows/templates/minimax/music-long.json +++ b/workflows/templates/minimax/music-long.json @@ -1,6 +1,6 @@ { "id": "MiniMaxMusicLong", - "description": "A piece longer than one Music 3 track, written as sections and joined. Music 3 is practical to about 300 s (stay at or under it); a longer piece is generated section by section, each section trimmed to its length, faded out a little, and crossfaded into the next. 'sections' is a list of {name, caption, lyrics, seconds, audio_duration}: 'caption' is that section's Music 3 prompt, 'lyrics' its tagged lyrics (empty for an instrumental), 'seconds' the length kept, and 'audio_duration' the generation ceiling - 1.5 times 'seconds', per the Music 3 skill, because the ceiling is not a target and a ceiling set at the wanted length cuts the end off and stretches the caption; the engine does no arithmetic, so write both. The loaded model is reused across sections. The sections are generated independently: section-to-section style continuity is NOT guaranteed - key, tempo, instrumentation and mix can all change at a seam, so repeat the same style tags in every caption, keep each section's caption describing the same band, and expect to listen at the joins. 'crossfade_ms' (1500 by default) overlaps the end of one section with the start of the next, which hides a change of sound better than a short fade but cannot hide a change of tempo; 'section_fade_out_ms' (400) rounds each section's trimmed end. A beat-aligned join through analyze_beats is not attempted. The joined piece is peak-normalized to -3 dBFS (Music 3 lands at or over full scale; see 'minimax/music'). Cost grows with the sum of the sections' 'audio_duration'.", + "description": "A piece longer than one Music 3 track, written as sections and joined. Music 3 is practical to about 300 s (stay at or under it); a longer piece is generated section by section, each section trimmed to its length, faded out a little, and crossfaded into the next. 'sections' is a list of {name, caption, lyrics, seconds, audio_duration}: 'caption' is that section's Music 3 prompt, 'lyrics' its tagged lyrics (empty for an instrumental), 'seconds' the length kept, and 'audio_duration' the generation ceiling - 1.5 times 'seconds', per the Music 3 skill, because the ceiling is not a target and a ceiling set at the wanted length cuts the end off and stretches the caption; the engine does no arithmetic, so write both. The loaded model is reused across sections. The sections are generated independently: section-to-section style continuity is NOT guaranteed - key, tempo, instrumentation and mix can all change at a seam, so repeat the same style tags in every caption, keep each section's caption describing the same band, and expect to listen at the joins. 'crossfade_ms' (1500 by default) overlaps the end of one section with the start of the next, which hides a change of sound better than a short fade but cannot hide a change of tempo; 'section_fade_out_ms' (400) rounds each section's trimmed end. A beat-aligned join through analyze_beats is not attempted. The joined piece is peak-normalized to -3 dBFS (Music 3 lands at or over full scale; see 'minimax/music'). The joined length is the sum of the sections' 'seconds' minus one crossfade per join. Cost grows with the sum of the sections' 'audio_duration'.", "cost_drivers": [ "sections" ], @@ -26,6 +26,12 @@ "seed": null }, "seed": "variable:seed", + "variable_constraints": { + "audio_duration": { + "max_frames": 300, + "reason": "Music 3 is practical to about 300 s; a section's generation ceiling over that is refused before the model loads (max_frames is the constraint block's plain upper bound, not a frame count here)" + } + }, "steps": [ { "name": "generate_section", diff --git a/workflows/templates/narration-from-takes.json b/workflows/templates/narration-from-takes.json new file mode 100644 index 000000000..c2a5a377d --- /dev/null +++ b/workflows/templates/narration-from-takes.json @@ -0,0 +1,100 @@ +{ + "id": "NarrationFromTakes", + "description": "A narration longer than one take, spoken in one cloned voice and joined without audible seams. Audio8-TTS holds about 95 s of reference, text and speech together (see 'voice-clone'), so a long text is cut into takes at sentence boundaries, each take is spoken with 'voice-clone's own generate_speech arguments under one 'reference_audio', 'reference_text' and 'seed', and the takes are joined in order. 'takes' is a list of strings, and it is a variable rather than a step because the engine expands 'for_each' before a run starts, so the list cannot come from a step of the same workflow: for a plain text use 'narration-long', which cuts the text with 'split_takes' and runs this template as a child workflow with the takes (a child expands its own 'for_each' at its own run). The default is three short takes to show the shape. Each take is followed by 'gap_seconds' of silence (0.3 by default, a first guess to be tuned by ear rather than a measured pause) cut to the take's own sample rate, and 'crossfade_ms' (30 by default) is the overlap at every join: the crossfade also overlaps each take into its gap, so the audible pause is about gap_seconds less one crossfade; the last take keeps its trailing gap, and 'slice_audio' trims it if it matters. The same seed does not make takes sound alike by itself - a different text is a different draw - so listen for a voice that drifts between takes and split into shorter takes if one does; a seam is also where a level step shows, so check the joined file's loudness before relying on it. Consent: clone only a voice you have the right to use - your own, or one whose owner has agreed - and never to impersonate or deceive. There is no default voice: 'reference_audio' (an asset: reference, at most 20 s) and 'reference_text' (exactly what is said in it) are required together, as in 'voice-clone'.", + "variables": { + "takes": [ + "Somebody finished the pistachio and nobody is saying anything.", + "The fridge hums, and nobody looks up from their phone.", + "It was a quiet kind of guilt." + ], + "reference_audio": "asset:voice.wav", + "reference_text": "The words spoken in the reference clip, exactly as said.", + "seed": null, + "gap_seconds": 0.3, + "crossfade_ms": 30 + }, + "seed": "variable:seed", + "steps": [ + { + "name": "speak", + "for_each": "variable:takes", + "task": { + "command": "generate_speech", + "arguments": { + "text": "item:", + "model_name": "Edge0/Audio8-TTS-Realistic-0.6b", + "reference_audio": "variable:reference_audio", + "reference_text": "variable:reference_text" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "silence_slice", + "for_each": "variable:takes", + "task": { + "command": "slice_audio", + "arguments": { + "audio": "previous_result:speak", + "start_seconds": 0, + "duration_seconds": "variable:gap_seconds" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "silence", + "for_each": "variable:takes", + "task": { + "command": "gain_audio", + "arguments": { + "audio": "previous_result:silence_slice", + "gain_db": -120 + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "gapped", + "for_each": "variable:takes", + "task": { + "command": "crossfade_audio", + "arguments": { + "audios": [ + "previous_result:speak", + "previous_result:silence" + ], + "crossfade_ms": "variable:crossfade_ms" + } + }, + "result": { + "content_type": "audio/wav", + "save": false + } + }, + { + "name": "narration", + "task": { + "command": "crossfade_audio", + "arguments": { + "audios": "gather:gapped", + "crossfade_ms": "variable:crossfade_ms" + } + }, + "result": { + "content_type": "audio/wav", + "subfolder": "final" + } + } + ], + "summary": "A narration longer than one take, spoken in one cloned voice and joined take by take." +} diff --git a/workflows/templates/narration-long.json b/workflows/templates/narration-long.json index 6f52c741b..efaf4d4d4 100644 --- a/workflows/templates/narration-long.json +++ b/workflows/templates/narration-long.json @@ -1,12 +1,10 @@ { "id": "NarrationLong", - "description": "A narration longer than one take, spoken in one cloned voice and joined without audible seams. Audio8-TTS holds about 95 s of reference, text and speech together (see 'voice-clone'), so a long text is cut into takes at sentence boundaries, each take is spoken with 'voice-clone's own generate_speech arguments under one 'reference_audio', 'reference_text' and 'seed', and the takes are joined in order. 'takes' is a list of strings, and it is a variable rather than a step because the engine expands 'for_each' before a run starts, so the cut has to exist first: run 'templates/split-takes' (the 'split_takes' task) on the text - 60 s a take by default, 2.5 words a second, so a take is about 150 words - and pass its 'takes' here. The default is three short takes to show the shape. Each take is followed by 'gap_seconds' of silence (0.3 by default, a first guess to be tuned by ear rather than a measured pause) cut to the take's own sample rate, and 'crossfade_ms' (30 by default) is the overlap at every join: the crossfade also overlaps each take into its gap, so the audible pause is about gap_seconds less one crossfade; the last take keeps its trailing gap, and 'slice_audio' trims it if it matters. The same seed does not make takes sound alike by itself - a different text is a different draw - so listen for a voice that drifts between takes and split into shorter takes if one does; a seam is also where a level step shows, so check the joined file's loudness before relying on it. Consent: clone only a voice you have the right to use - your own, or one whose owner has agreed - and never to impersonate or deceive. There is no default voice: 'reference_audio' (an asset: reference, at most 20 s) and 'reference_text' (exactly what is said in it) are required together, as in 'voice-clone'.", + "description": "A narration of any length from plain text, spoken in one cloned voice and joined without audible seams. Audio8-TTS holds about 95 s of reference, text and speech together (see 'voice-clone'), so the text is cut into takes at sentence boundaries by the 'split_takes' task (60 s a take by default, 2.5 words a second, so a take is about 150 words), and a child workflow, 'narration-from-takes', speaks each take under one 'reference_audio', 'reference_text' and 'seed' and joins them in order: the child expands its own 'for_each' over the takes it is handed. 'gap_seconds' of silence follows each take (0.3 by default, a first guess to be tuned by ear) and 'crossfade_ms' (30) is the overlap at every join; see 'narration-from-takes' for how the gap and the crossfade combine. Abbreviations such as Dr. and initials are not sentence ends. The same seed does not make takes sound alike by itself, so listen for a voice that drifts between takes and use a lower 'max_seconds' if one does; check the joined file's loudness before relying on it. Consent: clone only a voice you have the right to use - your own, or one whose owner has agreed - and never to impersonate or deceive. There is no default voice: 'reference_audio' (an asset: reference, at most 20 s) and 'reference_text' (exactly what is said in it) are required together, as in 'voice-clone'.", "variables": { - "takes": [ - "Somebody finished the pistachio and nobody is saying anything.", - "The fridge hums, and nobody looks up from their phone.", - "It was a quiet kind of guilt." - ], + "text": "Somebody finished the pistachio and nobody is saying anything. The fridge hums, and nobody looks up from their phone. It was a quiet kind of guilt.", + "max_seconds": 60, + "words_per_second": 2.5, "reference_audio": "asset:voice.wav", "reference_text": "The words spoken in the reference clip, exactly as said.", "seed": null, @@ -16,77 +14,30 @@ "seed": "variable:seed", "steps": [ { - "name": "speak", - "for_each": "variable:takes", + "name": "split", "task": { - "command": "generate_speech", + "command": "split_takes", "arguments": { - "text": "item:", - "model_name": "Edge0/Audio8-TTS-Realistic-0.6b", - "reference_audio": "variable:reference_audio", - "reference_text": "variable:reference_text" - } - }, - "result": { - "content_type": "audio/wav", - "save": false - } - }, - { - "name": "silence_slice", - "for_each": "variable:takes", - "task": { - "command": "slice_audio", - "arguments": { - "audio": "previous_result:speak", - "start_seconds": 0, - "duration_seconds": "variable:gap_seconds" + "text": "variable:text", + "max_seconds": "variable:max_seconds", + "words_per_second": "variable:words_per_second" } }, "result": { - "content_type": "audio/wav", + "content_type": "application/json", "save": false } }, { - "name": "silence", - "for_each": "variable:takes", - "task": { - "command": "gain_audio", + "name": "narrate", + "workflow": { + "path": "./narration-from-takes.json", "arguments": { - "audio": "previous_result:silence_slice", - "gain_db": -120 - } - }, - "result": { - "content_type": "audio/wav", - "save": false - } - }, - { - "name": "gapped", - "for_each": "variable:takes", - "task": { - "command": "crossfade_audio", - "arguments": { - "audios": [ - "previous_result:speak", - "previous_result:silence" - ], - "crossfade_ms": "variable:crossfade_ms" - } - }, - "result": { - "content_type": "audio/wav", - "save": false - } - }, - { - "name": "narration", - "task": { - "command": "crossfade_audio", - "arguments": { - "audios": "gather:gapped", + "takes": "previous_result:split.takes", + "reference_audio": "variable:reference_audio", + "reference_text": "variable:reference_text", + "seed": "variable:seed", + "gap_seconds": "variable:gap_seconds", "crossfade_ms": "variable:crossfade_ms" } }, @@ -96,5 +47,5 @@ } } ], - "summary": "A narration longer than one take, spoken in one cloned voice and joined take by take." + "summary": "A narration from plain text, cut into takes and spoken in one cloned voice." } diff --git a/workflows/templates/split-takes.json b/workflows/templates/split-takes.json index 0b62efa26..92e7ea616 100644 --- a/workflows/templates/split-takes.json +++ b/workflows/templates/split-takes.json @@ -1,6 +1,6 @@ { "id": "SplitTakes", - "description": "Cuts a long text into takes a speech model can say in one go, at sentence boundaries, and answers them as JSON: {takes, count, words}. Run it before 'narration-long', whose list of takes is the list it answers - the engine expands a 'for_each' before a run, so the list cannot come from a step of the same workflow. A take ends at a sentence boundary unless one sentence alone is over the limit, which is cut at its last comma that fits, else at a word. 'max_seconds' is the longest a take may run (60 keeps a voice-clone take, its reference and its speech inside Audio8-TTS's ~95 s context); 'words_per_second' turns it into words (2.5 default; written dialogue measured 2.5 to 4 words a second as filling a clip, so a text that is slow to say wants a lower figure).", + "description": "Cuts a long text into takes a speech model can say in one go, at sentence boundaries, and answers them as JSON: {takes, count, words}. 'narration-long' runs this task itself and speaks the takes, so reach for this template only to see or edit the takes first and then pass them to 'narration-from-takes' - the engine expands a 'for_each' before a run, so the list cannot come from a step of the same workflow. A take ends at a sentence boundary unless one sentence alone is over the limit, which is cut at its last comma that fits, else at a word. 'max_seconds' is the longest a take may run (60 keeps a voice-clone take, its reference and its speech inside Audio8-TTS's ~95 s context); 'words_per_second' turns it into words (2.5 default; written dialogue measured 2.5 to 4 words a second as filling a clip, so a text that is slow to say wants a lower figure).", "variables": { "text": "Somebody finished the pistachio and nobody is saying anything. The fridge hums. Nobody looks up from their phone, and nobody admits to the crime.", "max_seconds": 60, From 037f216e2629d8e01d2ad3050b96170e73ca2458 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 20:43:42 -0600 Subject: [PATCH 032/405] feat(h3): #1192 - an audio-only pass over a finished picture An H3 call with source_video and no mask and no hold_audio is now an audio pass: the source block encodes the source as for a regional edit, DwH3AudioPassStep (before denoise) widens num_condition_video_rows over every video row and pins them clean (t = 1.0) in the row plan, and DwH3ReleaseVideoStep (before after_denoise) puts the count back. No video row is ever stepped; the audio runs the stock audio schedule from full noise, attending to the real picture. The result plays the source's own frames (dw_held_video, swapped in by output_extraction like dw_held_audio), not the VAE round trip. listen_scale (1, 0.5, 0.25) encodes an area-resampled copy on a 32 px-snapped smaller canvas; num_frames, and so the audio length, stays the source's. Rules in h3_rules: is_audio_pass, and mask_problems refuses refine arguments on an audio pass, a listen_scale outside the three, and a listen_scale on any other step. A mask or hold_audio keeps today's meaning. The draft template is bench/workflows/add-sound.json (no LoRA, 12 steps, shifts 12/3), kept out of the catalog until Don listens. The look-hold and chain-check rules (family-independent) move from h3_rules.py to the new torch-free chain_rules.py, so h3_rules stays under the 1,100-line ceiling: the modules ratchet rises 249 -> 250 for it. The row-plan builder is shared (h3_hold_steps.row_timestep_plan) so the pass adds no duplicate block. Co-Authored-By: Claude Opus 5.5 --- bench/workflows/add-sound.json | 212 ++++++++ docs/ARCHITECTURE.md | 3 +- docs/ATTRIBUTIONS.md | 8 + docs/WORKFLOW_GUIDE.md | 38 +- dw/guides.py | 6 +- dw/hold_audio.py | 5 +- dw/output_extraction.py | 30 +- dw/pipeline_processors/chain.py | 8 +- dw/pipeline_processors/chain_check.py | 4 +- dw/pipeline_processors/chain_rules.py | 140 ++++++ dw/pipeline_processors/h3_hold_steps.py | 50 +- dw/pipeline_processors/h3_mask.py | 86 ++++ dw/pipeline_processors/h3_mask_steps.py | 135 +++++- dw/pipeline_processors/h3_rules.py | 219 ++++----- dw/pipeline_processors/pipeline_h3.py | 39 +- tests/test_chain.py | 2 +- tests/test_chain_check.py | 2 +- tests/test_h3_audio_pass.py | 619 ++++++++++++++++++++++++ tests/test_h3_mask.py | 12 +- 19 files changed, 1441 insertions(+), 177 deletions(-) create mode 100644 bench/workflows/add-sound.json create mode 100644 dw/pipeline_processors/chain_rules.py create mode 100644 tests/test_h3_audio_pass.py diff --git a/bench/workflows/add-sound.json b/bench/workflows/add-sound.json new file mode 100644 index 000000000..9f5f241e8 --- /dev/null +++ b/bench/workflows/add-sound.json @@ -0,0 +1,212 @@ +{ + "id": "MiniMaxH3AddSound", + "description": "DRAFT, not in the catalog (#1192): H3 audio over a finished picture. 'source_video' is the finished take (an 'asset:' or 'output:' video, a silent one is the usual case) and 'prompt' the take's own prompt with the sound written in: its overall_soundscape, its music and its dialogue lines. The one 'sound' step is the baseline t2va pipeline given the take as 'source_video' with no mask and no hold_audio, which makes it an audio pass: every video row is the take's own clean latent at every step and only the audio is generated, from full noise, attending to the real picture. The result plays the take's own frames, not a VAE round trip, under the new track. It runs with no distillation LoRA on the dialogue schedule (12 steps, shifts 12/3), since the turbo LoRA is what digitizes H3 voices (#984 B0). 'listen_scale' 0.5 or 0.25 encodes the take at that fraction of its canvas, snapped to 32 px, which is cheaper; the track still runs the take's length and the picture returned is full size. 'width', 'height' and 'num_frames' must be the take's, on the 17n + 5 grid. Kept under bench/workflows until Don has listened to it.", + "summary": "A new H3 soundtrack - sound, music, dialogue - over a finished take's own picture.", + "cost_drivers": [ + "num_frames", + "num_inference_steps", + "width", + "height", + "listen_scale", + "weights_dtype", + "lora_model_name", + "attention_backend", + "cache" + ], + "vram_estimate": { + "base_gb": 16.3, + "bytes_per_voxel": 28.71, + "bytes_per_guide_voxel": 28.71, + "voxel_variables": [ + "width", + "height", + "num_frames" + ], + "reason": "VAE decode's conv3d activation scales with the voxel count width * height * num_frames; base_gb and bytes_per_voxel are calibrated from two real runs: image-to-video's family baseline at 960x544x124 (on-demand VAE) succeeded at 18.03 GiB peak reserved (docs/WORKFLOW_GUIDE.md), and video-with-audio-768p at 1344x768x345 (the family's own num_frames ceiling) OOM'd mid-denoise needing >=25.82 GiB - 20.25 GiB already allocated plus a refused 5.57 GiB allocation, on a 23.56 GiB card (job 282172105da1, 2026-09-22) - closing the gap #265/#266 left the H3 family with nothing declared to check the engine's vram_estimate machinery against (#324)" + }, + "variables": { + "source_video": "asset:take.mp4", + "prompt": "prompt:minimax/fox_dawn_context_ir", + "num_inference_steps": 12, + "num_frames": 124, + "width": 960, + "height": 544, + "video_shift": 12.0, + "audio_shift": 3.0, + "weights_dtype": "{int4}", + "lora_scale": 1.0, + "lora_alpha": null, + "lora_model_name": null, + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "lora_adapter_name": "turbo", + "seed": 42, + "attention_backend": "sparse", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "cache": null, + "listen_scale": 1 + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "MiniMax-H3's pipeline refuses a `width` or `height` that is not a multiple of 32, and only after the weights have loaded; refused here instead, with no 'snap', since rounding would be a silent change to the size asked for" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "MiniMax-H3's pipeline refuses a `width` or `height` that is not a multiple of 32, and only after the weights have loaded; refused here instead, with no 'snap', since rounding would be a silent change to the size asked for" + } + }, + "seed": "variable:seed", + "steps": [ + { + "name": "sound", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "attention_backend": "variable:attention_backend", + "sparse_tau": "variable:sparse_tau", + "sparse_topk": "variable:sparse_topk", + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + }, + "cache": "variable:cache" + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "t2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "variable:lora_adapter_name", + "scale": "variable:lora_scale", + "alpha": "variable:lora_alpha" + } + ], + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "source_video": "variable:source_video", + "listen_scale": "variable:listen_scale", + "num_frames": "variable:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24 + } + } + ] +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 0b1175ca3..298198f5c 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -80,7 +80,7 @@ one to open. | JSON records beside media | `dw/media_types.py`: `JsonRecord`, `dw/result.py`: `Result.save_artifact` | A dict marked `JsonRecord` is saved whole as one `.json`, whatever content type the step declares, and is never split into one file per key. | `tests/test_face_track.py::test_json_record_is_saved_whole_under_a_video_content_type`, `tests/test_face_track.py::test_the_task_result_saves_crops_as_video_and_track_as_one_json` | | Reading a JSON record back | `dw/locations.py`: `load_json_record` | A task reads a saved JSON record (`fit_to_model`'s, `crop_face_track`'s) only through `load_json_record`: the path is confined like any media path, then the extension must be `.json`, then the size is checked against `MAX_JSON_SIZE`, and only then is it parsed. `fit._read_fit` and `face_track._read_track` call it, never `open`. | `tests/test_locations.py::TestLoadJsonRecord::test_traversal_out_of_the_roots_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_a_non_json_extension_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_an_oversized_file_is_refused_before_it_is_read`, `tests/test_fit_to_model.py::TestRestore::test_fit_path_with_traversal_is_refused` | | Shots in a joined video | `dw/shots.py` | Every `AudioVideo` constructor decides what to do with `shots`, and the sample spans are measured from the joined waveform. | `tests/test_shots.py::test_every_audio_video_constructor_site_is_decided`, `tests/test_shots.py::TestConcatVideosShots::test_an_overrun_track_is_measured_not_derived` | -| Chain per-segment check and re-roll | `dw/pipeline_processors/chain_check.py`: `check_segment`, `SegmentCheck`, `best_attempt`, `reroll_seed`; the keys' rule is `dw/pipeline_processors/h3_rules.py`: `chain_check_problem`; the attempt loop is `dw/pipeline_processors/chain.py`: `_checked_attempt`, `_render_attempt` | A checked segment is measured on the arithmetic `analyze_seams` uses (`dw/tasks/assess.py`: `seam_measures`, `frame_measures`) and `dw/texture_bands.py`, never a copy of it. Every attempt renders from the same `_ChainState`; nothing of an attempt (frames, audio join, shot, spill, look-hold state, segment 1's anchor or keystone) reaches the timeline until it is kept, and a re-roll runs on its own generator at `reroll_seed`, leaving the step's generator where attempt 0 left it. Segment 1 and a hard-cut seam are never checked. Absent `check` is off and absent `retries` is 0. The worst case is quoted by `dw/cost_scaling.py`: `_worst_case_retries`, 1 + (N - 1)(1 + retries) renders for N segments. | `tests/test_chain.py::TestCheckAndReroll::test_a_failed_segment_is_rerolled_with_a_new_seed_from_the_same_carry`, `tests/test_chain.py::TestCheckAndReroll::test_budget_exhausted_keeps_the_best_attempt_and_warns`, `tests/test_chain.py::TestCheckAndReroll::test_absent_check_is_todays_chain`, `tests/test_chain_check.py::test_the_estimate_quotes_the_worst_case_retries` | +| Chain per-segment check and re-roll | `dw/pipeline_processors/chain_check.py`: `check_segment`, `SegmentCheck`, `best_attempt`, `reroll_seed`; the keys' rule is `dw/pipeline_processors/chain_rules.py`: `chain_check_problem` (torch-free, beside the look hold's `chain_look_hold_problem`); the attempt loop is `dw/pipeline_processors/chain.py`: `_checked_attempt`, `_render_attempt` | A checked segment is measured on the arithmetic `analyze_seams` uses (`dw/tasks/assess.py`: `seam_measures`, `frame_measures`) and `dw/texture_bands.py`, never a copy of it. Every attempt renders from the same `_ChainState`; nothing of an attempt (frames, audio join, shot, spill, look-hold state, segment 1's anchor or keystone) reaches the timeline until it is kept, and a re-roll runs on its own generator at `reroll_seed`, leaving the step's generator where attempt 0 left it. Segment 1 and a hard-cut seam are never checked. Absent `check` is off and absent `retries` is 0. The worst case is quoted by `dw/cost_scaling.py`: `_worst_case_retries`, 1 + (N - 1)(1 + retries) renders for N segments. | `tests/test_chain.py::TestCheckAndReroll::test_a_failed_segment_is_rerolled_with_a_new_seed_from_the_same_carry`, `tests/test_chain.py::TestCheckAndReroll::test_budget_exhausted_keeps_the_best_attempt_and_warns`, `tests/test_chain.py::TestCheckAndReroll::test_absent_check_is_todays_chain`, `tests/test_chain_check.py::test_the_estimate_quotes_the_worst_case_retries` | | Overlap windows of a long video | `dw/tasks/windows.py`: `window_video`, `window_span` | Window `i` covers source frames `[i*stride - overlap, i*stride + stride)`, its synthetic frames repeat the first or last frame, and its audio is cut on the source's own `frames_to_samples` boundaries, so adjacent windows' strides tile the track with no sample lost or repeated. | `tests/test_window_video.py::TestAudio::test_strides_tile_the_source_track_exactly`, `tests/test_window_video.py::TestRealPath::test_a_path_argument_keeps_its_audio`, `tests/test_window_video.py::TestFileMatchesInMemory::test_every_window_equals_the_in_memory_window`, `tests/test_read_frame_range.py` | | Keep a span of a video's frames and audio | `dw/tasks/trim.py`: `trim_video`; `dw/shots.py`: `trimmed_shots` | Frames `[start_frame, start_frame + num_frames)` are kept and the audio is cut on `frames_to_samples` of each end at the track's own rate, so consecutive trims tile the track. A span past the clip's end is refused, never shortened. Shots are clipped to the span, sample side cleared. | `tests/test_trim_video.py::TestFramesAndAudio::test_consecutive_trims_tile_the_track`, `tests/test_trim_video.py::TestShots::test_shots_are_clipped_across_the_boundary` | | Join processed windows back to the source's length | `dw/tasks/windows.py`: `join_windows`; `dw/task_domains.py`: `window_count`, `window_count_problem`, `join_windows_errors` | Output is exactly the source's frames, each once: later windows' first `overlap` frames blend over the previous window's last on the open ramp `dissolve_videos` uses. The window count is `window_count`'s `ceil(source_frames / stride)` and nothing else (it is also `window_video`'s last index plus one). Audio is the source's own track; shots are one per window with `overlap_frames` on each seam, their sample side laid over that track by `dw/shots.py`'s `remeasured_shots` (the same rounding as `frames_to_samples`, so the spans are cumulative). | `tests/test_join_windows.py::TestRoundTrip::test_unprocessed_windows_rejoin_to_the_source`, `tests/test_join_windows.py::TestShots::test_spans_partition_the_frames_and_tile_the_audio`, `tests/test_join_windows.py::TestUint8Output::test_matches_the_float32_accumulator_exactly` | @@ -117,6 +117,7 @@ one to open. | H3 sparse attention | `dw/sparse_attention.py`: `sparse_call`, `install_sparse`, `install_sparse_for_call`, `apply_sparse_settings`, `SparseH3AttnProcessor`; wired from `dw/pipeline_processors/components.py`: `configure_components` and `dw/pipeline_processors/pipeline.py`: `apply_runtime_settings` | Whether a pipeline call runs sparse is decided once per call, all or nothing (the call's first kernel invocation included); it never changes diffusers' process-wide backend and never reaches `AttentionBackendName`; it installs by a processor swap plus forward hooks, never a `forward` wrapper; and `sparse_tau` is a runtime setting, not part of the weights identity (#931). | `tests/test_sparse_attention_processor.py`, `tests/test_sparse_attention_wiring.py`, `tests/test_sparse_attention_tiles.py`, `tests/test_sparse_attention_backends.py` | | H3 adapter partition | `dw/adapter_compatibility.py`; the server's index is `dw/server/deps.py` `lora_index` | The LoRA catalog decides first, on the server only: a `proven` entry's `workflow` refuses a step in the other partition, a `trial`/`rejected` entry's only warns. A file no entry places falls to its name: an FL2VA LoRA on a reference step is refused, and one whose name says neither `ref2v` nor `fl2v` is warned. | `tests/test_h3_adapters.py::TestWhatIsRefused::test_a_keyframe_adapter_on_the_reference_path`, `tests/test_h3_adapters.py::TestWhatIsRefused::test_an_unrecognised_name_is_a_warning_not_a_refusal`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_proven_entry_refuses_the_other_partition`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_weaker_entry_only_warns`, `tests/test_server_loras.py::TestTheCatalogPlacesAnAdapter::test_a_trial_entry_warns_and_promoting_it_refuses` | | H3 audio hold, refine, tiled refine, guide layout, guide noise, latent anchor and regional edit | Fourteen modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules, `refine_tiles` and the explicit `refine_sigmas` schedule among them, asked before validation and before the call; `tile_plan`, the tile layout on the 32 px grid, shared by validate and the run; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants; and the regional-edit rules `mask_problems`, `media_problems` and `edits_region`, the argument rules shared by validate and `_with_mask`; `media_problems` is asked by validate too, on the files that resolve then, and refuses a `source_video` off the `17n + 5` grid, which the pipeline would otherwise snap up over a padded source), `dw/pipeline_processors/h3_mask_cells.py` holds the mask-to-rows arithmetic (`binarized`, `regenerate_cells`, `keep_rows`, on Pillow's L conversion, torch's adaptive max pool and scipy's dilation, each imported where it runs), which `h3_mask.py` re-exports and `h3_mask_steps.py` reads from there so the blocks never import their importer, `dw/pipeline_processors/h3_hold.py` puts the hold and refine blocks in (`insert_audio_hold`, `core_denoise_sequences`, `refines`; `HELD_LATENTS_OUTPUT` and `encoded_hold_audio`, the hold block's encode handed back to it), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `insert_guides`), which imports `h3_hold` and never the reverse, `dw/pipeline_processors/h3_anchor.py` the latent statistics anchor (`insert_latent_anchor`, `anchor_refusal`; its block in `h3_anchor_steps.py`, imported only by `anchor_block()`; the arithmetic, `channel_statistics` and `match_channel_statistics`, in `h3_anchor_stats.py`, torch only, which `h3_anchor` re-exports and `h3_anchor_steps` reads from there, as `h3_mask_cells` is read, so the block never imports its importer), which imports `h3_hold` and `h3_guides` and never the reverse, and runs right after `after_denoise` in t2va/fl2va only (a no-op without `latent_anchor`, decided by the block itself; it declares `idle_without = "latent_anchor"` so the step-progress narrator leaves it unnamed then; `chain.anchor: "segment_1"` hands segments 2+ segment 1's frames in `chain.py`'s `_segment_arguments`, and `run_chain` sets `pipeline.latent_anchor_source` to `segment 1` for those segments only; `Pipeline._with_latent_anchor` emits one `log` line per anchored call naming that source, the quoted reference, or the step's frames; `dw/guides.py`'s `anchor_errors` is the `latent_anchor` validation check, on `h3_rules.chain_anchor_problem` and the guides step rule `_takes_guides_problem`, asked for the anchor's name, so a step passing `references` (ref2va) is refused as `guides` is; `Pipeline._with_latent_anchor` refuses `references` at run time as `_with_guides` does), and `dw/pipeline_processors/h3_mask.py` the regional edit (`insert_mask_blocks`, `edits_regions`), which imports `h3_hold` and never the reverse; the blocks themselves, and the arithmetic they run, are `dw/pipeline_processors/h3_hold_steps.py` (`encode_audio_span`, `refine_sigmas`, and `explicit_sigmas`, a step's `refine_sigmas` list run verbatim in place of the shift grid) `dw/pipeline_processors/h3_guide_steps.py` (`splice_guide_rows`, and `noised_guide_pixels`, the `guide_noise` pixels: the layout block's `_encode_guides` encodes each video guide's noised pixels in place of the clean ones when `guide_noise` is set, on `h3_rules.guide_noise_taper`, one encode per guide either way (PrefixStream's paired encode under one seed is that encode, so the clean one is not run); `guide_noise_errors`/`guide_noise_warnings` in `dw/guides.py` validate it on `h3_rules.guide_noise_problem` and `_takes_guides_problem`, and `Pipeline._with_guide_noise` drops a 0 and refuses at run time) and `dw/pipeline_processors/h3_mask_steps.py` (the source-latents and mask-blend blocks), which import diffusers when they load, so the three owners import them only inside `blocks()`, `guide_blocks()` and `h3_mask.blocks()` (#790), and `h3_guide_steps` imports `h3_hold_steps` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline_h3.py` - the `H3Arguments` mixin that gives `Pipeline` its per-call H3 argument wrappers `_with_held_audio`, `_with_guides`, `_with_guide_noise`, `_with_latent_anchor`, `_with_mask` and `_check_refine`, still named `Pipeline._with_*` - adds its blocks for a `guides` call, and `_with_mask` checks a regional edit's arguments and media and builds the regenerate mask for a `source_video` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/pipeline_processors/h3_tiles.py`: the tiled refine (the tile loop, the latent-space feather blend, the per-tile crop of one whole-canvas noise draw, the decode-free graph variant, and the one spatially tiled decode), which imports `h3_rules`, `h3_hold` and `media_types` (for `FrameList`; and `events`, `progress`) and is imported by `pipeline.py` and, lazily, `components.py` (which attaches the variant at load, behind `h3_hold.core_denoise_sequences`, and calls `insert_mask_blocks` at load), never by `h3_rules` or `h3_hold`, `dw/hold_audio.py`: `hold_audio_errors` (a video is held by its soundtrack), `refine_strength_errors`, `mask_errors` (Check `regional_edit`, which probes the source and mask through `ValidationContext.probe` the way `dw/guides.py` probes guide videos) | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and two more for a regional edit: the source block before `set_timesteps` and the blend block inside the `denoise` loop after `update`. All are no-ops without their arguments (`hold_audio` / `refine_strength`, `source_video`). The tiled refine adds a second graph built at load beside them (`h3_tiles.decode_free_blocks`: the decode and text-encoder blocks dropped, every kept block the graph's own instance, so hold and refine stay in) and swaps it into `pipeline._blocks` only for the duration of the one text-encode call and the tile calls (`swapped_blocks`, restored on failure). Every tile runs the hold block, but the track is encoded once per refine: the first tile is asked for the block's encode (`HELD_LATENTS_OUTPUT`) and every later tile's `hold_audio` carries it (`encoded_hold_audio`, an `EncodedAudioReference`), which the block writes in place of running the audio VAE. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_tiles.py::TestDecodeFree::test_variant_has_no_decode_and_shares_blocks`, `tests/test_h3_tiles.py::TestTileLoop::test_blocks_are_restored_when_a_tile_fails`, `tests/test_h3_tiles.py::TestTileLoop::test_the_track_is_encoded_by_one_tile_and_handed_to_the_rest`, `tests/test_h3_hold_audio.py::TestHoldBlock::test_a_carried_encode_is_written_without_encoding_again`, `tests/test_h3_tiles.py`, `tests/test_h3_rules.py` (`tile_plan` and the `refine_tiles` rules), `tests/test_h3_mask.py`, `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_refine.py::TestSigmaListOneOwner::test_the_runtime_message_is_the_owners`, `tests/test_h3_refine.py::TestSigmaListSchedule::test_the_block_runs_the_list_unshifted`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers`, `tests/test_h3_rules.py::test_the_block_owners_import_no_diffusers`, `tests/test_h3_anchor.py::TestInsert::test_follows_after_denoise`, `tests/test_h3_anchor.py::TestStep::test_without_an_anchor_the_latents_are_untouched`, `tests/test_h3_anchor.py::TestWithLatentAnchor::test_no_anchor_logs_nothing`, `tests/test_h3_anchor.py::TestWithLatentAnchor::test_a_chain_segment_logs_its_label_and_segment_1`, `tests/test_modular_progress.py::test_the_anchor_step_is_idle_without_latent_anchor`, `tests/test_h3_anchor.py::TestAnchorErrors::test_an_unknown_anchor_names_the_values`, `tests/test_h3_guide_noise.py` | +| H3 audio pass (#1192) | `dw/pipeline_processors/h3_rules.py`: `is_audio_pass` (a `source_video` with no mask and no `hold_audio`), the audio-pass rules inside `mask_problems` (`_audio_pass_problems`, `_listen_scale_problems`, `LISTEN_SCALES`) and `listen_canvas`; `dw/pipeline_processors/h3_mask.py`: `insert_mask_blocks` puts the pair in beside the source block, `passes_audio`, `listen_frames` (the area resample); `dw/pipeline_processors/h3_mask_steps.py`: `DwH3AudioPassStep` (before `denoise`), `DwH3ReleaseVideoStep` (before `after_denoise`), on `h3_hold_steps.row_timestep_plan`; `dw/pipeline_processors/pipeline_h3.py`: `with_audio_pass`; `dw/output_extraction.py`: `held_videos` | No video row is ever stepped: the pass block widens `num_condition_video_rows` over every video row after `set_timesteps` and pins them at t = 1.0 in the row plan, the source block's clean encode held at every step; the audio keeps the stock audio schedule from full noise. No refine argument runs with it. The result plays the full-size source frames (`HELD_VIDEO_OUTPUT`), never the decode, the way `dw_held_audio` stands in for a held track. `listen_scale` moves the encode canvas only: `num_frames`, and so the audio length, is the source's. | `tests/test_h3_audio_pass.py::test_the_output_frames_are_the_sources_own`, `tests/test_h3_audio_pass.py::test_listen_scale_half_runs_the_kept_rows_at_half_canvas`, `tests/test_h3_audio_pass.py::test_a_mask_or_hold_audio_keeps_todays_meaning`, `tests/test_h3_audio_pass.py::TestAudioPassBlock::test_the_loop_steps_audio_and_never_a_video_row` | | IC-LoRA reference scale | `workflows/templates/ltx2/` | The five conditioning templates run at `reference_downscale_factor: 1`, and the two upscalers at 2. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_reference_is_encoded_at_the_output_resolution`, `tests/test_ltx2_ic_loras.py::TestTheGenerativeUpscaleMatchesTheSameCard::test_it_loads_the_upscaler_at_factor_two` | | IC-LoRA numbers | `workflows/templates/ltx2/` | Every number in the IC-LoRA templates comes from the vendor's model card. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_strength_is_the_cards_default`, `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_defaults_are_the_trained_bucket` | | IC-LoRA prompt genre | `prompts/ltx2/` | A stored prompt tagged `ic-lora` is checked against its trained caption form, not against the T2V paragraph rule. | `tests/test_ltx_prompt_library.py::test_an_ic_lora_prompt_is_in_its_trained_form` | diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md index f23674cfa..059739bf8 100644 --- a/docs/ATTRIBUTIONS.md +++ b/docs/ATTRIBUTIONS.md @@ -163,6 +163,14 @@ grade of a segment's opening to the carried frame, an ease-out of contrast and saturation creep, an ease-out of faint texture, and a keyframe pre-compensated for the measured boost. +Smite79/MiniMax-H3-LongVideos (concepts only, under the licence above) and WanGP +(concept only, from [#984](https://github.com/dkackman/diffusers-workflow/issues/984)) +inspired the H3 audio pass +([dw/pipeline_processors/h3_mask.py](../dw/pipeline_processors/h3_mask.py); +[#1192](https://github.com/dkackman/diffusers-workflow/issues/1192)): a finished +picture's video latents locked clean while only the audio is generated over it, and +the listen pass on a smaller canvas. + leonardofrito/Directors-Cut (MIT; concept and schema shape, no code taken) inspired `export_timeline` ([dw/tasks/timeline_export.py](../dw/tasks/timeline_export.py); diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 1a42ded48..7b57c6254 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -2371,7 +2371,8 @@ frame count or empty mask there, the way it reads guide videos; a `previous_resu checked at the call. `hold_audio` is still required with `refine_strength`, since the refine holds the -soundtrack. It is normally the `source_video` itself: a video path (`.mp4`, `.mov`, `.mkv`, +soundtrack (a `source_video` with neither a mask nor `hold_audio` is the [audio +pass](#audio-over-a-finished-picture) instead). It is normally the `source_video` itself: a video path (`.mp4`, `.mov`, `.mkv`, `.webm`, `.avi`) is held by its own soundtrack, so the edit keeps the take's audio. A video with no audio stream is refused. An audio file (`.wav`, `.mp3`, `.flac`, `.ogg`) or an earlier step's audio-carrying result works too. @@ -2381,7 +2382,7 @@ Validation and the call both refuse: - `source_video` with `latents`, or on a chained step; - both mask kinds, `mask_video` and `mask_image`, on one step; - a mask with no `source_video`, or `mask_grow` with no mask; -- `source_video` with no `refine_strength`, or one outside (0, 1]; +- `source_video` with a mask or `hold_audio` and no `refine_strength`, or one outside (0, 1]; - `source_video` on a pipeline that is not H3 (`mask_image` there is left alone, since inpaint pipelines own that name); - a `source_video` whose frame count (and so the step's `num_frames`, inferred or @@ -2392,6 +2393,39 @@ Validation and the call both refuse: carry and an inferred size does not check; - any of it with `refine_tiles`, which does not combine with a regional edit. +#### Audio over a finished picture + +A `source_video` with no mask and no `hold_audio` is an audio pass: the picture is +finished and only the soundtrack is generated, from the prompt - its soundscape, its music +and its `` lines. Every video row of the source is held as the exact clean latent at +every step, pinned clean in the row plan the way a reference row is, so no video row is +ever stepped. The audio runs the stock audio schedule from full noise, attending to the +real picture from the first step. The step returns the source's own frames, not a VAE +round trip of them, under the new track. A silent source is the usual case and is fine. + +```json +{ + "source_video": "asset:silent_take.mp4", + "listen_scale": 1, + "prompt": "... overall_soundscape: ... [English] Over here. ...", + "num_inference_steps": 12, + "output": ["videos", "audio", "sampling_rate"] +} +``` + +Run it without a distillation LoRA, on the dialogue schedule (`num_inference_steps` 12, +`video_shift` 12, `audio_shift` 3): the turbo LoRA is what makes H3 voices sound +digital (#984). `listen_scale` (1, 0.5 or 0.25, default 1) encodes the source at that +fraction of its canvas, area-resampled and snapped to 32 px, which is cheaper per step. +The audio's length still follows the source's frame count, and the picture returned is the +full-size source. `width`, `height` and `num_frames` stay the source's, as for an edit. + +Validation and the call refuse `refine_strength`, `refine_sigmas` or `refine_tiles` on an +audio pass (no video row is re-denoised; pass `hold_audio` for a whole-source refine +instead), a `listen_scale` other than those three, a `listen_scale` on a step that is not +an audio pass, and an audio pass on a chained step or beside `latents`. A draft template +is `bench/workflows/add-sound.json`; it is not in the catalog yet. + ### Refining above 768p in tiles H3's refine runs at 1344x768 at most, and `upscale_h3_latents` reaches 1920x1088 (or diff --git a/dw/guides.py b/dw/guides.py index fba8290d7..083e623e5 100644 --- a/dw/guides.py +++ b/dw/guides.py @@ -47,6 +47,10 @@ ) from .for_each import render_path from .pipeline_processors.h3_guides import default_num_frames +from .pipeline_processors.chain_rules import ( + chain_check_problem, + chain_look_hold_problem, +) from .pipeline_processors.h3_rules import ( ANCHOR_NONE, CHAIN_CONTINUITY_MODES, @@ -63,8 +67,6 @@ RENDER_GRID, chain_anchor_problem, chain_carry_pick_problem, - chain_check_problem, - chain_look_hold_problem, chain_keystone_problem, guide_backoff_problems, guide_chain_problems, diff --git a/dw/hold_audio.py b/dw/hold_audio.py index 69488ad0a..ca393d8ba 100644 --- a/dw/hold_audio.py +++ b/dw/hold_audio.py @@ -38,7 +38,10 @@ resolve now, they are read here too, the way the guide-frame check reads its videos (`dw/guides.py`): the source's and the mask's size and frame count against the canvas (`h3_rules.media_problems`, which `Pipeline._with_mask` asks -again at the call), and a mask with no white pixel in it. +again at the call), and a mask with no white pixel in it. A `source_video` with +no mask and no `hold_audio` is the audio pass (#1192), whose rules - no refine +argument, a `listen_scale` of 1, 0.5 or 0.25 - `mask_problems` owns too; its +source may be silent, since a soundtrack is what it adds. """ from . import references diff --git a/dw/output_extraction.py b/dw/output_extraction.py index 9d51190ac..1c551e8b5 100644 --- a/dw/output_extraction.py +++ b/dw/output_extraction.py @@ -9,7 +9,11 @@ import torch from .media_types import AudioTrack, AudioVideo, rated_videos -from .pipeline_processors.h3_rules import HELD_AUDIO_OUTPUT, HELD_AUDIO_RATE_OUTPUT +from .pipeline_processors.h3_rules import ( + HELD_AUDIO_OUTPUT, + HELD_AUDIO_RATE_OUTPUT, + HELD_VIDEO_OUTPUT, +) logger = logging.getLogger("dw") @@ -36,6 +40,26 @@ # video, which plays in place of the decoded audio (the VAE round trip of it) HELD_AUDIO_KEYS = (HELD_AUDIO_OUTPUT,) HELD_AUDIO_RATE_KEYS = (HELD_AUDIO_RATE_OUTPUT,) +# A MiniMax-H3 audio pass: the source's own frames, which play in place of the +# decoded video (the VAE round trip of them) under the generated track +HELD_VIDEO_KEYS = (HELD_VIDEO_OUTPUT,) + + +def held_videos(videos, held): + """The batch `videos` with every video replaced by `held` - uint8 + `(frames, height, width, 3)` source frames - in the batch's own form: a + list of PIL frame lists (H3's default `output_type`), keeping the batch's + rate, or a float array in [0, 1] for `output_type='np'`.""" + held = numpy.asarray(held) + if isinstance(videos, numpy.ndarray): + return numpy.stack([held.astype(numpy.float32) / 255.0] * len(videos)) + if isinstance(videos, torch.Tensor): + pixels = torch.from_numpy(held).permute(0, 3, 1, 2).float() / 255.0 + return torch.stack([pixels] * len(videos)).to(videos.device) + from PIL import Image + + frames = [Image.fromarray(frame) for frame in held] + return rated_videos([list(frames) for _ in videos], getattr(videos, "fps", None)) def _frames_from_attributes(result): @@ -155,6 +179,10 @@ def modular_artifacts(result): return None consumed_keys = {video_key} + held_video_key, held_video = first_item(result, HELD_VIDEO_KEYS) + if held_video is not None: + videos = held_videos(videos, held_video) + consumed_keys.update(HELD_VIDEO_KEYS) audio_key, audio = first_item(result, MODULAR_AUDIO_KEYS) sample_rate = None diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index b05db28e7..e7573f066 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -68,6 +68,11 @@ from .h3_anchor import anchor_refusal from .h3_guides import guides_refusal from .h3_hold import hold_audio_reference +from .chain_rules import ( + chain_check_problem, + chain_look_hold_flags, + chain_look_hold_problem, +) from .h3_rules import ( ANCHOR_NONE, ANCHOR_SEGMENT_1, @@ -84,10 +89,7 @@ KEYSTONE_SEGMENT_1, chain_anchor_problem, chain_carry_pick_problem, - chain_check_problem, chain_keystone_problem, - chain_look_hold_flags, - chain_look_hold_problem, guide_backoff_problems, guide_chain_problems, ) diff --git a/dw/pipeline_processors/chain_check.py b/dw/pipeline_processors/chain_check.py index 3e032ed7e..75ac9a129 100644 --- a/dw/pipeline_processors/chain_check.py +++ b/dw/pipeline_processors/chain_check.py @@ -11,7 +11,7 @@ and is never checked; nor is a seam whose incoming shot is a hard cut. Measurement and choice only: nothing here renders, and nothing here touches -the timeline. The chain keys are validated by `h3_rules.chain_check_problem`. +the timeline. The chain keys are validated by `chain_rules.chain_check_problem`. """ import math @@ -69,7 +69,7 @@ class CheckResult: def segment_check(value): """A chain block's `check` as a SegmentCheck, or None when off: true is the defaults, an object overrides the fields it names. The value has - passed `h3_rules.chain_check_problem`.""" + passed `chain_rules.chain_check_problem`.""" if value is None or value is False: return None if value is True: diff --git a/dw/pipeline_processors/chain_rules.py b/dw/pipeline_processors/chain_rules.py new file mode 100644 index 000000000..2e6c0893a --- /dev/null +++ b/dw/pipeline_processors/chain_rules.py @@ -0,0 +1,140 @@ +"""The chain rules that are not MiniMax-H3's: the look hold (#1191) and the +per-segment check with its bounded re-roll (dw/pipeline_processors/chain_check.py). + +Torch-free, so validation (dw/guides.py) reads them without loading a model, +and the run (`ChainConfig` in dw/pipeline_processors/chain.py) asks the same +rules. Moved out of dw/pipeline_processors/h3_rules.py, which holds H3's own +rules, when that module reached the size ceiling (#1192). +""" + +import math + +from .. import references + +# The look hold (#1191): pixel corrections applied to each finished segment +# before its carry is taken. Family-independent +LOOK_HOLD_PARTS = ("pickup", "creep", "texture", "precompensate") +# Parts that act on a single carried keyframe, so need `last_frame` +LOOK_HOLD_KEYFRAME_PARTS = ("pickup", "precompensate") + + +def chain_look_hold_flags(chain): + """The four look-hold switches as {part: bool}, or None when off. `true` + turns on all four; an object names the ones it wants.""" + hold = chain.get("look_hold") + if hold is True: + return dict.fromkeys(LOOK_HOLD_PARTS, True) + if isinstance(hold, dict) and any(hold.get(part) is True for part in hold): + return {part: hold.get(part) is True for part in LOOK_HOLD_PARTS} + return None + + +def chain_look_hold_problem(chain): + """The reason a chain block's `look_hold` is refused, or None: a value that + is neither a boolean nor an object of the four boolean switches, or an + explicit `pickup` / `precompensate` on a continuity other than `last_frame` + (a guide or last_segment carry is a whole clip, not one keyframe - `true` + alone quietly skips those two there). Unresolved references are left to the + run.""" + if "look_hold" not in chain: + return None + hold = chain["look_hold"] + if isinstance(hold, str) and references.is_ref(references.UNRESOLVED, hold): + return None + expected = ( + "expected true, false or an object of " + + ", ".join(LOOK_HOLD_PARTS) + + " booleans" + ) + if isinstance(hold, bool): + return None + if not isinstance(hold, dict): + return f"invalid look_hold {hold!r} - {expected}" + unknown = sorted(set(hold) - set(LOOK_HOLD_PARTS)) + if unknown: + return f"look_hold has unknown key(s) {unknown} - {expected}" + for part, value in hold.items(): + if not isinstance(value, bool): + return f"look_hold {part} must be true or false, got {value!r}" + continuity = chain.get("continuity", "last_frame") + if isinstance(continuity, str) and references.is_ref( + references.UNRESOLVED, continuity + ): + return None + if continuity != "last_frame": + wrong = [p for p in LOOK_HOLD_KEYFRAME_PARTS if hold.get(p)] + if wrong: + return ( + f"look_hold {' and '.join(wrong)} need continuity 'last_frame' - " + f"the {continuity!r} carry is a whole clip, not one keyframe; " + "creep and texture work on any continuity" + ) + return None + + +# The per-segment check and its bounded re-roll (chain_check.py): the fields a +# `check` object may set (`face` is reserved, null only) and the most re-rolls +# a checked segment may take. Family-independent +CHAIN_CHECK_FIELDS = ("jump_ratio_max", "look_ratio_max", "texture_ratio_max", "face") +CHAIN_RETRIES_MAX = 3 + + +def _number_over(value, floor): + """Whether `value` is a finite number above `floor` (never a bool), or an + unresolved reference, which is left to the run.""" + if isinstance(value, str): + return references.is_ref(references.UNRESOLVED, value) + return ( + not isinstance(value, bool) + and isinstance(value, (int, float)) + and math.isfinite(value) + and value > floor + ) + + +def chain_check_problem(chain): + """The reason a chain block's `check` or `retries` is refused, or None. + `check` is true (the default ceilings), false, or an object of + `CHAIN_CHECK_FIELDS`: the ratio ceilings above 0, `texture_ratio_max` + above 1 (it bounds the ratio both ways) or null, `face` null. `retries` + is a whole number from 0 to `CHAIN_RETRIES_MAX`; above 0 it needs the + check on, as with no check nothing fails. Unresolved references are left + to the run, whose `ChainConfig` asks this same rule.""" + check = chain.get("check") + expected = "expected true, false or an object of " + ", ".join(CHAIN_CHECK_FIELDS) + if isinstance(check, dict): + unknown = sorted(set(check) - set(CHAIN_CHECK_FIELDS)) + if unknown: + return f"check has unknown key(s) {unknown} - {expected}" + for name in ("jump_ratio_max", "look_ratio_max"): + if name in check and not _number_over(check[name], 0): + return f"check {name} must be a number above 0, got {check[name]!r}" + texture = check.get("texture_ratio_max") + if texture is not None and not _number_over(texture, 1): + return ( + "check texture_ratio_max bounds the fine-band ratio both ways, " + f"so must be a number above 1 or null, got {texture!r}" + ) + if check.get("face") is not None: + return f"check face is reserved and must be null, got {check['face']!r}" + elif not ( + check is None + or isinstance(check, bool) + or (isinstance(check, str) and _number_over(check, 0)) + ): + return f"invalid check {check!r} - {expected}" + retries = chain.get("retries", 0) + if isinstance(retries, str) and _number_over(retries, 0): + return None + whole = isinstance(retries, int) and not isinstance(retries, bool) + if not whole or not 0 <= retries <= CHAIN_RETRIES_MAX: + return ( + f"retries must be a whole number between 0 and {CHAIN_RETRIES_MAX}, " + f"got {retries!r}" + ) + if retries and (check is None or check is False): + return ( + f"retries {retries} needs check on - a re-roll replaces a segment " + "its check failed, and with no check no segment fails" + ) + return None diff --git a/dw/pipeline_processors/h3_hold_steps.py b/dw/pipeline_processors/h3_hold_steps.py index 171599452..179c3148e 100644 --- a/dw/pipeline_processors/h3_hold_steps.py +++ b/dw/pipeline_processors/h3_hold_steps.py @@ -181,6 +181,32 @@ def encode_audio_span(encoder, components, waveform, num_latents): return fit_latents(latents, num_latents, silence[:, middle : middle + 1]) +def row_timestep_plan(block_state, device, condition_video_timestep): + """The row-timestep plan `set_timesteps` builds, over the state's + `timesteps` and `audio_timesteps`, with the conditioning video rows at + `condition_video_timestep(timestep)` and the reference audio rows clean + (1.0) - stock pins the video ones at max(t, keyframe_noise_aug).""" + return [ + tuple( + tensor.to(device) + for tensor in MiniMaxH3SetTimestepsStep.build_row_timesteps( + block_state.video_indices, + block_state.audio_indices, + block_state.num_condition_video_rows, + block_state.num_condition_audio_rows, + block_state.text_indices.numel(), + float(timestep), + float(audio_timestep), + condition_video_timestep(float(timestep)), + 1.0, + ) + ) + for timestep, audio_timestep in zip( + block_state.timesteps, block_state.audio_timesteps + ) + ] + + @dataclasses.dataclass class EncodedAudioReference(MiniMaxH3AudioReference): """A `hold_audio` reference carrying `latents`, the hold block's own @@ -414,25 +440,11 @@ def __call__(self, components, state): components.audio_scheduler.set_timesteps(sigmas=sigmas, device=device) block_state.timesteps = components.scheduler.timesteps block_state.audio_timesteps = components.audio_scheduler.timesteps - block_state.row_timestep_plan = [ - tuple( - tensor.to(device) - for tensor in MiniMaxH3SetTimestepsStep.build_row_timesteps( - block_state.video_indices, - block_state.audio_indices, - block_state.num_condition_video_rows, - block_state.num_condition_audio_rows, - block_state.text_indices.numel(), - float(timestep), - float(audio_timestep), - max(float(timestep), components.keyframe_noise_aug), - 1.0, - ) - ) - for timestep, audio_timestep in zip( - block_state.timesteps, block_state.audio_timesteps - ) - ] + block_state.row_timestep_plan = row_timestep_plan( + block_state, + device, + lambda timestep: max(timestep, components.keyframe_noise_aug), + ) # Only the generated rows: FL2VA and Ref2VA conditioning rows lead latents = block_state.latents diff --git a/dw/pipeline_processors/h3_mask.py b/dw/pipeline_processors/h3_mask.py index 6f90172d4..e8d3059f4 100644 --- a/dw/pipeline_processors/h3_mask.py +++ b/dw/pipeline_processors/h3_mask.py @@ -28,6 +28,31 @@ only, since it imports diffusers when it loads (#790). The argument rules shared with validation are dw/pipeline_processors/h3_rules.py (`mask_problems`, `media_problems`). + +The audio pass (#1192) is the same `source_video` with no mask and no +`hold_audio` (`h3_rules.is_audio_pass`): the picture is finished and only the +audio is generated, from the prompt and its `` lines. The source block +encodes the source over the generated rows as for an edit, and two more blocks +go into each sequence beside it: + +- `DwH3AudioPassStep`, before `denoise` - widens `num_condition_video_rows` + over every video row, so `update` steps none of them, and rebuilds the row + plan with every video row pinned at t = 1.0: the exact clean source latent at + every step, as a reference row is held. The audio rows keep the stock audio + schedule from full noise (strength 1 for audio), so they attend to the real + picture from the first step - which a re-noised video row, the regional + edit's way, would hide under noise at the steps where the audio takes shape; +- `DwH3ReleaseVideoStep`, before `after_denoise` - puts the count back, so the + decode finds the video rows where it slices them. + +No refine runs: `refine_strength` is refused on an audio pass. The decode is +still a VAE round trip of the kept rows, so the result plays the source's own +frames instead (`HELD_VIDEO_OUTPUT`, swapped in by dw/output_extraction.py the +way `dw_held_audio` is). `listen_scale` 0.5 or 0.25 encodes the kept rows from +an area-resampled copy of the source on a smaller canvas (`listen_frames`, +`h3_rules.listen_canvas`), which is cheaper per step; the audio's length +follows `num_frames`, which the scale leaves alone, and the picture returned +is still the full-size source. """ import logging @@ -35,6 +60,10 @@ from .h3_hold import core_denoise_sequences from .h3_rules import ( + AUDIO_PASS_BEFORE, + AUDIO_PASS_BLOCK, + RELEASE_VIDEO_BEFORE, + RELEASE_VIDEO_BLOCK, BLEND_AFTER, BLEND_BLOCK, REFINE_BEFORE, @@ -55,6 +84,46 @@ def blocks(): return h3_mask_steps.DwH3SourceLatentsStep, h3_mask_steps.DwH3MaskBlendStep +def audio_pass_blocks(): + """(DwH3AudioPassStep, DwH3ReleaseVideoStep), imported on first use as + `blocks()` imports its pair (#790).""" + from . import h3_mask_steps + + return h3_mask_steps.DwH3AudioPassStep, h3_mask_steps.DwH3ReleaseVideoStep + + +def listen_frames(frames, width, height): + """`(frames, height, width, 3)` uint8 source frames area-resampled to + `width` x `height` - the copy an audio pass at a `listen_scale` below 1 + encodes. torch is imported here, on first use.""" + import numpy as np + import torch + import torch.nn.functional as F + + pixels = torch.from_numpy(np.ascontiguousarray(frames)).permute(0, 3, 1, 2) + resized = F.interpolate(pixels.float(), size=(height, width), mode="area") + rounded = resized.round().clamp(0, 255).to(torch.uint8) + return rounded.permute(0, 2, 3, 1).contiguous().numpy() + + +def _insert_audio_pass(sequence, prefix, pass_block, release_block): + """The audio-pass pair into one sequence, or neither when an anchor is + missing. Returns whether the sequence has them.""" + sub_blocks = sequence.sub_blocks + pass_name, release_name = prefix + AUDIO_PASS_BLOCK, prefix + RELEASE_VIDEO_BLOCK + if pass_name in sub_blocks and release_name in sub_blocks: + return True + pass_before = prefix + AUDIO_PASS_BEFORE + release_before = prefix + RELEASE_VIDEO_BEFORE + if pass_before not in sub_blocks or release_before not in sub_blocks: + return False + names = list(sub_blocks) + sub_blocks.insert(pass_name, pass_block(), names.index(pass_before)) + names = list(sub_blocks) + sub_blocks.insert(release_name, release_block(), names.index(release_before)) + return True + + def _loop(sequence, prefix): """The sequence's denoise loop, or None - the step `refine` anchors to.""" loop = sequence.sub_blocks.get(prefix + REFINE_BEFORE) @@ -71,6 +140,7 @@ def insert_mask_blocks(pipeline): pipeline left with none refuses `source_video` at the call. """ source_block, blend_block = blocks() + pass_block, release_block = audio_pass_blocks() inserted = False for prefix, sequence in core_denoise_sequences(pipeline): sub_blocks = sequence.sub_blocks @@ -79,6 +149,7 @@ def insert_mask_blocks(pipeline): loop_names = list(loop.sub_blocks) if loop is not None else [] if source_name in sub_blocks and BLEND_BLOCK in loop_names: inserted = True + _insert_audio_pass(sequence, prefix, pass_block, release_block) continue if source_before not in sub_blocks or BLEND_AFTER not in loop_names: logger.warning( @@ -95,6 +166,12 @@ def insert_mask_blocks(pipeline): BLEND_BLOCK, blend_block(), loop_names.index(BLEND_AFTER) + 1 ) inserted = True + if not _insert_audio_pass(sequence, prefix, pass_block, release_block): + logger.warning( + f"MiniMax-H3 has a denoise sequence with no '{prefix}" + f"{AUDIO_PASS_BEFORE}'/'{prefix}{RELEASE_VIDEO_BEFORE}' step to " + f"anchor the audio pass to, so it cannot score a finished picture" + ) return inserted @@ -109,3 +186,12 @@ def edits_regions(pipeline): ): return True return False + + +def passes_audio(pipeline): + """Whether a pipeline's block graph carries the audio pass.""" + return any( + prefix + AUDIO_PASS_BLOCK in sequence.sub_blocks + and prefix + RELEASE_VIDEO_BLOCK in sequence.sub_blocks + for prefix, sequence in core_denoise_sequences(pipeline) + ) diff --git a/dw/pipeline_processors/h3_mask_steps.py b/dw/pipeline_processors/h3_mask_steps.py index 6c4559bee..8f0c02a4b 100644 --- a/dw/pipeline_processors/h3_mask_steps.py +++ b/dw/pipeline_processors/h3_mask_steps.py @@ -1,5 +1,5 @@ -"""The MiniMax-H3 regional-edit blocks (#1062) - what -dw/pipeline_processors/h3_mask.py inserts and describes. +"""The MiniMax-H3 regional-edit blocks (#1062) and the audio-pass blocks +(#1192) - what dw/pipeline_processors/h3_mask.py inserts and describes. This module imports diffusers when it loads, so nothing imports it at module level - `h3_mask.blocks()` imports it on first use (#790). @@ -21,9 +21,13 @@ from ..events import emit_log +from .h3_hold_steps import row_timestep_plan from .h3_mask_cells import keep_rows from .h3_rules import ( + AUDIO_PASS_ROWS, + HELD_VIDEO_OUTPUT, KEEP_ROWS, + LISTEN_SCALE_INPUT, MASK_GROW_DEFAULT, MASK_GROW_INPUT, REFINE_ROW_NOISE, @@ -37,7 +41,11 @@ def edit_summary(keep, grow, strength): """The log line naming what a `source_video` edit regenerates and how - hard - `keep` is the per-row bool the mask maps to, None for no mask.""" + hard - `keep` is the per-row bool the mask maps to, None for no mask. + With neither a mask nor a strength the call is an audio pass, which + `DwH3AudioPassStep` names itself.""" + if keep is None and strength is None: + return "Audio pass: encoding source_video to hold every video row" if keep is None: return ( f"Regional edit: no mask, re-denoising the whole source_video at " @@ -241,3 +249,124 @@ def __call__(self, components, block_state, i, t): rows.dtype ) return components, block_state + + +class DwH3AudioPassStep(ModularPipelineBlocks): + model_name = "minimax-h3" + + @property + def description(self): + return ( + "dw: an audio pass - with `source_video` and no mask or `hold_audio` - widens " + "`num_condition_video_rows` over every video row, so no video row is stepped, and pins them clean " + "in the row plan; the audio is generated from full noise on its own schedule, attending to the " + "source's picture. Does nothing on any other call." + ) + + @property + def inputs(self): + return [ + InputParam( + name=HELD_VIDEO_OUTPUT, + default=None, + description=( + "Internal: the source's own frames at full size, uint8 - set for an audio pass only, " + "and what the result plays in place of the decode." + ), + ), + InputParam( + name=LISTEN_SCALE_INPUT, + default=None, + description=( + "An audio pass only: 1, 0.5 or 0.25 - the kept rows run on the source's canvas scaled " + "by it (snapped to 32 px), cheaper and coarser; the picture returned is the source's " + "own, at full size, and the audio runs the source's length." + ), + ), + InputParam(name="latents", type_hint=torch.Tensor, required=True), + InputParam(name=SOURCE_ROWS, type_hint=torch.Tensor, default=None), + InputParam(name="num_condition_video_rows", type_hint=int, default=0), + InputParam(name="num_condition_audio_rows", type_hint=int, default=0), + InputParam(name="timesteps", type_hint=torch.Tensor, required=True), + InputParam(name="audio_timesteps", type_hint=torch.Tensor, required=True), + InputParam(name="video_indices", type_hint=torch.Tensor, required=True), + InputParam(name="audio_indices", type_hint=torch.Tensor, required=True), + InputParam(name="text_indices", type_hint=torch.Tensor, required=True), + InputParam(name="width", type_hint=int, required=True), + InputParam(name="height", type_hint=int, required=True), + ] + + @property + def intermediate_outputs(self): + return [ + OutputParam("num_condition_video_rows", type_hint=int), + OutputParam("row_timestep_plan", type_hint=list), + OutputParam(AUDIO_PASS_ROWS, type_hint=int), + ] + + @torch.no_grad() + def __call__(self, components, state): + block_state = self.get_block_state(state) + if getattr(block_state, HELD_VIDEO_OUTPUT) is None: + # The plan and the count stay as set_timesteps left them + return components, state + if getattr(block_state, SOURCE_ROWS) is None: + raise ValueError( + "an audio pass holds the source_video's rows, and the source " + "block encoded none" + ) + total = block_state.latents.shape[0] + start = block_state.num_condition_video_rows or 0 + block_state.num_condition_video_rows = total + setattr(block_state, AUDIO_PASS_ROWS, total - start) + # Every video row clean, t = 1.0 in H3's t = 1 - sigma: the exact + # source latent, held as a reference row is + block_state.row_timestep_plan = row_timestep_plan( + block_state, components._execution_device, lambda timestep: 1.0 + ) + scale = getattr(block_state, LISTEN_SCALE_INPUT) or 1 + emit_log( + f"Audio pass: holding all {total - start} video rows of the source " + f"at {block_state.width}x{block_state.height} (listen_scale {scale}) " + f"and generating the audio over {len(block_state.timesteps)} steps", + held_video_rows=total - start, + listen_scale=scale, + canvas=[block_state.width, block_state.height], + ) + self.set_block_state(state, block_state) + return components, state + + +class DwH3ReleaseVideoStep(ModularPipelineBlocks): + model_name = "minimax-h3" + + @property + def description(self): + return ( + "dw: puts `num_condition_video_rows` back to what it was before an audio pass widened it, so " + "`after_denoise` finds the generated video rows where it slices them." + ) + + @property + def inputs(self): + return [ + InputParam(name="num_condition_video_rows", type_hint=int, default=0), + InputParam(name=AUDIO_PASS_ROWS, type_hint=int, default=0), + ] + + @property + def intermediate_outputs(self): + return [ + OutputParam("num_condition_video_rows", type_hint=int), + OutputParam(AUDIO_PASS_ROWS, type_hint=int), + ] + + @torch.no_grad() + def __call__(self, components, state): + block_state = self.get_block_state(state) + held = getattr(block_state, AUDIO_PASS_ROWS) or 0 + if held: + block_state.num_condition_video_rows -= held + setattr(block_state, AUDIO_PASS_ROWS, 0) + self.set_block_state(state, block_state) + return components, state diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index 7cdeb3ab1..ae8ae0029 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -92,7 +92,8 @@ def refine_problems(arguments, canvas=None): "refine_strength re-denoises the 'latents' it is passed - pass the " "upscaled latents, e.g. 'previous_result:up'" ) - if arguments.get(HOLD_AUDIO_INPUT) is None: + # An audio pass refuses refine_strength itself (h3_audio_pass.py) + if arguments.get(HOLD_AUDIO_INPUT) is None and not is_audio_pass(arguments): problems.append( "refine_strength re-denoises the video only, so it needs 'hold_audio' " "to keep a soundtrack - e.g. the base pass's 'previous_result:base.audio'" @@ -842,7 +843,17 @@ def guide_end_problem(frame, length, num_frames): MASK_IMAGE_INPUT = "mask_image" MASK_GROW_INPUT = "mask_grow" MASK_INPUTS = (MASK_VIDEO_INPUT, MASK_IMAGE_INPUT) -REGIONAL_EDIT_INPUTS = (SOURCE_VIDEO_INPUT, *MASK_INPUTS, MASK_GROW_INPUT) +# The audio pass (#1192): a `source_video` with no mask and no `hold_audio` +# keeps every video row and generates only the audio; `listen_scale` runs the +# kept rows on a smaller canvas (dw/pipeline_processors/h3_mask.py) +LISTEN_SCALE_INPUT = "listen_scale" +LISTEN_SCALES = (1, 0.5, 0.25) +REGIONAL_EDIT_INPUTS = ( + SOURCE_VIDEO_INPUT, + *MASK_INPUTS, + MASK_GROW_INPUT, + LISTEN_SCALE_INPUT, +) # Patch cells a mask is widened by past what it covers, so the edit's seam # lands in regenerated rows rather than against a kept one MASK_GROW_DEFAULT = 1 @@ -861,6 +872,18 @@ def guide_end_problem(frame, length, num_frames): # The refine block's re-noise of the generated rows, packed - the noise the # blend re-noises the kept rows with, so they ride the same trajectory REFINE_ROW_NOISE = "dw_refine_row_noise" +# The audio pass's blocks: one before `denoise` widens the conditioning rows +# over every video row, one before `after_denoise` puts the count back +AUDIO_PASS_BLOCK = "dw_audio_pass" +AUDIO_PASS_BEFORE = REFINE_BEFORE +RELEASE_VIDEO_BLOCK = "dw_release_video" +RELEASE_VIDEO_BEFORE = RELEASE_BEFORE +AUDIO_PASS_ROWS = "dw_audio_pass_rows" +# The source's own frames at full size, uint8 - what the result plays in +# place of the decode (dw/output_extraction.py), as dw_held_audio is for audio +HELD_VIDEO_OUTPUT = "dw_held_video" +# The refine arguments an audio pass has no use for: no video row is re-denoised +AUDIO_PASS_REFUSES = (REFINE_STRENGTH_INPUT, REFINE_SIGMAS_INPUT, REFINE_TILES_INPUT) def is_h3_step(pipeline): @@ -886,6 +909,14 @@ def _given(arguments, name): return arguments.get(name) is not None +def is_audio_pass(arguments): + """A `source_video` with no mask and no `hold_audio`: the audio pass, which + keeps every video row and generates the audio (#1192).""" + names = (SOURCE_VIDEO_INPUT, *MASK_INPUTS, HOLD_AUDIO_INPUT) + given = [_given(arguments, name) for name in names] + return given[0] and not any(given[1:]) + + def mask_problems(arguments, chained=False): """Why this step's regional edit - `source_video`, `mask_video` or `mask_image`, `mask_grow` - cannot run, as a list; empty when it can or @@ -897,9 +928,10 @@ def mask_problems(arguments, chained=False): masks = [name for name in MASK_INPUTS if _given(arguments, name)] source = _given(arguments, SOURCE_VIDEO_INPUT) grow = arguments.get(MASK_GROW_INPUT) + audio_pass = is_audio_pass(arguments) + problems = [] if audio_pass else _listen_scale_problems(arguments) if not masks and not source and grow is None: - return [] - problems = [] + return problems if len(masks) > 1: problems.append( "pass mask_video or mask_image, not both - mask_image holds one mask " @@ -918,7 +950,9 @@ def mask_problems(arguments, chained=False): problems.append( f"mask_grow is a whole number of patch cells, 0 or more, not {grow!r}" ) - if source and arguments.get(REFINE_STRENGTH_INPUT) is None: + if audio_pass: + problems.extend(_audio_pass_problems(arguments)) + elif source and not _given(arguments, REFINE_STRENGTH_INPUT): problems.append( "source_video is re-denoised from 'refine_strength' - pass one, near " "1.0 to replace what the mask covers, lower to keep its layout" @@ -926,13 +960,55 @@ def mask_problems(arguments, chained=False): if source and _given(arguments, "latents"): problems.append("source_video stands in for 'latents' - pass one or the other") if source and chained: + what = "an audio pass scores" if audio_pass else "a regional edit re-denoises" problems.append( - "a regional edit re-denoises one source_video, and a chained step " - "renders several segments - use an unchained step" + f"{what} one source_video, and a chained step renders several " + f"segments - use an unchained step" ) return problems +def _listen_scale_problems(arguments): + """A `listen_scale` on a step that is no audio pass: what makes it one is + said, since a mask or `hold_audio` is the usual reason it is not.""" + if not _given(arguments, LISTEN_SCALE_INPUT): + return [] + return [ + "listen_scale sets the canvas of an audio pass - a 'source_video' with " + "no mask_video, mask_image or hold_audio - and this step is not one" + ] + + +def _audio_pass_problems(arguments): + """Why this audio pass's own arguments cannot run: a refine argument (no + video row is re-denoised) or a `listen_scale` not in LISTEN_SCALES.""" + problems = [ + f"{name} re-denoises video, and an audio pass (a source_video with no " + f"mask and no hold_audio) keeps every video row - drop it, or pass " + f"hold_audio to re-denoise the source under its own soundtrack" + for name in AUDIO_PASS_REFUSES + if _given(arguments, name) + ][:1] + scale = arguments.get(LISTEN_SCALE_INPUT) + if scale is not None and not _deferred(scale): + if isinstance(scale, bool) or scale not in LISTEN_SCALES: + problems.append( + f"listen_scale is one of {', '.join(map(str, LISTEN_SCALES))} - " + f"the audio pass's canvas against the source's - not {scale!r}" + ) + return problems + + +def listen_canvas(width, height, scale): + """The (width, height) an audio pass at `scale` runs its kept rows on: the + source's canvas scaled and snapped to the nearest TILE_GRID multiple, ties + up, never below one cell of it.""" + return tuple( + max(TILE_GRID, int(side * scale / TILE_GRID + 0.5) * TILE_GRID) + for side in (width, height) + ) + + def media_problems(name, shape, width, height, num_frames=None): """Why a `source_video` or mask of `shape` - (frames, height, width) - does not fit a `width` x `height` canvas, as a list. `num_frames` is the @@ -962,132 +1038,3 @@ def media_problems(name, shape, width, height, num_frames=None): f"pad it to {on_grid}" ) return problems - - -# The look hold (#1191): pixel corrections applied to each finished segment -# before its carry is taken. Family-independent -LOOK_HOLD_PARTS = ("pickup", "creep", "texture", "precompensate") -# Parts that act on a single carried keyframe, so need `last_frame` -LOOK_HOLD_KEYFRAME_PARTS = ("pickup", "precompensate") - - -def chain_look_hold_flags(chain): - """The four look-hold switches as {part: bool}, or None when off. `true` - turns on all four; an object names the ones it wants.""" - hold = chain.get("look_hold") - if hold is True: - return dict.fromkeys(LOOK_HOLD_PARTS, True) - if isinstance(hold, dict) and any(hold.get(part) is True for part in hold): - return {part: hold.get(part) is True for part in LOOK_HOLD_PARTS} - return None - - -def chain_look_hold_problem(chain): - """The reason a chain block's `look_hold` is refused, or None: a value that - is neither a boolean nor an object of the four boolean switches, or an - explicit `pickup` / `precompensate` on a continuity other than `last_frame` - (a guide or last_segment carry is a whole clip, not one keyframe - `true` - alone quietly skips those two there). Unresolved references are left to the - run.""" - if "look_hold" not in chain: - return None - hold = chain["look_hold"] - if isinstance(hold, str) and references.is_ref(references.UNRESOLVED, hold): - return None - expected = ( - "expected true, false or an object of " - + ", ".join(LOOK_HOLD_PARTS) - + " booleans" - ) - if isinstance(hold, bool): - return None - if not isinstance(hold, dict): - return f"invalid look_hold {hold!r} - {expected}" - unknown = sorted(set(hold) - set(LOOK_HOLD_PARTS)) - if unknown: - return f"look_hold has unknown key(s) {unknown} - {expected}" - for part, value in hold.items(): - if not isinstance(value, bool): - return f"look_hold {part} must be true or false, got {value!r}" - continuity = chain.get("continuity", "last_frame") - if isinstance(continuity, str) and references.is_ref( - references.UNRESOLVED, continuity - ): - return None - if continuity != "last_frame": - wrong = [p for p in LOOK_HOLD_KEYFRAME_PARTS if hold.get(p)] - if wrong: - return ( - f"look_hold {' and '.join(wrong)} need continuity 'last_frame' - " - f"the {continuity!r} carry is a whole clip, not one keyframe; " - "creep and texture work on any continuity" - ) - return None - - -# The per-segment check and its bounded re-roll (chain_check.py): the fields a -# `check` object may set (`face` is reserved, null only) and the most re-rolls -# a checked segment may take. Family-independent -CHAIN_CHECK_FIELDS = ("jump_ratio_max", "look_ratio_max", "texture_ratio_max", "face") -CHAIN_RETRIES_MAX = 3 - - -def _number_over(value, floor): - """Whether `value` is a finite number above `floor` (never a bool), or an - unresolved reference, which is left to the run.""" - if isinstance(value, str): - return references.is_ref(references.UNRESOLVED, value) - return ( - not isinstance(value, bool) - and isinstance(value, (int, float)) - and math.isfinite(value) - and value > floor - ) - - -def chain_check_problem(chain): - """The reason a chain block's `check` or `retries` is refused, or None. - `check` is true (the default ceilings), false, or an object of - `CHAIN_CHECK_FIELDS`: the ratio ceilings above 0, `texture_ratio_max` - above 1 (it bounds the ratio both ways) or null, `face` null. `retries` - is a whole number from 0 to `CHAIN_RETRIES_MAX`; above 0 it needs the - check on, as with no check nothing fails. Unresolved references are left - to the run, whose `ChainConfig` asks this same rule.""" - check = chain.get("check") - expected = "expected true, false or an object of " + ", ".join(CHAIN_CHECK_FIELDS) - if isinstance(check, dict): - unknown = sorted(set(check) - set(CHAIN_CHECK_FIELDS)) - if unknown: - return f"check has unknown key(s) {unknown} - {expected}" - for name in ("jump_ratio_max", "look_ratio_max"): - if name in check and not _number_over(check[name], 0): - return f"check {name} must be a number above 0, got {check[name]!r}" - texture = check.get("texture_ratio_max") - if texture is not None and not _number_over(texture, 1): - return ( - "check texture_ratio_max bounds the fine-band ratio both ways, " - f"so must be a number above 1 or null, got {texture!r}" - ) - if check.get("face") is not None: - return f"check face is reserved and must be null, got {check['face']!r}" - elif not ( - check is None - or isinstance(check, bool) - or (isinstance(check, str) and _number_over(check, 0)) - ): - return f"invalid check {check!r} - {expected}" - retries = chain.get("retries", 0) - if isinstance(retries, str) and _number_over(retries, 0): - return None - whole = isinstance(retries, int) and not isinstance(retries, bool) - if not whole or not 0 <= retries <= CHAIN_RETRIES_MAX: - return ( - f"retries must be a whole number between 0 and {CHAIN_RETRIES_MAX}, " - f"got {retries!r}" - ) - if retries and (check is None or check is False): - return ( - f"retries {retries} needs check on - a re-roll replaces a segment " - "its check failed, and with no check no segment fails" - ) - return None diff --git a/dw/pipeline_processors/pipeline_h3.py b/dw/pipeline_processors/pipeline_h3.py index 1b4719fce..b02306d3d 100644 --- a/dw/pipeline_processors/pipeline_h3.py +++ b/dw/pipeline_processors/pipeline_h3.py @@ -11,7 +11,7 @@ from .h3_anchor import anchor_refusal, prepared_anchor from .h3_guides import guide_frames_array, guides_refusal, prepared_guide from .h3_hold import hold_audio_reference, holds_audio, refines -from .h3_mask import binarized, edits_regions +from .h3_mask import binarized, edits_regions, listen_frames, passes_audio from .h3_rules import ( GUIDES_INPUT, GUIDE_LIMIT, @@ -20,7 +20,9 @@ LATENT_ANCHOR_STRENGTH_INPUT, HELD_AUDIO_OUTPUT, HELD_AUDIO_RATE_OUTPUT, + HELD_VIDEO_OUTPUT, HOLD_AUDIO_INPUT, + LISTEN_SCALE_INPUT, MASK_IMAGE_INPUT, MASK_INPUTS, REFINE_SIGMAS_INPUT, @@ -29,6 +31,8 @@ REGENERATE_MASK, SOURCE_VIDEO_INPUT, edits_region, + is_audio_pass, + listen_canvas, anchor_strength_problem, guide_noise_problem, mask_problems, @@ -48,6 +52,35 @@ def latent_anchor_source(chain_source, value): return "the frames given in the step's arguments" +def with_audio_pass(pipeline, arguments, where): + """An audio pass's arguments (dw/pipeline_processors/h3_mask.py): the + full-size source as `HELD_VIDEO_OUTPUT`, asked for beside `videos` so + the result plays it in place of the decode, and at a `listen_scale` + below 1 the canvas scaled and the source area-resampled onto it. The + frame count - and so the audio's length - is the source's either way. + + Raises: + ValueError: If this pipeline has no audio-pass blocks + """ + if not passes_audio(pipeline): + raise ValueError( + f"{where}: a source_video with no mask and no hold_audio is an " + f"audio pass, and this pipeline has no block to hold its picture" + ) + source = arguments[SOURCE_VIDEO_INPUT] + arguments[HELD_VIDEO_OUTPUT] = source + scale = arguments.get(LISTEN_SCALE_INPUT) + if scale is not None and scale != 1: + width, height = listen_canvas(arguments["width"], arguments["height"], scale) + arguments["width"], arguments["height"] = width, height + arguments[SOURCE_VIDEO_INPUT] = listen_frames(source, width, height) + output = arguments.get("output") + names = [output] if isinstance(output, str) else list(output or []) + if "videos" in names: + arguments["output"] = names + [HELD_VIDEO_OUTPUT] + return arguments + + class H3Arguments: """The H3 argument wrappers, mixed into `Pipeline` (which supplies `name`, `pipeline`, `base_dir`, `segment_label` and `latent_anchor_source`).""" @@ -211,6 +244,8 @@ def _with_mask(self, arguments): (dw/pipeline_processors/h3_mask.py). A step without `width`, `height` or `num_frames` renders the source's. + An audio pass (no mask, no `hold_audio`) carries the source's own + frames for the result (`with_audio_pass`). Raises: ValueError: If this pipeline cannot take a regional edit, the @@ -278,6 +313,8 @@ def _with_mask(self, arguments): arguments[REGENERATE_MASK] = mask if problems: raise ValueError(f"{where}: {problems[0]}") + if is_audio_pass(arguments): + return with_audio_pass(self.pipeline, arguments, where) return arguments def _check_refine(self, arguments): diff --git a/tests/test_chain.py b/tests/test_chain.py index e7f0f8f67..053a8349c 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -1547,7 +1547,7 @@ def test_segment_one_is_left_as_rendered(self): assert (numpy.asarray(out.frames[0]) == raw).all() def test_validation_refuses_bad_shapes_and_guide_keyframe_parts(self): - from dw.pipeline_processors.h3_rules import chain_look_hold_problem + from dw.pipeline_processors.chain_rules import chain_look_hold_problem assert chain_look_hold_problem({}) is None assert chain_look_hold_problem({"look_hold": True}) is None diff --git a/tests/test_chain_check.py b/tests/test_chain_check.py index f0fa848ba..76ab13dd4 100644 --- a/tests/test_chain_check.py +++ b/tests/test_chain_check.py @@ -15,7 +15,7 @@ check_segment, segment_check, ) -from dw.pipeline_processors.h3_rules import chain_check_problem +from dw.pipeline_processors.chain_rules import chain_check_problem def noise(seed, size=32): diff --git a/tests/test_h3_audio_pass.py b/tests/test_h3_audio_pass.py new file mode 100644 index 000000000..ec4c3c33b --- /dev/null +++ b/tests/test_h3_audio_pass.py @@ -0,0 +1,619 @@ +"""MiniMax-H3 audio pass (#1192): a `source_video` with no mask and no +`hold_audio` keeps every video row and generates only the audio. The rules, +the block insertion, the two blocks on a synthetic pipeline state, the call's +argument preparation and the result's frames.""" + +import json +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pytest + +from dw.hold_audio import mask_errors +from dw.pipeline_processors import h3_mask +from dw.pipeline_processors.h3_hold import core_denoise_sequences +from dw.pipeline_processors.h3_rules import ( + AUDIO_PASS_BLOCK, + AUDIO_PASS_ROWS, + HELD_VIDEO_OUTPUT, + LISTEN_SCALE_INPUT, + REGENERATE_MASK, + RELEASE_VIDEO_BLOCK, + SOURCE_BLOCK, + SOURCE_ROWS, + edits_region, + is_audio_pass, + listen_canvas, + mask_problems, + refine_problems, +) + +torch = pytest.importorskip("torch") +minimax = pytest.importorskip("diffusers.modular_pipelines.minimax_h3") +from diffusers.modular_pipelines.minimax_h3.denoise import ( # noqa: E402 + MiniMaxH3LoopSchedulerStep, +) +from diffusers.modular_pipelines.modular_pipeline import PipelineState # noqa: E402 +from diffusers.schedulers import MiniMaxH3Scheduler # noqa: E402 + +ROOT = Path(__file__).resolve().parent.parent +WORKFLOWS = ("t2va", "fl2va", "ref2va") +# A 128x64 source, 5 frames (the test canvas, not H3's real grid) +WIDTH, HEIGHT, FRAMES = 128, 64, 5 + + +def source_frames(width=WIDTH, height=HEIGHT, frames=FRAMES, seed=0): + rng = np.random.default_rng(seed) + return rng.integers(0, 256, (frames, height, width, 3), dtype=np.uint8) + + +def pipeline_with_blocks(): + pipeline = minimax.MiniMaxH3Blocks().get_workflow("t2va").init_pipeline() + h3_mask.insert_mask_blocks(pipeline) + return pipeline + + +def calling_step(pipeline, chained=False): + return SimpleNamespace( + name="sound", + pipeline=pipeline, + base_dir=None, + segment_label="segment 1" if chained else None, + ) + + +def with_mask(pipeline, arguments, chained=False): + from dw.pipeline_processors.pipeline import Pipeline + + return Pipeline._with_mask(calling_step(pipeline, chained), arguments) + + +def audio_pass_arguments(**overrides): + arguments = { + "source_video": source_frames(), + "prompt": "a voice says [English] hello", + "output": ["videos", "audio", "sampling_rate"], + } + arguments.update(overrides) + return {k: v for k, v in arguments.items() if v is not None} + + +# The three the brief names + + +def test_the_output_frames_are_the_sources_own(): + """The call carries the full-size source, the pipeline returns it beside + its decode, and the result plays the source's pixels - not the decode's - + under the generated track, at the rate H3 reports.""" + from dw.media_types import rated_videos + from dw.output_extraction import modular_artifacts + from PIL import Image + + source = source_frames() + arguments = with_mask( + pipeline_with_blocks(), audio_pass_arguments(source_video=source) + ) + assert arguments["output"] == [ + "videos", + "audio", + "sampling_rate", + HELD_VIDEO_OUTPUT, + ] + + # What the pipeline would hand back: a VAE round trip that is not the source + decoded = [Image.fromarray(np.zeros((HEIGHT, WIDTH, 3), np.uint8))] * FRAMES + generated = torch.randn(1, 2, 4800) + result = { + "videos": rated_videos([decoded], 24), + "audio": generated, + "sampling_rate": 48000, + HELD_VIDEO_OUTPUT: arguments[HELD_VIDEO_OUTPUT], + } + artifacts = modular_artifacts(result) + + assert len(artifacts) == 1 + played = artifacts[0] + assert np.array_equal(np.stack([np.asarray(f) for f in played.frames]), source) + assert played.fps == 24 + assert torch.equal(played.audio, generated[0]) + assert played.sample_rate == 48000 + + +def test_listen_scale_half_runs_the_kept_rows_at_half_canvas(): + source = source_frames() + arguments = with_mask( + pipeline_with_blocks(), + audio_pass_arguments(source_video=source, listen_scale=0.5), + ) + assert (arguments["width"], arguments["height"]) == (WIDTH // 2, HEIGHT // 2) + assert arguments["source_video"].shape == (FRAMES, HEIGHT // 2, WIDTH // 2, 3) + assert arguments["source_video"].dtype == np.uint8 + # The audio's length follows the frame count, which the scale leaves alone + assert arguments["num_frames"] == FRAMES + # The picture handed back is still the full-size source + assert np.array_equal(arguments[HELD_VIDEO_OUTPUT], source) + + +def test_a_mask_or_hold_audio_keeps_todays_meaning(): + edit = { + "source_video": "asset:take.mp4", + "mask_image": "asset:m.png", + "refine_strength": 0.9, + "hold_audio": "asset:take.mp4", + "num_inference_steps": 5, + } + assert not is_audio_pass(edit) + assert mask_problems(edit) == [] and refine_problems(edit) == [] + + whole = {k: v for k, v in edit.items() if k != "mask_image"} + assert not is_audio_pass(whole) + assert mask_problems(whole) == [] and refine_problems(whole) == [] + + # A mask with no hold_audio is still refused for its soundtrack + unheld = {k: v for k, v in edit.items() if k != "hold_audio"} + assert not is_audio_pass(unheld) + assert any("hold_audio" in p for p in refine_problems(unheld)) + + # hold_audio with no refine_strength is still refused for the strength + no_strength = {k: v for k, v in whole.items() if k != "refine_strength"} + assert any("refine_strength" in p for p in mask_problems(no_strength)) + + # And the call: a regional edit gets no held video and keeps its canvas + from PIL import Image + + mask = np.zeros((HEIGHT, WIDTH, 3), dtype=np.uint8) + mask[:8, :8] = 255 + arguments = with_mask( + pipeline_with_blocks(), + audio_pass_arguments( + mask_image=Image.fromarray(mask), refine_strength=0.5, hold_audio=object() + ), + ) + assert HELD_VIDEO_OUTPUT not in arguments + assert REGENERATE_MASK in arguments + assert arguments["output"] == ["videos", "audio", "sampling_rate"] + assert (arguments["width"], arguments["height"]) == (WIDTH, HEIGHT) + + +# Rules + + +class TestRules: + def test_a_source_with_no_mask_and_no_hold_is_an_audio_pass(self): + assert is_audio_pass({"source_video": "asset:take.mp4"}) + assert is_audio_pass({"source_video": "asset:take.mp4", "hold_audio": None}) + assert not is_audio_pass({"source_video": None}) + assert not is_audio_pass({"prompt": "x"}) + + def test_an_audio_pass_needs_no_refine_strength(self): + assert mask_problems({"source_video": "asset:take.mp4"}) == [] + + @pytest.mark.parametrize("scale", [1, 1.0, 0.5, 0.25, "variable:scale"]) + def test_each_listen_scale_is_accepted(self, scale): + arguments = {"source_video": "asset:take.mp4", LISTEN_SCALE_INPUT: scale} + assert mask_problems(arguments) == [] + + @pytest.mark.parametrize("scale", [0.3, 2, 0, True, "half"]) + def test_another_listen_scale_is_refused(self, scale): + arguments = {"source_video": "asset:take.mp4", LISTEN_SCALE_INPUT: scale} + problems = mask_problems(arguments) + assert len(problems) == 1 + assert "listen_scale is one of 1, 0.5, 0.25" in problems[0] + + @pytest.mark.parametrize( + "name, value", + [("refine_strength", 0.9), ("refine_sigmas", [1.0, 0.0])], + ) + def test_a_refine_argument_is_refused_once(self, name, value): + arguments = {"source_video": "asset:take.mp4", name: value} + problems = mask_problems(arguments) + assert len(problems) == 1 + assert name in problems[0] and "audio pass" in problems[0] + + def test_refine_does_not_also_demand_hold_audio_on_an_audio_pass(self): + arguments = { + "source_video": "asset:take.mp4", + "refine_strength": 0.9, + "num_inference_steps": 5, + } + assert not any("hold_audio" in p for p in refine_problems(arguments)) + + def test_listen_scale_on_a_regional_edit_is_refused(self): + arguments = { + "source_video": "asset:take.mp4", + "mask_image": "asset:m.png", + "refine_strength": 0.9, + LISTEN_SCALE_INPUT: 0.5, + } + problems = mask_problems(arguments) + assert len(problems) == 1 and "listen_scale" in problems[0] + + def test_listen_scale_without_a_source_is_refused(self): + problems = mask_problems({LISTEN_SCALE_INPUT: 0.5}) + assert len(problems) == 1 and "audio pass" in problems[0] + assert edits_region({LISTEN_SCALE_INPUT: 0.5}, True) + assert not edits_region({LISTEN_SCALE_INPUT: 0.5}, False) + + def test_a_chained_audio_pass_is_refused(self): + problems = mask_problems({"source_video": "asset:take.mp4"}, chained=True) + assert len(problems) == 1 and "chain" in problems[0] + + def test_latents_beside_the_source_is_refused(self): + problems = mask_problems({"source_video": "asset:take.mp4", "latents": "x"}) + assert len(problems) == 1 and "latents" in problems[0] + + def test_the_listen_canvas_snaps_to_32_px(self): + assert listen_canvas(960, 544, 1) == (960, 544) + assert listen_canvas(960, 544, 0.5) == (480, 288) + assert listen_canvas(960, 544, 0.25) == (256, 128) + assert listen_canvas(48, 32, 0.25) == (32, 32) + + +def h3_step(**arguments): + return { + "name": "sound", + "pipeline": { + "configuration": {"component_type": "ModularPipeline"}, + "from_pretrained_arguments": {"workflow": "t2va"}, + "arguments": arguments, + }, + } + + +class TestStatic: + def test_an_audio_pass_validates(self): + assert mask_errors({"steps": [h3_step(source_video="asset:take.mp4")]}) == [] + + def test_a_bad_listen_scale_is_refused_at_the_source_path(self): + errors = mask_errors( + {"steps": [h3_step(source_video="asset:take.mp4", listen_scale=0.3)]} + ) + assert len(errors) == 1 + assert errors[0]["path"].endswith("source_video") + assert "listen_scale" in errors[0]["message"] + + def test_a_silent_source_is_fine_for_an_audio_pass(self, tmp_path): + # Adding a soundtrack is the point, so a source with none is no problem + from dw.media import probe_metadata + import av + + path = tmp_path / "silent.mp4" + with av.open(str(path), "w") as container: + stream = container.add_stream("libx264", rate=24) + stream.width, stream.height, stream.pix_fmt = 64, 48, "yuv420p" + for _ in range(5): + frame = av.VideoFrame.from_ndarray( + np.zeros((48, 64, 3), np.uint8), format="rgb24" + ) + container.mux(stream.encode(frame)) + container.mux(stream.encode()) + errors = mask_errors( + {"steps": [h3_step(source_video="silent.mp4")]}, + None, + str(tmp_path), + probe=probe_metadata, + ) + assert errors == [] + + +# Insertion + + +@pytest.fixture +def built(): + blocks_ = minimax.MiniMaxH3Blocks() + result = [("whole", blocks_.init_pipeline())] + for name in WORKFLOWS: + result.append((name, blocks_.get_workflow(name).init_pipeline())) + return result + + +class TestInsertion: + def test_the_pair_lands_before_denoise_and_after_denoise(self, built): + for label, pipeline in built: + assert not h3_mask.passes_audio(pipeline), label + assert h3_mask.insert_mask_blocks(pipeline) is True, label + assert h3_mask.passes_audio(pipeline), label + for prefix, sequence in core_denoise_sequences(pipeline): + names = list(sequence.sub_blocks) + at = names.index(prefix + AUDIO_PASS_BLOCK) + assert names[at + 1] == prefix + "denoise", label + assert names.index(prefix + SOURCE_BLOCK) < at, label + release = names.index(prefix + RELEASE_VIDEO_BLOCK) + assert at < release < names.index(prefix + "after_denoise"), label + + def test_a_second_call_adds_nothing(self, built): + for label, pipeline in built: + h3_mask.insert_mask_blocks(pipeline) + before = [list(s.sub_blocks) for _, s in core_denoise_sequences(pipeline)] + h3_mask.insert_mask_blocks(pipeline) + after = [list(s.sub_blocks) for _, s in core_denoise_sequences(pipeline)] + assert after == before, label + + def test_the_inserted_blocks_are_the_step_classes(self): + from dw.pipeline_processors.h3_mask_steps import ( + DwH3AudioPassStep, + DwH3ReleaseVideoStep, + ) + + assert h3_mask.audio_pass_blocks() == (DwH3AudioPassStep, DwH3ReleaseVideoStep) + + def test_listen_scale_is_a_pipeline_input(self, built): + for label, pipeline in built: + h3_mask.insert_mask_blocks(pipeline) + names = [param.name for param in pipeline._blocks.inputs] + assert LISTEN_SCALE_INPUT in names, label + + def test_the_hold_and_the_pass_coexist(self): + from dw.pipeline_processors.h3_hold import insert_audio_hold + + pipeline = minimax.MiniMaxH3Blocks().get_workflow("fl2va").init_pipeline() + insert_audio_hold(pipeline) + h3_mask.insert_mask_blocks(pipeline) + names = list(pipeline._blocks.sub_blocks) + assert names.index("denoise.dw_refine_schedule") < names.index( + "denoise." + AUDIO_PASS_BLOCK + ) + + +# The blocks, on a synthetic state + + +VIDEO_ROWS, AUDIO_ROWS, CONDITION, TEXT, CHANNELS = 6, 4, 2, 3, 8 + + +def audio_pass_state(held=True, condition=CONDITION, steps=4): + """A state as `set_timesteps` leaves it: the stock video (shift 12) and + audio (shift 3) schedules, `condition` leading video rows, the source + rows encoded behind them.""" + video = MiniMaxH3Scheduler(shift=12.0) + audio = MiniMaxH3Scheduler(shift=3.0) + video.set_timesteps(steps) + audio.set_timesteps(steps) + total = condition + VIDEO_ROWS + state = PipelineState() + generator = torch.Generator().manual_seed(0) + state.set("latents", torch.randn(total, CHANNELS, generator=generator)) + state.set("audio_latents", torch.randn(AUDIO_ROWS, CHANNELS, generator=generator)) + state.set(SOURCE_ROWS, state.get("latents")[condition:].clone()) + state.set("num_condition_video_rows", condition) + state.set("num_condition_audio_rows", 0) + state.set("timesteps", video.timesteps) + state.set("audio_timesteps", audio.timesteps) + state.set("video_indices", torch.arange(total)) + state.set("audio_indices", torch.arange(total, total + AUDIO_ROWS)) + state.set( + "text_indices", torch.arange(total + AUDIO_ROWS, total + AUDIO_ROWS + TEXT) + ) + state.set("width", WIDTH) + state.set("height", HEIGHT) + state.set("row_timestep_plan", ["stock"]) + state.set(HELD_VIDEO_OUTPUT, source_frames() if held else None) + components = SimpleNamespace( + _execution_device="cpu", scheduler=video, audio_scheduler=audio + ) + return components, state + + +def run_pass(components, state): + from dw.pipeline_processors.h3_mask_steps import DwH3AudioPassStep + + return DwH3AudioPassStep()(components, state) + + +def row_timesteps(plan_entry): + timesteps, indices = plan_entry + return timesteps[indices] + + +class TestAudioPassBlock: + def test_without_a_held_video_it_does_nothing(self): + components, state = audio_pass_state(held=False) + run_pass(components, state) + assert state.get("num_condition_video_rows") == CONDITION + assert state.get("row_timestep_plan") == ["stock"] + + def test_every_video_row_becomes_a_conditioning_row(self): + components, state = audio_pass_state() + run_pass(components, state) + assert state.get("num_condition_video_rows") == CONDITION + VIDEO_ROWS + assert state.get(AUDIO_PASS_ROWS) == VIDEO_ROWS + + def test_the_plan_pins_video_clean_and_steps_audio_from_full_noise(self): + components, state = audio_pass_state() + run_pass(components, state) + plan = state.get("row_timestep_plan") + audio_timesteps = state.get("audio_timesteps") + assert len(plan) == len(state.get("timesteps")) + total = CONDITION + VIDEO_ROWS + for step, entry in enumerate(plan): + rows = row_timesteps(entry) + assert torch.all(rows[:total] == 1.0), step + assert torch.allclose( + rows[total : total + AUDIO_ROWS], + audio_timesteps[step].float().expand(AUDIO_ROWS), + ), step + # The stock audio schedule starts from noise: t = 1 - sigma = 0 + assert float(audio_timesteps[0]) == 0.0 + + def test_no_source_rows_is_refused(self): + components, state = audio_pass_state() + state.set(SOURCE_ROWS, None) + with pytest.raises(ValueError, match="source"): + run_pass(components, state) + + def test_the_loop_steps_audio_and_never_a_video_row(self): + """`update`, the diffusers loop step, over the widened count: the + video rows end the loop as the exact source latent, and the audio rows + move.""" + components, state = audio_pass_state() + run_pass(components, state) + latents_before = state.get("latents").clone() + audio_before = state.get("audio_latents").clone() + update = MiniMaxH3LoopSchedulerStep() + block_state = SimpleNamespace( + latents=state.get("latents"), + audio_latents=state.get("audio_latents"), + num_condition_video_rows=state.get("num_condition_video_rows"), + num_condition_audio_rows=0, + audio_timesteps=state.get("audio_timesteps"), + ) + generator = torch.Generator().manual_seed(1) + for i, t in enumerate(state.get("timesteps")): + block_state.noise_pred = torch.randn( + 1, *block_state.latents.shape, generator=generator + ) + block_state.audio_noise_pred = torch.randn( + 1, *block_state.audio_latents.shape, generator=generator + ) + update(components, block_state, i, t) + assert torch.equal(block_state.latents, latents_before) + assert not torch.equal(block_state.audio_latents, audio_before) + + def test_the_release_puts_the_count_back(self): + from dw.pipeline_processors.h3_mask_steps import DwH3ReleaseVideoStep + + components, state = audio_pass_state() + run_pass(components, state) + DwH3ReleaseVideoStep()(components, state) + assert state.get("num_condition_video_rows") == CONDITION + assert state.get(AUDIO_PASS_ROWS) == 0 + + def test_the_release_is_a_no_op_off_the_audio_pass(self): + from dw.pipeline_processors.h3_mask_steps import DwH3ReleaseVideoStep + + components, state = audio_pass_state(held=False) + run_pass(components, state) + DwH3ReleaseVideoStep()(components, state) + assert state.get("num_condition_video_rows") == CONDITION + + def test_it_emits_what_it_held(self): + from dw.events import RunContext, activate_context, deactivate_context + + components, state = audio_pass_state() + events = [] + token = activate_context(RunContext(on_event=events.append)) + try: + run_pass(components, state) + finally: + deactivate_context(token) + logs = [e for e in events if e.get("event") == "log"] + assert len(logs) == 1 + assert logs[0]["held_video_rows"] == VIDEO_ROWS + assert logs[0]["listen_scale"] == 1 + assert "Audio pass" in logs[0]["message"] + + +# The call + + +class TestTheCall: + def test_an_unsized_audio_pass_renders_the_sources_canvas(self): + arguments = with_mask(pipeline_with_blocks(), audio_pass_arguments()) + assert (arguments["width"], arguments["height"], arguments["num_frames"]) == ( + WIDTH, + HEIGHT, + FRAMES, + ) + assert arguments["source_video"].shape == (FRAMES, HEIGHT, WIDTH, 3) + + def test_listen_scale_one_leaves_the_canvas(self): + arguments = with_mask( + pipeline_with_blocks(), audio_pass_arguments(listen_scale=1) + ) + assert (arguments["width"], arguments["height"]) == (WIDTH, HEIGHT) + assert arguments["source_video"] is arguments[HELD_VIDEO_OUTPUT] + + def test_a_quarter_listen_scale_snaps_the_canvas(self): + arguments = with_mask( + pipeline_with_blocks(), audio_pass_arguments(listen_scale=0.25) + ) + assert (arguments["width"], arguments["height"]) == (32, 32) + + def test_the_step_canvas_is_still_the_sources(self): + with pytest.raises(ValueError, match="must match"): + with_mask( + pipeline_with_blocks(), + audio_pass_arguments(width=WIDTH // 2, listen_scale=0.5), + ) + + def test_no_videos_output_asks_for_no_held_video(self): + arguments = with_mask( + pipeline_with_blocks(), audio_pass_arguments(output=["audio"]) + ) + assert arguments["output"] == ["audio"] + + def test_a_refine_strength_is_refused_at_the_call(self): + with pytest.raises(ValueError, match="audio pass"): + with_mask(pipeline_with_blocks(), audio_pass_arguments(refine_strength=0.5)) + + def test_a_pipeline_without_the_pass_blocks_refuses(self, monkeypatch): + from dw.pipeline_processors import pipeline_h3 + + monkeypatch.setattr(pipeline_h3, "passes_audio", lambda pipeline: False) + with pytest.raises(ValueError, match="audio pass"): + with_mask(pipeline_with_blocks(), audio_pass_arguments()) + + def test_area_resampling_keeps_a_flat_colour(self): + flat = np.full((3, 64, 128, 3), 77, dtype=np.uint8) + resized = h3_mask.listen_frames(flat, 32, 16) + assert resized.shape == (3, 16, 32, 3) + assert np.all(resized == 77) + + def test_area_resampling_averages_each_cell(self): + frames = np.zeros((1, 2, 2, 3), dtype=np.uint8) + frames[0, 0, 0] = 200 + assert h3_mask.listen_frames(frames, 1, 1)[0, 0, 0, 0] == 50 + + +class TestHeldVideos: + def test_an_np_batch_is_replaced_as_floats(self): + from dw.output_extraction import held_videos + + source = source_frames() + videos = np.zeros((2, FRAMES, HEIGHT, WIDTH, 3), dtype=np.float32) + held = held_videos(videos, source) + assert held.shape == videos.shape + assert np.allclose(held[1] * 255, source) + + def test_a_result_without_the_held_video_is_todays(self): + from dw.output_extraction import modular_artifacts + + frames = [[object()]] + artifacts = modular_artifacts({"videos": frames}) + assert artifacts == frames + + +# The draft template + + +DRAFT = ROOT / "bench" / "workflows" / "add-sound.json" + + +class TestDraftTemplate: + def definition(self): + return json.loads(DRAFT.read_text(encoding="utf-8")) + + def test_it_is_not_in_the_catalog(self): + assert not ( + ROOT / "workflows" / "templates" / "minimax" / "add-sound.json" + ).exists() + + def test_it_validates(self): + from dw.workflow import Workflow + + assert Workflow(self.definition(), None, str(DRAFT)).validation_errors({}) == [] + + def test_it_runs_without_a_distillation_lora_on_the_dialogue_schedule(self): + variables = self.definition()["variables"] + assert variables["lora_model_name"] is None + assert variables["num_inference_steps"] == 12 + assert (variables["video_shift"], variables["audio_shift"]) == (12.0, 3.0) + assert variables["listen_scale"] == 1 + + def test_it_is_an_audio_pass(self): + step = self.definition()["steps"][0]["pipeline"]["arguments"] + assert "source_video" in step + for name in ("hold_audio", "mask_image", "mask_video", "refine_strength"): + assert name not in step diff --git a/tests/test_h3_mask.py b/tests/test_h3_mask.py index d87125509..9cff038f5 100644 --- a/tests/test_h3_mask.py +++ b/tests/test_h3_mask.py @@ -578,6 +578,8 @@ def args(**overrides): "mask_video": "asset:mask", "mask_grow": 1, "refine_strength": 0.9, + # Without it a source and no mask is an audio pass (#1192) + "hold_audio": "asset:clip", } arguments.update(overrides) return {k: v for k, v in arguments.items() if v is not None} @@ -842,6 +844,8 @@ def probed_errors(tmp_path, **arguments): from dw.media import probe_metadata arguments.setdefault("refine_strength", 0.9) + # Without it a source and no mask is an audio pass (#1192) + arguments.setdefault("hold_audio", "asset:clip.mp4") return mask_errors( {"steps": [h3_step(**arguments)]}, None, str(tmp_path), probe=probe_metadata ) @@ -1045,8 +1049,9 @@ def realized(self, default, value): variables = {"refine_strength": default} set_variables({"refine_strength": value}, variables) step = h3_step( - **args(refine_strength="variable:refine_strength"), - hold_audio="asset:clip.mp4", + **args( + refine_strength="variable:refine_strength", hold_audio="asset:clip.mp4" + ), num_inference_steps=5, ) return replace_variables(step, variables) @@ -1058,8 +1063,7 @@ def errors(self, step): def test_a_literal_string_is_refused_at_its_path(self): step = h3_step( - **args(refine_strength="0.8"), - hold_audio="asset:clip.mp4", + **args(refine_strength="0.8", hold_audio="asset:clip.mp4"), num_inference_steps=5, ) errors = self.errors(step) From e716fd2ca66d0aeae3de99cc0b630417ce75d617 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 20:47:36 -0600 Subject: [PATCH 033/405] fix(bench): S4 slices the music bed at its declared sample rate The music_bed step hands over a raw waveform, which carries no rate of its own; slice_audio needs sample_rate beside it. Rerun ids are fresh so the failed runs' cached shots do not serve the noise pair. Co-Authored-By: Claude Opus 5.5 --- bench/arms/noise.json | 7 ++++--- bench/workflows/s4-montage.json | 1 + 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/bench/arms/noise.json b/bench/arms/noise.json index c4a2200e3..2fda626ff 100644 --- a/bench/arms/noise.json +++ b/bench/arms/noise.json @@ -49,7 +49,8 @@ "scene": "s4-montage", "workflow": "s4-montage.json", "arguments": {}, - "seed": 1166 + "seed": 1166, + "workflow_id_suffix": "-noise-a2" }, { "name": "noise-b-s4", @@ -57,7 +58,7 @@ "workflow": "s4-montage.json", "arguments": {}, "seed": 1166, - "workflow_id_suffix": "-noise-b" + "workflow_id_suffix": "-noise-b2" }, { "name": "noise-a-s5", @@ -89,4 +90,4 @@ "seed": 1166, "workflow_id_suffix": "-noise-b" } -] \ No newline at end of file +] diff --git a/bench/workflows/s4-montage.json b/bench/workflows/s4-montage.json index e5b3d97cf..8fae1bc3d 100644 --- a/bench/workflows/s4-montage.json +++ b/bench/workflows/s4-montage.json @@ -434,6 +434,7 @@ "command": "slice_audio", "arguments": { "audio": "previous_result:music_bed", + "sample_rate": "variable:sample_rate", "start_frame": "variable:score_start_frame", "num_frames": "variable:total_frames", "fps": "variable:fps" From b71275c428ae9cc0674a4454473d00ad13093932 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 20:48:47 -0600 Subject: [PATCH 034/405] feat(bench): driver measure mode re-measures a record without re-running Co-Authored-By: Claude Opus 5.5 --- bench/driver.py | 41 +++++++++++++++++++ tests/test_bench_driver.py | 83 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) diff --git a/bench/driver.py b/bench/driver.py index c95fdb3ed..ea78ccf02 100644 --- a/bench/driver.py +++ b/bench/driver.py @@ -249,6 +249,30 @@ def flush(): return records +def measure_records( + records, scenes, arms, server, token, workspace=None, out_path=None +): + """Measure again, without running again, every succeeded record, with the + scene crops and moments as they are now (an arm's own moments, when an + arms file names one, win). Other records pass through untouched.""" + moments = {arm.name: arm.moments for arm in arms} + for record in records: + if record.get("status") != "succeeded": + continue + job = http("GET", f"/api/jobs/{record['job_id']}", server, token) + record["measures"] = measure( + job, + scenes[record["scene"]], + server, + token, + workspace, + moments.get(record["arm"]), + ) + if out_path: + Path(out_path).write_text(json.dumps(records, indent=2)) + return records + + def report(records): lines = [] keys = sorted({k for r in records for k in r.get("measures", {})}) @@ -292,6 +316,12 @@ def main(argv=None): run.add_argument("--measure", action="store_true") run.add_argument("--out") run.add_argument("--timeout", type=int, default=7200) + meas = sub.add_parser( + "measure", help="measure a record's succeeded jobs again; rewrites it" + ) + meas.add_argument("record") + meas.add_argument("--scenes", default=str(HERE / "scenes.json")) + meas.add_argument("--arms", help="arms file whose per-arm moments apply") rep = sub.add_parser("report", help="markdown tables from records") rep.add_argument("records", nargs="+") args = parser.parse_args(argv) @@ -302,6 +332,17 @@ def main(argv=None): print() return 0 token = _token(args) + if args.command == "measure": + measure_records( + json.loads(Path(args.record).read_text()), + load_scenes(args.scenes), + load_arms(args.arms) if args.arms else [], + args.server, + token, + args.workspace, + args.record, + ) + return 0 records = run_arms( load_arms(args.arms), load_scenes(args.scenes) if args.measure else {}, diff --git a/tests/test_bench_driver.py b/tests/test_bench_driver.py index 75233ed8b..f94ecef6c 100644 --- a/tests/test_bench_driver.py +++ b/tests/test_bench_driver.py @@ -201,3 +201,86 @@ def fake(method, path, server, token, payload=None, params=None): job = {"id": "j", "manifest": [{"files": ["o.mp4"], "subfolder": "final"}]} driver.measure(job, scene, "http://h", "tok", moments=["frame:9"]) assert calls[-1]["at"] == ["frame:9"] + + +def _fake_server(compare_calls): + def fake(method, path, server, token, payload=None, params=None): + if path.startswith("/api/jobs/"): + return { + "id": path.rsplit("/", 1)[1], + "status": "succeeded", + "started_at": 0, + "finished_at": 600, + "manifest": [{"subfolder": "final", "files": ["out/x.mp4"]}], + "output_kinds": {"out/x.mp4": "video"}, + } + if path.endswith("/assess"): + return {"seams": [{"jump_ratio": 2.0, "look_ratio": 1.0}]} + compare_calls.append(payload) + cell = {"texture": {"luma": {"fine": 10.0, "mid": 10.0, "coarse": 10.0}}} + return {"texture": [{"cells": [cell, cell]}]} + + return fake + + +def test_measure_records_remeasures_succeeded_jobs_with_current_crops( + monkeypatch, tmp_path +): + calls = [] + monkeypatch.setattr(driver, "http", _fake_server(calls)) + scenes = tmp_path / "scenes.json" + scenes.write_text( + json.dumps( + [{"name": "s", "crop": [5, 6, 7, 8], "moments": ["frame:3", "frame:9"]}] + ) + ) + rec = tmp_path / "rec.json" + rec.write_text( + json.dumps( + [ + { + "arm": "a", + "scene": "s", + "seed": 1, + "job_id": "j1", + "status": "succeeded", + "measures": {"stale": 1.0}, + }, + { + "arm": "b", + "scene": "s", + "seed": 1, + "job_id": "j2", + "status": "failed", + "error": "boom", + }, + ] + ) + ) + token = tmp_path / "t" + token.write_text("tok") + argv = ["--token-file", str(token), "measure", str(rec), "--scenes", str(scenes)] + assert driver.main(argv) == 0 + saved = json.loads(rec.read_text()) + assert saved[0]["measures"]["seam_jump_mean"] == 2.0 + assert "stale" not in saved[0]["measures"] + assert saved[0]["measures"]["wall_minutes"] == 10.0 + assert saved[1] == { + "arm": "b", + "scene": "s", + "seed": 1, + "job_id": "j2", + "status": "failed", + "error": "boom", + } + assert calls[0]["crop"] == [5, 6, 7, 8] and calls[0]["at"] == ["frame:3", "frame:9"] + + +def test_measure_records_takes_arm_moments_from_the_arms_file(monkeypatch, tmp_path): + calls = [] + monkeypatch.setattr(driver, "http", _fake_server(calls)) + scenes = {"s": driver.Scene("s", [0, 0, 9, 9], ["frame:1", "frame:2"])} + arms = [driver.Arm("a", "s", "w.json", {}, 1, moments=["frame:62", "frame:99"])] + records = [{"arm": "a", "scene": "s", "job_id": "j1", "status": "succeeded"}] + driver.measure_records(records, scenes, arms, "http://h", "tok") + assert calls[0]["at"] == ["frame:62", "frame:99"] From 195b25c351ea9c78bfb7526044e9077b2746475e Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:09:50 -0600 Subject: [PATCH 035/405] bench: noise floor and cheap-setting checks Twelve noise runs and six cheap-setting runs, the S4 pair rerun after the sample_rate fix, crops verified (S4 and S6 moved) and re-measured, and the 6-segment ordering checked against #1175. test_scenes_are_well_formed read the crop as a corner box; it is [x, y, w, h]. Co-Authored-By: Claude Opus 5.5 --- bench/records/cheap-checks.json | 128 +++++++++++++++ bench/records/noise-floor.json | 254 +++++++++++++++++++++++++++++ bench/records/noise-floor.md | 274 ++++++++++++++++++++++++++++++++ bench/scenes.json | 24 +-- tests/test_bench_workflows.py | 4 +- 5 files changed, 670 insertions(+), 14 deletions(-) create mode 100644 bench/records/cheap-checks.json create mode 100644 bench/records/noise-floor.json create mode 100644 bench/records/noise-floor.md diff --git a/bench/records/cheap-checks.json b/bench/records/cheap-checks.json new file mode 100644 index 000000000..2a702d49c --- /dev/null +++ b/bench/records/cheap-checks.json @@ -0,0 +1,128 @@ +[ + { + "arm": "cheap-a-s1", + "scene": "s1-static", + "seed": 1166, + "job_id": "a438bffb764f", + "status": "succeeded", + "error": null, + "output": "BenchS1Static/20261104-003017-655187be/final/BenchS1Static-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 0.882, + "seam_jump_max": 1.0, + "seam_look_mean": 1.6219999999999999, + "seam_look_max": 2.2, + "seam_level_step_db_mean": 8.854000000000001, + "seam_level_step_db_max": 17.77, + "texture_fine_seg_last": 0.502764996926132, + "texture_mid_seg_last": 0.17135204694593162, + "texture_coarse_seg_last": 0.007668630503602891, + "wall_minutes": 10.919583070278168 + } + }, + { + "arm": "cheap-b-s1", + "scene": "s1-static", + "seed": 1166, + "job_id": "7e18115259a6", + "status": "succeeded", + "error": null, + "output": "BenchS1Static/20261104-005919-f3588d64/final/BenchS1Static-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 0.8140000000000001, + "seam_jump_max": 0.95, + "seam_look_mean": 1.392, + "seam_look_max": 2.54, + "seam_level_step_db_mean": 16.613999999999997, + "seam_level_step_db_max": 27.02, + "texture_fine_seg_last": 0.8544273526151175, + "texture_mid_seg_last": 0.3867239073129147, + "texture_coarse_seg_last": 0.0022898952157175668, + "wall_minutes": 10.515459883213044 + } + }, + { + "arm": "cheap-c-s1", + "scene": "s1-static", + "seed": 1166, + "job_id": "9cc328702d45", + "status": "succeeded", + "error": null, + "output": "BenchS1Static/20261104-011029-6cf2a1b0/final/BenchS1Static-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.162, + "seam_jump_max": 1.63, + "seam_look_mean": 1.404, + "seam_look_max": 2.83, + "seam_level_step_db_mean": 7.5280000000000005, + "seam_level_step_db_max": 16.58, + "texture_fine_seg_last": 0.18974276936522114, + "texture_mid_seg_last": 0.10538200739962529, + "texture_coarse_seg_last": 0.035110393085034976, + "wall_minutes": 13.661045277118683 + } + }, + { + "arm": "cheap-a-s2", + "scene": "s2-arc", + "seed": 1166, + "job_id": "80eb57300971", + "status": "succeeded", + "error": null, + "output": "BenchS2Arc/20261104-014743-a12ce1e7/final/BenchS2Arc-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.3619999999999999, + "seam_jump_max": 2.09, + "seam_look_mean": 1.286, + "seam_look_max": 1.74, + "seam_level_step_db_mean": 1.4620000000000002, + "seam_level_step_db_max": 3.43, + "texture_fine_seg_last": 3.083963956401898, + "texture_mid_seg_last": 2.2932110623147106, + "texture_coarse_seg_last": 1.4297775601219735, + "wall_minutes": 10.56614822546641 + } + }, + { + "arm": "cheap-b-s2", + "scene": "s2-arc", + "seed": 1166, + "job_id": "ebf09ad9fa12", + "status": "succeeded", + "error": null, + "output": "BenchS2Arc/20261104-020905-75a1b6b3/final/BenchS2Arc-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.166, + "seam_jump_max": 1.63, + "seam_look_mean": 1.6940000000000002, + "seam_look_max": 3.56, + "seam_level_step_db_mean": 1.59, + "seam_level_step_db_max": 4.06, + "texture_fine_seg_last": 2.883403862056931, + "texture_mid_seg_last": 1.7066904530022595, + "texture_coarse_seg_last": 0.10520650290810232, + "wall_minutes": 10.593446962038676 + } + }, + { + "arm": "cheap-c-s2", + "scene": "s2-arc", + "seed": 1166, + "job_id": "10cbd9b21a72", + "status": "succeeded", + "error": null, + "output": "BenchS2Arc/20261104-023013-fa9a28f1/final/BenchS2Arc-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.8539999999999999, + "seam_jump_max": 2.87, + "seam_look_mean": 4.27, + "seam_look_max": 6.89, + "seam_level_step_db_mean": 1.6260000000000001, + "seam_level_step_db_max": 3.35, + "texture_fine_seg_last": 2.5632527861842003, + "texture_mid_seg_last": 2.040688435068713, + "texture_coarse_seg_last": 2.0723558586423962, + "wall_minutes": 13.248008183638255 + } + } +] \ No newline at end of file diff --git a/bench/records/noise-floor.json b/bench/records/noise-floor.json new file mode 100644 index 000000000..2c7df3999 --- /dev/null +++ b/bench/records/noise-floor.json @@ -0,0 +1,254 @@ +[ + { + "arm": "noise-a-s1", + "scene": "s1-static", + "seed": 1166, + "job_id": "00e4000a3ffb", + "status": "succeeded", + "error": null, + "output": "BenchS1Static/20261104-000916-390b57bc/final/BenchS1Static-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 0.9311111111111112, + "seam_jump_max": 1.09, + "seam_look_mean": 1.528888888888889, + "seam_look_max": 2.61, + "seam_level_step_db_mean": 9.856666666666667, + "seam_level_step_db_max": 36.56, + "texture_fine_seg_last": 1.1178899576395858, + "texture_mid_seg_last": 0.40150993772889665, + "texture_coarse_seg_last": 0.04875222289931379, + "wall_minutes": 21.064961115519207 + } + }, + { + "arm": "noise-b-s1", + "scene": "s1-static", + "seed": 1166, + "job_id": "b4e31bb17593", + "status": "succeeded", + "error": null, + "output": "BenchS1Static-noise-b/20261104-004113-f663789a/final/BenchS1Static-noise-b-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 0.9311111111111112, + "seam_jump_max": 1.09, + "seam_look_mean": 1.528888888888889, + "seam_look_max": 2.61, + "seam_level_step_db_mean": 9.856666666666667, + "seam_level_step_db_max": 36.56, + "texture_fine_seg_last": 1.1178899576395858, + "texture_mid_seg_last": 0.40150993772889665, + "texture_coarse_seg_last": 0.04875222289931379, + "wall_minutes": 18.098664323488872 + } + }, + { + "arm": "noise-a-s2", + "scene": "s2-arc", + "seed": 1166, + "job_id": "04978dc6f338", + "status": "succeeded", + "error": null, + "output": "BenchS2Arc/20261104-010950-2f5ecfe0/final/BenchS2Arc-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.1944444444444444, + "seam_jump_max": 2.09, + "seam_look_mean": 1.2844444444444445, + "seam_look_max": 2.71, + "seam_level_step_db_mean": 2.035555555555556, + "seam_level_step_db_max": 5.54, + "texture_fine_seg_last": 3.5575967638583803, + "texture_mid_seg_last": 2.8636397627256116, + "texture_coarse_seg_last": 1.9746980403751424, + "wall_minutes": 20.045715339978535 + } + }, + { + "arm": "noise-b-s2", + "scene": "s2-arc", + "seed": 1166, + "job_id": "51d7b09236f6", + "status": "succeeded", + "error": null, + "output": "BenchS2Arc-noise-b/20261104-012953-430a6737/final/BenchS2Arc-noise-b-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.1944444444444444, + "seam_jump_max": 2.09, + "seam_look_mean": 1.2844444444444445, + "seam_look_max": 2.71, + "seam_level_step_db_mean": 2.035555555555556, + "seam_level_step_db_max": 5.54, + "texture_fine_seg_last": 3.5575967638583803, + "texture_mid_seg_last": 2.8636397627256116, + "texture_coarse_seg_last": 1.9746980403751424, + "wall_minutes": 17.843897787729897 + } + }, + { + "arm": "noise-a-s3", + "scene": "s3-dialogue-walk", + "seed": 1166, + "job_id": "3c76d1966485", + "status": "succeeded", + "error": null, + "output": "BenchS3DialogueWalk/20261104-015817-5942f052/final/BenchS3DialogueWalk-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.478, + "seam_jump_max": 1.83, + "seam_look_mean": 1.3780000000000001, + "seam_look_max": 1.61, + "seam_level_step_db_mean": 2.7700000000000005, + "seam_level_step_db_max": 8.81, + "texture_fine_seg_last": 0.17510029217190348, + "texture_mid_seg_last": 0.22613031907347164, + "texture_coarse_seg_last": 0.2075222005860499, + "wall_minutes": 10.790832726160685 + } + }, + { + "arm": "noise-b-s3", + "scene": "s3-dialogue-walk", + "seed": 1166, + "job_id": "d2095670cd89", + "status": "succeeded", + "error": null, + "output": "BenchS3DialogueWalk-noise-b/20261104-021940-da114475/final/BenchS3DialogueWalk-noise-b-chained_keyframe_to_video_audio.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.2120000000000002, + "seam_jump_max": 1.6, + "seam_look_mean": 1.324, + "seam_look_max": 1.87, + "seam_level_step_db_mean": 3.0, + "seam_level_step_db_max": 8.65, + "texture_fine_seg_last": 0.17818822030424697, + "texture_mid_seg_last": 0.20762310554096494, + "texture_coarse_seg_last": 0.43956359806218753, + "wall_minutes": 10.548045885562896 + } + }, + { + "arm": "noise-a-s4", + "scene": "s4-montage", + "seed": 1166, + "job_id": "568a3da301c4", + "status": "succeeded", + "error": null, + "output": "BenchS4Montage-noise-a2/20261104-024801-94372b51/final/BenchS4Montage-noise-a2-film.14-0.0.mp4", + "measures": { + "seam_jump_mean": 20.476666666666667, + "seam_jump_max": 22.69, + "seam_look_mean": 13.473333333333331, + "seam_look_max": 18.24, + "seam_level_step_db_mean": 3.4966666666666666, + "seam_level_step_db_max": 4.42, + "texture_fine_seg_last": 1.7140760391646614, + "texture_mid_seg_last": 0.5798479202075943, + "texture_coarse_seg_last": 1.214317730531653, + "wall_minutes": 19.715835420290627 + } + }, + { + "arm": "noise-b-s4", + "scene": "s4-montage", + "seed": 1166, + "job_id": "511b37c9bd70", + "status": "succeeded", + "error": null, + "output": "BenchS4Montage-noise-b2/20261104-024801-6c692da9/final/BenchS4Montage-noise-b2-film.14-0.0.mp4", + "measures": { + "seam_jump_mean": 20.476666666666667, + "seam_jump_max": 22.69, + "seam_look_mean": 13.473333333333331, + "seam_look_max": 18.24, + "seam_level_step_db_mean": 3.4966666666666666, + "seam_level_step_db_max": 4.42, + "texture_fine_seg_last": 1.7140760391646614, + "texture_mid_seg_last": 0.5798479202075943, + "texture_coarse_seg_last": 1.214317730531653, + "wall_minutes": 20.75935315291087 + } + }, + { + "arm": "noise-a-s5", + "scene": "s5-ltx-action", + "seed": 1166, + "job_id": "a91df8d3684c", + "status": "succeeded", + "error": null, + "output": "BenchS5LtxAction/20261104-005131-8fc124d3/final/BenchS5LtxAction-chained_image_to_video.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.245, + "seam_jump_max": 1.36, + "seam_look_mean": 2.085, + "seam_look_max": 2.43, + "seam_level_step_db_mean": 2.145, + "seam_level_step_db_max": 2.85, + "texture_fine_seg_last": 0.545653955553654, + "texture_mid_seg_last": 0.48674276499700386, + "texture_coarse_seg_last": 1.5416578973723583, + "wall_minutes": 3.1932440121968586 + } + }, + { + "arm": "noise-b-s5", + "scene": "s5-ltx-action", + "seed": 1166, + "job_id": "813a4ec70859", + "status": "succeeded", + "error": null, + "output": "BenchS5LtxAction-noise-b/20261104-005443-a7939f5d/final/BenchS5LtxAction-noise-b-chained_image_to_video.0-0.0.mp4", + "measures": { + "seam_jump_mean": 1.245, + "seam_jump_max": 1.36, + "seam_look_mean": 2.085, + "seam_look_max": 2.43, + "seam_level_step_db_mean": 2.1399999999999997, + "seam_level_step_db_max": 2.84, + "texture_fine_seg_last": 0.545653955553654, + "texture_mid_seg_last": 0.48674276499700386, + "texture_coarse_seg_last": 1.5416578973723583, + "wall_minutes": 1.9625712672869364 + } + }, + { + "arm": "noise-a-s6", + "scene": "s6-narration", + "seed": 1166, + "job_id": "54240901f86a", + "status": "succeeded", + "error": null, + "output": "BenchS6Narration/20261104-005645-a4669737/final/BenchS6Narration-film.9-0.0.mp4", + "measures": { + "seam_jump_mean": 40.2, + "seam_jump_max": 45.33, + "seam_look_mean": 70.635, + "seam_look_max": 82.75, + "seam_level_step_db_mean": 7.79, + "seam_level_step_db_max": 13.26, + "texture_fine_seg_last": 1.4010987900525822, + "texture_mid_seg_last": 1.2212536933691236, + "texture_coarse_seg_last": 0.8530977746815804, + "wall_minutes": 1.5750840504964192 + } + }, + { + "arm": "noise-b-s6", + "scene": "s6-narration", + "seed": 1166, + "job_id": "5670fac7bc34", + "status": "succeeded", + "error": null, + "output": "BenchS6Narration-noise-b/20261104-005815-12f0dc21/final/BenchS6Narration-noise-b-film.9-0.0.mp4", + "measures": { + "seam_jump_mean": 40.2, + "seam_jump_max": 45.33, + "seam_look_mean": 70.635, + "seam_look_max": 82.75, + "seam_level_step_db_mean": 7.8, + "seam_level_step_db_max": 13.28, + "texture_fine_seg_last": 1.4010987900525822, + "texture_mid_seg_last": 1.2212536933691236, + "texture_coarse_seg_last": 0.8530977746815804, + "wall_minutes": 1.314908997217814 + } + } +] \ No newline at end of file diff --git a/bench/records/noise-floor.md b/bench/records/noise-floor.md new file mode 100644 index 000000000..3a3cb31d1 --- /dev/null +++ b/bench/records/noise-floor.md @@ -0,0 +1,274 @@ +# Noise floor and cheap-setting checks + +Task 4 of the long-form media plan. Every scene's baseline arm was run twice on +lem (bench server, both cards) at seed 1166; the second run carries a distinct +inline workflow id (`-noise-b`) so the step cache cannot serve it. The floor +for a measure is the absolute difference of the two runs (`bench/verdict.py` +`noise_floor`). Records: `bench/records/noise-floor.json` (12 runs), +`bench/records/cheap-checks.json` (6 runs). Every measure is lower-is-better; +`texture_*_seg_last` is `|ln(last/first)|` of the crop's band energy. + +## Reading the floor + +- **The engine is deterministic for a seeded run.** S1, S2, S4, S5 and S6 give + bit-identical picture measures across the two runs, so their floor is exactly + 0 on every seam and texture measure (S5 and S6 differ by 0.005-0.02 dB in the + seam level step, an audio-encode wobble). Only S3, whose picture follows a + spoken line per segment, moves between runs. +- **A floor of 0 means the verdict has no tie zone.** `verdict()` calls any + difference past 0 a win or a loss. That is right for "did this arm change the + result", and says nothing about how much another seed would move the same + measure; the floor measures reproducibility of a run, not seed-to-seed + variance. A bench arm that must hold across seeds needs a second seed, which + this task did not run. +- **S3 is the only non-zero floor**: `texture_coarse_seg_last` 0.23 (0.21 vs + 0.44) and `seam_jump_mean` 0.27 are the large ones; use them for S3 verdicts. +- The two S4 runs ran side by side on the two cards and agree exactly. + +## S4 rerun + +The first S4 pair (jobs `0d08eccff9dc`, `14a8b42f73f0`) failed at step 11 of 15, +the `soundtrack` `slice_audio` step, with "slice_audio needs 'sample_rate' with +a raw waveform". Cause: the bench workflow (not the engine). `music_bed`'s +result hands over a raw Music 3 waveform that carries no rate of its own (the +result's `sample_rate: 44100` only labels the file at save time, and +`save: false` here), and `slice_audio` was given no `sample_rate`. The +`templates/minimax/music.json` and `music-video.json` templates pass +`sample_rate` beside a raw waveform for the same reason. Fix: `"sample_rate": +"variable:sample_rate"` on the `soundtrack` step (commit e716fd2c). The pair was +rerun with fresh ids (`-noise-a2`, `-noise-b2`) so the first pair's cached shots +did not serve it. Both succeeded in about 20 minutes each, not the 70 the +launch estimated. + +## Crop check (ruling R10) + +Frames were read through `GET /api/gallery/{name}/frames` at each scene's +moments, on a job of that scene, with the crop drawn over them. A crop must +stay clear of every subject at each moment, keep a 128 px short side (the +coarse band's longest wavelength; a smaller crop returns no coarse value), and +stay off the frame's border. + +| scene | crop before | decision | why | +|---|---|---|---| +| S1, S2 | `[0,0,240,150]` | unchanged (already verified under #1175) | | +| S3 | `[0,0,240,150]` | unchanged, now verified | shop fronts, top left; the walker stays in the middle third in all three moments and a 12-frame contact sheet | +| S4 | `[0,0,240,150]` | `[500,16,160,130]` | the seated man's vest and the standing man's shoulder reach the old crop's lower right corner at frame 62; the new crop is wall in all three shots (a sconce at frame 62, brick at 186 and 434) | +| S5 | `[0,0,240,150]` | unchanged, now verified | sky above the horizon; the cyclist never leaves the lower half over a 12-frame contact sheet. Plain sky holds little fine texture, so S5's fine-band drift is a weak signal | +| S6 | `[0,0,240,150]` | `[690,340,190,135]` | the old crop held the page border and, at frame 240, the lighthouse finial. A smaller interior crop was tried and returned no coarse band; the new one is lower right, clear of the lighthouse and the climber in all three frames (at frame 240 the beam's faint haze touches its top edge, which is lighting, not subject) | + +All six scenes are `crop_verified: true`. S3, S4, S5 and S6 were measured again +with `driver.py measure` (S4's runs were measured with the new crop once +they existed); S1 and S2 crops did not move, so their records stand. + +## Tables + +One table per scene: both runs, the floor (`noise_floor(a, b)`), job ids, +output names, and wall minutes per output minute. S1/S2 are 10 segments +(1042 frames), S3 6 segments (634), S4 four shots under one bed (496), +S5 LTX (359), S6 narration (288). Wall time includes model load. + +### S1 static (`s1-static`) + +| measure | run a | run b | floor | +|---|---|---|---| +| seam_jump_mean | 0.9311 | 0.9311 | 0.0000 | +| seam_jump_max | 1.0900 | 1.0900 | 0.0000 | +| seam_look_mean | 1.5289 | 1.5289 | 0.0000 | +| seam_look_max | 2.6100 | 2.6100 | 0.0000 | +| seam_level_step_db_mean | 9.8567 | 9.8567 | 0.0000 | +| seam_level_step_db_max | 36.5600 | 36.5600 | 0.0000 | +| texture_fine_seg_last | 1.1179 | 1.1179 | 0.0000 | +| texture_mid_seg_last | 0.4015 | 0.4015 | 0.0000 | +| texture_coarse_seg_last | 0.0488 | 0.0488 | 0.0000 | + +| | run a | run b | +|---|---|---| +| job | `00e4000a3ffb` | `b4e31bb17593` | +| output | `BenchS1Static/20261104-000916-390b57bc/final/BenchS1Static-chained_keyframe_to_video_audio.0-0.0.mp4` | `BenchS1Static-noise-b/20261104-004113-f663789a/final/BenchS1Static-noise-b-chained_keyframe_to_video_audio.0-0.0.mp4` | +| wall min | 21.1 | 18.1 | +| output min | 0.72 | 0.72 | +| wall min per output min | 29.1 | 25.0 | + +### S2 arc (`s2-arc`) + +| measure | run a | run b | floor | +|---|---|---|---| +| seam_jump_mean | 1.1944 | 1.1944 | 0.0000 | +| seam_jump_max | 2.0900 | 2.0900 | 0.0000 | +| seam_look_mean | 1.2844 | 1.2844 | 0.0000 | +| seam_look_max | 2.7100 | 2.7100 | 0.0000 | +| seam_level_step_db_mean | 2.0356 | 2.0356 | 0.0000 | +| seam_level_step_db_max | 5.5400 | 5.5400 | 0.0000 | +| texture_fine_seg_last | 3.5576 | 3.5576 | 0.0000 | +| texture_mid_seg_last | 2.8636 | 2.8636 | 0.0000 | +| texture_coarse_seg_last | 1.9747 | 1.9747 | 0.0000 | + +| | run a | run b | +|---|---|---| +| job | `04978dc6f338` | `51d7b09236f6` | +| output | `BenchS2Arc/20261104-010950-2f5ecfe0/final/BenchS2Arc-chained_keyframe_to_video_audio.0-0.0.mp4` | `BenchS2Arc-noise-b/20261104-012953-430a6737/final/BenchS2Arc-noise-b-chained_keyframe_to_video_audio.0-0.0.mp4` | +| wall min | 20.0 | 17.8 | +| output min | 0.72 | 0.72 | +| wall min per output min | 27.7 | 24.7 | + +### S3 dialogue walk (`s3-dialogue-walk`) + +| measure | run a | run b | floor | +|---|---|---|---| +| seam_jump_mean | 1.4780 | 1.2120 | 0.2660 | +| seam_jump_max | 1.8300 | 1.6000 | 0.2300 | +| seam_look_mean | 1.3780 | 1.3240 | 0.0540 | +| seam_look_max | 1.6100 | 1.8700 | 0.2600 | +| seam_level_step_db_mean | 2.7700 | 3.0000 | 0.2300 | +| seam_level_step_db_max | 8.8100 | 8.6500 | 0.1600 | +| texture_fine_seg_last | 0.1751 | 0.1782 | 0.0031 | +| texture_mid_seg_last | 0.2261 | 0.2076 | 0.0185 | +| texture_coarse_seg_last | 0.2075 | 0.4396 | 0.2320 | + +| | run a | run b | +|---|---|---| +| job | `3c76d1966485` | `d2095670cd89` | +| output | `BenchS3DialogueWalk/20261104-015817-5942f052/final/BenchS3DialogueWalk-chained_keyframe_to_video_audio.0-0.0.mp4` | `BenchS3DialogueWalk-noise-b/20261104-021940-da114475/final/BenchS3DialogueWalk-noise-b-chained_keyframe_to_video_audio.0-0.0.mp4` | +| wall min | 10.8 | 10.5 | +| output min | 0.44 | 0.44 | +| wall min per output min | 24.5 | 24.0 | + +### S4 montage (`s4-montage`) + +| measure | run a | run b | floor | +|---|---|---|---| +| seam_jump_mean | 20.4767 | 20.4767 | 0.0000 | +| seam_jump_max | 22.6900 | 22.6900 | 0.0000 | +| seam_look_mean | 13.4733 | 13.4733 | 0.0000 | +| seam_look_max | 18.2400 | 18.2400 | 0.0000 | +| seam_level_step_db_mean | 3.4967 | 3.4967 | 0.0000 | +| seam_level_step_db_max | 4.4200 | 4.4200 | 0.0000 | +| texture_fine_seg_last | 1.7141 | 1.7141 | 0.0000 | +| texture_mid_seg_last | 0.5798 | 0.5798 | 0.0000 | +| texture_coarse_seg_last | 1.2143 | 1.2143 | 0.0000 | + +| | run a | run b | +|---|---|---| +| job | `568a3da301c4` | `511b37c9bd70` | +| output | `BenchS4Montage-noise-a2/20261104-024801-94372b51/final/BenchS4Montage-noise-a2-film.14-0.0.mp4` | `BenchS4Montage-noise-b2/20261104-024801-6c692da9/final/BenchS4Montage-noise-b2-film.14-0.0.mp4` | +| wall min | 19.7 | 20.8 | +| output min | 0.34 | 0.34 | +| wall min per output min | 57.2 | 60.3 | + +### S5 LTX action (`s5-ltx-action`) + +| measure | run a | run b | floor | +|---|---|---|---| +| seam_jump_mean | 1.2450 | 1.2450 | 0.0000 | +| seam_jump_max | 1.3600 | 1.3600 | 0.0000 | +| seam_look_mean | 2.0850 | 2.0850 | 0.0000 | +| seam_look_max | 2.4300 | 2.4300 | 0.0000 | +| seam_level_step_db_mean | 2.1450 | 2.1400 | 0.0050 | +| seam_level_step_db_max | 2.8500 | 2.8400 | 0.0100 | +| texture_fine_seg_last | 0.5457 | 0.5457 | 0.0000 | +| texture_mid_seg_last | 0.4867 | 0.4867 | 0.0000 | +| texture_coarse_seg_last | 1.5417 | 1.5417 | 0.0000 | + +| | run a | run b | +|---|---|---| +| job | `a91df8d3684c` | `813a4ec70859` | +| output | `BenchS5LtxAction/20261104-005131-8fc124d3/final/BenchS5LtxAction-chained_image_to_video.0-0.0.mp4` | `BenchS5LtxAction-noise-b/20261104-005443-a7939f5d/final/BenchS5LtxAction-noise-b-chained_image_to_video.0-0.0.mp4` | +| wall min | 3.2 | 2.0 | +| output min | 0.25 | 0.25 | +| wall min per output min | 12.8 | 7.9 | + +### S6 narration (`s6-narration`) + +| measure | run a | run b | floor | +|---|---|---|---| +| seam_jump_mean | 40.2000 | 40.2000 | 0.0000 | +| seam_jump_max | 45.3300 | 45.3300 | 0.0000 | +| seam_look_mean | 70.6350 | 70.6350 | 0.0000 | +| seam_look_max | 82.7500 | 82.7500 | 0.0000 | +| seam_level_step_db_mean | 7.7900 | 7.8000 | 0.0100 | +| seam_level_step_db_max | 13.2600 | 13.2800 | 0.0200 | +| texture_fine_seg_last | 1.4011 | 1.4011 | 0.0000 | +| texture_mid_seg_last | 1.2213 | 1.2213 | 0.0000 | +| texture_coarse_seg_last | 0.8531 | 0.8531 | 0.0000 | + +| | run a | run b | +|---|---|---| +| job | `54240901f86a` | `5670fac7bc34` | +| output | `BenchS6Narration/20261104-005645-a4669737/final/BenchS6Narration-film.9-0.0.mp4` | `BenchS6Narration-noise-b/20261104-005815-12f0dc21/final/BenchS6Narration-noise-b-film.9-0.0.mp4` | +| wall min | 1.6 | 1.3 | +| output min | 0.20 | 0.20 | +| wall min per output min | 7.9 | 6.6 | + +Wall minutes per output minute, across the scenes: S1 25-29, S2 25-28, S3 24, +S4 57-60 (four H3 shots plus a music bed), S5 8-13, S6 7-8. The first run of a +pair in S5 and S6 paid model load; the second did not. + +## Cheap-setting check (6 segments, #1175 ordering) + +Arms A (control), B (`guide_noise` 0.4) and C (`chain.anchor: "segment_1"`) at +6 segments, moments frame 62 / 277 / 583, against #1175's 10-segment ordering +(`docs/proposals/complete/h3-guide-chain-drift-complete.md`, Stage 3). Jobs in +`bench/records/cheap-checks.json`. + +| scene | measure | A | B | C | #1175 (10 segments) | +|---|---|---|---|---|---| +| S1 static | `texture_fine_seg_last` | 0.503 | 0.854 | 0.190 | A 1.00, B 1.47 (rougher), C 0.40 | +| S1 static | `seam_look_mean` | 1.622 | 1.392 | 1.404 | A 1.67, B 1.11, C 1.27 | +| S2 arc | `seam_jump_mean` | 1.362 | 1.166 | 1.854 | A 1.13, B 1.09, C 2.20 | +| S2 arc | `seam_look_mean` | 1.286 | 1.694 | 4.270 | A 0.98, B 1.07, C 3.62 | + +`verdict(A, arm, floor, primary)` with the S1 and S2 floors (all 0, so any +difference counts): + +| scene | arm | primary | verdict | reading | +|---|---|---|---|---| +| S1 | B | `texture_fine_seg_last` | loses | B is rougher, as in #1175 | +| S1 | B | `seam_look_mean` | mixed | wins the look, loses texture | +| S1 | C | `texture_fine_seg_last` | mixed | C beats A on texture, worse elsewhere (seam jump 1.16 vs 0.88) | +| S1 | C | `seam_look_mean` | mixed | | +| S2 | B | `seam_jump_mean` | mixed | best jump of the three, look slightly worse than A | +| S2 | B | `seam_look_mean` | loses | 1.69 vs 1.29, as B 1.07 vs A 0.98 was slightly worse at 10 segments | +| S2 | C | `seam_jump_mean` | loses | worst | +| S2 | C | `seam_look_mean` | loses | worst by a wide margin | + +**Verdict: 6 segments reproduce #1175's ordering on every row**: on the static +scene C beats A on texture and B is rougher than both, and B and C both +improve on A's look; on the arc, C is worst on both seam measures and B has the +best jump, with a look slightly worse than A. The bench may use 6 segments. +The values are not the 10-segment values (the texture drift is smaller at 6 +segments, as it has fewer segments to compound over), so a 6-segment bench +result is a ranking, not a number to compare with #1175's tables. The 960x544 +size and the draft tier were not separately checked; the 6-segment arms ran at +the bench workflows' 960x544 and draft settings, and the ordering held there. + +## Chain check thresholds (ruling R19) + +`dw/pipeline_processors/chain_check.py` measures each segment with +`check_segment`: the seam's `jump_ratio` and `look_ratio` against the segment +before, and the whole-frame fine-band luma against segment 1's. The 28 seams of +the unchecked S1, S2 and S3 noise-run chains (S1 and S2 run a, S3 runs a and b; +S1 and S2 run b are bit-identical to run a) were measured with that function on +the decoded output, each segment's kept frames against the one before: + +| measure | P50 | P99 | max | set | +|---|---|---|---|---| +| `jump_ratio` | 1.10 | 2.02 | 2.08 | `JUMP_RATIO_MAX` 25 -> 4.0 | +| `look_ratio` | 1.28 | 2.67 | 2.71 | `LOOK_RATIO_MAX` 12 -> 5.5 | +| `texture_ratio` (either way) | | 2.99 | 3.13 | kept 2.0, provisional | + +The margin is a factor of 2 on the P99, rounded up to half a point. 4.0 stays +below an intended cut's 9.9-20.3 and above a dissolve's 1.8; 5.5 stays well +under the 19.2 a saturation-1.6 grade reads. + +The texture ceiling is not set from these runs, for two reasons. The two +same-seed runs of S1 and S2 are identical, so they show no spread to take a +floor from. And the texture ratio of an unchecked chain is the drift the check +exists to catch (1.07 rising to 2.43 over S1's ten segments, 1.04 falling to +0.32 over S2's), so its percentile measures the defect and a ceiling set at it +would pass the defect. It needs runs of a checked chain, or a repeat over +several seeds, to set. + +The three scenes are one subject type each and one seed; the P99 of 28 values +is the maximum, in effect, so a faster-moving or darker scene may read above +these ceilings and be re-rolled on a good segment. `check` is opt-in per chain, +and a workflow can name its own ceilings. diff --git a/bench/scenes.json b/bench/scenes.json index 580d48928..88a46beca 100644 --- a/bench/scenes.json +++ b/bench/scenes.json @@ -52,22 +52,22 @@ "Maybe that is what I was afraid of.", "Well. Let's see if the door still sticks." ], - "crop_verified": false + "crop_verified": true }, { "name": "s4-montage", "crop": [ - 0, - 0, - 240, - 150 + 500, + 16, + 160, + 130 ], "moments": [ "frame:62", "frame:186", "frame:434" ], - "crop_verified": false + "crop_verified": true }, { "name": "s5-ltx-action", @@ -82,15 +82,15 @@ "frame:180", "frame:299" ], - "crop_verified": false + "crop_verified": true }, { "name": "s6-narration", "crop": [ - 0, - 0, - 240, - 150 + 690, + 340, + 190, + 135 ], "moments": [ "frame:48", @@ -102,6 +102,6 @@ "By the time he reached the lamp, the sea had already gone dark.", "Then the light turned, and the whole coast could see him." ], - "crop_verified": false + "crop_verified": true } ] diff --git a/tests/test_bench_workflows.py b/tests/test_bench_workflows.py index f28b1d699..3ce67e4a2 100644 --- a/tests/test_bench_workflows.py +++ b/tests/test_bench_workflows.py @@ -72,8 +72,8 @@ def test_every_arm_names_a_scene_and_a_workflow_that_exist(): def test_scenes_are_well_formed(): for scene in driver.load_scenes(BENCH / "scenes.json").values(): - x0, y0, x1, y1 = scene.crop - assert 0 <= x0 < x1 and 0 <= y0 < y1 + x, y, width, height = scene.crop # the media routes' [x, y, w, h] + assert x >= 0 and y >= 0 and width >= 128 and height >= 128 assert len(scene.moments) == 3 assert all(m.startswith("frame:") for m in scene.moments) frames = [int(m.split(":")[1]) for m in scene.moments] From 7979b0c9af5ddd8a38c606a63eb628792458d1a6 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:09:50 -0600 Subject: [PATCH 036/405] feat(chain): jump and look ceilings from the noise floor JUMP_RATIO_MAX 25 -> 4.0 and LOOK_RATIO_MAX 12 -> 5.5: twice the P99 of 28 seams of the unchecked S1-S3 chains (bench/records/noise-floor.md). The texture ceiling stays provisional - the noise runs show no spread and the unchecked chains' own texture ratio is the drift to be caught. Co-Authored-By: Claude Opus 5.5 --- docs/WORKFLOW_GUIDE.md | 10 +++++--- dw/pipeline_processors/chain_check.py | 35 +++++++++++++++------------ tests/test_chain_check.py | 4 +-- 3 files changed, 28 insertions(+), 21 deletions(-) diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 7b57c6254..f6adc539d 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -2628,13 +2628,15 @@ state: | Measure | Against | Default ceiling | Fails when | |---|---|---|---| -| `jump_ratio` | the previous segment: the largest frame change across the seam over the larger of the two segments' own typical change, floored at 2 grey levels (`analyze_seams`'s measure) | `jump_ratio_max` 25 | above the ceiling | -| `look_ratio` | the previous segment: the largest saturation, lightness or contrast step across the seam over that measure's own rate inside either segment (`analyze_seams`'s measure) | `look_ratio_max` 12 | above the ceiling | +| `jump_ratio` | the previous segment: the largest frame change across the seam over the larger of the two segments' own typical change, floored at 2 grey levels (`analyze_seams`'s measure) | `jump_ratio_max` 4 | above the ceiling | +| `look_ratio` | the previous segment: the largest saturation, lightness or contrast step across the seam over that measure's own rate inside either segment (`analyze_seams`'s measure) | `look_ratio_max` 5.5 | above the ceiling | | `texture_ratio` | segment 1: the mean fine-band (2-8 px) luma texture of 4 evenly spaced frames, whole frame | `texture_ratio_max` 2 | above it or below its inverse (doubled or halved); `null` skips it | `"check": true` uses those ceilings; `"check": {"jump_ratio_max": 20}` overrides the ones -it names. `face` is reserved and must be `null`. The ceilings are provisional, set to -catch gross failures only until the bench's noise floor sets them. A static segment's own +it names. `face` is reserved and must be `null`. The jump and look ceilings are twice the +99th percentile of the bench's unchecked S1-S3 chains (`bench/records/noise-floor.md`); +the texture ceiling is still provisional, because those chains' own texture ratio is the +drift it exists to catch. A static segment's own motion is ~0, so the jump is divided by the floor rather than by ~0. Segment 1 has no seam and is never checked, nor is a seam whose incoming shot is a hard cut (a chain's own segments never are). diff --git a/dw/pipeline_processors/chain_check.py b/dw/pipeline_processors/chain_check.py index 75ac9a129..1b2baa8b6 100644 --- a/dw/pipeline_processors/chain_check.py +++ b/dw/pipeline_processors/chain_check.py @@ -19,22 +19,27 @@ import numpy -# PROVISIONAL: the default ceilings, to be set from the 99th percentile of -# each measure across the noise-floor runs of S1-S3, plus margin, once -# bench/records/noise-floor.md has them. Until then they are set to catch -# gross failures only: -# - jump_ratio: `seam_frame_jump`'s own threshold (dw/assessment_rules.py), -# above every intended cut the stage D runs measured (9.9-20.3; a dissolve -# reads about 1.8) - a chain seam over it is a cut nobody asked for -# - look_ratio: no assessment rule reads it yet; its calibration +# The default ceilings, set from the 99th percentile of each seam measure +# across the noise-floor runs of S1-S3 (bench/records/noise-floor.md, "Chain +# check thresholds"; 28 seams of unchecked chains, measured with this +# module's own arithmetic), doubled for margin and rounded up to half a point: +# - jump_ratio: P99 2.02 -> 4.0. Well under `seam_frame_jump`'s own threshold +# (25, dw/assessment_rules.py) and the 9.9-20.3 the stage D runs measured +# for an intended cut, above a dissolve's 1.8 - a chain seam over it is a +# cut nobody asked for +# - look_ratio: P99 2.67 -> 5.5. The calibration # (docs/proposals/complete/seam-look-measures-complete.md) read 0.64 for a -# clip joined to itself, 19.2 for a saturation-1.6 grade (the size of -# join Kinoloom reported) and 47.5 for an exposure step - 12 is ~19x the -# control and under the visible grade -# - texture_ratio: the segment's fine-band luma against segment 1's, either -# way - halved (blur) or doubled (crust) -JUMP_RATIO_MAX = 25.0 -LOOK_RATIO_MAX = 12.0 +# clip joined to itself and 19.2 for a saturation-1.6 grade, so 5.5 is +# well under a visible grade +# - texture_ratio: still PROVISIONAL, 2.0, the segment's fine-band luma +# against segment 1's, either way - halved (blur) or doubled (crust). The +# noise runs cannot set it: the two same-seed runs of S1 and S2 are +# identical (no spread), and the unchecked chains' own ratio is the drift +# the check exists to catch (P99 3.0x, up to 2.4x in S1 and 0.32x in S2 +# by segment 10), so a percentile of it would raise the ceiling to the +# very defect. It is set from a checked chain's runs, not these +JUMP_RATIO_MAX = 4.0 +LOOK_RATIO_MAX = 5.5 TEXTURE_RATIO_MAX = 2.0 # The measures a check can fail, in the order a failure names them diff --git a/tests/test_chain_check.py b/tests/test_chain_check.py index 76ab13dd4..33b2a258d 100644 --- a/tests/test_chain_check.py +++ b/tests/test_chain_check.py @@ -61,11 +61,11 @@ def test_a_seamless_segment_passes(): def test_a_static_shot_with_near_zero_motion_does_not_fail_on_jump(): # Both shots static: their own typical delta is ~0, under the assess # floor, so the seam's change is divided by the floor and not by ~0 - result = check_segment(grey(100), grey(130), grey(100), SegmentCheck()) + result = check_segment(grey(100), grey(105), grey(100), SegmentCheck()) assert "jump_ratio" not in result.failed assert result.measures["typical_delta"] == 2.0 - assert result.measures["jump_ratio"] == pytest.approx(15.0) + assert result.measures["jump_ratio"] == pytest.approx(2.5) def test_hard_cut_seams_are_never_checked(): From 92984b85a99017d21e69950f96af2d3dfb4ea088 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:12:47 -0600 Subject: [PATCH 037/405] bench: baseline seeds arms; S1-S3 take the seed argument S1-S3 hard-coded the top-level seed, so a seed argument changed nothing. It now reads variable:seed (default still 1166). Co-Authored-By: Claude Opus 5.5 --- bench/arms/baseline-seeds.json | 82 +++++++++++++++++++++++++++ bench/workflows/s1-static.json | 2 +- bench/workflows/s2-arc.json | 2 +- bench/workflows/s3-dialogue-walk.json | 2 +- 4 files changed, 85 insertions(+), 3 deletions(-) create mode 100644 bench/arms/baseline-seeds.json diff --git a/bench/arms/baseline-seeds.json b/bench/arms/baseline-seeds.json new file mode 100644 index 000000000..01cf08914 --- /dev/null +++ b/bench/arms/baseline-seeds.json @@ -0,0 +1,82 @@ +[ + { + "name": "control-s1-seed2", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": {}, + "seed": 2 + }, + { + "name": "control-s2-seed2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": {}, + "seed": 2 + }, + { + "name": "control-s3-seed2", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": {}, + "seed": 2 + }, + { + "name": "control-s4-seed2", + "scene": "s4-montage", + "workflow": "s4-montage.json", + "arguments": {}, + "seed": 2 + }, + { + "name": "control-s5-seed2", + "scene": "s5-ltx-action", + "workflow": "s5-ltx-action.json", + "arguments": {}, + "seed": 2 + }, + { + "name": "control-s6-seed2", + "scene": "s6-narration", + "workflow": "s6-narration.json", + "arguments": {}, + "seed": 2 + }, + { + "name": "best-s1-seed1166", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "anchor": "segment_1", + "guide_noise": 0.4 + }, + "seed": 1166 + }, + { + "name": "best-s1-seed2", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "anchor": "segment_1", + "guide_noise": 0.4 + }, + "seed": 2 + }, + { + "name": "best-s2-seed1166", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "guide_noise": 0.4 + }, + "seed": 1166 + }, + { + "name": "best-s2-seed2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "guide_noise": 0.4 + }, + "seed": 2 + } +] diff --git a/bench/workflows/s1-static.json b/bench/workflows/s1-static.json index dda9c2972..17ead1228 100644 --- a/bench/workflows/s1-static.json +++ b/bench/workflows/s1-static.json @@ -41,7 +41,7 @@ "reason": "H3 refuses width/height not a multiple of 32" } }, - "seed": 1166, + "seed": "variable:seed", "steps": [ { "name": "chained_keyframe_to_video_audio", diff --git a/bench/workflows/s2-arc.json b/bench/workflows/s2-arc.json index 9dd057352..dd4c8be04 100644 --- a/bench/workflows/s2-arc.json +++ b/bench/workflows/s2-arc.json @@ -41,7 +41,7 @@ "reason": "H3 refuses width/height not a multiple of 32" } }, - "seed": 1166, + "seed": "variable:seed", "steps": [ { "name": "chained_keyframe_to_video_audio", diff --git a/bench/workflows/s3-dialogue-walk.json b/bench/workflows/s3-dialogue-walk.json index b737659ef..182ded766 100644 --- a/bench/workflows/s3-dialogue-walk.json +++ b/bench/workflows/s3-dialogue-walk.json @@ -49,7 +49,7 @@ "reason": "H3 refuses width/height not a multiple of 32" } }, - "seed": 1166, + "seed": "variable:seed", "steps": [ { "name": "chained_keyframe_to_video_audio", From 4a31b372f78cf2008b38db471eb09e2193c1d5f5 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:24:53 -0600 Subject: [PATCH 038/405] fix(h3): #1192 - the audio pass keeps the source's frame rate and aspect Review fixes on the audio-only pass (controller ruling R25): - The source's frame rate is read (h3_guides.source_frame_rate: the rate a FrameList, AudioVideo or FittedVideo carries, else its soundtrack's length) and handed back as dw_held_video_fps. A source at another rate is conditioned on a copy retimed to 24 fps over its own duration (h3_rules.retimed_indices, nearest frame, count snapped up onto 17n + 5); the result plays the source's own frames at its own rate and the track is trimmed to its duration (output_extraction.held_duration_audio). A 24 fps source is conditioned on as it is. A source with no rate and no soundtrack is refused. Validate reads the probed rate, so a 30 fps source off the grid is not refused for it. - _with_mask decides is_audio_pass before the mask is popped. - listen_canvas snaps the shorter side and derives the longer from the source aspect (960x544 at 0.25: 224x128, not 256x128). - listen_scale is refused at validate on any step that is no H3 audio pass, a non-H3 pipeline included. - ARCHITECTURE names the four regional-edit/audio-pass blocks; WORKFLOW_GUIDE says a no-mask whole-source refine needs hold_audio. Co-Authored-By: Claude Opus 5.5 --- docs/ARCHITECTURE.md | 4 +- docs/WORKFLOW_GUIDE.md | 17 +- dw/hold_audio.py | 40 +++-- dw/output_extraction.py | 56 ++++++- dw/pipeline_processors/h3_guides.py | 27 ++++ dw/pipeline_processors/h3_mask.py | 6 + dw/pipeline_processors/h3_mask_steps.py | 9 ++ dw/pipeline_processors/h3_rules.py | 40 +++-- dw/pipeline_processors/pipeline_h3.py | 63 ++++++-- tests/test_h3_audio_pass.py | 206 +++++++++++++++++++++++- 10 files changed, 412 insertions(+), 56 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 298198f5c..15117dfd8 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -116,8 +116,8 @@ one to open. | Acceleration warnings | `dw/acceleration_warnings.py`: `acceleration_warnings`, `CACHE_MIN_USEFUL_STEPS`, `CUDA_ONLY_ATTENTION_BACKENDS` (the last entry of `WARNING_CHECKS`) | An acceleration setting that can't pay off (a `first_block`/`mag` cache under `CACHE_MIN_USEFUL_STEPS`, any `taylorseer` cache, `compile.fullgraph` with a cache, a cache on a pipeline class with no transformer or whose transformer lacks `enable_cache()`, a CUDA-only `attention_backend` on a non-CUDA device) is a warning and never a refusal, and an unresolvable value (steps, backend, device) leaves it silent. | `tests/test_validation.py::TestAccelerationWarnings::test_warnings_never_refuse`, `tests/test_validation.py::TestAccelerationWarnings::test_steps_that_resolve_to_nothing_are_silent`, `tests/test_validation.py::TestAccelerationWarnings::test_an_unknown_device_is_silent`, `tests/test_validation.py::TestAccelerationWarnings::test_an_unresolvable_backend_is_silent_on_mps` | | H3 sparse attention | `dw/sparse_attention.py`: `sparse_call`, `install_sparse`, `install_sparse_for_call`, `apply_sparse_settings`, `SparseH3AttnProcessor`; wired from `dw/pipeline_processors/components.py`: `configure_components` and `dw/pipeline_processors/pipeline.py`: `apply_runtime_settings` | Whether a pipeline call runs sparse is decided once per call, all or nothing (the call's first kernel invocation included); it never changes diffusers' process-wide backend and never reaches `AttentionBackendName`; it installs by a processor swap plus forward hooks, never a `forward` wrapper; and `sparse_tau` is a runtime setting, not part of the weights identity (#931). | `tests/test_sparse_attention_processor.py`, `tests/test_sparse_attention_wiring.py`, `tests/test_sparse_attention_tiles.py`, `tests/test_sparse_attention_backends.py` | | H3 adapter partition | `dw/adapter_compatibility.py`; the server's index is `dw/server/deps.py` `lora_index` | The LoRA catalog decides first, on the server only: a `proven` entry's `workflow` refuses a step in the other partition, a `trial`/`rejected` entry's only warns. A file no entry places falls to its name: an FL2VA LoRA on a reference step is refused, and one whose name says neither `ref2v` nor `fl2v` is warned. | `tests/test_h3_adapters.py::TestWhatIsRefused::test_a_keyframe_adapter_on_the_reference_path`, `tests/test_h3_adapters.py::TestWhatIsRefused::test_an_unrecognised_name_is_a_warning_not_a_refusal`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_proven_entry_refuses_the_other_partition`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_weaker_entry_only_warns`, `tests/test_server_loras.py::TestTheCatalogPlacesAnAdapter::test_a_trial_entry_warns_and_promoting_it_refuses` | -| H3 audio hold, refine, tiled refine, guide layout, guide noise, latent anchor and regional edit | Fourteen modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules, `refine_tiles` and the explicit `refine_sigmas` schedule among them, asked before validation and before the call; `tile_plan`, the tile layout on the 32 px grid, shared by validate and the run; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants; and the regional-edit rules `mask_problems`, `media_problems` and `edits_region`, the argument rules shared by validate and `_with_mask`; `media_problems` is asked by validate too, on the files that resolve then, and refuses a `source_video` off the `17n + 5` grid, which the pipeline would otherwise snap up over a padded source), `dw/pipeline_processors/h3_mask_cells.py` holds the mask-to-rows arithmetic (`binarized`, `regenerate_cells`, `keep_rows`, on Pillow's L conversion, torch's adaptive max pool and scipy's dilation, each imported where it runs), which `h3_mask.py` re-exports and `h3_mask_steps.py` reads from there so the blocks never import their importer, `dw/pipeline_processors/h3_hold.py` puts the hold and refine blocks in (`insert_audio_hold`, `core_denoise_sequences`, `refines`; `HELD_LATENTS_OUTPUT` and `encoded_hold_audio`, the hold block's encode handed back to it), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `insert_guides`), which imports `h3_hold` and never the reverse, `dw/pipeline_processors/h3_anchor.py` the latent statistics anchor (`insert_latent_anchor`, `anchor_refusal`; its block in `h3_anchor_steps.py`, imported only by `anchor_block()`; the arithmetic, `channel_statistics` and `match_channel_statistics`, in `h3_anchor_stats.py`, torch only, which `h3_anchor` re-exports and `h3_anchor_steps` reads from there, as `h3_mask_cells` is read, so the block never imports its importer), which imports `h3_hold` and `h3_guides` and never the reverse, and runs right after `after_denoise` in t2va/fl2va only (a no-op without `latent_anchor`, decided by the block itself; it declares `idle_without = "latent_anchor"` so the step-progress narrator leaves it unnamed then; `chain.anchor: "segment_1"` hands segments 2+ segment 1's frames in `chain.py`'s `_segment_arguments`, and `run_chain` sets `pipeline.latent_anchor_source` to `segment 1` for those segments only; `Pipeline._with_latent_anchor` emits one `log` line per anchored call naming that source, the quoted reference, or the step's frames; `dw/guides.py`'s `anchor_errors` is the `latent_anchor` validation check, on `h3_rules.chain_anchor_problem` and the guides step rule `_takes_guides_problem`, asked for the anchor's name, so a step passing `references` (ref2va) is refused as `guides` is; `Pipeline._with_latent_anchor` refuses `references` at run time as `_with_guides` does), and `dw/pipeline_processors/h3_mask.py` the regional edit (`insert_mask_blocks`, `edits_regions`), which imports `h3_hold` and never the reverse; the blocks themselves, and the arithmetic they run, are `dw/pipeline_processors/h3_hold_steps.py` (`encode_audio_span`, `refine_sigmas`, and `explicit_sigmas`, a step's `refine_sigmas` list run verbatim in place of the shift grid) `dw/pipeline_processors/h3_guide_steps.py` (`splice_guide_rows`, and `noised_guide_pixels`, the `guide_noise` pixels: the layout block's `_encode_guides` encodes each video guide's noised pixels in place of the clean ones when `guide_noise` is set, on `h3_rules.guide_noise_taper`, one encode per guide either way (PrefixStream's paired encode under one seed is that encode, so the clean one is not run); `guide_noise_errors`/`guide_noise_warnings` in `dw/guides.py` validate it on `h3_rules.guide_noise_problem` and `_takes_guides_problem`, and `Pipeline._with_guide_noise` drops a 0 and refuses at run time) and `dw/pipeline_processors/h3_mask_steps.py` (the source-latents and mask-blend blocks), which import diffusers when they load, so the three owners import them only inside `blocks()`, `guide_blocks()` and `h3_mask.blocks()` (#790), and `h3_guide_steps` imports `h3_hold_steps` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline_h3.py` - the `H3Arguments` mixin that gives `Pipeline` its per-call H3 argument wrappers `_with_held_audio`, `_with_guides`, `_with_guide_noise`, `_with_latent_anchor`, `_with_mask` and `_check_refine`, still named `Pipeline._with_*` - adds its blocks for a `guides` call, and `_with_mask` checks a regional edit's arguments and media and builds the regenerate mask for a `source_video` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/pipeline_processors/h3_tiles.py`: the tiled refine (the tile loop, the latent-space feather blend, the per-tile crop of one whole-canvas noise draw, the decode-free graph variant, and the one spatially tiled decode), which imports `h3_rules`, `h3_hold` and `media_types` (for `FrameList`; and `events`, `progress`) and is imported by `pipeline.py` and, lazily, `components.py` (which attaches the variant at load, behind `h3_hold.core_denoise_sequences`, and calls `insert_mask_blocks` at load), never by `h3_rules` or `h3_hold`, `dw/hold_audio.py`: `hold_audio_errors` (a video is held by its soundtrack), `refine_strength_errors`, `mask_errors` (Check `regional_edit`, which probes the source and mask through `ValidationContext.probe` the way `dw/guides.py` probes guide videos) | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and two more for a regional edit: the source block before `set_timesteps` and the blend block inside the `denoise` loop after `update`. All are no-ops without their arguments (`hold_audio` / `refine_strength`, `source_video`). The tiled refine adds a second graph built at load beside them (`h3_tiles.decode_free_blocks`: the decode and text-encoder blocks dropped, every kept block the graph's own instance, so hold and refine stay in) and swaps it into `pipeline._blocks` only for the duration of the one text-encode call and the tile calls (`swapped_blocks`, restored on failure). Every tile runs the hold block, but the track is encoded once per refine: the first tile is asked for the block's encode (`HELD_LATENTS_OUTPUT`) and every later tile's `hold_audio` carries it (`encoded_hold_audio`, an `EncodedAudioReference`), which the block writes in place of running the audio VAE. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_tiles.py::TestDecodeFree::test_variant_has_no_decode_and_shares_blocks`, `tests/test_h3_tiles.py::TestTileLoop::test_blocks_are_restored_when_a_tile_fails`, `tests/test_h3_tiles.py::TestTileLoop::test_the_track_is_encoded_by_one_tile_and_handed_to_the_rest`, `tests/test_h3_hold_audio.py::TestHoldBlock::test_a_carried_encode_is_written_without_encoding_again`, `tests/test_h3_tiles.py`, `tests/test_h3_rules.py` (`tile_plan` and the `refine_tiles` rules), `tests/test_h3_mask.py`, `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_refine.py::TestSigmaListOneOwner::test_the_runtime_message_is_the_owners`, `tests/test_h3_refine.py::TestSigmaListSchedule::test_the_block_runs_the_list_unshifted`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers`, `tests/test_h3_rules.py::test_the_block_owners_import_no_diffusers`, `tests/test_h3_anchor.py::TestInsert::test_follows_after_denoise`, `tests/test_h3_anchor.py::TestStep::test_without_an_anchor_the_latents_are_untouched`, `tests/test_h3_anchor.py::TestWithLatentAnchor::test_no_anchor_logs_nothing`, `tests/test_h3_anchor.py::TestWithLatentAnchor::test_a_chain_segment_logs_its_label_and_segment_1`, `tests/test_modular_progress.py::test_the_anchor_step_is_idle_without_latent_anchor`, `tests/test_h3_anchor.py::TestAnchorErrors::test_an_unknown_anchor_names_the_values`, `tests/test_h3_guide_noise.py` | -| H3 audio pass (#1192) | `dw/pipeline_processors/h3_rules.py`: `is_audio_pass` (a `source_video` with no mask and no `hold_audio`), the audio-pass rules inside `mask_problems` (`_audio_pass_problems`, `_listen_scale_problems`, `LISTEN_SCALES`) and `listen_canvas`; `dw/pipeline_processors/h3_mask.py`: `insert_mask_blocks` puts the pair in beside the source block, `passes_audio`, `listen_frames` (the area resample); `dw/pipeline_processors/h3_mask_steps.py`: `DwH3AudioPassStep` (before `denoise`), `DwH3ReleaseVideoStep` (before `after_denoise`), on `h3_hold_steps.row_timestep_plan`; `dw/pipeline_processors/pipeline_h3.py`: `with_audio_pass`; `dw/output_extraction.py`: `held_videos` | No video row is ever stepped: the pass block widens `num_condition_video_rows` over every video row after `set_timesteps` and pins them at t = 1.0 in the row plan, the source block's clean encode held at every step; the audio keeps the stock audio schedule from full noise. No refine argument runs with it. The result plays the full-size source frames (`HELD_VIDEO_OUTPUT`), never the decode, the way `dw_held_audio` stands in for a held track. `listen_scale` moves the encode canvas only: `num_frames`, and so the audio length, is the source's. | `tests/test_h3_audio_pass.py::test_the_output_frames_are_the_sources_own`, `tests/test_h3_audio_pass.py::test_listen_scale_half_runs_the_kept_rows_at_half_canvas`, `tests/test_h3_audio_pass.py::test_a_mask_or_hold_audio_keeps_todays_meaning`, `tests/test_h3_audio_pass.py::TestAudioPassBlock::test_the_loop_steps_audio_and_never_a_video_row` | +| H3 audio hold, refine, tiled refine, guide layout, guide noise, latent anchor and regional edit | Fourteen modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules, `refine_tiles` and the explicit `refine_sigmas` schedule among them, asked before validation and before the call; `tile_plan`, the tile layout on the 32 px grid, shared by validate and the run; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants; and the regional-edit rules `mask_problems`, `media_problems` and `edits_region`, the argument rules shared by validate and `_with_mask`; `media_problems` is asked by validate too, on the files that resolve then, and refuses a `source_video` off the `17n + 5` grid, which the pipeline would otherwise snap up over a padded source), `dw/pipeline_processors/h3_mask_cells.py` holds the mask-to-rows arithmetic (`binarized`, `regenerate_cells`, `keep_rows`, on Pillow's L conversion, torch's adaptive max pool and scipy's dilation, each imported where it runs), which `h3_mask.py` re-exports and `h3_mask_steps.py` reads from there so the blocks never import their importer, `dw/pipeline_processors/h3_hold.py` puts the hold and refine blocks in (`insert_audio_hold`, `core_denoise_sequences`, `refines`; `HELD_LATENTS_OUTPUT` and `encoded_hold_audio`, the hold block's encode handed back to it), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `insert_guides`), which imports `h3_hold` and never the reverse, `dw/pipeline_processors/h3_anchor.py` the latent statistics anchor (`insert_latent_anchor`, `anchor_refusal`; its block in `h3_anchor_steps.py`, imported only by `anchor_block()`; the arithmetic, `channel_statistics` and `match_channel_statistics`, in `h3_anchor_stats.py`, torch only, which `h3_anchor` re-exports and `h3_anchor_steps` reads from there, as `h3_mask_cells` is read, so the block never imports its importer), which imports `h3_hold` and `h3_guides` and never the reverse, and runs right after `after_denoise` in t2va/fl2va only (a no-op without `latent_anchor`, decided by the block itself; it declares `idle_without = "latent_anchor"` so the step-progress narrator leaves it unnamed then; `chain.anchor: "segment_1"` hands segments 2+ segment 1's frames in `chain.py`'s `_segment_arguments`, and `run_chain` sets `pipeline.latent_anchor_source` to `segment 1` for those segments only; `Pipeline._with_latent_anchor` emits one `log` line per anchored call naming that source, the quoted reference, or the step's frames; `dw/guides.py`'s `anchor_errors` is the `latent_anchor` validation check, on `h3_rules.chain_anchor_problem` and the guides step rule `_takes_guides_problem`, asked for the anchor's name, so a step passing `references` (ref2va) is refused as `guides` is; `Pipeline._with_latent_anchor` refuses `references` at run time as `_with_guides` does), and `dw/pipeline_processors/h3_mask.py` the regional edit (`insert_mask_blocks`, `edits_regions`), which imports `h3_hold` and never the reverse; the blocks themselves, and the arithmetic they run, are `dw/pipeline_processors/h3_hold_steps.py` (`encode_audio_span`, `refine_sigmas`, and `explicit_sigmas`, a step's `refine_sigmas` list run verbatim in place of the shift grid) `dw/pipeline_processors/h3_guide_steps.py` (`splice_guide_rows`, and `noised_guide_pixels`, the `guide_noise` pixels: the layout block's `_encode_guides` encodes each video guide's noised pixels in place of the clean ones when `guide_noise` is set, on `h3_rules.guide_noise_taper`, one encode per guide either way (PrefixStream's paired encode under one seed is that encode, so the clean one is not run); `guide_noise_errors`/`guide_noise_warnings` in `dw/guides.py` validate it on `h3_rules.guide_noise_problem` and `_takes_guides_problem`, and `Pipeline._with_guide_noise` drops a 0 and refuses at run time) and `dw/pipeline_processors/h3_mask_steps.py` (the source-latents and mask-blend blocks), which import diffusers when they load, so the three owners import them only inside `blocks()`, `guide_blocks()` and `h3_mask.blocks()` (#790), and `h3_guide_steps` imports `h3_hold_steps` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline_h3.py` - the `H3Arguments` mixin that gives `Pipeline` its per-call H3 argument wrappers `_with_held_audio`, `_with_guides`, `_with_guide_noise`, `_with_latent_anchor`, `_with_mask` and `_check_refine`, still named `Pipeline._with_*` - adds its blocks for a `guides` call, and `_with_mask` checks a regional edit's arguments and media and builds the regenerate mask for a `source_video` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/pipeline_processors/h3_tiles.py`: the tiled refine (the tile loop, the latent-space feather blend, the per-tile crop of one whole-canvas noise draw, the decode-free graph variant, and the one spatially tiled decode), which imports `h3_rules`, `h3_hold` and `media_types` (for `FrameList`; and `events`, `progress`) and is imported by `pipeline.py` and, lazily, `components.py` (which attaches the variant at load, behind `h3_hold.core_denoise_sequences`, and calls `insert_mask_blocks` at load), never by `h3_rules` or `h3_hold`, `dw/hold_audio.py`: `hold_audio_errors` (a video is held by its soundtrack), `refine_strength_errors`, `mask_errors` (Check `regional_edit`, which probes the source and mask through `ValidationContext.probe` the way `dw/guides.py` probes guide videos) | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and four more for a regional edit and the audio pass: the source block before `set_timesteps`, the blend block inside the `denoise` loop after `update`, the audio-pass block (`DwH3AudioPassStep`) before `denoise` and the release block (`DwH3ReleaseVideoStep`) before `after_denoise`. All are no-ops without their arguments (`hold_audio` / `refine_strength`, `source_video`, `dw_held_video`). The tiled refine adds a second graph built at load beside them (`h3_tiles.decode_free_blocks`: the decode and text-encoder blocks dropped, every kept block the graph's own instance, so hold and refine stay in) and swaps it into `pipeline._blocks` only for the duration of the one text-encode call and the tile calls (`swapped_blocks`, restored on failure). Every tile runs the hold block, but the track is encoded once per refine: the first tile is asked for the block's encode (`HELD_LATENTS_OUTPUT`) and every later tile's `hold_audio` carries it (`encoded_hold_audio`, an `EncodedAudioReference`), which the block writes in place of running the audio VAE. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_tiles.py::TestDecodeFree::test_variant_has_no_decode_and_shares_blocks`, `tests/test_h3_tiles.py::TestTileLoop::test_blocks_are_restored_when_a_tile_fails`, `tests/test_h3_tiles.py::TestTileLoop::test_the_track_is_encoded_by_one_tile_and_handed_to_the_rest`, `tests/test_h3_hold_audio.py::TestHoldBlock::test_a_carried_encode_is_written_without_encoding_again`, `tests/test_h3_tiles.py`, `tests/test_h3_rules.py` (`tile_plan` and the `refine_tiles` rules), `tests/test_h3_mask.py`, `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_refine.py::TestSigmaListOneOwner::test_the_runtime_message_is_the_owners`, `tests/test_h3_refine.py::TestSigmaListSchedule::test_the_block_runs_the_list_unshifted`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers`, `tests/test_h3_rules.py::test_the_block_owners_import_no_diffusers`, `tests/test_h3_anchor.py::TestInsert::test_follows_after_denoise`, `tests/test_h3_anchor.py::TestStep::test_without_an_anchor_the_latents_are_untouched`, `tests/test_h3_anchor.py::TestWithLatentAnchor::test_no_anchor_logs_nothing`, `tests/test_h3_anchor.py::TestWithLatentAnchor::test_a_chain_segment_logs_its_label_and_segment_1`, `tests/test_modular_progress.py::test_the_anchor_step_is_idle_without_latent_anchor`, `tests/test_h3_anchor.py::TestAnchorErrors::test_an_unknown_anchor_names_the_values`, `tests/test_h3_guide_noise.py` | +| H3 audio pass (#1192) | `dw/pipeline_processors/h3_rules.py`: `is_audio_pass` (a `source_video` with no mask and no `hold_audio`), the audio-pass rules inside `mask_problems` (`_audio_pass_problems`, `_listen_scale_problems`, `LISTEN_SCALES`) and `listen_canvas`; `dw/pipeline_processors/h3_mask.py`: `insert_mask_blocks` puts the pair in beside the source block, `passes_audio`, `listen_frames` (the area resample); `dw/pipeline_processors/h3_mask_steps.py`: `DwH3AudioPassStep` (before `denoise`), `DwH3ReleaseVideoStep` (before `after_denoise`), on `h3_hold_steps.row_timestep_plan`; `dw/pipeline_processors/pipeline_h3.py`: `with_audio_pass`; `dw/output_extraction.py`: `held_videos` | No video row is ever stepped: the pass block widens `num_condition_video_rows` over every video row after `set_timesteps` and pins them at t = 1.0 in the row plan, the source block's clean encode held at every step; the audio keeps the stock audio schedule from full noise. No refine argument runs with it. The result plays the full-size source frames (`HELD_VIDEO_OUTPUT`), never the decode, the way `dw_held_audio` stands in for a held track. `listen_scale` moves the encode canvas only, keeping the source's aspect (`listen_canvas` snaps the shorter side and derives the longer). The result keeps the source's own frame rate (`HELD_VIDEO_RATE_OUTPUT`, read by `h3_guides.source_frame_rate`), never relabelled to 24: a source at another rate is conditioned on a copy retimed to 24 fps over its own duration (`h3_rules.retimed_indices`, nearest frame, the count snapped up onto the `17n + 5` grid), and the generated track is trimmed to the source's duration (`output_extraction.held_duration_audio`); a 24 fps source is conditioned on as it is. A source with no rate and no soundtrack to time it is refused. | `tests/test_h3_audio_pass.py::test_the_output_frames_are_the_sources_own`, `tests/test_h3_audio_pass.py::test_listen_scale_half_runs_the_kept_rows_at_half_canvas`, `tests/test_h3_audio_pass.py::test_a_mask_or_hold_audio_keeps_todays_meaning`, `tests/test_h3_audio_pass.py::TestAudioPassBlock::test_the_loop_steps_audio_and_never_a_video_row`, `tests/test_h3_audio_pass.py::TestSourceFrameRate::test_a_30_fps_source_plays_at_30_under_audio_of_its_own_length`, `tests/test_h3_audio_pass.py::TestSourceFrameRate::test_a_24_fps_source_is_unchanged` | | IC-LoRA reference scale | `workflows/templates/ltx2/` | The five conditioning templates run at `reference_downscale_factor: 1`, and the two upscalers at 2. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_reference_is_encoded_at_the_output_resolution`, `tests/test_ltx2_ic_loras.py::TestTheGenerativeUpscaleMatchesTheSameCard::test_it_loads_the_upscaler_at_factor_two` | | IC-LoRA numbers | `workflows/templates/ltx2/` | Every number in the IC-LoRA templates comes from the vendor's model card. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_strength_is_the_cards_default`, `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_defaults_are_the_trained_bucket` | | IC-LoRA prompt genre | `prompts/ltx2/` | A stored prompt tagged `ic-lora` is checked against its trained caption form, not against the T2V paragraph rule. | `tests/test_ltx_prompt_library.py::test_an_ic_lora_prompt_is_in_its_trained_form` | diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index f6adc539d..4be5af53f 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -2310,7 +2310,9 @@ it over the generated rows and re-denoises from `refine_strength`, which is requ in (0, 1]: 1.0 regenerates the masked region from pure noise, and about 0.9 replaces an object in it (see [Choosing refine_strength for a regional edit](#choosing-refine_strength-for-a-regional-edit)). (Without a `source_video` the refine's range stays (0, 1).) With no -mask the whole source is refined. +mask and a `hold_audio`, the whole source is refined under that soundtrack. With no mask +and no `hold_audio`, the call is not a refine at all: it is the audio pass (see [Audio over +a finished picture](#audio-over-a-finished-picture)). ```json { @@ -2417,13 +2419,22 @@ Run it without a distillation LoRA, on the dialogue schedule (`num_inference_ste `video_shift` 12, `audio_shift` 3): the turbo LoRA is what makes H3 voices sound digital (#984). `listen_scale` (1, 0.5 or 0.25, default 1) encodes the source at that fraction of its canvas, area-resampled and snapped to 32 px, which is cheaper per step. -The audio's length still follows the source's frame count, and the picture returned is the +The scale keeps the source's aspect: the shorter side is snapped and the longer derived from +it. The audio's length still follows the source's duration, and the picture returned is the full-size source. `width`, `height` and `num_frames` stay the source's, as for an edit. +The result keeps the source's own frame rate. A source at 24 fps is conditioned on as it +is. A source at another rate (30 fps, say) is conditioned on a copy retimed to H3's 24 fps +over the same duration, nearest frame, its count snapped up onto the `17n + 5` grid. The +step returns the source's own frames at the source's own rate, with the track trimmed to the +source's duration, so its own frame count need not be on the grid. The rate comes from the +file or the step result that supplied the video, or else from the length of its soundtrack. +A source with neither is refused. + Validation and the call refuse `refine_strength`, `refine_sigmas` or `refine_tiles` on an audio pass (no video row is re-denoised; pass `hold_audio` for a whole-source refine instead), a `listen_scale` other than those three, a `listen_scale` on a step that is not -an audio pass, and an audio pass on a chained step or beside `latents`. A draft template +an audio pass (on any pipeline, H3 or not), and an audio pass on a chained step or beside `latents`. A draft template is `bench/workflows/add-sound.json`; it is not in the catalog yet. ### Refining above 768p in tiles diff --git a/dw/hold_audio.py b/dw/hold_audio.py index ca393d8ba..4d87b3f40 100644 --- a/dw/hold_audio.py +++ b/dw/hold_audio.py @@ -49,6 +49,7 @@ from .for_each import render_path from .pipeline_processors.h3_rules import ( HOLD_AUDIO_INPUT, + LISTEN_SCALE_INPUT, MASK_IMAGE_INPUT, MASK_INPUTS, REFINE_SIGMAS_INPUT, @@ -57,6 +58,7 @@ REGIONAL_EDIT_INPUTS, SOURCE_VIDEO_INPUT, edits_region, + is_audio_pass, is_h3_step, mask_problems, media_problems, @@ -254,23 +256,23 @@ def _mask_video_white(path): def _video_shape(name, value, base_dir, probe): - """(problem, (frames, height, width), path) for a regional edit's video - argument: why it is no video, its shape when the probe reads one, and the - confined path it resolved to; Nones for what cannot be told yet.""" + """(problem, (frames, height, width), path, fps) for a regional edit's + video argument: why it is no video, its shape when the probe reads one, + the confined path it resolved to and the rate it declares; Nones for what + cannot be told yet.""" path = resolve_probe_path(value, base_dir, name) info = probe(path) if path is not None else None if not isinstance(info, dict): - return None, None, None + return None, None, None, None if info.get("kind") != "video": - return ( - f"{name} must be a video, and this is {info.get('kind') or 'unreadable'}", - None, - path, - ) + kind = info.get("kind") or "unreadable" + return f"{name} must be a video, and this is {kind}", None, path, None frames, height, width = (info.get(k) for k in ("frame_count", "height", "width")) + fps = info.get("fps") + fps = fps if isinstance(fps, (int, float)) and fps > 0 else None if not all(isinstance(v, int) and v > 0 for v in (frames, height, width)): - return None, None, path - return None, (frames, height, width), path + return None, None, path, fps + return None, (frames, height, width), path, fps def _media_problems(arguments, base_dir, probe): @@ -284,9 +286,11 @@ def _media_problems(arguments, base_dir, probe): width, height, num_frames = ( arguments.get(k) for k in ("width", "height", "num_frames") ) - problem, shape, _ = _video_shape( + problem, shape, _, fps = _video_shape( SOURCE_VIDEO_INPUT, arguments.get(SOURCE_VIDEO_INPUT), base_dir, probe ) + # An audio pass retimes a source at another rate onto the grid (R25) + fps = fps if is_audio_pass(arguments) else None if problem: problems.append((SOURCE_VIDEO_INPUT, problem)) if shape is not None: @@ -297,7 +301,7 @@ def _media_problems(arguments, base_dir, probe): if shape is not None and None not in canvas[:2]: problems += [ (SOURCE_VIDEO_INPUT, p) - for p in media_problems(SOURCE_VIDEO_INPUT, shape, *canvas) + for p in media_problems(SOURCE_VIDEO_INPUT, shape, *canvas, fps) ] name = next((n for n in MASK_INPUTS if arguments.get(n) is not None), None) if name is None: @@ -308,7 +312,9 @@ def _media_problems(arguments, base_dir, probe): read = _mask_image_shape(path) if path is not None else None mask_shape, white = read if read is not None else (None, None) else: - problem, mask_shape, path = _video_shape(name, arguments[name], base_dir, probe) + problem, mask_shape, path, _ = _video_shape( + name, arguments[name], base_dir, probe + ) if problem: problems.append((name, problem)) elif path is not None: @@ -341,8 +347,10 @@ def mask_errors(workflow_definition, source_indices=None, base_dir=None, probe=N steps, source_indices, "pipeline" ): arguments = pipeline.get("arguments") - if not isinstance(arguments, dict) or not edits_region( - arguments, is_h3_step(pipeline) + # listen_scale is no diffusers argument, so it is dw's on any step + if not isinstance(arguments, dict) or not ( + edits_region(arguments, is_h3_step(pipeline)) + or arguments.get(LISTEN_SCALE_INPUT) is not None ): continue given = [n for n in REGIONAL_EDIT_INPUTS if arguments.get(n) is not None] diff --git a/dw/output_extraction.py b/dw/output_extraction.py index 1c551e8b5..0303e1833 100644 --- a/dw/output_extraction.py +++ b/dw/output_extraction.py @@ -8,11 +8,20 @@ import numpy import torch -from .media_types import AudioTrack, AudioVideo, rated_videos +from .media_types import ( + AudioTrack, + AudioVideo, + FittedVideo, + fit_codec_padding, + rated_videos, + sample_axis, +) from .pipeline_processors.h3_rules import ( + H3_FPS, HELD_AUDIO_OUTPUT, HELD_AUDIO_RATE_OUTPUT, HELD_VIDEO_OUTPUT, + HELD_VIDEO_RATE_OUTPUT, ) logger = logging.getLogger("dw") @@ -43,23 +52,47 @@ # A MiniMax-H3 audio pass: the source's own frames, which play in place of the # decoded video (the VAE round trip of them) under the generated track HELD_VIDEO_KEYS = (HELD_VIDEO_OUTPUT,) +HELD_VIDEO_RATE_KEYS = (HELD_VIDEO_RATE_OUTPUT,) -def held_videos(videos, held): +def held_videos(videos, held, fps=None): """The batch `videos` with every video replaced by `held` - uint8 `(frames, height, width, 3)` source frames - in the batch's own form: a - list of PIL frame lists (H3's default `output_type`), keeping the batch's - rate, or a float array in [0, 1] for `output_type='np'`.""" + list of PIL frame lists (H3's default `output_type`), or a float array in + [0, 1] for `output_type='np'`. They play at `fps`, the source's own rate, + when it is given - never relabelled to the batch's 24 (R25).""" held = numpy.asarray(held) if isinstance(videos, numpy.ndarray): - return numpy.stack([held.astype(numpy.float32) / 255.0] * len(videos)) + stacked = numpy.stack([held.astype(numpy.float32) / 255.0] * len(videos)) + return FittedVideo(stacked, fps) if fps else stacked if isinstance(videos, torch.Tensor): pixels = torch.from_numpy(held).permute(0, 3, 1, 2).float() / 255.0 return torch.stack([pixels] * len(videos)).to(videos.device) from PIL import Image frames = [Image.fromarray(frame) for frame in held] - return rated_videos([list(frames) for _ in videos], getattr(videos, "fps", None)) + fps = fps or getattr(videos, "fps", None) + return rated_videos([list(frames) for _ in videos], fps) + + +def held_duration_audio(audio, frame_count, fps, sample_rate): + """Each generated track of an audio pass over a retimed source, trimmed + to the source's own `frame_count / fps` seconds: the conditioning ran on + the H3 grid, snapped up past the source's end, so the track runs long by + up to a grid step - more than `fit_codec_padding`'s tolerance - and a + short one is padded within it.""" + expected = round(frame_count / fps * sample_rate) + fitted = [] + for track in audio: + axis = sample_axis(track) + if axis is not None and track.shape[axis] > expected: + trim = [slice(None)] * track.ndim + trim[axis] = slice(None, expected) + track = track[tuple(trim)] + else: + track = fit_codec_padding(track, frame_count, fps, sample_rate) + fitted.append(track) + return fitted def _frames_from_attributes(result): @@ -180,9 +213,10 @@ def modular_artifacts(result): consumed_keys = {video_key} held_video_key, held_video = first_item(result, HELD_VIDEO_KEYS) + _, held_fps = first_item(result, HELD_VIDEO_RATE_KEYS) if held_video is not None: - videos = held_videos(videos, held_video) - consumed_keys.update(HELD_VIDEO_KEYS) + videos = held_videos(videos, held_video, held_fps) + consumed_keys.update(HELD_VIDEO_KEYS + HELD_VIDEO_RATE_KEYS) audio_key, audio = first_item(result, MODULAR_AUDIO_KEYS) sample_rate = None @@ -199,6 +233,12 @@ def modular_artifacts(result): audio, sample_rate = [held[0]] * len(videos), held_rate consumed_keys.update(HELD_AUDIO_KEYS + HELD_AUDIO_RATE_KEYS) + # An audio pass over a source not at 24 fps: the track was generated + # against its retimed conditioning, and plays under its own frames + retimed = held_video is not None and held_fps and held_fps != H3_FPS + if retimed and audio is not None and sample_rate: + audio = held_duration_audio(audio, len(held_video), held_fps, sample_rate) + artifacts = frames_with_audio(videos, audio, sample_rate) # Keys the video/audio pairing above did not consume still need to be saved, not diff --git a/dw/pipeline_processors/h3_guides.py b/dw/pipeline_processors/h3_guides.py index 86ac51f4e..6f872c97e 100644 --- a/dw/pipeline_processors/h3_guides.py +++ b/dw/pipeline_processors/h3_guides.py @@ -195,6 +195,33 @@ def guide_frames_array(value): return np.ascontiguousarray(value) +def source_frame_rate(value, frame_count): + """The rate a video argument plays at: the rate it carries at any level a + step's result wraps it in (a FrameList's, an AudioVideo's, a + FittedVideo's), else the rate its own soundtrack's length gives its + `frame_count` frames, else None - what `guide_frames_array` drops.""" + from ..media_types import sample_axis + + seconds = None + while value is not None: + fps = getattr(value, "fps", None) + if isinstance(fps, (int, float)) and not isinstance(fps, bool) and fps > 0: + return fps + audio, rate = getattr(value, "audio", None), getattr(value, "sample_rate", None) + axis = sample_axis(audio) if hasattr(audio, "ndim") else None + if seconds is None and axis is not None and rate and audio.shape[axis]: + seconds = audio.shape[axis] / rate + if hasattr(value, "value") and hasattr(value, "position"): + value = value.value + elif hasattr(value, "frames") and hasattr(value, "audio"): + value = value.frames + elif isinstance(value, (list, tuple)) and len(value) == 1: + value = value[0] + else: + break + return frame_count / seconds if seconds else None + + def prepared_guide(where, index, guide, base_dir): """One `guides` entry checked and read: its clip as uint8 frames cut to a whole-latent length, with its soundtrack when it asks for one.""" diff --git a/dw/pipeline_processors/h3_mask.py b/dw/pipeline_processors/h3_mask.py index e8d3059f4..c22df8f42 100644 --- a/dw/pipeline_processors/h3_mask.py +++ b/dw/pipeline_processors/h3_mask.py @@ -53,6 +53,12 @@ `h3_rules.listen_canvas`), which is cheaper per step; the audio's length follows `num_frames`, which the scale leaves alone, and the picture returned is still the full-size source. + +The result plays at the source's own rate (`HELD_VIDEO_RATE_OUTPUT`), never +relabelled to H3's 24. A source at another rate is conditioned on a copy +retimed to 24 fps over its own duration (`h3_rules.retimed_indices`), so the +track generated covers the source's real length, and the result trims it to +that length under the source's own frames (R25). """ import logging diff --git a/dw/pipeline_processors/h3_mask_steps.py b/dw/pipeline_processors/h3_mask_steps.py index 8f0c02a4b..319f8c4bb 100644 --- a/dw/pipeline_processors/h3_mask_steps.py +++ b/dw/pipeline_processors/h3_mask_steps.py @@ -26,6 +26,7 @@ from .h3_rules import ( AUDIO_PASS_ROWS, HELD_VIDEO_OUTPUT, + HELD_VIDEO_RATE_OUTPUT, KEEP_ROWS, LISTEN_SCALE_INPUT, MASK_GROW_DEFAULT, @@ -274,6 +275,14 @@ def inputs(self): "and what the result plays in place of the decode." ), ), + InputParam( + name=HELD_VIDEO_RATE_OUTPUT, + default=None, + description=( + "Internal: the frame rate those frames play at - the source's own, which the " + "result keeps." + ), + ), InputParam( name=LISTEN_SCALE_INPUT, default=None, diff --git a/dw/pipeline_processors/h3_rules.py b/dw/pipeline_processors/h3_rules.py index ae8ae0029..a91d1c130 100644 --- a/dw/pipeline_processors/h3_rules.py +++ b/dw/pipeline_processors/h3_rules.py @@ -882,6 +882,8 @@ def guide_end_problem(frame, length, num_frames): # The source's own frames at full size, uint8 - what the result plays in # place of the decode (dw/output_extraction.py), as dw_held_audio is for audio HELD_VIDEO_OUTPUT = "dw_held_video" +# The rate those frames play at: the source's own, never H3's 24 (R25) +HELD_VIDEO_RATE_OUTPUT = "dw_held_video_fps" # The refine arguments an audio pass has no use for: no video row is re-denoised AUDIO_PASS_REFUSES = (REFINE_STRENGTH_INPUT, REFINE_SIGMAS_INPUT, REFINE_TILES_INPUT) @@ -1001,18 +1003,36 @@ def _audio_pass_problems(arguments): def listen_canvas(width, height, scale): """The (width, height) an audio pass at `scale` runs its kept rows on: the - source's canvas scaled and snapped to the nearest TILE_GRID multiple, ties - up, never below one cell of it.""" - return tuple( - max(TILE_GRID, int(side * scale / TILE_GRID + 0.5) * TILE_GRID) - for side in (width, height) - ) + shorter side scaled and snapped to the nearest TILE_GRID multiple, ties + up, the longer derived from the source's aspect and snapped the same way, + neither below one cell of the grid. Snapping each side on its own would + squeeze 960x544 at 0.25 to 256x128, 13% off its aspect.""" + + def snapped(side): + return max(TILE_GRID, int(side / TILE_GRID + 0.5) * TILE_GRID) + + short = snapped(min(width, height) * scale) + long = snapped(short * max(width, height) / min(width, height)) + return (long, short) if width >= height else (short, long) -def media_problems(name, shape, width, height, num_frames=None): +def retimed_indices(frame_count, fps): + """The source frame each conditioning frame of an audio pass shows: the + source's `frame_count` frames at `fps` resampled in time, nearest frame, + to H3_FPS over the same duration, the count snapped up onto RENDER_GRID + as `num_frames` is (the frames past the end repeat the last). The video + rows then encode a 24 fps picture of the same moments, and the audio + generated against them covers the source's real duration (R25).""" + count = aligned(max(1, round(frame_count / fps * H3_FPS)), RENDER_GRID) + return [min(frame_count - 1, int(i * fps / H3_FPS + 0.5)) for i in range(count)] + + +def media_problems(name, shape, width, height, num_frames=None, fps=None): """Why a `source_video` or mask of `shape` - (frames, height, width) - does not fit a `width` x `height` canvas, as a list. `num_frames` is the - step's frame count, when it is known; a `mask_image` (one frame) fits any.""" + step's frame count, when it is known; a `mask_image` (one frame) fits any. + `fps` is an audio pass's source rate: at any rate but H3_FPS the source is + retimed onto the grid (`retimed_indices`), so its own count need not be.""" problems = [] if (width, height) != (shape[2], shape[1]): problems.append( @@ -1024,7 +1044,9 @@ def media_problems(name, shape, width, height, num_frames=None): f"{name} has {shape[0]} frames, and the step renders num_frames=" f"{num_frames} - they must match" ) - elif num_frames is not None and name == SOURCE_VIDEO_INPUT: + elif ( + num_frames is not None and name == SOURCE_VIDEO_INPUT and fps in (None, H3_FPS) + ): # Off the 17n + 5 grid the pipeline snaps num_frames up and the source # is padded to fit, so the mask drifts in time and the output is longer # than the source: refused here, where the count can still be changed diff --git a/dw/pipeline_processors/pipeline_h3.py b/dw/pipeline_processors/pipeline_h3.py index b02306d3d..a50d90cc7 100644 --- a/dw/pipeline_processors/pipeline_h3.py +++ b/dw/pipeline_processors/pipeline_h3.py @@ -9,7 +9,12 @@ """ from .h3_anchor import anchor_refusal, prepared_anchor -from .h3_guides import guide_frames_array, guides_refusal, prepared_guide +from .h3_guides import ( + guide_frames_array, + guides_refusal, + prepared_guide, + source_frame_rate, +) from .h3_hold import hold_audio_reference, holds_audio, refines from .h3_mask import binarized, edits_regions, listen_frames, passes_audio from .h3_rules import ( @@ -21,6 +26,8 @@ HELD_AUDIO_OUTPUT, HELD_AUDIO_RATE_OUTPUT, HELD_VIDEO_OUTPUT, + HELD_VIDEO_RATE_OUTPUT, + H3_FPS, HOLD_AUDIO_INPUT, LISTEN_SCALE_INPUT, MASK_IMAGE_INPUT, @@ -33,6 +40,7 @@ edits_region, is_audio_pass, listen_canvas, + retimed_indices, anchor_strength_problem, guide_noise_problem, mask_problems, @@ -52,12 +60,18 @@ def latent_anchor_source(chain_source, value): return "the frames given in the step's arguments" -def with_audio_pass(pipeline, arguments, where): +def with_audio_pass(pipeline, arguments, where, fps): """An audio pass's arguments (dw/pipeline_processors/h3_mask.py): the - full-size source as `HELD_VIDEO_OUTPUT`, asked for beside `videos` so - the result plays it in place of the decode, and at a `listen_scale` - below 1 the canvas scaled and the source area-resampled onto it. The - frame count - and so the audio's length - is the source's either way. + full-size source as `HELD_VIDEO_OUTPUT` and its rate `fps` as + `HELD_VIDEO_RATE_OUTPUT`, asked for beside `videos` so the result plays + the source's own frames at the source's own rate in place of the decode. + + The conditioning is what the kept rows encode. At any rate but H3_FPS it + is the source retimed to 24 fps over its own duration (`retimed_indices`), + and `num_frames` that count, so the audio generated covers the source's + real length; a 24 fps source is conditioned on as it is (R25). At a + `listen_scale` below 1 the canvas is scaled and the conditioning + area-resampled onto it. Raises: ValueError: If this pipeline has no audio-pass blocks @@ -69,15 +83,29 @@ def with_audio_pass(pipeline, arguments, where): ) source = arguments[SOURCE_VIDEO_INPUT] arguments[HELD_VIDEO_OUTPUT] = source + arguments[HELD_VIDEO_RATE_OUTPUT] = fps + if fps != H3_FPS: + indices = retimed_indices(len(source), fps) + arguments[SOURCE_VIDEO_INPUT] = source[indices] + arguments["num_frames"] = len(indices) + emit_log( + f"Audio pass: the source runs at {fps:g} fps - conditioning on " + f"{len(indices)} frames retimed to {H3_FPS} fps; the result plays " + f"its own {len(source)} frames at {fps:g} fps", + source_fps=fps, + conditioning_frames=len(indices), + ) scale = arguments.get(LISTEN_SCALE_INPUT) if scale is not None and scale != 1: width, height = listen_canvas(arguments["width"], arguments["height"], scale) arguments["width"], arguments["height"] = width, height - arguments[SOURCE_VIDEO_INPUT] = listen_frames(source, width, height) + arguments[SOURCE_VIDEO_INPUT] = listen_frames( + arguments[SOURCE_VIDEO_INPUT], width, height + ) output = arguments.get("output") names = [output] if isinstance(output, str) else list(output or []) if "videos" in names: - arguments["output"] = names + [HELD_VIDEO_OUTPUT] + arguments["output"] = names + [HELD_VIDEO_OUTPUT, HELD_VIDEO_RATE_OUTPUT] return arguments @@ -266,13 +294,25 @@ def _with_mask(self, arguments): problems = mask_problems(arguments, chained=self.segment_label is not None) if problems: raise ValueError(f"{where}: {problems[0]}") + # Decided before the mask is popped below, so an edit is never misread + audio_pass = is_audio_pass(arguments) arguments = dict(arguments) try: source = guide_frames_array(arguments[SOURCE_VIDEO_INPUT]) except ValueError as e: raise ValueError(f"{where}: {SOURCE_VIDEO_INPUT} is not a video: {e}") - arguments[SOURCE_VIDEO_INPUT] = source frames, height, width = source.shape[:3] + fps = None + if audio_pass: + fps = source_frame_rate(arguments[SOURCE_VIDEO_INPUT], frames) + if fps is None: + raise ValueError( + f"{where}: an audio pass plays the source_video's own frames " + f"at its own frame rate, and this source carries no frame " + f"rate and no soundtrack to time it by - pass a video file " + f"that declares its rate, or a step's video result" + ) + arguments[SOURCE_VIDEO_INPUT] = source for key, value in ( ("width", width), ("height", height), @@ -286,6 +326,7 @@ def _with_mask(self, arguments): arguments["width"], arguments["height"], arguments["num_frames"], + fps, ) name = next((n for n in MASK_INPUTS if arguments.get(n) is not None), None) if name is not None: @@ -313,8 +354,8 @@ def _with_mask(self, arguments): arguments[REGENERATE_MASK] = mask if problems: raise ValueError(f"{where}: {problems[0]}") - if is_audio_pass(arguments): - return with_audio_pass(self.pipeline, arguments, where) + if audio_pass: + return with_audio_pass(self.pipeline, arguments, where, fps) return arguments def _check_refine(self, arguments): diff --git a/tests/test_h3_audio_pass.py b/tests/test_h3_audio_pass.py index ec4c3c33b..8e2a52fbd 100644 --- a/tests/test_h3_audio_pass.py +++ b/tests/test_h3_audio_pass.py @@ -17,6 +17,7 @@ AUDIO_PASS_BLOCK, AUDIO_PASS_ROWS, HELD_VIDEO_OUTPUT, + HELD_VIDEO_RATE_OUTPUT, LISTEN_SCALE_INPUT, REGENERATE_MASK, RELEASE_VIDEO_BLOCK, @@ -48,6 +49,14 @@ def source_frames(width=WIDTH, height=HEIGHT, frames=FRAMES, seed=0): return rng.integers(0, 256, (frames, height, width, 3), dtype=np.uint8) +def rated(frames, fps=24): + """Frames carrying the rate they play at, as a loaded video file's do - + an audio pass refuses a source with no rate (R25).""" + from dw.media_types import FrameList + + return FrameList(list(frames), fps) + + def pipeline_with_blocks(): pipeline = minimax.MiniMaxH3Blocks().get_workflow("t2va").init_pipeline() h3_mask.insert_mask_blocks(pipeline) @@ -71,7 +80,7 @@ def with_mask(pipeline, arguments, chained=False): def audio_pass_arguments(**overrides): arguments = { - "source_video": source_frames(), + "source_video": rated(source_frames()), "prompt": "a voice says [English] hello", "output": ["videos", "audio", "sampling_rate"], } @@ -92,13 +101,14 @@ def test_the_output_frames_are_the_sources_own(): source = source_frames() arguments = with_mask( - pipeline_with_blocks(), audio_pass_arguments(source_video=source) + pipeline_with_blocks(), audio_pass_arguments(source_video=rated(source)) ) assert arguments["output"] == [ "videos", "audio", "sampling_rate", HELD_VIDEO_OUTPUT, + HELD_VIDEO_RATE_OUTPUT, ] # What the pipeline would hand back: a VAE round trip that is not the source @@ -109,6 +119,7 @@ def test_the_output_frames_are_the_sources_own(): "audio": generated, "sampling_rate": 48000, HELD_VIDEO_OUTPUT: arguments[HELD_VIDEO_OUTPUT], + HELD_VIDEO_RATE_OUTPUT: arguments[HELD_VIDEO_RATE_OUTPUT], } artifacts = modular_artifacts(result) @@ -124,7 +135,7 @@ def test_listen_scale_half_runs_the_kept_rows_at_half_canvas(): source = source_frames() arguments = with_mask( pipeline_with_blocks(), - audio_pass_arguments(source_video=source, listen_scale=0.5), + audio_pass_arguments(source_video=rated(source), listen_scale=0.5), ) assert (arguments["width"], arguments["height"]) == (WIDTH // 2, HEIGHT // 2) assert arguments["source_video"].shape == (FRAMES, HEIGHT // 2, WIDTH // 2, 3) @@ -245,9 +256,10 @@ def test_latents_beside_the_source_is_refused(self): def test_the_listen_canvas_snaps_to_32_px(self): assert listen_canvas(960, 544, 1) == (960, 544) - assert listen_canvas(960, 544, 0.5) == (480, 288) - assert listen_canvas(960, 544, 0.25) == (256, 128) - assert listen_canvas(48, 32, 0.25) == (32, 32) + assert listen_canvas(960, 544, 0.5) == (512, 288) + assert listen_canvas(960, 544, 0.25) == (224, 128) + assert listen_canvas(544, 960, 0.25) == (128, 224) + assert listen_canvas(64, 32, 0.25) == (64, 32) def h3_step(**arguments): @@ -344,6 +356,8 @@ def test_listen_scale_is_a_pipeline_input(self, built): h3_mask.insert_mask_blocks(pipeline) names = [param.name for param in pipeline._blocks.inputs] assert LISTEN_SCALE_INPUT in names, label + # The held frames and their rate are inputs, so the call returns them + assert HELD_VIDEO_OUTPUT in names and HELD_VIDEO_RATE_OUTPUT in names, label def test_the_hold_and_the_pass_coexist(self): from dw.pipeline_processors.h3_hold import insert_audio_hold @@ -529,7 +543,7 @@ def test_a_quarter_listen_scale_snaps_the_canvas(self): arguments = with_mask( pipeline_with_blocks(), audio_pass_arguments(listen_scale=0.25) ) - assert (arguments["width"], arguments["height"]) == (32, 32) + assert (arguments["width"], arguments["height"]) == (64, 32) def test_the_step_canvas_is_still_the_sources(self): with pytest.raises(ValueError, match="must match"): @@ -617,3 +631,181 @@ def test_it_is_an_audio_pass(self): assert "source_video" in step for name in ("hold_audio", "mask_image", "mask_video", "refine_strength"): assert name not in step + + +# The source's own frame rate (R25): conditioning is retimed to H3's 24 fps, +# the result plays the source's own frames at the source's own rate + + +def pass_result(arguments, seconds, sample_rate=48000): + """What the pipeline would hand back for `arguments`: a decode that is not + the source, `seconds` of generated audio, and the held keys echoed.""" + from dw.media_types import rated_videos + from dw.pipeline_processors.h3_rules import HELD_VIDEO_RATE_OUTPUT + from PIL import Image + + count = arguments["num_frames"] + canvas = (arguments["height"], arguments["width"], 3) + decoded = [Image.fromarray(np.zeros(canvas, np.uint8))] * count + generated = torch.randn(1, 2, round(seconds * sample_rate)) + return { + "videos": rated_videos([decoded], 24), + "audio": generated, + "sampling_rate": sample_rate, + HELD_VIDEO_OUTPUT: arguments[HELD_VIDEO_OUTPUT], + HELD_VIDEO_RATE_OUTPUT: arguments[HELD_VIDEO_RATE_OUTPUT], + }, generated + + +class TestSourceFrameRate: + def test_a_30_fps_source_plays_at_30_under_audio_of_its_own_length(self): + from dw.output_extraction import modular_artifacts + from dw.pipeline_processors.h3_rules import RENDER_GRID + from dw.variable_constraints import aligned + + # 124 frames at 30 fps is 4.13 s; at a relabelled 24 it would be 5.17 s + source = source_frames(frames=124) + arguments = with_mask( + pipeline_with_blocks(), audio_pass_arguments(source_video=rated(source, 30)) + ) + conditioning = aligned(round(124 / 30 * 24), RENDER_GRID) + assert arguments["num_frames"] == conditioning == 107 + assert arguments["source_video"].shape[0] == conditioning + assert np.array_equal(arguments[HELD_VIDEO_OUTPUT], source) + + result, _ = pass_result(arguments, conditioning / 24) + (played,) = modular_artifacts(result) + assert played.fps == 30 + assert len(played.frames) == 124 + assert np.array_equal(np.stack([np.asarray(f) for f in played.frames]), source) + assert played.audio.shape[-1] == round(124 / 30 * 48000) + + def test_a_24_fps_source_is_unchanged(self): + from dw.output_extraction import modular_artifacts + + source = source_frames() + arguments = with_mask( + pipeline_with_blocks(), audio_pass_arguments(source_video=rated(source, 24)) + ) + assert arguments["num_frames"] == FRAMES + assert arguments["source_video"] is arguments[HELD_VIDEO_OUTPUT] + assert np.array_equal(arguments["source_video"], source) + + result, generated = pass_result(arguments, 0.1) + (played,) = modular_artifacts(result) + assert played.fps == 24 + assert torch.equal(played.audio, generated[0]) + + @pytest.mark.parametrize( + "frames, fps", + [(124, 30), (100, 30), (60, 25), (240, 60), (50, 12), (124, 23.976)], + ) + def test_the_retimed_conditioning_lands_on_the_h3_grid(self, frames, fps): + from dw.pipeline_processors.h3_rules import RENDER_GRID, retimed_indices + from dw.variable_constraints import aligned + + indices = retimed_indices(frames, fps) + assert len(indices) == aligned(len(indices), RENDER_GRID) + # It covers the source's duration, with less than one grid step over + assert len(indices) >= round(frames / fps * 24) + assert len(indices) - round(frames / fps * 24) < 17 + # Nearest-frame in time, never past the source's last frame + assert all(0 <= i < frames for i in indices) + assert indices == sorted(indices) + assert indices[0] == 0 + + def test_the_conditioning_frames_are_the_nearest_source_frames(self): + source = source_frames(width=32, height=32, frames=124) + arguments = with_mask( + pipeline_with_blocks(), + audio_pass_arguments(source_video=rated(source, 30)), + ) + for i in (0, 1, 4, 50, 98): + nearest = min(123, int(i * 30 / 24 + 0.5)) + assert np.array_equal(arguments["source_video"][i], source[nearest]) + + def test_a_soundtrack_times_a_source_that_carries_no_rate(self): + from dw.media_types import AudioVideo + from dw.pipeline_processors.h3_rules import HELD_VIDEO_RATE_OUTPUT + + source = source_frames(frames=124) + clip = AudioVideo(list(source), np.zeros((2, 124 * 1600)), 48000) + arguments = with_mask( + pipeline_with_blocks(), audio_pass_arguments(source_video=clip) + ) + assert arguments[HELD_VIDEO_RATE_OUTPUT] == pytest.approx(30) + assert arguments["num_frames"] == 107 + + def test_a_source_with_no_rate_and_no_soundtrack_is_refused(self): + with pytest.raises(ValueError, match="frame rate"): + with_mask( + pipeline_with_blocks(), + audio_pass_arguments(source_video=source_frames()), + ) + + def test_an_off_grid_source_at_another_rate_is_not_refused_for_the_grid(self): + # The conditioning count is retimed onto the grid, so the source's own + # count need not be on it; at 24 fps it still must + arguments = with_mask( + pipeline_with_blocks(), + audio_pass_arguments(source_video=rated(source_frames(frames=100), 30)), + ) + assert arguments["num_frames"] == 90 + with pytest.raises(ValueError, match="17n"): + with_mask( + pipeline_with_blocks(), + audio_pass_arguments(source_video=rated(source_frames(frames=100), 24)), + ) + + def test_validate_reads_the_rate_for_the_grid_check(self, tmp_path): + (tmp_path / "take.mp4").write_bytes(b"") + + def probe(fps): + return lambda path: { + "kind": "video", + "frame_count": 100, + "width": WIDTH, + "height": HEIGHT, + "fps": fps, + } + + step = {"steps": [h3_step(source_video="take.mp4")]} + assert mask_errors(step, None, str(tmp_path), probe=probe(30)) == [] + errors = mask_errors(step, None, str(tmp_path), probe=probe(24)) + assert len(errors) == 1 and "17n" in errors[0]["message"] + + +class TestReviewFindings: + def test_a_mask_edit_is_never_read_as_an_audio_pass(self): + # A mask with no hold_audio: _check_refine refuses it before the call + # today, but _with_mask itself must not take it for an audio pass + from PIL import Image + + mask = np.zeros((HEIGHT, WIDTH, 3), dtype=np.uint8) + mask[:8, :8] = 255 + arguments = with_mask( + pipeline_with_blocks(), + audio_pass_arguments(mask_image=Image.fromarray(mask), refine_strength=0.5), + ) + assert HELD_VIDEO_OUTPUT not in arguments + assert REGENERATE_MASK in arguments + + @pytest.mark.parametrize("width, height", [(960, 544), (1344, 768), (544, 960)]) + def test_a_quarter_listen_canvas_keeps_the_aspect(self, width, height): + canvas = listen_canvas(width, height, 0.25) + assert all(side % 32 == 0 and side >= 32 for side in canvas) + error = abs(canvas[0] / canvas[1] - width / height) / (width / height) + assert error < 0.07 + + def test_listen_scale_on_a_non_h3_step_is_refused_at_validate(self): + step = { + "name": "still", + "pipeline": { + "configuration": {"component_type": "FluxPipeline"}, + "arguments": {"prompt": "x", LISTEN_SCALE_INPUT: 0.5}, + }, + } + errors = mask_errors({"steps": [step]}) + assert errors + assert all(e["path"].endswith(LISTEN_SCALE_INPUT) for e in errors) + assert any("MiniMax-H3" in e["message"] for e in errors) From d0cd8490034f42dad008c5d7fb0419a6a50ad5e0 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:28:31 -0600 Subject: [PATCH 039/405] fix(templates): music-long gives slice_audio the section's sample rate Co-Authored-By: Claude Opus 5.5 --- tests/test_text_takes.py | 16 ++++++++++++++++ workflows/templates/minimax/music-long.json | 1 + 2 files changed, 17 insertions(+) diff --git a/tests/test_text_takes.py b/tests/test_text_takes.py index 7ad3383cf..e5f6182d9 100644 --- a/tests/test_text_takes.py +++ b/tests/test_text_takes.py @@ -196,3 +196,19 @@ def test_music_long_refuses_a_section_over_300_seconds(): definition["variables"]["sections"][0]["audio_duration"] = 301 errors = Workflow(definition, str(path.parent), str(path)).validation_errors() assert any("300" in json.dumps(e) for e in errors), errors + + +def test_music_long_slice_audio_steps_carry_a_sample_rate(): + import json + from pathlib import Path + + root = Path(__file__).resolve().parent.parent / "workflows" / "templates" + definition = json.loads((root / "minimax" / "music-long.json").read_text()) + slices = [ + s + for s in definition["steps"] + if s.get("task", {}).get("command") == "slice_audio" + ] + assert slices + for step in slices: + assert "sample_rate" in step["task"]["arguments"], step["name"] diff --git a/workflows/templates/minimax/music-long.json b/workflows/templates/minimax/music-long.json index 3e2491eca..91684e1fa 100644 --- a/workflows/templates/minimax/music-long.json +++ b/workflows/templates/minimax/music-long.json @@ -69,6 +69,7 @@ "command": "slice_audio", "arguments": { "audio": "previous_result:generate_section", + "sample_rate": 44100, "start_seconds": 0, "duration_seconds": "item:seconds" } From d80af3543dbed6347d0d2f60f00d041e034435bf Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:28:36 -0600 Subject: [PATCH 040/405] docs(chain): the jump ceiling's comment matches its value Co-Authored-By: Claude Opus 5.5 --- bench/records/noise-floor.md | 2 +- dw/pipeline_processors/chain_check.py | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/bench/records/noise-floor.md b/bench/records/noise-floor.md index 3a3cb31d1..99105f6c4 100644 --- a/bench/records/noise-floor.md +++ b/bench/records/noise-floor.md @@ -256,7 +256,7 @@ the decoded output, each segment's kept frames against the one before: | `look_ratio` | 1.28 | 2.67 | 2.71 | `LOOK_RATIO_MAX` 12 -> 5.5 | | `texture_ratio` (either way) | | 2.99 | 3.13 | kept 2.0, provisional | -The margin is a factor of 2 on the P99, rounded up to half a point. 4.0 stays +The margin is about a factor of 2 on the P99 (jump 2.02 -> 4.0; look 2.67 -> 5.5, rounded up to half a point). 4.0 stays below an intended cut's 9.9-20.3 and above a dissolve's 1.8; 5.5 stays well under the 19.2 a saturation-1.6 grade reads. diff --git a/dw/pipeline_processors/chain_check.py b/dw/pipeline_processors/chain_check.py index 1b2baa8b6..03e45be60 100644 --- a/dw/pipeline_processors/chain_check.py +++ b/dw/pipeline_processors/chain_check.py @@ -22,7 +22,8 @@ # The default ceilings, set from the 99th percentile of each seam measure # across the noise-floor runs of S1-S3 (bench/records/noise-floor.md, "Chain # check thresholds"; 28 seams of unchecked chains, measured with this -# module's own arithmetic), doubled for margin and rounded up to half a point: +# module's own arithmetic). The jump ceiling is about twice its P99 (2.02), +# so 4.0; the look ceiling is doubled and rounded up to half a point: # - jump_ratio: P99 2.02 -> 4.0. Well under `seam_frame_jump`'s own threshold # (25, dw/assessment_rules.py) and the 9.9-20.3 the stage D runs measured # for an intended cut, above a dissolve's 1.8 - a chain seam over it is a From 3e3e78455e1a1af7266c014e49e9769968a62297 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:31:12 -0600 Subject: [PATCH 041/405] bench: counter-round arms (round 1 and controls) Co-Authored-By: Claude Opus 5.5 --- bench/arms/counters-controls.json | 152 ++++++++++ bench/arms/counters-round1.json | 411 ++++++++++++++++++++++++++ bench/workflows/lf-s1-static.json | 212 +++++++++++++ bench/workflows/lf-s2-arc.json | 212 +++++++++++++ bench/workflows/s1-static.json | 23 +- bench/workflows/s2-arc.json | 23 +- bench/workflows/s3-dialogue-walk.json | 23 +- tests/test_bench_workflows.py | 34 +++ 8 files changed, 1081 insertions(+), 9 deletions(-) create mode 100644 bench/arms/counters-controls.json create mode 100644 bench/arms/counters-round1.json create mode 100644 bench/workflows/lf-s1-static.json create mode 100644 bench/workflows/lf-s2-arc.json diff --git a/bench/arms/counters-controls.json b/bench/arms/counters-controls.json new file mode 100644 index 000000000..a42b50bd0 --- /dev/null +++ b/bench/arms/counters-controls.json @@ -0,0 +1,152 @@ +[ + { + "name": "ctl-guide-s1-s1166", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-guide-s1-s2", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide" + }, + "seed": 2, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-lastframe-s1-s1166", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "last_frame" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-lastframe-s1-s2", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "last_frame" + }, + "seed": 2, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-guide-s2-s1166", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-guide-s2-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide" + }, + "seed": 2, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-lastframe-s2-s1166", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "last_frame" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-lastframe-s2-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "last_frame" + }, + "seed": 2, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-guide-s3-s1166", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "ctl-guide-s3-s2", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide" + }, + "seed": 2, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + } +] diff --git a/bench/arms/counters-round1.json b/bench/arms/counters-round1.json new file mode 100644 index 000000000..441c31c52 --- /dev/null +++ b/bench/arms/counters-round1.json @@ -0,0 +1,411 @@ +[ + { + "name": "r1-anchor25-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.25 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor50-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.5 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor75-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.75 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-backoff17-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "guide_backoff": 17 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-keystone-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "keystone": "segment_1" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-lookhold-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "look_hold": true + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-antiburn-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "antiburn_model_name": "Imagineer13/H3-Anti-Burn-LoRA" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-lf-carrypick-s1", + "scene": "s1-static", + "workflow": "lf-s1-static.json", + "arguments": { + "segments": 6, + "continuity": "last_frame", + "carry_pick_n": 4 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-lf-lookhold-s1", + "scene": "s1-static", + "workflow": "s1-static.json", + "arguments": { + "segments": 6, + "continuity": "last_frame", + "look_hold": true + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor25-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.25 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor50-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.5 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor75-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.75 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-backoff17-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "guide_backoff": 17 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-keystone-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "keystone": "segment_1" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-lookhold-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "look_hold": true + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-antiburn-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "antiburn_model_name": "Imagineer13/H3-Anti-Burn-LoRA" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-lf-carrypick-s2", + "scene": "s2-arc", + "workflow": "lf-s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "last_frame", + "carry_pick_n": 4 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-lf-lookhold-s2", + "scene": "s2-arc", + "workflow": "s2-arc.json", + "arguments": { + "segments": 6, + "continuity": "last_frame", + "look_hold": true + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor25-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.25 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor50-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.5 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-anchor75-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "anchor": "segment_1", + "anchor_strength": 0.75 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-backoff17-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "guide_backoff": 17 + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-keystone-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "keystone": "segment_1" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-lookhold-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "look_hold": true + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + }, + { + "name": "r1-antiburn-s3", + "scene": "s3-dialogue-walk", + "workflow": "s3-dialogue-walk.json", + "arguments": { + "segments": 6, + "continuity": "guide", + "antiburn_model_name": "Imagineer13/H3-Anti-Burn-LoRA" + }, + "seed": 1166, + "moments": [ + "frame:62", + "frame:277", + "frame:583" + ] + } +] diff --git a/bench/workflows/lf-s1-static.json b/bench/workflows/lf-s1-static.json new file mode 100644 index 000000000..0ee321bc3 --- /dev/null +++ b/bench/workflows/lf-s1-static.json @@ -0,0 +1,212 @@ +{ + "id": "BenchLfS1Static", + "description": "Long-form bench S1 with last_frame continuity and carry_pick: the s1-static workflow with the guide-only keys (guide_frames, guide_backoff, keystone) removed and chain.carry_pick = {sharpest: carry_pick_n} (default 4). The counter-round last_frame carry_pick arms run here; every other last_frame arm runs the plain s1-static workflow with continuity=last_frame.", + "variables": { + "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous static shot. The shot opens exactly on : a young woman in a plain dark grey sweater stands centered in front of a flat painted wall of bold diagonal rainbow stripes with a black circle at upper left and a teal square at lower right. She breathes calmly, blinks, shifts her weight slightly and turns her head a little left and right, staying in place at the center of the frame. The painted wall stays exactly as it is, evenly lit by soft studio light. The camera is locked off on a tripod and never moves.\n\noverall_soundscape: Quiet studio room tone, the soft rustle of a wool sweater, calm breathing.\n\nnon_diegetic_music: None.", + "image": "asset:bench/static-stripes.jpg", + "num_frames": 124, + "segments": 10, + "continuity": "last_frame", + "width": 960, + "height": 544, + "num_inference_steps": 5, + "video_shift": 6, + "audio_shift": 3, + "weights_dtype": "{int4}", + "seed": 1166, + "anchor": "none", + "guide_noise": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "attention_backend": "sparse", + "anchor_strength": null, + "look_hold": null, + "antiburn_model_name": null, + "antiburn_weight_name": "h3_antiburn_universal_v1.safetensors", + "antiburn_scale": 0.6, + "carry_pick_n": 4 + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + } + }, + "seed": "variable:seed", + "steps": [ + { + "name": "chained_keyframe_to_video_audio", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "attention_backend": "variable:attention_backend", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "fl2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "turbo", + "scale": 1 + }, + { + "model_name": "variable:antiburn_model_name", + "weight_name": "variable:antiburn_weight_name", + "adapter_name": "antiburn", + "scale": "variable:antiburn_scale" + } + ], + "chain": { + "segments": "variable:segments", + "continuity": "variable:continuity", + "anchor": "variable:anchor", + "segment_argument": "image", + "trim_frames": 2, + "crossfade_ms": 80, + "fps": 24, + "frame_snap": "constraint:num_frames", + "anchor_strength": "variable:anchor_strength", + "look_hold": "variable:look_hold", + "carry_pick": { + "sharpest": "variable:carry_pick_n" + } + }, + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "image": "variable:image", + "num_frames": "variable:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "guide_noise": "variable:guide_noise", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "final" + } + } + ] +} diff --git a/bench/workflows/lf-s2-arc.json b/bench/workflows/lf-s2-arc.json new file mode 100644 index 000000000..81b759543 --- /dev/null +++ b/bench/workflows/lf-s2-arc.json @@ -0,0 +1,212 @@ +{ + "id": "BenchLfS2Arc", + "description": "Long-form bench S2 with last_frame continuity and carry_pick: the s2-arc workflow with the guide-only keys (guide_frames, guide_backoff, keystone) removed and chain.carry_pick = {sharpest: carry_pick_n} (default 4). The counter-round last_frame carry_pick arms run here; every other last_frame arm runs the plain s2-arc workflow with continuity=last_frame.", + "variables": { + "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a bronze statue of a seated man on a stone plinth in a sunlit public garden, a red brick wall with ivy, tall cypress trees and a white gazebo behind it. The camera travels in a slow, steady orbit around the statue, arcing to the right at a constant speed and keeping the statue centered, so the brick wall, the cypresses, the gazebo and the garden behind it sweep past and change with the angle. Warm late-afternoon sunlight, leaves stirring gently in a light breeze.\n\noverall_soundscape: Outdoor garden ambience: light breeze in the leaves, distant birdsong, faint far-off traffic.\n\nnon_diegetic_music: None.", + "image": "asset:bench/arc-statue.jpg", + "num_frames": 124, + "segments": 10, + "continuity": "last_frame", + "width": 960, + "height": 544, + "num_inference_steps": 5, + "video_shift": 6, + "audio_shift": 3, + "weights_dtype": "{int4}", + "seed": 1166, + "anchor": "none", + "guide_noise": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "attention_backend": "sparse", + "anchor_strength": null, + "look_hold": null, + "antiburn_model_name": null, + "antiburn_weight_name": "h3_antiburn_universal_v1.safetensors", + "antiburn_scale": 0.6, + "carry_pick_n": 4 + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "H3 refuses width/height not a multiple of 32" + } + }, + "seed": "variable:seed", + "steps": [ + { + "name": "chained_keyframe_to_video_audio", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "attention_backend": "variable:attention_backend", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "fl2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "turbo", + "scale": 1 + }, + { + "model_name": "variable:antiburn_model_name", + "weight_name": "variable:antiburn_weight_name", + "adapter_name": "antiburn", + "scale": "variable:antiburn_scale" + } + ], + "chain": { + "segments": "variable:segments", + "continuity": "variable:continuity", + "anchor": "variable:anchor", + "segment_argument": "image", + "trim_frames": 2, + "crossfade_ms": 80, + "fps": 24, + "frame_snap": "constraint:num_frames", + "anchor_strength": "variable:anchor_strength", + "look_hold": "variable:look_hold", + "carry_pick": { + "sharpest": "variable:carry_pick_n" + } + }, + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "image": "variable:image", + "num_frames": "variable:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "guide_noise": "variable:guide_noise", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "final" + } + } + ] +} diff --git a/bench/workflows/s1-static.json b/bench/workflows/s1-static.json index 17ead1228..5dad47dcc 100644 --- a/bench/workflows/s1-static.json +++ b/bench/workflows/s1-static.json @@ -1,6 +1,6 @@ { "id": "BenchS1Static", - "description": "Long-form bench S1 (static): the #1175 control (arm A), reproduced exactly - same workflow, prompt, still, seed 1166 and arguments as probe-1166's drift-chain. #1175 drift probe: templates/minimax/chained-segments as shipped (turbo-4step-keyframe, fl2va) with guide continuity, plus two probe variables: `anchor` (chain.anchor, arm C) and `guide_noise` (H3 call argument, arm B; null is dropped before the call, so the default is today's call). Defaults are the control (arm A).", + "description": "Long-form bench S1 (static): the #1175 control (arm A), reproduced exactly - same workflow, prompt, still, seed 1166 and arguments as probe-1166's drift-chain. #1175 drift probe: templates/minimax/chained-segments as shipped (turbo-4step-keyframe, fl2va) with guide continuity, plus two probe variables: `anchor` (chain.anchor, arm C) and `guide_noise` (H3 call argument, arm B; null is dropped before the call, so the default is today's call). Defaults are the control (arm A). Counter-round knobs (all null/off by default, so the defaults are still the control): anchor_strength, guide_backoff, keystone, look_hold (chain keys) and antiburn_model_name (set to minimax-h3's Imagineer13/H3-Anti-Burn-LoRA to load it at antiburn_scale on every segment).", "variables": { "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous static shot. The shot opens exactly on : a young woman in a plain dark grey sweater stands centered in front of a flat painted wall of bold diagonal rainbow stripes with a black circle at upper left and a teal square at lower right. She breathes calmly, blinks, shifts her weight slightly and turns her head a little left and right, staying in place at the center of the frame. The painted wall stays exactly as it is, evenly lit by soft studio light. The camera is locked off on a tripod and never moves.\n\noverall_soundscape: Quiet studio room tone, the soft rustle of a wool sweater, calm breathing.\n\nnon_diegetic_music: None.", "image": "asset:bench/static-stripes.jpg", @@ -19,7 +19,14 @@ "guide_noise": null, "lora_model_name": "lightx2v/Minimax-h3-Turbo", "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", - "attention_backend": "sparse" + "attention_backend": "sparse", + "anchor_strength": null, + "guide_backoff": null, + "keystone": null, + "look_hold": null, + "antiburn_model_name": null, + "antiburn_weight_name": "h3_antiburn_universal_v1.safetensors", + "antiburn_scale": 0.6 }, "variable_constraints": { "num_frames": { @@ -153,6 +160,12 @@ "weight_name": "variable:lora_weight_name", "adapter_name": "turbo", "scale": 1 + }, + { + "model_name": "variable:antiburn_model_name", + "weight_name": "variable:antiburn_weight_name", + "adapter_name": "antiburn", + "scale": "variable:antiburn_scale" } ], "chain": { @@ -164,7 +177,11 @@ "trim_frames": 2, "crossfade_ms": 80, "fps": 24, - "frame_snap": "constraint:num_frames" + "frame_snap": "constraint:num_frames", + "anchor_strength": "variable:anchor_strength", + "guide_backoff": "variable:guide_backoff", + "keystone": "variable:keystone", + "look_hold": "variable:look_hold" }, "scheduler": { "shift": "variable:video_shift" diff --git a/bench/workflows/s2-arc.json b/bench/workflows/s2-arc.json index dd4c8be04..ff9f632d3 100644 --- a/bench/workflows/s2-arc.json +++ b/bench/workflows/s2-arc.json @@ -1,6 +1,6 @@ { "id": "BenchS2Arc", - "description": "Long-form bench S2 (arc): the #1175 control (arm A), reproduced exactly - same workflow, prompt, still, seed 1166 and arguments as probe-1166's drift-chain. #1175 drift probe: templates/minimax/chained-segments as shipped (turbo-4step-keyframe, fl2va) with guide continuity, plus two probe variables: `anchor` (chain.anchor, arm C) and `guide_noise` (H3 call argument, arm B; null is dropped before the call, so the default is today's call). Defaults are the control (arm A).", + "description": "Long-form bench S2 (arc): the #1175 control (arm A), reproduced exactly - same workflow, prompt, still, seed 1166 and arguments as probe-1166's drift-chain. #1175 drift probe: templates/minimax/chained-segments as shipped (turbo-4step-keyframe, fl2va) with guide continuity, plus two probe variables: `anchor` (chain.anchor, arm C) and `guide_noise` (H3 call argument, arm B; null is dropped before the call, so the default is today's call). Defaults are the control (arm A). Counter-round knobs (all null/off by default, so the defaults are still the control): anchor_strength, guide_backoff, keystone, look_hold (chain keys) and antiburn_model_name (set to minimax-h3's Imagineer13/H3-Anti-Burn-LoRA to load it at antiburn_scale on every segment).", "variables": { "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a bronze statue of a seated man on a stone plinth in a sunlit public garden, a red brick wall with ivy, tall cypress trees and a white gazebo behind it. The camera travels in a slow, steady orbit around the statue, arcing to the right at a constant speed and keeping the statue centered, so the brick wall, the cypresses, the gazebo and the garden behind it sweep past and change with the angle. Warm late-afternoon sunlight, leaves stirring gently in a light breeze.\n\noverall_soundscape: Outdoor garden ambience: light breeze in the leaves, distant birdsong, faint far-off traffic.\n\nnon_diegetic_music: None.", "image": "asset:bench/arc-statue.jpg", @@ -19,7 +19,14 @@ "guide_noise": null, "lora_model_name": "lightx2v/Minimax-h3-Turbo", "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", - "attention_backend": "sparse" + "attention_backend": "sparse", + "anchor_strength": null, + "guide_backoff": null, + "keystone": null, + "look_hold": null, + "antiburn_model_name": null, + "antiburn_weight_name": "h3_antiburn_universal_v1.safetensors", + "antiburn_scale": 0.6 }, "variable_constraints": { "num_frames": { @@ -153,6 +160,12 @@ "weight_name": "variable:lora_weight_name", "adapter_name": "turbo", "scale": 1 + }, + { + "model_name": "variable:antiburn_model_name", + "weight_name": "variable:antiburn_weight_name", + "adapter_name": "antiburn", + "scale": "variable:antiburn_scale" } ], "chain": { @@ -164,7 +177,11 @@ "trim_frames": 2, "crossfade_ms": 80, "fps": 24, - "frame_snap": "constraint:num_frames" + "frame_snap": "constraint:num_frames", + "anchor_strength": "variable:anchor_strength", + "guide_backoff": "variable:guide_backoff", + "keystone": "variable:keystone", + "look_hold": "variable:look_hold" }, "scheduler": { "shift": "variable:video_shift" diff --git a/bench/workflows/s3-dialogue-walk.json b/bench/workflows/s3-dialogue-walk.json index 182ded766..a8f7abaf8 100644 --- a/bench/workflows/s3-dialogue-walk.json +++ b/bench/workflows/s3-dialogue-walk.json @@ -1,6 +1,6 @@ { "id": "BenchS3DialogueWalk", - "description": "Long-form bench S3 (dialogue walk): templates/minimax/chained-segments with continuity 'guide' and carry_audio, six segments, a cast portrait (asset:bench/walker.png) and one short spoken line per segment through `prompts`. Each line is sized to a segment (see plugins/dw/skills/minimax-h3/references/dialogue.md). `anchor` and `guide_noise` are the drift-probe knobs; their defaults are today's call.", + "description": "Long-form bench S3 (dialogue walk): templates/minimax/chained-segments with continuity 'guide' and carry_audio, six segments, a cast portrait (asset:bench/walker.png) and one short spoken line per segment through `prompts`. Each line is sized to a segment (see plugins/dw/skills/minimax-h3/references/dialogue.md). `anchor` and `guide_noise` are the drift-probe knobs; their defaults are today's call. Counter-round knobs (all null/off by default, so the defaults are still the control): anchor_strength, guide_backoff, keystone, look_hold (chain keys) and antiburn_model_name (set to minimax-h3's Imagineer13/H3-Anti-Burn-LoRA to load it at antiburn_scale on every segment).", "variables": { "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. She walks toward the camera down the cobblestone street, slow and steady, glancing up at the lamps. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] I told myself I would not come back here. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", "image": "asset:bench/walker.png", @@ -27,7 +27,14 @@ "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. A faint smile crosses her face as she keeps walking toward the camera. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] Funny how a place waits for you without asking. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. Her expression softens and her pace eases, the golden light warm along her shoulder. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] Maybe that is what I was afraid of. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None.", "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action, one continuous shot. The shot opens exactly on : a woman in her early thirties with shoulder-length dark hair, in a long olive coat, walks toward the camera down a quiet cobblestone street at golden hour. She squares her shoulders and walks on, the camera keeping her centered in frame. The woman (S1), a warm, low female alto voice, calm and unhurried, says [English] Well. Let's see if the door still sticks. The camera holds a steady medium shot at eye level, keeping pace with her as she walks.\n\noverall_soundscape: Soft footsteps on cobblestones, a faint breeze, distant street ambience.\n\nnon_diegetic_music: None." - ] + ], + "anchor_strength": null, + "guide_backoff": null, + "keystone": null, + "look_hold": null, + "antiburn_model_name": null, + "antiburn_weight_name": "h3_antiburn_universal_v1.safetensors", + "antiburn_scale": 0.6 }, "variable_constraints": { "num_frames": { @@ -161,6 +168,12 @@ "weight_name": "variable:lora_weight_name", "adapter_name": "turbo", "scale": 1 + }, + { + "model_name": "variable:antiburn_model_name", + "weight_name": "variable:antiburn_weight_name", + "adapter_name": "antiburn", + "scale": "variable:antiburn_scale" } ], "chain": { @@ -174,7 +187,11 @@ "crossfade_ms": 80, "fps": 24, "frame_snap": "constraint:num_frames", - "carry_audio": true + "carry_audio": true, + "anchor_strength": "variable:anchor_strength", + "guide_backoff": "variable:guide_backoff", + "keystone": "variable:keystone", + "look_hold": "variable:look_hold" }, "scheduler": { "shift": "variable:video_shift" diff --git a/tests/test_bench_workflows.py b/tests/test_bench_workflows.py index 3ce67e4a2..7eee9a60a 100644 --- a/tests/test_bench_workflows.py +++ b/tests/test_bench_workflows.py @@ -91,3 +91,37 @@ def test_a_workflow_declaring_prompts_references_it(): if "prompts" in (definition.get("variables") or {}): steps = json.dumps(definition.get("steps")) assert "variable:prompts" in steps, path.name + + +ARM_FILES = sorted((BENCH / "arms").glob("*.json")) + + +@pytest.mark.parametrize("path", ARM_FILES, ids=lambda p: p.stem) +def test_every_arm_file_loads_and_each_arm_validates(path): + """Each arm names a scene and workflow that exist, passes only variables + its workflow declares, and validates with those arguments.""" + from dw.workflow import Workflow + + scenes = driver.load_scenes(BENCH / "scenes.json") + arms = driver.load_arms(path) + assert arms + names = [a.name for a in arms] + assert len(names) == len(set(names)), "arm names must be unique" + for arm in arms: + assert arm.scene in scenes, arm.name + workflow_path = BENCH / "workflows" / arm.workflow + definition = json.loads(workflow_path.read_text(encoding="utf-8")) + undeclared = set(arm.arguments) - set(definition["variables"]) + assert not undeclared, (arm.name, sorted(undeclared)) + errors = Workflow(definition, None, str(workflow_path)).validation_errors( + arguments=dict(arm.arguments) + ) + assert errors == [], (arm.name, errors) + + +def test_the_counter_round_arms_exist(): + controls = driver.load_arms(BENCH / "arms" / "counters-controls.json") + round1 = driver.load_arms(BENCH / "arms" / "counters-round1.json") + assert {a.seed for a in controls} == {1166, 2} + assert {a.arguments["continuity"] for a in controls} == {"guide", "last_frame"} + assert all(a.arguments["segments"] == 6 for a in controls + round1) From 31e7df7f1ded09e029d92ba6f41dfd793cf8e208 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:33:51 -0600 Subject: [PATCH 042/405] bench: counter-round moments follow each geometry (backoff, last_frame) Co-Authored-By: Claude Opus 5.5 --- bench/arms/counters-controls.json | 16 ++++++++-------- bench/arms/counters-round1.json | 28 ++++++++++++++-------------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/bench/arms/counters-controls.json b/bench/arms/counters-controls.json index a42b50bd0..ec2ac3a01 100644 --- a/bench/arms/counters-controls.json +++ b/bench/arms/counters-controls.json @@ -40,8 +40,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { @@ -55,8 +55,8 @@ "seed": 2, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { @@ -100,8 +100,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { @@ -115,8 +115,8 @@ "seed": 2, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { diff --git a/bench/arms/counters-round1.json b/bench/arms/counters-round1.json index 441c31c52..7b1a77aea 100644 --- a/bench/arms/counters-round1.json +++ b/bench/arms/counters-round1.json @@ -62,8 +62,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:234", + "frame:498" ] }, { @@ -126,8 +126,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { @@ -142,8 +142,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { @@ -209,8 +209,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:234", + "frame:498" ] }, { @@ -273,8 +273,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { @@ -289,8 +289,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:307", + "frame:673" ] }, { @@ -356,8 +356,8 @@ "seed": 1166, "moments": [ "frame:62", - "frame:277", - "frame:583" + "frame:234", + "frame:498" ] }, { From dcc587706efa6424b3a50480a0765f4cbb69a925 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 21:53:58 -0600 Subject: [PATCH 043/405] feat(templates): #1169 - repair one chain segment in place between its neighbours templates/minimax/repair-segment re-renders interior segment k of a finished H3 chain (run with save_segments + keep_segments) as one fl2va render held at both ends: segment k-1's last 22 frames as a guide at frame 0 and segment k+1's frames as a guide ending at the render's end (39 by default, so the tail hold outnumbers the head), both holding their audio. The guide spans are trimmed off and the repair spliced back between its neighbours; the take keeps its frame count exactly. A pure workflow cannot do it: picking segments k-1 and k+1 from `segment` and every offset and trim needs arithmetic on the files' frame counts. The new chain_repair_cut task (dw/tasks/chain_repair.py) finds the segment files from any one of them, does the geometry (repair_geometry, pure) and cuts the guides, the keyframe and the before/after lists. The 17-frame grid puts the tail guide at num_frames - tail_frames, always 17j, and leaves 12 (mod 17) new frames between the guides, so the repair also replaces segment k+1's first 12 frames (overrun) and the splice takes k+1 back from there. Default render 175 for an interior segment of a 124/22 chain; too small is refused naming the smallest. Segments 1 and N are refused (one-sided repair). concat_videos gains continuous_joins: seams recorded without hard_cut, so analyze_seams measures both repaired seams, butt-joined audio, refused with seam_fade_ms > 0 or audio_bleed_ms. No H3 engine change: a guide ending at num_frames already fits (pinned in test_h3_guides). The template's VRAM formula folds the two run-time guide clips into the voxel rate (38.72 bytes), since the per-guide charge would price each at num_frames and refuse the default on a 24 GB card; listed TEMPLATE_SPECIFIC. Compact listing budget 13_250 -> 13_450 (measured 13_432). Modules ratchet +1 (chain_repair.py). Concepts only from H3MD Director (AGPL-3.0) and roycho87/geiru_context_suite (GPL-3.0), credited in ATTRIBUTIONS. Co-Authored-By: Claude Opus 5.5 --- bench/records/for-don.md | 2 + docs/ARCHITECTURE.md | 1 + docs/ATTRIBUTIONS.md | 14 +- docs/TASKS.md | 50 +++ docs/WORKFLOW_GUIDE.md | 14 + dw/tasks/chain_repair.py | 305 ++++++++++++++ dw/tasks/concat_videos.py | 49 ++- dw/tasks/task.py | 1 + tests/test_catalog_structure.py | 21 +- tests/test_chain_repair.py | 387 ++++++++++++++++++ tests/test_concat_videos.py | 64 +++ tests/test_h3_guides.py | 13 + tests/test_h3_sparse_defaults.py | 7 +- tests/test_step_cache.py | 3 + tests/test_vram_inheritance.py | 10 +- workflows/templates/minimax/README.md | 1 + .../templates/minimax/repair-segment.json | 303 ++++++++++++++ 17 files changed, 1230 insertions(+), 15 deletions(-) create mode 100644 dw/tasks/chain_repair.py create mode 100644 tests/test_chain_repair.py create mode 100644 workflows/templates/minimax/repair-segment.json diff --git a/bench/records/for-don.md b/bench/records/for-don.md index ad4ebc517..b8734619b 100644 --- a/bench/records/for-don.md +++ b/bench/records/for-don.md @@ -5,3 +5,5 @@ Things only you can check. - [ ] Import one exported `.otioz` (`export_timeline`, #1197) in DaVinci Resolve 18.5+ once a bench cut exists: File > Import > Timeline, then check one clip per shot, the rate (23.976 as 24000/1001) and that each clip plays its span. Nothing here can run Resolve. - Compact catalog listing budget raised to 13_250 (tests) for the new split-takes, narration-long, narration-from-takes and music-long templates. Recommendation: keep (discovery). - Arch ratchet module count grows on feat/long-form-media (presets, chain_pick, chain_look, chain_check, timeline_export, text_takes, ...). Re-baseline at merge needs your ok. Recommendation: approve. +- Compact catalog listing budget raised again to 13_450 (tests) for `minimax/repair-segment` (#1169, task 14), measured 13_432. Recommendation: keep (discovery). +- `minimax/repair-segment` declares its own VRAM formula (folded guide frames, 38.72 bytes/voxel) and is listed in `TEMPLATE_SPECIFIC` in `tests/test_vram_inheritance.py`; it is derived, not measured. The Step 3 bench run should record its real peak. Recommendation: replace with the measured figure after the bench. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 15117dfd8..36a17a278 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -83,6 +83,7 @@ one to open. | Chain per-segment check and re-roll | `dw/pipeline_processors/chain_check.py`: `check_segment`, `SegmentCheck`, `best_attempt`, `reroll_seed`; the keys' rule is `dw/pipeline_processors/chain_rules.py`: `chain_check_problem` (torch-free, beside the look hold's `chain_look_hold_problem`); the attempt loop is `dw/pipeline_processors/chain.py`: `_checked_attempt`, `_render_attempt` | A checked segment is measured on the arithmetic `analyze_seams` uses (`dw/tasks/assess.py`: `seam_measures`, `frame_measures`) and `dw/texture_bands.py`, never a copy of it. Every attempt renders from the same `_ChainState`; nothing of an attempt (frames, audio join, shot, spill, look-hold state, segment 1's anchor or keystone) reaches the timeline until it is kept, and a re-roll runs on its own generator at `reroll_seed`, leaving the step's generator where attempt 0 left it. Segment 1 and a hard-cut seam are never checked. Absent `check` is off and absent `retries` is 0. The worst case is quoted by `dw/cost_scaling.py`: `_worst_case_retries`, 1 + (N - 1)(1 + retries) renders for N segments. | `tests/test_chain.py::TestCheckAndReroll::test_a_failed_segment_is_rerolled_with_a_new_seed_from_the_same_carry`, `tests/test_chain.py::TestCheckAndReroll::test_budget_exhausted_keeps_the_best_attempt_and_warns`, `tests/test_chain.py::TestCheckAndReroll::test_absent_check_is_todays_chain`, `tests/test_chain_check.py::test_the_estimate_quotes_the_worst_case_retries` | | Overlap windows of a long video | `dw/tasks/windows.py`: `window_video`, `window_span` | Window `i` covers source frames `[i*stride - overlap, i*stride + stride)`, its synthetic frames repeat the first or last frame, and its audio is cut on the source's own `frames_to_samples` boundaries, so adjacent windows' strides tile the track with no sample lost or repeated. | `tests/test_window_video.py::TestAudio::test_strides_tile_the_source_track_exactly`, `tests/test_window_video.py::TestRealPath::test_a_path_argument_keeps_its_audio`, `tests/test_window_video.py::TestFileMatchesInMemory::test_every_window_equals_the_in_memory_window`, `tests/test_read_frame_range.py` | | Keep a span of a video's frames and audio | `dw/tasks/trim.py`: `trim_video`; `dw/shots.py`: `trimmed_shots` | Frames `[start_frame, start_frame + num_frames)` are kept and the audio is cut on `frames_to_samples` of each end at the track's own rate, so consecutive trims tile the track. A span past the clip's end is refused, never shortened. Shots are clipped to the span, sample side cleared. | `tests/test_trim_video.py::TestFramesAndAudio::test_consecutive_trims_tile_the_track`, `tests/test_trim_video.py::TestShots::test_shots_are_clipped_across_the_boundary` | +| Repair one chain segment in place (#1169) | `dw/tasks/chain_repair.py`: `repair_geometry` (pure), `segment_files`, `chain_repair_cut`; `dw/tasks/concat_videos.py`: `concat_videos` (its continuous_joins argument); `workflows/templates/minimax/repair-segment.json` | The repair render holds segment k-1's tail at frame 0 and segment k+1 at its end (at `num_frames - tail_frames`, always 17j); the kept span plus segment k+1 from the overrun on leaves the take's frame count exactly. Interior segments only. The splice's seams are recorded without `hard_cut`, so `analyze_seams` measures them. | `tests/test_chain_repair.py::TestGeometry::test_the_splice_keeps_the_take_length_and_the_grid`, `tests/test_chain_repair.py::TestCut::test_the_cut_and_the_splice`, `tests/test_concat_videos.py::TestContinuousJoins::test_continuous_joins_record_no_hard_cut` | | Join processed windows back to the source's length | `dw/tasks/windows.py`: `join_windows`; `dw/task_domains.py`: `window_count`, `window_count_problem`, `join_windows_errors` | Output is exactly the source's frames, each once: later windows' first `overlap` frames blend over the previous window's last on the open ramp `dissolve_videos` uses. The window count is `window_count`'s `ceil(source_frames / stride)` and nothing else (it is also `window_video`'s last index plus one). Audio is the source's own track; shots are one per window with `overlap_frames` on each seam, their sample side laid over that track by `dw/shots.py`'s `remeasured_shots` (the same rounding as `frames_to_samples`, so the spans are cumulative). | `tests/test_join_windows.py::TestRoundTrip::test_unprocessed_windows_rejoin_to_the_source`, `tests/test_join_windows.py::TestShots::test_spans_partition_the_frames_and_tile_the_audio`, `tests/test_join_windows.py::TestUint8Output::test_matches_the_float32_accumulator_exactly` | | A `join_windows` list of the wrong length, before the run | `dw/window_count_errors.py`: `window_count_errors`, wired into `validation_errors` | Owned by the task, never a template: any `join_windows` step whose `source` is probeable header-only (`asset:`/`output:` or a readable literal path), with literal `num_frames`/`overlap` and an expanded `videos` list, is checked against `window_count_problem` - the same rule the run refuses with. Anything not knowable stays silent for the run. | `tests/test_window_count_errors.py` | | Fit to a model and restore | `dw/tasks/fit.py`: `fit_to_model`, `restore_to_source` | fit_to_model's record is the whole of what restore_to_source needs: the scale is inferred from the output's size and must be uniform, and restore returns the source's size times that scale and at most its frame count. | `tests/test_fit_to_model.py` | diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md index 059739bf8..f12f3c21d 100644 --- a/docs/ATTRIBUTIONS.md +++ b/docs/ATTRIBUTIONS.md @@ -132,7 +132,13 @@ segment's per-channel mean and standard deviation before decode [dw/pipeline_processors/h3_anchor.py](../dw/pipeline_processors/h3_anchor.py); [#1166](https://github.com/dkackman/diffusers-workflow/issues/1166)). The diagnosis it answers, colour and contrast drifting from segment to segment of -an H3 chain, was described by Lumifel and by roycho87 (NerdNotes). +an H3 chain, was described by Lumifel and by roycho87 (NerdNotes). It also +inspired repairing one segment of a finished chain in place, held to both of its +neighbours, and its rules: no tail hold on the last segment, a new seed for the +re-render, and the chain's own text encoder, length and sizes kept +([templates/minimax/repair-segment](../workflows/templates/minimax/repair-segment.json), +[dw/tasks/chain_repair.py](../dw/tasks/chain_repair.py); +[#1169](https://github.com/dkackman/diffusers-workflow/issues/1169)). ### [roycho87/geiru_context_suite](https://github.com/roycho87/geiru_context_suite) @@ -147,7 +153,11 @@ same suite's idea of a first-segment keyframe, and NerdNotes' (no licence, so concepts only), inspired `keystone` ([#1187](https://github.com/dkackman/diffusers-workflow/issues/1187)): segment 1's frame 0 as every later segment's keyframe, pulling the chain back toward its -clean start. +clean start. Its Anchoring node, a source's head held in a shot's last frames +with its audio, gave the repair's tail hold and the rule that the tail hold +outnumbers the head (39 frames against 22 in +[templates/minimax/repair-segment](../workflows/templates/minimax/repair-segment.json); +[#1169](https://github.com/dkackman/diffusers-workflow/issues/1169)). harlixay7/Kinoloom (MIT; concept only, no code taken) inspired `carry_pick` ([dw/pipeline_processors/chain_pick.py](../dw/pipeline_processors/chain_pick.py); diff --git a/docs/TASKS.md b/docs/TASKS.md index 5813e1ae4..564b74296 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -627,6 +627,7 @@ workflow guide). | `match_levels` | No | Even the shots' loudness out before joining - `"rms"` for perceived level (the measurement `get_gallery_metadata` reports as `mean_dbfs`), `"peak"` for the loudest sample. Off by default | | `match_levels_dbfs` | No | The level `match_levels` moves every shot to (default: -1 dBFS for `peak`, -20 dBFS for `rms`). A shot that would clip at the target is held at -0.5 dBFS peak instead, reported as a `match_levels_held` warning with a per-shot log event | | `sample_rate` | No | The joined soundtrack's rate. Default: the highest among the inputs, the rest resampled up with a warning (#108) | +| `continuous_joins` | No | `true` when the joins continue one take rather than cut between shots, as a chain segment repaired in place and spliced back (`chain_repair_cut`). Each seam is recorded without `hard_cut`, so `analyze_seams` measures its frame jump as it measures a chain's own seams, and the audio is butt-joined with no declick fade. Refused beside `seam_fade_ms` above 0 or `audio_bleed_ms` (default: `false`, every seam a cut) | Shots an earlier run wrote are joined by path or URL without regenerating them. A file's track, and a `previous_result:` shot's, is fitted to its frames' @@ -913,6 +914,55 @@ comes back as one shot named after its file (else `video 1`). A span past the cl shortened or padded), as are a `num_frames` of `0` or less and a negative `start_frame`; `validate_workflow` catches these on literal values. +### chain_repair_cut + +The pieces that re-render one segment of a finished chain in place (#1169), +for [templates/minimax/repair-segment](../workflows/templates/minimax/repair-segment.json). +A chain run with `save_segments` and `keep_segments` leaves each segment as a +playable mp4 beside the take; this command finds them from any one of them, +counts their frames, and cuts what a repair render of segment `k` needs: + +```json +{ + "task": { + "command": "chain_repair_cut", + "arguments": { + "chain_output": "output:minimax/MiniMaxH3I2VChained/latest/final/MiniMaxH3I2VChained-chained_keyframe_to_video_audio.0.segment-000.mp4", + "segment": 3, + "num_frames": 175, + "head_frames": 22, + "tail_frames": 39 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `chain_output` | Yes | Any one of the chain's segment files (`..segment-NNN.mp4`); the others are found beside it, and a gap in their numbering is refused | +| `segment` | Yes | The segment to re-render, 1-based. Only 2 to N-1: the first has no segment before it and the last none after it, and are refused | +| `num_frames` | Yes | The repair render's length, on the 17n + 5 grid: at least `head_frames` + segment k's frames + `tail_frames`, rounded up onto the grid. A smaller one is refused naming the smallest; a larger one reaches further into segment k+1 | +| `head_frames` | No | Segment k-1's last frames, held at the render's frame 0: a whole-latent length, 17m + 5 from 22 (default `22`) | +| `tail_frames` | No | Segment k+1's frames held as a guide ending at the render's last frame, a whole-latent length (default `39`, so the tail hold outnumbers the head) | +| `width`, `height` | No | The render's canvas; when given, refused unless it is the segment files' own size | + +It returns a record for later steps (`previous_result:cut.`): +`head_guide` and `tail_guide` (clips with their audio, for two `guides` +entries with `"audio": true`), `keyframe` (the head guide's first frame, the +fl2va keyframe a guide chain gives its own segments), `tail_frame` (where the +tail guide is placed: `num_frames - tail_frames`, always on 17j), `keep_start` +and `keep_frames` (the render span to keep, between the guides), `before` +(segment files 1 to k-1) and `after` (segment k+1 from `overrun` on, then the +files after it), with `overrun`, `segment_frames` and `total_frames`. + +The frames between the two guides number 12 more than a multiple of 17 +(17n + 5 less two 17m + 5 guides), and an interior segment of a 17n + 5 chain +with a 22- or 39-frame carry is a multiple of 17, so the new frames replace +segment k and the first `overrun` frames of segment k+1 (12 at the smallest +render). The tail guide is segment k+1 from `overrun` on, and `after` takes +segment k+1 back from the same frame, so `before`, the kept span and `after` +come to the take's own frame count exactly. + ### window_video Cut one overlapping, fixed-length window out of a long video, so a diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 4be5af53f..12570e695 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -2631,6 +2631,20 @@ See [workflows/templates/ltx2/chained-segments.json](../workflows/templates/ltx2 [workflows/templates/minimax/chain-matched-to-audio.json](../workflows/templates/minimax/chain-matched-to-audio.json) (audio-matched lip-sync of arbitrary length). +One bad segment of a finished H3 chain can be re-rendered in place instead of +re-rolling the chain, when the chain kept its segment files (`save_segments` and +`keep_segments`): +[workflows/templates/minimax/repair-segment.json](../workflows/templates/minimax/repair-segment.json) +renders segment k once with segment k-1's last frames as a guide at frame 0 and segment +k+1's frames as a guide ending at the render's end, both holding their audio, then trims +the guides off and splices the result between its neighbours. The take keeps its frame +count; the new frames also replace the first 12 frames of segment k+1 (the 17-frame +grid's remainder, [`chain_repair_cut`](TASKS.md#chain_repair_cut)). The splice is a +`concat_videos` with `continuous_joins`, so `analyze_seams` checks both repaired seams as +it checks a chain's own. Interior segments only: the first and the last need a +one-sided repair. The chain's settings (canvas, steps, LoRA, counters) are passed to it +as arguments, and the seed is new by default. + #### Checking each segment: `check` and `retries` With `check` on, each segment after the first is measured before anything of it reaches diff --git a/dw/tasks/chain_repair.py b/dw/tasks/chain_repair.py new file mode 100644 index 000000000..cd084622f --- /dev/null +++ b/dw/tasks/chain_repair.py @@ -0,0 +1,305 @@ +"""chain_repair_cut: the pieces that re-render one segment of a finished +chain in place (#1169, `templates/minimax/repair-segment`). + +A guide chain run with `save_segments` + `keep_segments` leaves each segment +as a playable mp4, `..segment-NNN.mp4`, holding the frames +it put on the timeline (chain_spill.py). Segment k is repaired by one H3 render +held at both ends: segment k-1's last `head_frames` frames as a guide at frame +0, segment k+1's frames as a guide ending at the render's last frame, both with +their audio. The render's new frames between the guides replace segment k, and +the splice puts it back between its neighbours. + +The geometry is the 17-frame grid's. A guide starts at frame 17j and is 17m + 5 +frames long, and a render is 17n + 5, so the tail guide ending at the render's +end starts at `num_frames - tail_frames`, always on 17j - and the frames between +the guides number 12 (mod 17). An interior segment of a 17n + 5 chain with a +22- or 39-frame carry is a multiple of 17 long, so the new frames cannot be +exactly segment k: they run `overrun` frames on into segment k+1 (12 at the +smallest render), the tail guide is segment k+1's frames from there, and the +splice takes segment k+1 back from that frame on. The take keeps its frame +count exactly; segments other than k and the head of k+1 are the files' own +frames, re-encoded once by the join. + +A workflow has no arithmetic, so this command does it (`repair_geometry`, +pure) and cuts the media; the template composes it with the H3 step, +`trim_video` and `concat_videos(continuous_joins=true)`, which keeps the two +repaired seams measured by analyze_seams rather than skipped as cuts. + +Concepts only from H3MD Director (AGPL-3.0) and roycho87/geiru_context_suite +(GPL-3.0): a tail hold that outnumbers the head hold, no tail hold on the last +segment, a new seed for a re-render (docs/ATTRIBUTIONS.md). +""" + +import logging +import os +import re + +from ..locations import validate_media_path +from ..media import probe_metadata +from ..pipeline_processors.h3_rules import GUIDE_FRAMES_PER_CHUNK +from ..security import validate_path +from ..task_domains import POSITIVE, check_arguments, whole_number +from .registry import register_command +from .trim import trim_video +from .video_utils import load_audio_video + +logger = logging.getLogger("dw") + +COMMAND = "chain_repair_cut" + +# The name SegmentSpill gives a segment file (chain_spill.py) +_SEGMENT_FILE = re.compile(r"^(?P.+)\.segment-(?P\d{3})\.mp4$") +# The shortest whole-latent guide a hold is made of +MIN_HOLD = 22 + + +def _hold_problem(name, frames): + if frames < MIN_HOLD or (frames - 5) % GUIDE_FRAMES_PER_CHUNK: + return ( + f"{COMMAND}: '{name}' must be a whole-latent guide length (17m + 5, " + f"from {MIN_HOLD}: 22, 39, 56), got {frames}" + ) + return None + + +def repair_geometry(segment_frames, segment, num_frames, head_frames, tail_frames): + """Where the repair render's guides sit and how the splice is cut. + + Args: + segment_frames: Each segment file's frame count, in chain order + segment: The segment to repair, 1-based; not the first or the last + num_frames: The repair render's length, on the 17n + 5 grid + head_frames: Frames of segment k-1's tail held at frame 0 + tail_frames: Frames of segment k+1 held at the render's end + + Returns: + A dict: `head_start` (where in segment k-1 the head guide starts), + `tail_start` (where in segment k+1 the tail guide starts, which is + also `overrun`, the frames of k+1 the repair replaces), `tail_frame` + (the render frame the tail guide is placed at), `keep_start` and + `keep_frames` (the render span spliced in), `minimum_num_frames` + and `total_frames` (the take's, unchanged by the splice). + + Raises: + ValueError: naming the argument and the value that would work. + """ + count = len(segment_frames) + if count < 3: + raise ValueError( + f"{COMMAND}: the chain has {count} segment file(s); repairing one in " + "place needs a segment on each side of it, so at least 3" + ) + if segment < 2 or segment > count - 1: + raise ValueError( + f"{COMMAND}: 'segment' must be 2 to {count - 1} of this {count}-segment " + f"chain, got {segment}. Segment 1 has no segment before it to hold " + f"and segment {count} none after it - the first and the last need a " + "one-sided repair (one guide), which this command does not cut" + ) + for name, frames in (("head_frames", head_frames), ("tail_frames", tail_frames)): + problem = _hold_problem(name, frames) + if problem: + raise ValueError(problem) + if (num_frames - 5) % GUIDE_FRAMES_PER_CHUNK: + raise ValueError( + f"{COMMAND}: 'num_frames' must be on H3's 17n + 5 grid, got {num_frames}" + ) + + before, replaced, after = segment_frames[segment - 2 : segment + 1] + if before < head_frames: + raise ValueError( + f"{COMMAND}: segment {segment - 1} has {before} frames, fewer than " + f"the {head_frames}-frame head guide" + ) + shortest = head_frames + replaced + tail_frames + over = (5 - shortest) % GUIDE_FRAMES_PER_CHUNK + minimum = shortest + over + overrun = num_frames - shortest + if overrun < 0: + raise ValueError( + f"{COMMAND}: 'num_frames' {num_frames} cannot hold segment {segment} " + f"({replaced} frames) between a {head_frames}-frame head and a " + f"{tail_frames}-frame tail guide - pass num_frames={minimum}" + ) + if overrun + tail_frames > after: + raise ValueError( + f"{COMMAND}: the tail guide would be segment {segment + 1}'s frames " + f"{overrun} to {overrun + tail_frames}, but it has {after} - " + f"lower num_frames (the smallest is {minimum}) or tail_frames" + ) + return { + "head_start": before - head_frames, + "tail_start": overrun, + "overrun": overrun, + "tail_frame": num_frames - tail_frames, + "keep_start": head_frames, + "keep_frames": replaced + overrun, + "minimum_num_frames": minimum, + "total_frames": sum(segment_frames), + } + + +def segment_files(chain_output): + """Every segment file of the chain `chain_output` names one of, in order.""" + path = validate_media_path(chain_output, what="chain_output") + match = _SEGMENT_FILE.match(os.path.basename(path)) + if match is None: + raise ValueError( + f"{COMMAND}: 'chain_output' must name one of a chain's segment files " + "(..segment-000.mp4, left by a chain run with " + f"save_segments and keep_segments), got {os.path.basename(path)!r}" + ) + directory = os.path.dirname(path) + found = {} + for name in os.listdir(directory): + sibling = _SEGMENT_FILE.match(name) + if sibling and sibling["base"] == match["base"]: + found[int(sibling["index"])] = validate_path( + os.path.join(directory, name), directory + ) + indices = sorted(found) + if indices != list(range(len(indices))): + missing = sorted(set(range(indices[-1] + 1)) - set(indices)) + raise ValueError( + f"{COMMAND}: the chain's segment files are not all there - " + f"segment-{missing[0]:03d} is missing beside {match['base']}" + ) + return [found[index] for index in indices] + + +def _frame_count(path): + info = probe_metadata(path) or {} + count = info.get("frame_count") + if not isinstance(count, int) or count <= 0: + raise ValueError(f"{COMMAND}: cannot count the frames of {path}") + return count, info + + +def chain_repair_cut( + chain_output, + segment, + num_frames, + head_frames=22, + tail_frames=39, + width=None, + height=None, +): + """Task command: cut a finished chain's segment files for a repair of one. + + Args: + chain_output: Any one of the chain's segment files (an `output:` + reference) - the others are found beside it + segment: The segment to re-render, 1-based; 2 to N-1 + num_frames: The repair render's length - at least segment k's frames + plus both guides, on the 17n + 5 grid; a smaller one is refused + naming the smallest that fits + head_frames: Segment k-1's last frames held at the render's frame 0 + (17m + 5 from 22) + tail_frames: Segment k+1's frames held at the render's end - more + than head_frames by default (39), the tail hold outnumbering the + head + width, height: The render's canvas, when given checked against the + segment files' own size - a repair renders on the chain's canvas + + Returns: + A dict for the template's later steps: `head_guide` and `tail_guide` + (AudioVideo clips with their audio), `keyframe` (the head guide's + first frame, the fl2va keyframe a guide chain gives its segments), + `tail_frame`, `keep_start`, `keep_frames`, `before` (segment files + 1..k-1) and `after` (segment k+1 from the overrun on, then the files + after it), with the geometry's numbers beside them. + """ + segment = whole_number(segment, "segment", COMMAND, required=True) + num_frames = whole_number(num_frames, "num_frames", COMMAND, required=True) + head_frames = whole_number(head_frames, "head_frames", COMMAND, required=True) + tail_frames = whole_number(tail_frames, "tail_frames", COMMAND, required=True) + width = whole_number(width, "width", COMMAND) + height = whole_number(height, "height", COMMAND) + check_arguments( + COMMAND, + segment=segment, + num_frames=num_frames, + head_frames=head_frames, + tail_frames=tail_frames, + width=width, + height=height, + ) + + paths = segment_files(chain_output) + counts, infos = zip(*(_frame_count(path) for path in paths)) + geometry = repair_geometry( + list(counts), segment, num_frames, head_frames, tail_frames + ) + _check_canvas(infos[segment - 1], width, height, paths[segment - 1]) + + previous = load_audio_video(paths[segment - 2]) + following = load_audio_video(paths[segment]) + for clip, path in ((previous, paths[segment - 2]), (following, paths[segment])): + if clip.audio is None: + raise ValueError( + f"{COMMAND}: {os.path.basename(path)} has no soundtrack, and the " + "repair holds the neighbours' audio over both guides" + ) + head_guide = trim_video(previous, geometry["head_start"], head_frames) + tail_guide = trim_video(following, geometry["tail_start"], tail_frames) + rest = trim_video( + following, geometry["tail_start"], counts[segment] - geometry["tail_start"] + ) + logger.info( + f"{COMMAND}: segment {segment} of {len(paths)} - render {num_frames} " + f"frames, head guide {head_frames} at 0, tail guide {tail_frames} at " + f"{geometry['tail_frame']}, {geometry['overrun']} frames into segment " + f"{segment + 1}" + ) + return { + **geometry, + "segments": len(paths), + "segment_frames": list(counts), + "head_guide": head_guide, + "tail_guide": tail_guide, + "keyframe": head_guide.frames[0], + "before": list(paths[: segment - 1]), + "after": [rest, *paths[segment + 1 :]], + } + + +def _check_canvas(info, width, height, path): + for name, wanted in (("width", width), ("height", height)): + if wanted is None: + continue + own = info.get(name) + if own is not None and own != wanted: + raise ValueError( + f"{COMMAND}: '{name}' is {wanted}, but the chain was rendered at " + f"{own} ({os.path.basename(path)}) - a repair renders on the " + "chain's canvas, so the splice joins frames of one size" + ) + + +@register_command( + COMMAND, + implementation="dw.tasks.chain_repair.chain_repair_cut", + domains={ + name: POSITIVE + for name in ( + "segment", + "num_frames", + "head_frames", + "tail_frames", + "width", + "height", + ) + }, + whole_numbers=( + "segment", + "num_frames", + "head_frames", + "tail_frames", + "width", + "height", + ), + media_arguments=("chain_output",), +) +def _handle_chain_repair_cut(task, arguments, previous_pipelines): + """Cut a finished chain's segment files for an in-place repair of one""" + return chain_repair_cut(**arguments) diff --git a/dw/tasks/concat_videos.py b/dw/tasks/concat_videos.py index 64963827f..4691ef7a7 100644 --- a/dw/tasks/concat_videos.py +++ b/dw/tasks/concat_videos.py @@ -44,6 +44,7 @@ def concat_videos( match_levels=None, match_levels_dbfs=None, sample_rate=None, + continuous_joins=False, ): """Concatenate a list of videos into a single AudioVideo. @@ -115,6 +116,13 @@ def concat_videos( highest rate among the inputs is chosen and the rest are resampled up to it, with a warning naming which. Give this to pin the target instead (#108) + continuous_joins: The joins continue one take rather than cut + between shots - a chain segment repaired in place and spliced + back between its neighbours (#1169). Each seam is recorded + without `hard_cut`, so analyze_seams measures its frame jump as + it does a chain's own seams, and its audio is butt-joined with no + declick fade. A cut device beside it (`seam_fade_ms` above 0, + `audio_bleed_ms`) is refused. Default false: every seam is a cut Returns: One AudioVideo; its audio is None when no input video carries any @@ -122,6 +130,7 @@ def concat_videos( if not isinstance(videos, list) or not videos: raise ValueError("concat_videos needs a non-empty list of videos") + seam_fade_ms = _continuous_seam_fade(continuous_joins, seam_fade_ms, audio_bleed_ms) _warn_bleed_without_seam(videos, audio_bleed_ms) names, videos, clips, waveforms, sample_rate = _prepare_inputs( videos, match_levels, match_levels_dbfs, sample_rate @@ -139,6 +148,7 @@ def concat_videos( audio_bleed_ms, audio_bleed_gain_db, seam_fade_ms, + continuous_joins, ) _warn_fps_override(videos, names, fps) @@ -159,6 +169,7 @@ def _joined_tracks( audio_bleed_ms, audio_bleed_gain_db, seam_fade_ms, + continuous_joins=False, ): """The prepared inputs joined end to end, as (frames, audio, shots): each video's frames after its head trim, its soundtrack joined at the @@ -189,6 +200,7 @@ def _joined_tracks( len(frames) - start_frame, start_sample if has_audio else None, sample_rate, + hard_cut=not continuous_joins, ) ) @@ -221,6 +233,31 @@ def _joined_tracks( return frames, audio, shots +def _continuous_seam_fade(continuous_joins, seam_fade_ms, audio_bleed_ms): + """The seam fade the joins take: 0 - a plain butt join - for continuous + joins, which refuse a cut's devices; the caller's own otherwise.""" + if not isinstance(continuous_joins, bool): + raise ValueError( + f"concat_videos needs 'continuous_joins' as true or false, got " + f"{continuous_joins!r}" + ) + if not continuous_joins: + return seam_fade_ms + if seam_fade_ms: + raise ValueError( + "concat_videos: 'continuous_joins' joins one take with no fade, so " + f"'seam_fade_ms' ({seam_fade_ms}) cannot be set beside it - a fade " + "is a cut's device" + ) + if audio_bleed_ms: + raise ValueError( + "concat_videos: 'continuous_joins' joins one take, so " + f"'audio_bleed_ms' ({audio_bleed_ms}) cannot be set beside it - a " + "bleed covers a cut" + ) + return 0 + + def _record_seam_fade(shot, applied): """Record on a seam's first shot the fade its join realized: the caller's own edit, recorded beside hard_cut so analyze_seams can tell a requested @@ -368,7 +405,15 @@ def _silence_channels(waveforms): def _shot_records_for( - video, name, index, head_trim, start_frame, frame_count, start_sample, sample_rate + video, + name, + index, + head_trim, + start_frame, + frame_count, + start_sample, + sample_rate, + hard_cut=True, ): """The shot records one input contributes to the joined picture.""" inner = getattr(video, "shots", None) @@ -382,7 +427,7 @@ def _shot_records_for( ) else: video_shots = [shot_record(name, start_frame, frame_count, start_sample or 0)] - if index and video_shots: + if index and video_shots and hard_cut: # Every seam this step draws is a cut it chose to make, unlike a # chain's inner segments (continuity is expected there) - marking # it lets analyze_seams tell the two apart (#466) diff --git a/dw/tasks/task.py b/dw/tasks/task.py index 3e11590f2..04c365c1a 100644 --- a/dw/tasks/task.py +++ b/dw/tasks/task.py @@ -24,6 +24,7 @@ from . import cuts # noqa: F401 - registers plan_cuts from . import trim # noqa: F401 - registers trim_video from . import text_takes # noqa: F401 - registers split_takes +from . import chain_repair # noqa: F401 - registers chain_repair_cut # The handlers by family, split out of this module when it outgrew the size # ceiling (#790). Each registers its commands on import, so these lines are diff --git a/tests/test_catalog_structure.py b/tests/test_catalog_structure.py index b6bc33bd0..674f35778 100644 --- a/tests/test_catalog_structure.py +++ b/tests/test_catalog_structure.py @@ -306,8 +306,9 @@ def test_per_entry_problems_reports_a_mismatched_entries_count(): "workflows/templates/assemble-and-score.json": {"video"}, # the name of the sub-workflow step that makes the clip "workflows/templates/best-of-n-to-video.json": {"video"}, - # attribute_voices' argument, fed the transcript rather than passed - "workflows/templates/attribute-lines.json": {"lines"}, + # attribute_voices' argument, fed the transcript rather than passed; and + # 'segment', a value of 'timestamps' (repair-segment's variable) + "workflows/templates/attribute-lines.json": {"lines", "segment"}, # the argument the composed image-to-video workflow receives "workflows/templates/compose-workflows.json": {"image"}, # a 'result' field, and the point is that this workflow omits it @@ -329,8 +330,13 @@ def test_per_entry_problems_reports_a_mismatched_entries_count(): # pair_audio's 'fit' argument and its value 'video', and the per-entry # 'num_frames' of a shot "workflows/templates/minimax/music-video.json": {"fit", "num_frames", "video"}, - # fields of the plan's entries, which music-video's shots then take - "workflows/templates/minimax/music-video-cuts.json": {"num_frames", "prompt"}, + # fields of the plan's entries, which music-video's shots then take; and + # 'segment', a value of 'timestamps' (repair-segment's variable) + "workflows/templates/minimax/music-video-cuts.json": { + "num_frames", + "prompt", + "segment", + }, # the pipeline's 'cache' block it writes inline, not the H3 templates' # variable of that name "workflows/templates/step-caching.json": {"cache"}, @@ -560,8 +566,11 @@ def test_no_stale_entry_in_the_allowlist(): # entries, the catalog's only long narration, its text splitter and its # multi-section music, with no cost blocks yet. Then to 13_250 (measured # 13_236) for `narration-from-takes`: `narration-long` now takes plain text and -# the take-list template keeps its own entry. -COMPACT_BUDGET = 13_250 +# the take-list template keeps its own entry. Then to 13_450 (measured +# 13_432) for `minimax/repair-segment` (#1169, task 14): the catalog's only +# in-place repair of one chain segment, 25 variables because it carries the +# chain's own settings by argument. +COMPACT_BUDGET = 13_450 FILTERED_BUDGET = 1_500 diff --git a/tests/test_chain_repair.py b/tests/test_chain_repair.py new file mode 100644 index 000000000..b70121dea --- /dev/null +++ b/tests/test_chain_repair.py @@ -0,0 +1,387 @@ +"""chain_repair_cut: re-render one chain segment in place between its +neighbours (#1169) - the splice arithmetic and the cut.""" + +import os + +import numpy +import pytest +from PIL import Image + +from dw.media_types import AudioVideo +from dw.tasks.chain_repair import chain_repair_cut, repair_geometry, segment_files +from dw.tasks.concat_videos import concat_videos +from dw.tasks.task import _COMMAND_REGISTRY +from dw.tasks.trim import trim_video +from dw.task_domains import frames_to_samples + +# A 5-segment guide chain at the chained-segments template's defaults: 124 +# frames a segment, 22 carried, so every segment after the first is 102 +CHAIN = [124, 102, 102, 102, 102] + + +def splice_length(segment_frames, segment, geometry): + """The take's frame count after the splice the template makes: the + segments before k, the kept span of the render, segment k+1 from the + overrun on, and the segments after it.""" + k = segment + return ( + sum(segment_frames[: k - 1]) + + geometry["keep_frames"] + + segment_frames[k] + - geometry["tail_start"] + + sum(segment_frames[k + 1 :]) + ) + + +class TestGeometry: + def test_default_chain_middle_segment(self): + geometry = repair_geometry(CHAIN, 3, 175, 22, 39) + assert geometry["minimum_num_frames"] == 175 + assert geometry["overrun"] == 12 + assert geometry["tail_frame"] == 136 + assert geometry["head_start"] == 80 + assert geometry["keep_start"] == 22 + assert geometry["keep_frames"] == 114 + assert splice_length(CHAIN, 3, geometry) == sum(CHAIN) + + def test_equal_holds_match_the_parked_plan(self): + # plan v1 on #1169: 124/22, segment 3 of 5 -> render 158, tail at 136 + geometry = repair_geometry(CHAIN, 3, 158, 22, 22) + assert geometry["minimum_num_frames"] == 158 + assert geometry["tail_frame"] == 136 + assert geometry["tail_start"] == 12 + + @pytest.mark.parametrize( + "frames, segment", + [ + (CHAIN, 2), + (CHAIN, 4), + ([124, 102, 102], 2), + ([345, 306, 306, 200], 2), # a short last segment (duration_seconds) + ([345, 306, 306, 200], 3), + ([124, 122, 122, 122], 2), # last_frame continuity, trim_frames 2 + ([124, 85, 85, 85], 3), # guide_frames 39 + ], + ) + @pytest.mark.parametrize("head, tail", [(22, 22), (22, 39), (39, 56)]) + def test_the_splice_keeps_the_take_length_and_the_grid( + self, frames, segment, head, tail + ): + shortest = head + frames[segment - 1] + tail + num_frames = shortest + (5 - shortest) % 17 + try: + geometry = repair_geometry(frames, segment, num_frames, head, tail) + except ValueError as error: + # Only a neighbour too short for the tail guide may refuse + assert "tail guide would be" in str(error) + return + assert splice_length(frames, segment, geometry) == sum(frames) + assert geometry["total_frames"] == sum(frames) + assert geometry["tail_frame"] % 17 == 0 + assert geometry["tail_frame"] + tail == num_frames + assert 0 <= geometry["overrun"] < 17 + # The kept span sits exactly between the two guides + assert geometry["keep_start"] == head + assert ( + geometry["keep_start"] + geometry["keep_frames"] == geometry["tail_frame"] + ) + + def test_a_longer_render_reaches_further_into_the_next_segment(self): + geometry = repair_geometry(CHAIN, 3, 192, 22, 39) + assert geometry["overrun"] == 29 + assert splice_length(CHAIN, 3, geometry) == sum(CHAIN) + + @pytest.mark.parametrize("segment", [1, 5, 0, 6]) + def test_the_first_and_last_segment_are_refused(self, segment): + with pytest.raises(ValueError, match="one-sided repair"): + repair_geometry(CHAIN, segment, 175, 22, 39) + + def test_a_render_too_short_names_the_smallest(self): + with pytest.raises(ValueError, match="pass num_frames=175"): + repair_geometry(CHAIN, 3, 158, 22, 39) + + def test_an_off_grid_render_is_refused(self): + with pytest.raises(ValueError, match="17n \\+ 5"): + repair_geometry(CHAIN, 3, 176, 22, 39) + + @pytest.mark.parametrize("name, head, tail", [("head", 21, 39), ("tail", 22, 30)]) + def test_a_hold_off_the_latent_grid_is_refused(self, name, head, tail): + with pytest.raises(ValueError, match=f"{name}_frames"): + repair_geometry(CHAIN, 3, 175, head, tail) + + def test_a_short_next_segment_is_refused(self): + with pytest.raises(ValueError, match="tail guide would be"): + repair_geometry([124, 102, 102, 40], 3, 175, 22, 39) + + def test_two_segments_are_too_few(self): + with pytest.raises(ValueError, match="at least 3"): + repair_geometry([124, 102], 2, 175, 22, 39) + + +FPS = 24 +RATE = 48000 + + +def write_chain(directory, lengths, base="Wf-step.0", size=(64, 32)): + """Segment files as SegmentSpill names them, each frame's red channel its + index on the take and its audio a ramp of the take's sample index.""" + from dw.writers import encode_video + import torch + + paths, start = [], 0 + for index, count in enumerate(lengths): + frames = numpy.zeros((count, size[1], size[0], 3), dtype=numpy.uint8) + frames[..., 0] = (numpy.arange(start, start + count) % 250)[:, None, None] + samples = frames_to_samples(count, FPS, RATE) + audio = numpy.full((1, samples), 0.1 + 0.05 * index, dtype=numpy.float32) + path = os.path.join(directory, f"{base}.segment-{index:03d}.mp4") + encode_video( + torch.from_numpy(frames), + fps=FPS, + output_path=path, + audio=torch.from_numpy(audio), + audio_sample_rate=RATE, + ) + paths.append(path) + start += count + return paths + + +@pytest.fixture +def chain(tmp_path): + return write_chain(str(tmp_path), [40, 51, 51, 51]) + + +class TestSegmentFiles: + def test_finds_every_sibling_in_order(self, chain, tmp_path): + # Another chain's files, and an unrelated file, beside them + write_chain(str(tmp_path), [5], base="Wf-other.0") + (tmp_path / "Wf-step.0-0.0.mp4").write_bytes(b"") + assert segment_files(chain[2]) == chain + + def test_a_file_that_is_no_segment_is_refused(self, tmp_path): + path = tmp_path / "take.mp4" + path.write_bytes(b"") + with pytest.raises(ValueError, match="save_segments and keep_segments"): + segment_files(str(path)) + + def test_a_missing_segment_is_refused(self, chain): + os.remove(chain[1]) + with pytest.raises(ValueError, match="segment-001 is missing"): + segment_files(chain[0]) + + +class TestCut: + def test_the_cut_and_the_splice(self, chain): + # 51-frame segments: head 22 + 51 + tail 22 = 95 -> 107, overrun 12 + cut = chain_repair_cut(chain[0], 2, 107, 22, 22, width=64, height=32) + assert cut["segments"] == 4 + assert cut["segment_frames"] == [40, 51, 51, 51] + assert cut["overrun"] == 12 and cut["tail_frame"] == 85 + assert len(cut["head_guide"].frames) == 22 + assert len(cut["tail_guide"].frames) == 22 + assert cut["head_guide"].audio is not None + assert cut["tail_guide"].audio is not None + assert cut["keyframe"] is cut["head_guide"].frames[0] + assert cut["before"] == chain[:1] + assert len(cut["after"]) == 2 + assert len(cut["after"][0].frames) == 51 - 12 + assert cut["after"][1:] == chain[3:] + + # A stand-in render: the head guide, new frames, the tail guide + new = [Image.new("RGB", (64, 32), (255, 255, 255))] * cut["keep_frames"] + render_frames = ( + list(cut["head_guide"].frames) + new + list(cut["tail_guide"].frames) + ) + samples = frames_to_samples(len(render_frames), FPS, RATE) + render = AudioVideo( + render_frames, + numpy.zeros((1, samples), dtype=numpy.float32), + RATE, + fps=FPS, + ) + assert len(render.frames) == 107 + kept = trim_video(render, cut["keep_start"], cut["keep_frames"]) + before = concat_videos(cut["before"], fps=FPS, continuous_joins=True) + after = concat_videos(cut["after"], fps=FPS, continuous_joins=True) + take = concat_videos([before, kept, after], fps=FPS, continuous_joins=True) + assert len(take.frames) == sum(cut["segment_frames"]) + assert not any(shot.get("hard_cut") for shot in take.shots) + # The take resumes on segment 3's own frame 12 right after the repair + resume = 40 + cut["keep_frames"] + red = take.frames[resume].getpixel((10, 10))[0] + assert abs(red - (40 + 51 + 12)) <= 3 + + def test_a_canvas_other_than_the_chains_is_refused(self, chain): + with pytest.raises(ValueError, match="chain's canvas"): + chain_repair_cut(chain[0], 2, 107, 22, 22, width=960, height=544) + + def test_the_last_segment_is_refused(self, chain): + with pytest.raises(ValueError, match="one-sided repair"): + chain_repair_cut(chain[0], 4, 107, 22, 22) + + def test_registered(self): + assert "chain_repair_cut" in _COMMAND_REGISTRY + + +TEMPLATE = ( + __import__("pathlib").Path(__file__).resolve().parent.parent + / "workflows" + / "templates" + / "minimax" + / "repair-segment.json" +) + + +def template_errors(arguments=None): + import json + + from dw.workflow import Workflow + + definition = json.loads(TEMPLATE.read_text()) + workflow = Workflow(definition, str(TEMPLATE.parent), str(TEMPLATE)) + return workflow.validation_errors(arguments) + + +class TestTemplate: + def test_validates_with_its_defaults(self): + assert template_errors() == [] + + def test_validates_with_sample_arguments(self): + assert ( + template_errors( + { + "segment": 3, + "num_frames": 192, + "tail_frames": 56, + "prompt": "A glass figure turns towards the light.", + "seed": 7, + } + ) + == [] + ) + + def test_segment_1_is_refused_before_the_run(self): + errors = template_errors({"segment": 1}) + assert any("segment" in str(error) for error in errors), errors + + def test_the_splice_is_continuous_and_the_guides_hold_audio(self): + import json + + steps = {s["name"]: s for s in json.loads(TEMPLATE.read_text())["steps"]} + assert steps["cut"]["task"]["command"] == "chain_repair_cut" + guides = steps["repair"]["pipeline"]["arguments"]["guides"] + assert [g["frame"] for g in guides] == [0, "previous_result:cut.tail_frame"] + assert all(g["audio"] is True for g in guides) + for name in ("before", "after", "splice"): + arguments = steps[name]["task"]["arguments"] + assert steps[name]["task"]["command"] == "concat_videos" + assert arguments["continuous_joins"] is True + assert steps["splice"]["task"]["arguments"]["videos"] == [ + "previous_result:before", + "previous_result:keep", + "previous_result:after", + ] + + def test_the_default_render_fits_the_default_chain(self): + import json + + variables = json.loads(TEMPLATE.read_text())["variables"] + geometry = repair_geometry( + CHAIN, + variables["segment"], + variables["num_frames"], + variables["head_frames"], + variables["tail_frames"], + ) + assert geometry["minimum_num_frames"] == variables["num_frames"] + assert variables["tail_frames"] > variables["head_frames"] + assert variables["seed"] is None + + +def test_the_cut_feeds_the_template_steps_in_a_workflow(chain, tmp_path, monkeypatch): + """The template's task steps as a workflow, with a stand-in for the H3 + render: a record's list and number fields reach trim_video and + concat_videos as one value each, and the splice keeps the take's length.""" + import av + + from dw.workflow import Workflow + + monkeypatch.setenv("DW_TRUST_WORKFLOWS", "1") + definition = { + "id": "repair-wiring", + "steps": [ + { + "name": "cut", + "task": { + "command": "chain_repair_cut", + "arguments": { + "chain_output": chain[0], + "segment": 2, + "num_frames": 107, + "tail_frames": 22, + }, + }, + }, + { + # Stands in for the render: its kept span is the head guide's + # frames from overrun on (a span of the right shape) + "name": "keep", + "task": { + "command": "trim_video", + "arguments": { + "video": "previous_result:cut.head_guide", + "start_frame": "previous_result:cut.overrun", + "num_frames": 10, + "fps": 24, + }, + }, + }, + { + "name": "before", + "task": { + "command": "concat_videos", + "arguments": { + "videos": "previous_result:cut.before", + "fps": 24, + "continuous_joins": True, + }, + }, + }, + { + "name": "after", + "task": { + "command": "concat_videos", + "arguments": { + "videos": "previous_result:cut.after", + "fps": 24, + "continuous_joins": True, + }, + }, + }, + { + "name": "splice", + "task": { + "command": "concat_videos", + "arguments": { + "videos": [ + "previous_result:before", + "previous_result:keep", + "previous_result:after", + ], + "fps": 24, + "continuous_joins": True, + }, + }, + "result": {"content_type": "video/mp4", "fps": 24}, + }, + ], + } + out = tmp_path / "out" + Workflow(definition, str(out), "").run({}) + saved = [p for p in out.rglob("*.mp4")] + assert len(saved) == 1, saved + with av.open(str(saved[0])) as container: + frames = sum(1 for _ in container.decode(video=0)) + # segment 1 (40) + the 10-frame stand-in + segments 3 (51 - 12) and 4 (51) + assert frames == 40 + 10 + 39 + 51 diff --git a/tests/test_concat_videos.py b/tests/test_concat_videos.py index f76c0d1c2..5a8055918 100644 --- a/tests/test_concat_videos.py +++ b/tests/test_concat_videos.py @@ -1110,3 +1110,67 @@ def test_fps_overriding_an_input_rate_warns(): deactivate_context(token) assert len(messages) == 1 assert "8 fps" in messages[0] and "24" in messages[0] + + +class TestContinuousJoins: + """#1169: a repaired chain segment is spliced back between its neighbours. + Those joins continue the take rather than cut it, so they must not be + recorded hard_cut - analyze_seams skips seam_frame_jump at a hard cut.""" + + def test_default_joins_are_hard_cuts(self): + result = concat_videos([audio_video(4, 0.1), audio_video(4, 0.2)], fps=4) + assert result.shots[1]["hard_cut"] is True + + def test_continuous_joins_record_no_hard_cut(self): + result = concat_videos( + [audio_video(4, 0.1), audio_video(4, 0.2), audio_video(4, 0.3)], + fps=4, + continuous_joins=True, + ) + assert [shot.get("hard_cut") for shot in result.shots] == [None] * 3 + assert all("seam_fade_ms" not in shot for shot in result.shots) + + def test_continuous_joins_butt_join_the_audio_untouched(self): + result = concat_videos( + [audio_video(4, 0.1), audio_video(4, 0.2)], + fps=4, + continuous_joins=True, + ) + # No declick ramp: every sample keeps its input's level + assert numpy.allclose(result.audio[:, :100], 0.1) + assert numpy.allclose(result.audio[:, 100:], 0.2) + + def test_an_inner_join_keeps_its_own_seams_continuous(self): + inner = concat_videos( + [audio_video(4, 0.1), audio_video(4, 0.2)], fps=4, continuous_joins=True + ) + outer = concat_videos( + [inner, audio_video(4, 0.3)], fps=4, continuous_joins=True + ) + assert len(outer.shots) == 3 + assert not any(shot.get("hard_cut") for shot in outer.shots) + + def test_the_seams_are_measured_by_analyze_seams(self): + from dw.tasks.assess import analyze_seams + + result = concat_videos( + [audio_video(4, 0.1), audio_video(4, 0.2)], fps=4, continuous_joins=True + ) + answer = analyze_seams(result) + assert answer["seams"][0]["hard_cut"] is False + + @pytest.mark.parametrize("extra", [{"seam_fade_ms": 50}, {"audio_bleed_ms": 300}]) + def test_a_cut_device_is_refused(self, extra): + with pytest.raises(ValueError, match="continuous_joins"): + concat_videos( + [audio_video(4, 0.1), audio_video(4, 0.2)], + fps=4, + continuous_joins=True, + **extra, + ) + + def test_a_non_boolean_is_refused(self): + with pytest.raises(ValueError, match="continuous_joins"): + concat_videos( + [audio_video(4, 0.1), audio_video(4, 0.2)], continuous_joins="yes" + ) diff --git a/tests/test_h3_guides.py b/tests/test_h3_guides.py index fc8fac52d..a6000fc19 100644 --- a/tests/test_h3_guides.py +++ b/tests/test_h3_guides.py @@ -199,6 +199,19 @@ def test_past_the_end_is_refused(self): assert "past the end" in guide_end_problem(119, 22, 124) assert guide_end_problem(0, 125, 124) + @pytest.mark.parametrize( + "num_frames, tail", [(158, 22), (175, 39), (192, 39), (209, 56), (345, 39)] + ) + def test_a_tail_guide_sits_at_the_end_of_any_render(self, num_frames, tail): + # #1169's repair holds the next segment at the render's end: on the + # 17n + 5 grid, num_frames - tail is always a chunk boundary, the + # guide fits, and its audio ends with the render's own audio + frame = num_frames - tail + assert guide_frame_problem(frame) is None + assert guide_end_problem(frame, tail, num_frames) is None + start, _pad, count = guide_audio_span(frame, tail) + assert start + count == guide_audio_latents(num_frames) + # 2. Rotary origin diff --git a/tests/test_h3_sparse_defaults.py b/tests/test_h3_sparse_defaults.py index 3b2afd2d2..40daaf2a5 100644 --- a/tests/test_h3_sparse_defaults.py +++ b/tests/test_h3_sparse_defaults.py @@ -27,9 +27,10 @@ def _read(path): ) -def test_twenty_one_h3_templates(): - # 20 since minimax/regional-edit (#1078), 21 since minimax/face-repair (#1099) - assert len(H3) == 21 +def test_twenty_two_h3_templates(): + # 20 since minimax/regional-edit (#1078), 21 since minimax/face-repair + # (#1099), 22 since minimax/repair-segment (#1169) + assert len(H3) == 22 @pytest.mark.parametrize("path", H3) diff --git a/tests/test_step_cache.py b/tests/test_step_cache.py index 0dad01a5e..1a096da02 100644 --- a/tests/test_step_cache.py +++ b/tests/test_step_cache.py @@ -809,6 +809,9 @@ def test_normalized_downstream_false_when_nothing_references_it(): "minimax/chained-segments.json": { "chained_keyframe_to_video_audio": "0d1cd2532b5c19d97ba165ddb4c715c9ead5e05c2ab6ba4db9bb182e012ec4c0" }, + "minimax/repair-segment.json": { + "repair": "0d1cd2532b5c19d97ba165ddb4c715c9ead5e05c2ab6ba4db9bb182e012ec4c0" + }, "minimax/composable-references.json": { "video_reference_to_video_audio": "ba9915580e6a8017d8c68efbddea776985d107222546f84cca18e319298243bf" }, diff --git a/tests/test_vram_inheritance.py b/tests/test_vram_inheritance.py index 7f0d8275a..f23e59d95 100644 --- a/tests/test_vram_inheritance.py +++ b/tests/test_vram_inheritance.py @@ -346,8 +346,14 @@ def _numbers(definition): # Templates measured on their own: they carry a formula that differs from # their identity's, because their bound is a different point on the card. Only # a name here may differ; every other template on the identity carries the -# identity-level formula, or declares none and inherits it -TEMPLATE_SPECIFIC = {"templates/ltx2/reference-sheet"} +# identity-level formula, or declares none and inherits it. +# minimax/repair-segment is not measured on its own: its two guides are clips +# cut at run time, which the per-guide charge would price at num_frames each, +# so it folds their frames into the identity's voxel rate (#1169) +TEMPLATE_SPECIFIC = { + "templates/ltx2/reference-sheet", + "templates/minimax/repair-segment", +} def disagreements(catalog, specific=()): diff --git a/workflows/templates/minimax/README.md b/workflows/templates/minimax/README.md index ff023d916..d220fe184 100644 --- a/workflows/templates/minimax/README.md +++ b/workflows/templates/minimax/README.md @@ -162,6 +162,7 @@ segments means half the compounding. | [chain-matched-to-audio.json](chain-matched-to-audio.json) | `match_audio`: a supplied track decides the length, and the final video is muxed against the original, seamless track | | [chain-video-continuity.json](chain-video-continuity.json) | `last_segment` continuity: the previous segment's tail rides along as a video reference, carrying motion and voice across the seam | | [chain-matched-and-aligned.json](chain-matched-and-aligned.json) | Everything together for a long soundtrack-driven take, with per-segment prompts and crash-safe `save_segments` | +| [repair-segment.json](repair-segment.json) | Re-rendering one interior segment of a finished chain (run with `save_segments` + `keep_segments`) between its neighbours, held to both by guides with their audio, and splicing it back with continuous joins | ## Going long: cuts (digital shorts) diff --git a/workflows/templates/minimax/repair-segment.json b/workflows/templates/minimax/repair-segment.json new file mode 100644 index 000000000..6f0ecc24f --- /dev/null +++ b/workflows/templates/minimax/repair-segment.json @@ -0,0 +1,303 @@ +{ + "id": "MiniMaxH3RepairSegment", + "description": "Re-renders one bad segment of a finished MiniMax-H3 chain in place and splices it back, so the rest of the take is kept rather than re-rolled (#1169). The chain must have been run with 'save_segments' and 'keep_segments' on its chain block, which leave each segment as a playable mp4 beside the take; 'chain_output' is an 'output:' reference to any one of those files (..segment-000.mp4) and the rest are found beside it. 'segment' is 1-based and must be an interior one, 2 to N-1: the first segment has nothing before it and the last nothing after it, so both need a one-sided repair this template does not make, and are refused. 'chain_repair_cut' counts the segment files and cuts the guides: the repair is one fl2va render with segment k-1's last 'head_frames' frames held at frame 0 (its first frame is the keyframe, as a guide chain gives its own segments) and segment k+1's frames held as a guide ending at the render's last frame, both with their audio, so picture and soundtrack run on from one neighbour into the other. A guide starts on the 17-frame grid and a render is 17n + 5 frames, so the frames between the guides number 12 more than a multiple of 17 while an interior segment of a 124/22 chain is 102: the new frames replace segment k and the first 12 frames of segment k+1, the tail guide is segment k+1's frames from 12 on, and the splice takes segment k+1 back from frame 12. 'num_frames' is the render's length - at least 'head_frames' + the segment's frames + 'tail_frames', rounded up onto the grid; the default 175 fits an interior segment of the chained-segments defaults (124 frames, guide_frames 22) with the 22/39 holds, and a value too small is refused before the model loads, naming the one to pass. The tail hold outnumbers the head hold by default (39 against 22). After the render the guide spans are trimmed off picture and audio ('trim_video') and 'concat_videos' with 'continuous_joins' joins segments 1..k-1, the repair and segments k+1..N: the take keeps its frame count exactly, and its two repaired seams are recorded as continuous rather than as cuts, so 'analyze_seams' measures their frame jump as it measures the chain's own seams. Every other frame is the segment files' own, through one more h264 encode, not byte-identical. Pass the chain's own settings: its canvas ('width'/'height', checked against the segment files), steps, shifts, LoRA, weights_dtype and its counters ('guide_noise'; 'latent_anchor', an 'output:' of segment 1's file, for a chain run with anchor segment_1), since a workflow cannot read another run's workflow - the defaults are the chained-segments template's. The text encoder and its truncation are the chain's. 'seed' defaults to a new one: a re-render at the chain's seed would not redraw the segment's noise anyway, since a chain runs one generator through every segment. 'prompt' should describe this segment's stretch of the take, timed for the render's length. Concepts only from H3MD Director and roycho87/geiru_context_suite (docs/ATTRIBUTIONS.md).", + "summary": "Re-render one interior segment of a finished H3 chain between its neighbours and splice it back.", + "vram_estimate": { + "base_gb": 16.3, + "bytes_per_voxel": 38.72, + "voxel_variables": [ + "width", + "height", + "num_frames" + ], + "reason": "one H3 render: the t2va calibration (#324) - peak activation memory scales with width * height * num_frames at 28.71 bytes a voxel - plus its two guides, each encoded on the target canvas at about the clip's own cost per voxel (docs/proposals/h3-guide-vram.md, #694). The guides are clips cut at run time, which the per-guide charge cannot count before the run and would price at num_frames each, so their frames are folded into the voxel rate instead: 28.71 * (175 + 22 + 39) / 175 = 38.72 bytes, exact at the default 175-frame render with the 22/39 holds and slightly over for a longer render with the same holds" + }, + "cost_drivers": [ + "num_frames", + "num_inference_steps", + "width", + "height", + "weights_dtype", + "lora_model_name", + "attention_backend", + "cache" + ], + "variables": { + "chain_output": "output:minimax/MiniMaxH3I2VChained/latest/final/MiniMaxH3I2VChained-chained_keyframe_to_video_audio.0.segment-000.mp4", + "segment": 2, + "prompt": "For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.\n\nintegrated_multimodal_description: [Shot 1] Cinematic, live-action macro photography with a shallow depth of field, one continuous shot. The shot opens exactly on : a translucent blown glass figure fills the frame against a dark, softly graded background, its surface catching cool rim light along every curve and throwing small caustics onto the surface beneath it. Its tentacles writhe without pause, curling and uncurling in slow deliberate coils, the glass flexing as though molten while staying perfectly clear. The figure (S1), a deep cavernous male voice with a long tail of reverb, unhurried and very low in pitch, speaks steadily throughout, [English] Rise now. The old ones are listening. Its central mass swells and contracts with the cadence of the words, and the caustics on the surface below ripple in time. The camera holds a static shot, letting the tentacles move through the frame edges.\n\noverall_soundscape: A dark, near-silent room tone under a faint low-frequency hum. The tentacle movement carries a fine, dry glass-on-glass squeak and an occasional soft chime as two limbs touch, continuous and unhurried from the first frame to the last.\n\nnon_diegetic_music: A sustained low drone of bowed double bass and detuned synthesiser, almost subsonic, holding an even intensity with no discernible melody or pulse.", + "seed": null, + "num_frames": 175, + "head_frames": 22, + "tail_frames": 39, + "width": 960, + "height": 544, + "num_inference_steps": 5, + "video_shift": 6.0, + "audio_shift": 3.0, + "weights_dtype": "{int4}", + "lora_scale": 1.0, + "lora_alpha": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "lora_adapter_name": "turbo", + "guide_noise": null, + "latent_anchor": null, + "latent_anchor_strength": null, + "attention_backend": "sparse", + "sparse_tau": 0.5, + "sparse_topk": 0.15, + "cache": null + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "min_frames": 124, + "max_frames": 345, + "snap": "up", + "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps" + }, + "segment": { + "min_frames": 2, + "reason": "segment 1 has no segment before it to hold - repairing the first segment (or the last) needs a one-sided repair with one guide, which this template does not make; the last segment is refused when the run counts the chain's files" + }, + "width": { + "modulus": 32, + "remainder": 0, + "reason": "MiniMax-H3's pipeline refuses a `width` or `height` that is not a multiple of 32, and only after the weights have loaded; refused here instead, with no 'snap', since rounding would be a silent change to the size asked for" + }, + "height": { + "modulus": 32, + "remainder": 0, + "reason": "MiniMax-H3's pipeline refuses a `width` or `height` that is not a multiple of 32, and only after the weights have loaded; refused here instead, with no 'snap', since rounding would be a silent change to the size asked for" + } + }, + "seed": "variable:seed", + "steps": [ + { + "name": "cut", + "task": { + "command": "chain_repair_cut", + "arguments": { + "chain_output": "variable:chain_output", + "segment": "variable:segment", + "num_frames": "variable:num_frames", + "head_frames": "variable:head_frames", + "tail_frames": "variable:tail_frames", + "width": "variable:width", + "height": "variable:height" + } + } + }, + { + "name": "repair", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "attention_backend": "variable:attention_backend", + "sparse_tau": "variable:sparse_tau", + "sparse_topk": "variable:sparse_topk", + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + }, + "cache": "variable:cache" + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "fl2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "variable:lora_adapter_name", + "scale": "variable:lora_scale", + "alpha": "variable:lora_alpha" + } + ], + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "image": "previous_result:cut.keyframe", + "guides": [ + { + "video": "previous_result:cut.head_guide", + "frame": 0, + "audio": true + }, + { + "video": "previous_result:cut.tail_guide", + "frame": "previous_result:cut.tail_frame", + "audio": true + } + ], + "num_frames": "variable:num_frames", + "width": "variable:width", + "height": "variable:height", + "num_inference_steps": "variable:num_inference_steps", + "guide_noise": "variable:guide_noise", + "latent_anchor": "variable:latent_anchor", + "latent_anchor_strength": "variable:latent_anchor_strength", + "output": [ + "videos", + "audio", + "sampling_rate" + ] + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "intermediate" + } + }, + { + "name": "keep", + "task": { + "command": "trim_video", + "arguments": { + "video": "previous_result:repair", + "start_frame": "previous_result:cut.keep_start", + "num_frames": "previous_result:cut.keep_frames", + "fps": 24 + } + } + }, + { + "name": "before", + "task": { + "command": "concat_videos", + "arguments": { + "videos": "previous_result:cut.before", + "fps": 24, + "continuous_joins": true + } + } + }, + { + "name": "after", + "task": { + "command": "concat_videos", + "arguments": { + "videos": "previous_result:cut.after", + "fps": 24, + "continuous_joins": true + } + } + }, + { + "name": "splice", + "task": { + "command": "concat_videos", + "arguments": { + "videos": [ + "previous_result:before", + "previous_result:keep", + "previous_result:after" + ], + "fps": 24, + "continuous_joins": true + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "final" + } + } + ] +} From dc1eef84f8605420f256eb83c74ccb1d8815498a Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 3 Nov 2026 22:18:45 -0600 Subject: [PATCH 044/405] fix(templates): #1169 - repair splices the take itself; holds capped for VRAM Review fixes on the in-place segment repair: - chain_repair_splice cuts the take, not the segment files: outside the repaired span the output keeps the take's frames, frame count (a duration_seconds take is shorter than its files) and track sample for sample, so every other seam keeps its crossfade or declick and a match_audio take keeps its own track whole. The span's audio is the render's, blended in and out of the take inside the span over the window the chain used at those seams. - A chain with keep_segments names each segment file on the take's shots (segment_file); the cut finds the files from them, and finds the take from a segment file through the run manifest. The by-name fallback refuses siblings that disagree in rate or size. Neighbours are counted from their decode. - head_frames is held at 22 and tail_frames at 22 or 39, and the folded VRAM rate is taken at the shortest render (42.84), so the estimate is an upper bound; the reason no longer calls it exact. - concat_videos(continuous_joins) keeps the default declick. - keep_segments variable on the four H3 chain templates; the schema takes a variable for save_segments/keep_segments. Co-Authored-By: Claude Opus 5.5 --- bench/records/for-don.md | 4 +- docs/ARCHITECTURE.md | 2 +- docs/TASKS.md | 57 +- docs/WORKFLOW_GUIDE.md | 17 +- dw/pipeline_processors/chain.py | 5 + dw/runs.py | 33 +- dw/shots.py | 7 +- dw/tasks/chain_repair.py | 484 +++++++++++++--- dw/tasks/concat_videos.py | 15 +- dw/workflow_schema.json | 14 +- tests/test_catalog_structure.py | 8 +- tests/test_chain.py | 20 + tests/test_chain_repair.py | 540 ++++++++++++++---- tests/test_concat_videos.py | 21 +- tests/test_shots.py | 2 + workflows/templates/minimax/README.md | 2 +- .../minimax/chain-matched-and-aligned.json | 6 +- .../minimax/chain-matched-to-audio.json | 7 +- .../minimax/chain-video-continuity.json | 7 +- .../templates/minimax/chained-segments.json | 7 +- .../templates/minimax/repair-segment.json | 75 +-- 21 files changed, 1034 insertions(+), 299 deletions(-) diff --git a/bench/records/for-don.md b/bench/records/for-don.md index b8734619b..be4814b17 100644 --- a/bench/records/for-don.md +++ b/bench/records/for-don.md @@ -6,4 +6,6 @@ Things only you can check. - Compact catalog listing budget raised to 13_250 (tests) for the new split-takes, narration-long, narration-from-takes and music-long templates. Recommendation: keep (discovery). - Arch ratchet module count grows on feat/long-form-media (presets, chain_pick, chain_look, chain_check, timeline_export, text_takes, ...). Re-baseline at merge needs your ok. Recommendation: approve. - Compact catalog listing budget raised again to 13_450 (tests) for `minimax/repair-segment` (#1169, task 14), measured 13_432. Recommendation: keep (discovery). -- `minimax/repair-segment` declares its own VRAM formula (folded guide frames, 38.72 bytes/voxel) and is listed in `TEMPLATE_SPECIFIC` in `tests/test_vram_inheritance.py`; it is derived, not measured. The Step 3 bench run should record its real peak. Recommendation: replace with the measured figure after the bench. +- `minimax/repair-segment` declares its own VRAM formula (folded guide frames, 42.84 bytes/voxel = 28.71 * (124 + 61) / 124, an upper bound with head_frames held at 22 and tail_frames at most 39) and is listed in `TEMPLATE_SPECIFIC` in `tests/test_vram_inheritance.py`; it is derived, not measured, and over-projects a long render (about 0.9 GB at 192 frames on 1344x768). The Step 3 bench run should record its real peak. Recommendation: replace with the measured figure after the bench. +- Compact catalog listing budget raised to 13_500 (tests) for the #1169 review fixes (repair-segment's `chain_take`, `keep_segments` on four H3 chain templates), measured 13_476. Recommendation: keep (discovery). +- `save_segments`/`keep_segments` in the workflow schema now accept a `variable:` reference (they were boolean only). Recommendation: keep. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 36a17a278..72bda57c9 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -83,7 +83,7 @@ one to open. | Chain per-segment check and re-roll | `dw/pipeline_processors/chain_check.py`: `check_segment`, `SegmentCheck`, `best_attempt`, `reroll_seed`; the keys' rule is `dw/pipeline_processors/chain_rules.py`: `chain_check_problem` (torch-free, beside the look hold's `chain_look_hold_problem`); the attempt loop is `dw/pipeline_processors/chain.py`: `_checked_attempt`, `_render_attempt` | A checked segment is measured on the arithmetic `analyze_seams` uses (`dw/tasks/assess.py`: `seam_measures`, `frame_measures`) and `dw/texture_bands.py`, never a copy of it. Every attempt renders from the same `_ChainState`; nothing of an attempt (frames, audio join, shot, spill, look-hold state, segment 1's anchor or keystone) reaches the timeline until it is kept, and a re-roll runs on its own generator at `reroll_seed`, leaving the step's generator where attempt 0 left it. Segment 1 and a hard-cut seam are never checked. Absent `check` is off and absent `retries` is 0. The worst case is quoted by `dw/cost_scaling.py`: `_worst_case_retries`, 1 + (N - 1)(1 + retries) renders for N segments. | `tests/test_chain.py::TestCheckAndReroll::test_a_failed_segment_is_rerolled_with_a_new_seed_from_the_same_carry`, `tests/test_chain.py::TestCheckAndReroll::test_budget_exhausted_keeps_the_best_attempt_and_warns`, `tests/test_chain.py::TestCheckAndReroll::test_absent_check_is_todays_chain`, `tests/test_chain_check.py::test_the_estimate_quotes_the_worst_case_retries` | | Overlap windows of a long video | `dw/tasks/windows.py`: `window_video`, `window_span` | Window `i` covers source frames `[i*stride - overlap, i*stride + stride)`, its synthetic frames repeat the first or last frame, and its audio is cut on the source's own `frames_to_samples` boundaries, so adjacent windows' strides tile the track with no sample lost or repeated. | `tests/test_window_video.py::TestAudio::test_strides_tile_the_source_track_exactly`, `tests/test_window_video.py::TestRealPath::test_a_path_argument_keeps_its_audio`, `tests/test_window_video.py::TestFileMatchesInMemory::test_every_window_equals_the_in_memory_window`, `tests/test_read_frame_range.py` | | Keep a span of a video's frames and audio | `dw/tasks/trim.py`: `trim_video`; `dw/shots.py`: `trimmed_shots` | Frames `[start_frame, start_frame + num_frames)` are kept and the audio is cut on `frames_to_samples` of each end at the track's own rate, so consecutive trims tile the track. A span past the clip's end is refused, never shortened. Shots are clipped to the span, sample side cleared. | `tests/test_trim_video.py::TestFramesAndAudio::test_consecutive_trims_tile_the_track`, `tests/test_trim_video.py::TestShots::test_shots_are_clipped_across_the_boundary` | -| Repair one chain segment in place (#1169) | `dw/tasks/chain_repair.py`: `repair_geometry` (pure), `segment_files`, `chain_repair_cut`; `dw/tasks/concat_videos.py`: `concat_videos` (its continuous_joins argument); `workflows/templates/minimax/repair-segment.json` | The repair render holds segment k-1's tail at frame 0 and segment k+1 at its end (at `num_frames - tail_frames`, always 17j); the kept span plus segment k+1 from the overrun on leaves the take's frame count exactly. Interior segments only. The splice's seams are recorded without `hard_cut`, so `analyze_seams` measures them. | `tests/test_chain_repair.py::TestGeometry::test_the_splice_keeps_the_take_length_and_the_grid`, `tests/test_chain_repair.py::TestCut::test_the_cut_and_the_splice`, `tests/test_concat_videos.py::TestContinuousJoins::test_continuous_joins_record_no_hard_cut` | +| Repair one chain segment in place (#1169) | `dw/tasks/chain_repair.py`: `repair_geometry` (pure), `chain_repair_cut`, `chain_repair_splice`; `dw/pipeline_processors/chain.py` (`segment_file` on a kept segment's shot); `workflows/templates/minimax/repair-segment.json` | The repair render holds segment k-1's tail at frame 0 and segment k+1 at its end (at `num_frames - tail_frames`, always 17j). The splice cuts the take itself, never the segment files: outside the repaired span the output is the take's frames, frame count and track. The segment files are found from the take's shots, not by name. Interior segments only. The repaired seams are not `hard_cut`, so `analyze_seams` measures them. | `tests/test_chain_repair.py::TestGeometry::test_the_splice_keeps_the_take_length_and_the_grid`, `tests/test_chain_repair.py::TestSplice::test_the_take_outside_the_span`, `tests/test_chain_repair.py::TestSplice::test_a_duration_take_keeps_its_frame_count`, `tests/test_chain.py::TestSaveSegments::test_kept_segment_files_are_named_on_the_take_shots` | | Join processed windows back to the source's length | `dw/tasks/windows.py`: `join_windows`; `dw/task_domains.py`: `window_count`, `window_count_problem`, `join_windows_errors` | Output is exactly the source's frames, each once: later windows' first `overlap` frames blend over the previous window's last on the open ramp `dissolve_videos` uses. The window count is `window_count`'s `ceil(source_frames / stride)` and nothing else (it is also `window_video`'s last index plus one). Audio is the source's own track; shots are one per window with `overlap_frames` on each seam, their sample side laid over that track by `dw/shots.py`'s `remeasured_shots` (the same rounding as `frames_to_samples`, so the spans are cumulative). | `tests/test_join_windows.py::TestRoundTrip::test_unprocessed_windows_rejoin_to_the_source`, `tests/test_join_windows.py::TestShots::test_spans_partition_the_frames_and_tile_the_audio`, `tests/test_join_windows.py::TestUint8Output::test_matches_the_float32_accumulator_exactly` | | A `join_windows` list of the wrong length, before the run | `dw/window_count_errors.py`: `window_count_errors`, wired into `validation_errors` | Owned by the task, never a template: any `join_windows` step whose `source` is probeable header-only (`asset:`/`output:` or a readable literal path), with literal `num_frames`/`overlap` and an expanded `videos` list, is checked against `window_count_problem` - the same rule the run refuses with. Anything not knowable stays silent for the run. | `tests/test_window_count_errors.py` | | Fit to a model and restore | `dw/tasks/fit.py`: `fit_to_model`, `restore_to_source` | fit_to_model's record is the whole of what restore_to_source needs: the scale is inferred from the output's size and must be uniform, and restore returns the source's size times that scale and at most its frame count. | `tests/test_fit_to_model.py` | diff --git a/docs/TASKS.md b/docs/TASKS.md index 564b74296..35acaa810 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -627,7 +627,7 @@ workflow guide). | `match_levels` | No | Even the shots' loudness out before joining - `"rms"` for perceived level (the measurement `get_gallery_metadata` reports as `mean_dbfs`), `"peak"` for the loudest sample. Off by default | | `match_levels_dbfs` | No | The level `match_levels` moves every shot to (default: -1 dBFS for `peak`, -20 dBFS for `rms`). A shot that would clip at the target is held at -0.5 dBFS peak instead, reported as a `match_levels_held` warning with a per-shot log event | | `sample_rate` | No | The joined soundtrack's rate. Default: the highest among the inputs, the rest resampled up with a warning (#108) | -| `continuous_joins` | No | `true` when the joins continue one take rather than cut between shots, as a chain segment repaired in place and spliced back (`chain_repair_cut`). Each seam is recorded without `hard_cut`, so `analyze_seams` measures its frame jump as it measures a chain's own seams, and the audio is butt-joined with no declick fade. Refused beside `seam_fade_ms` above 0 or `audio_bleed_ms` (default: `false`, every seam a cut) | +| `continuous_joins` | No | `true` when the joins continue one take rather than cut between shots, for example pieces of one chain take joined back up. Each seam is recorded without `hard_cut`, so `analyze_seams` measures its frame jump as it measures a chain's own seams, and the audio keeps the default few-ms declick (`seam_fade_ms: 0` butt-joins it). Refused beside `seam_fade_ms` above 0 or `audio_bleed_ms` (default: `false`, every seam a cut) | Shots an earlier run wrote are joined by path or URL without regenerating them. A file's track, and a `previous_result:` shot's, is fitted to its frames' @@ -919,15 +919,17 @@ shortened or padded), as are a `num_frames` of `0` or less and a negative The pieces that re-render one segment of a finished chain in place (#1169), for [templates/minimax/repair-segment](../workflows/templates/minimax/repair-segment.json). A chain run with `save_segments` and `keep_segments` leaves each segment as a -playable mp4 beside the take; this command finds them from any one of them, -counts their frames, and cuts what a repair render of segment `k` needs: +playable mp4 beside the take, and names each file on the take's shots +(`segment_file`, in the run manifest). This command finds the files from the +take, and cuts the guides a repair render of segment `k` needs and where the +repair sits on the take: ```json { "task": { "command": "chain_repair_cut", "arguments": { - "chain_output": "output:minimax/MiniMaxH3I2VChained/latest/final/MiniMaxH3I2VChained-chained_keyframe_to_video_audio.0.segment-000.mp4", + "chain_take": "output:minimax/MiniMaxH3I2VChained/latest/final/.mp4", "segment": 3, "num_frames": 175, "head_frames": 22, @@ -939,7 +941,8 @@ counts their frames, and cuts what a repair render of segment `k` needs: | Argument | Required | Description | | -------- | -------- | ----------- | -| `chain_output` | Yes | Any one of the chain's segment files (`..segment-NNN.mp4`); the others are found beside it, and a gap in their numbering is refused | +| `chain_take` | One of the two | The chain's final video. Its shots (the run manifest beside it) give each segment's place on the take and name its segment file | +| `chain_output` | One of the two | One of the chain's segment files. Alone, the take is found through the run manifest that names it. Beside a take whose shots name no files (copied away from its run), the others are found by name beside it, and refused when they disagree in rate or size (two runs mixed in one directory) or leave a gap | | `segment` | Yes | The segment to re-render, 1-based. Only 2 to N-1: the first has no segment before it and the last none after it, and are refused | | `num_frames` | Yes | The repair render's length, on the 17n + 5 grid: at least `head_frames` + segment k's frames + `tail_frames`, rounded up onto the grid. A smaller one is refused naming the smallest; a larger one reaches further into segment k+1 | | `head_frames` | No | Segment k-1's last frames, held at the render's frame 0: a whole-latent length, 17m + 5 from 22 (default `22`) | @@ -947,21 +950,45 @@ counts their frames, and cuts what a repair render of segment `k` needs: | `width`, `height` | No | The render's canvas; when given, refused unless it is the segment files' own size | It returns a record for later steps (`previous_result:cut.`): -`head_guide` and `tail_guide` (clips with their audio, for two `guides` -entries with `"audio": true`), `keyframe` (the head guide's first frame, the -fl2va keyframe a guide chain gives its own segments), `tail_frame` (where the -tail guide is placed: `num_frames - tail_frames`, always on 17j), `keep_start` -and `keep_frames` (the render span to keep, between the guides), `before` -(segment files 1 to k-1) and `after` (segment k+1 from `overrun` on, then the -files after it), with `overrun`, `segment_frames` and `total_frames`. +`head_guide` and `tail_guide` (clips with their audio, cut from the segment +files, for two `guides` entries with `"audio": true`), `keyframe` (the head +guide's first frame, the fl2va keyframe a guide chain gives its own +segments), `tail_frame` (where the tail guide is placed: `num_frames - +tail_frames`, always on 17j), `keep_start` (the render frame the kept span +starts at), `chain_take`, and `start_frame`/`end_frame` (the take's frames +the span replaces), with `overrun`, `segment_frames`, `segment_files` and +`total_frames` (the take's). The neighbours are counted from their decode. The frames between the two guides number 12 more than a multiple of 17 (17n + 5 less two 17m + 5 guides), and an interior segment of a 17n + 5 chain with a 22- or 39-frame carry is a multiple of 17, so the new frames replace segment k and the first `overrun` frames of segment k+1 (12 at the smallest -render). The tail guide is segment k+1 from `overrun` on, and `after` takes -segment k+1 back from the same frame, so `before`, the kept span and `after` -come to the take's own frame count exactly. +render), and the take resumes on segment k+1's frame `overrun`. + +### chain_repair_splice + +Puts a repair render's span into the take in place (#1169): the second half +of `chain_repair_cut`. + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `chain_take` | Yes | The take, `previous_result:cut.chain_take` | +| `video` | Yes | The repair render | +| `segment` | Yes | The segment repaired, 1-based | +| `start_frame`, `end_frame` | Yes | The take's frames the span replaces, from the cut | +| `keep_start` | Yes | The render frame the span starts at, from the cut | + +Apart from frames `[start_frame, end_frame)` the result is the take: its +frames, its frame count (a `duration_seconds` take, shorter than its segment +files, stays its own length) and its track, sample for sample, so every other +seam keeps its crossfade or declick. A `match_audio` take keeps its own track +whole under the repaired picture. Otherwise the span's audio is the render's +own, at the take's rate, blended in from the take's track and back into it +inside the span, over the window the chain blended those seams with (the +shot's `crossfade_ms`, else the few-ms declick). Its shots are the take's, +with the span as one `segment k (repaired)` shot and the next shot starting +where the take resumes, none of them `hard_cut`, so `analyze_seams` measures +both repaired seams. ### window_video diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 12570e695..439dbca13 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -2633,16 +2633,17 @@ See [workflows/templates/ltx2/chained-segments.json](../workflows/templates/ltx2 One bad segment of a finished H3 chain can be re-rendered in place instead of re-rolling the chain, when the chain kept its segment files (`save_segments` and -`keep_segments`): +`keep_segments`, which the H3 chain templates set from their `keep_segments` variable): [workflows/templates/minimax/repair-segment.json](../workflows/templates/minimax/repair-segment.json) renders segment k once with segment k-1's last frames as a guide at frame 0 and segment -k+1's frames as a guide ending at the render's end, both holding their audio, then trims -the guides off and splices the result between its neighbours. The take keeps its frame -count; the new frames also replace the first 12 frames of segment k+1 (the 17-frame -grid's remainder, [`chain_repair_cut`](TASKS.md#chain_repair_cut)). The splice is a -`concat_videos` with `continuous_joins`, so `analyze_seams` checks both repaired seams as -it checks a chain's own. Interior segments only: the first and the last need a -one-sided repair. The chain's settings (canvas, steps, LoRA, counters) are passed to it +k+1's frames as a guide ending at the render's end, both holding their audio, then +splices the new frames into the take itself (`chain_take`). Outside the repaired span the +output is the take: its frame count, its picture and its soundtrack, every other seam's +blend included. The new frames also replace the first 12 frames of segment k+1 (the +17-frame grid's remainder, [`chain_repair_cut`](TASKS.md#chain_repair_cut)). The two +repaired seams are not cuts, so `analyze_seams` checks them as it checks a chain's own. +Interior segments only: the first and the last need a one-sided repair. The chain's +settings (canvas, steps, LoRA, counters) are passed to it as arguments, and the seed is new by default. #### Checking each segment: `check` and `retries` diff --git a/dw/pipeline_processors/chain.py b/dw/pipeline_processors/chain.py index e7573f066..5a125685e 100644 --- a/dw/pipeline_processors/chain.py +++ b/dw/pipeline_processors/chain.py @@ -35,6 +35,7 @@ import gc import logging import math +import os import sys from dataclasses import dataclass @@ -404,6 +405,10 @@ def run_chain(pipeline, chain_definition, arguments): _on_timeline_audio(segment_audio, segment, config, segment_rate), segment_rate, ) + if config.keep_segments: + # The file as written ('-2' deduped on a rerun), which is + # how repair-segment finds a take's own files (#1169) + shots[-1]["segment_file"] = os.path.basename(spill.paths[-1]) else: frames.extend(kept_frames) diff --git a/dw/runs.py b/dw/runs.py index 86cb7e0b6..2e2718264 100644 --- a/dw/runs.py +++ b/dw/runs.py @@ -926,6 +926,24 @@ def record_kept_shots(directory, file_name, shots, provenance=None): MANIFEST_SEARCH_DEPTH = 8 +def manifest_beside(path): + """(run directory, manifest) of the run a file named by its absolute path + sits in - the nearest parent within MANIFEST_SEARCH_DEPTH holding a + readable manifest - or None.""" + run_dir = os.path.dirname(os.path.abspath(path)) + for _ in range(MANIFEST_SEARCH_DEPTH): + if os.path.isfile(os.path.join(run_dir, MANIFEST_FILE_NAME)): + break + parent = os.path.dirname(run_dir) + if parent == run_dir: + return None + run_dir = parent + else: + return None + manifest = _read_manifest(run_dir) + return None if manifest is None else (run_dir, manifest) + + def shots_beside(path): """The shots a run's manifest records for a file named by its absolute path. @@ -939,19 +957,10 @@ def shots_beside(path): from .shots import shots_for_file path = os.path.abspath(path) - run_dir = os.path.dirname(path) - for _ in range(MANIFEST_SEARCH_DEPTH): - if os.path.isfile(os.path.join(run_dir, MANIFEST_FILE_NAME)): - break - parent = os.path.dirname(run_dir) - if parent == run_dir: - return None - run_dir = parent - else: - return None - manifest = _read_manifest(run_dir) - if manifest is None: + found = manifest_beside(path) + if found is None: return None + run_dir, manifest = found own = os.path.relpath(path, run_dir).replace(os.sep, "/") for entry in manifest.get("steps") or []: if not isinstance(entry, dict) or entry.get("reused"): diff --git a/dw/shots.py b/dw/shots.py index a0a449863..72fd64546 100644 --- a/dw/shots.py +++ b/dw/shots.py @@ -42,6 +42,9 @@ `crossfade_ms` as it does `seam_fade_ms` (#660). `trimmed_shots` drops every seam attribute from a shot its cut falls inside: that seam is not in the kept span (#674) +- `segment_file` - a chain run with `save_segments` and `keep_segments` names + each segment's file on its shot, as written (a rerun's '-2' dedupe + included), beside the take - how `chain_repair_cut` finds them (#1169) Every other `AudioVideo` constructor either carries the list (same frames), rescales it (`interpolate_frames`), re-measures the sample side for a new @@ -51,7 +54,9 @@ of a longer source, #601), or clips the list to a kept span (`trim_video`, via `trimmed_shots(shots, head_trim, keep_frames)`, sample side cleared; a clip with no shots carries one spanning the kept frames, #627). -`tests/test_shots.py` fails on a constructor site nobody decided for. +`chain_repair_splice` builds its list from the take's: the repaired span +becomes one `(repaired)` shot and the shot after it starts where the take +resumes (#1169). `tests/test_shots.py` fails on a constructor site nobody decided for. `Result.save` keeps each file's shots as plain data in `saved_shots` (path -> shots), so a step cache hit's stripped copy still reports them. The manifest diff --git a/dw/tasks/chain_repair.py b/dw/tasks/chain_repair.py index cd084622f..c477e2a52 100644 --- a/dw/tasks/chain_repair.py +++ b/dw/tasks/chain_repair.py @@ -1,13 +1,24 @@ -"""chain_repair_cut: the pieces that re-render one segment of a finished -chain in place (#1169, `templates/minimax/repair-segment`). +"""chain_repair_cut and chain_repair_splice: re-render one segment of a +finished chain in place (#1169, `templates/minimax/repair-segment`). A guide chain run with `save_segments` + `keep_segments` leaves each segment -as a playable mp4, `..segment-NNN.mp4`, holding the frames -it put on the timeline (chain_spill.py). Segment k is repaired by one H3 render -held at both ends: segment k-1's last `head_frames` frames as a guide at frame -0, segment k+1's frames as a guide ending at the render's last frame, both with -their audio. The render's new frames between the guides replace segment k, and -the splice puts it back between its neighbours. +as a playable mp4 beside the take, `..segment-NNN.mp4`, +holding the frames it put on the timeline (chain_spill.py), and names each +file on the take's shots (`segment_file`, in the run manifest). Segment k is +repaired by one H3 render held at both ends: segment k-1's last `head_frames` +frames as a guide at frame 0, segment k+1's frames as a guide ending at the +render's last frame, both with their audio. The render's new frames between +the guides replace segment k, and the splice puts them into the take. + +The splice is cut from the take itself, never rebuilt from the segment files: +apart from the repaired span the output IS the take - its frame count (a +duration_seconds take is shorter than its files, cut on replay), its +soundtrack (a match_audio take's own track is kept whole; a generated one +keeps every seam's crossfade or declick bit for bit) and its picture. The +segment files are only the guides' material. The repaired span's audio is the +render's own, blended in and out of the take's track inside the span over +the window the chain blended that seam with (the shot's `crossfade_ms`, or +the declick a guide chain's held audio gets). The geometry is the 17-frame grid's. A guide starts at frame 17j and is 17m + 5 frames long, and a render is 17n + 5, so the tail guide ending at the render's @@ -16,14 +27,10 @@ 22- or 39-frame carry is a multiple of 17 long, so the new frames cannot be exactly segment k: they run `overrun` frames on into segment k+1 (12 at the smallest render), the tail guide is segment k+1's frames from there, and the -splice takes segment k+1 back from that frame on. The take keeps its frame -count exactly; segments other than k and the head of k+1 are the files' own -frames, re-encoded once by the join. +take resumes at that frame. -A workflow has no arithmetic, so this command does it (`repair_geometry`, -pure) and cuts the media; the template composes it with the H3 step, -`trim_video` and `concat_videos(continuous_joins=true)`, which keeps the two -repaired seams measured by analyze_seams rather than skipped as cuts. +A workflow has no arithmetic, so the cut does it (`repair_geometry`, pure); +the template composes cut, the H3 step and the splice. Concepts only from H3MD Director (AGPL-3.0) and roycho87/geiru_context_suite (GPL-3.0): a tail hold that outnumbers the head hold, no tail hold on the last @@ -34,18 +41,32 @@ import os import re +import numpy + +from ..dsp import as_channels_samples, equal_power_ramps, resample_waveform from ..locations import validate_media_path from ..media import probe_metadata +from ..media_types import AudioVideo from ..pipeline_processors.h3_rules import GUIDE_FRAMES_PER_CHUNK +from ..runs import manifest_beside, shots_beside from ..security import validate_path -from ..task_domains import POSITIVE, check_arguments, whole_number +from ..shots import shot_record, shots_for_file +from ..task_domains import ( + NON_NEGATIVE, + POSITIVE, + check_arguments, + frames_to_samples, + whole_number, +) +from .joins import DECLICK_MS from .registry import register_command from .trim import trim_video -from .video_utils import load_audio_video +from .video_utils import VideoFileReference, frames_as_pil_list, load_audio_video logger = logging.getLogger("dw") COMMAND = "chain_repair_cut" +SPLICE = "chain_repair_splice" # The name SegmentSpill gives a segment file (chain_spill.py) _SEGMENT_FILE = re.compile(r"^(?P.+)\.segment-(?P\d{3})\.mp4$") @@ -140,7 +161,9 @@ def repair_geometry(segment_frames, segment, num_frames, head_frames, tail_frame def segment_files(chain_output): - """Every segment file of the chain `chain_output` names one of, in order.""" + """Every segment file named like the one `chain_output` names, in order - + the fallback for a take whose shots name no files (no manifest beside + it), checked by the caller against each other.""" path = validate_media_path(chain_output, what="chain_output") match = _SEGMENT_FILE.match(os.path.basename(path)) if match is None: @@ -167,34 +190,164 @@ def segment_files(chain_output): return [found[index] for index in indices] -def _frame_count(path): +def _take_naming(segment_path): + """The take whose run manifest names `segment_path` on one of its shots, + or None.""" + found = manifest_beside(segment_path) + if found is None: + return None + run_dir, manifest = found + name, directory = os.path.basename(segment_path), os.path.dirname(segment_path) + for entry in manifest.get("steps") or []: + if not isinstance(entry, dict) or entry.get("reused"): + continue + files = entry.get("files") or [] + for file in files: + take = validate_path(os.path.join(run_dir, file), run_dir) + shots = shots_for_file(entry.get("shots"), file, files) or [] + if os.path.dirname(take) == directory and any( + shot.get("segment_file") == name for shot in shots + ): + return take + return None + + +def _named_files(take, shots): + """The segment files the take's shots name, beside it - or None when a + shot names none.""" + if not shots or not all(shot.get("segment_file") for shot in shots): + return None + directory = os.path.dirname(take) + paths = [ + validate_path(os.path.join(directory, shot["segment_file"]), directory) + for shot in shots + ] + for path in paths: + if not os.path.isfile(path): + raise ValueError( + f"{COMMAND}: the take's segment file {os.path.basename(path)} is " + "gone - a repair needs every one of them kept" + ) + return paths + + +def _locate(chain_take, chain_output): + """(take path, take shots or None, segment file paths).""" + if chain_take is None and chain_output is None: + raise ValueError( + f"{COMMAND}: pass 'chain_take' (an 'output:' of the chain's final " + "video) or 'chain_output' (one of its segment files)" + ) + output = ( + None + if chain_output is None + else validate_media_path(chain_output, what="chain_output") + ) + if chain_take is None: + chain_take = _take_naming(output) + if chain_take is None: + raise ValueError( + f"{COMMAND}: no run manifest beside {os.path.basename(output)} " + "names it on a take's shots - pass 'chain_take', the chain's " + "final video, as well" + ) + take = validate_media_path(chain_take, what="chain_take") + shots = shots_beside(take) + paths = _named_files(take, shots) + if paths is not None: + if output is not None and output not in paths: + raise ValueError( + f"{COMMAND}: 'chain_output' {os.path.basename(output)} is not one " + f"of the segment files {os.path.basename(take)}'s shots name" + ) + return take, shots, paths + if output is None: + raise ValueError( + f"{COMMAND}: {os.path.basename(take)}'s shots name no segment files " + "(a chain run before they were recorded, or a take copied away from " + "its run) - pass 'chain_output', one of its segment files, as well" + ) + paths = segment_files(output) + if shots is not None and len(shots) != len(paths): + shots = None + _check_siblings_agree(paths) + return take, shots, paths + + +def _check_siblings_agree(paths): + """Refuse segment files found by name that disagree in rate or size - two + runs' files mixed in one directory by the flat layout.""" + seen = {} + for path in paths: + info = probe_metadata(path) or {} + own = (info.get("fps"), info.get("width"), info.get("height")) + for key, value in seen.items(): + if value != own: + raise ValueError( + f"{COMMAND}: segment files {os.path.basename(key)} and " + f"{os.path.basename(path)} disagree (fps, width, height: " + f"{value} against {own}) - files of two runs are mixed; " + "pass the take from a run directory" + ) + seen = {path: own} + + +def _header_frames(path): info = probe_metadata(path) or {} count = info.get("frame_count") if not isinstance(count, int) or count <= 0: raise ValueError(f"{COMMAND}: cannot count the frames of {path}") - return count, info + return count + + +def _take_spans(take, shots, paths, segment, neighbours): + """[(start, frames)] of every segment on the take, and its frame count: + the take's shots, else the segment files' counts (the neighbours' from + their decode) cut to the take's length.""" + if shots is not None: + spans = [(int(s["start_frame"]), int(s["num_frames"])) for s in shots] + return spans, sum(frames for _, frames in spans) + counts = [ + neighbours.get(index) or _header_frames(path) + for index, path in enumerate(paths) + ] + total = _header_frames(take) + if not sum(counts[:-1]) < total <= sum(counts): + raise ValueError( + f"{COMMAND}: {os.path.basename(take)} has {total} frames, and its " + f"segment files hold {sum(counts)} - they are not this take's" + ) + spans, start = [], 0 + for count in counts: + spans.append((start, min(count, total - start))) + start += count + return spans, total def chain_repair_cut( - chain_output, - segment, - num_frames, + chain_take=None, + chain_output=None, + segment=None, + num_frames=None, head_frames=22, tail_frames=39, width=None, height=None, ): - """Task command: cut a finished chain's segment files for a repair of one. + """Task command: cut a finished chain's guides for a repair of one segment, + and where the repair sits on the take. Args: - chain_output: Any one of the chain's segment files (an `output:` - reference) - the others are found beside it + chain_take: The chain's final video (an `output:` reference). Found + from `chain_output` through the run manifest when not given + chain_output: One of the chain's segment files - needed only when the + take's shots name none (no manifest beside it); then the others + are found beside it by name segment: The segment to re-render, 1-based; 2 to N-1 num_frames: The repair render's length - at least segment k's frames plus both guides, on the 17n + 5 grid; a smaller one is refused naming the smallest that fits head_frames: Segment k-1's last frames held at the render's frame 0 - (17m + 5 from 22) tail_frames: Segment k+1's frames held at the render's end - more than head_frames by default (39), the tail hold outnumbering the head @@ -204,10 +357,9 @@ def chain_repair_cut( Returns: A dict for the template's later steps: `head_guide` and `tail_guide` (AudioVideo clips with their audio), `keyframe` (the head guide's - first frame, the fl2va keyframe a guide chain gives its segments), - `tail_frame`, `keep_start`, `keep_frames`, `before` (segment files - 1..k-1) and `after` (segment k+1 from the overrun on, then the files - after it), with the geometry's numbers beside them. + first frame), `tail_frame`, `keep_start`, `chain_take` (the take's + path) and `start_frame`/`end_frame` (the take's frames the render's + span replaces), with the geometry's numbers beside them. """ segment = whole_number(segment, "segment", COMMAND, required=True) num_frames = whole_number(num_frames, "num_frames", COMMAND, required=True) @@ -225,13 +377,10 @@ def chain_repair_cut( height=height, ) - paths = segment_files(chain_output) - counts, infos = zip(*(_frame_count(path) for path in paths)) - geometry = repair_geometry( - list(counts), segment, num_frames, head_frames, tail_frames - ) - _check_canvas(infos[segment - 1], width, height, paths[segment - 1]) - + take, shots, paths = _locate(chain_take, chain_output) + if not 2 <= segment <= len(paths) - 1: + # The geometry's own refusal, before anything is decoded + repair_geometry([0] * len(paths), segment, num_frames, 22, 22) previous = load_audio_video(paths[segment - 2]) following = load_audio_video(paths[segment]) for clip, path in ((previous, paths[segment - 2]), (following, paths[segment])): @@ -240,35 +389,58 @@ def chain_repair_cut( f"{COMMAND}: {os.path.basename(path)} has no soundtrack, and the " "repair holds the neighbours' audio over both guides" ) + _check_canvas(previous.frames[0].size, width, height, paths[segment - 2]) + spans, total = _take_spans( + take, + shots, + paths, + segment, + {segment - 2: len(previous.frames), segment: len(following.frames)}, + ) + counts = [frames for _, frames in spans] + for index, clip in ((segment - 2, previous), (segment, following)): + last = index == len(spans) - 1 + if counts[index] > len(clip.frames) or ( + not last and counts[index] != len(clip.frames) + ): + raise ValueError( + f"{COMMAND}: the take's segment {index + 1} is {counts[index]} " + f"frames, but {os.path.basename(paths[index])} holds " + f"{len(clip.frames)} - they are not the same run's" + ) + counts[segment - 2] = len(previous.frames) + counts[segment] = len(following.frames) + geometry = repair_geometry(counts, segment, num_frames, head_frames, tail_frames) + start = spans[segment - 1][0] + end = min(start + geometry["keep_frames"], total) + head_guide = trim_video(previous, geometry["head_start"], head_frames) tail_guide = trim_video(following, geometry["tail_start"], tail_frames) - rest = trim_video( - following, geometry["tail_start"], counts[segment] - geometry["tail_start"] - ) logger.info( f"{COMMAND}: segment {segment} of {len(paths)} - render {num_frames} " f"frames, head guide {head_frames} at 0, tail guide {tail_frames} at " - f"{geometry['tail_frame']}, {geometry['overrun']} frames into segment " - f"{segment + 1}" + f"{geometry['tail_frame']}; take frames {start}..{end - 1} of {total} " + "replaced" ) return { **geometry, + "keep_frames": end - start, + "total_frames": total, "segments": len(paths), - "segment_frames": list(counts), + "segment_frames": counts, + "segment_files": list(paths), + "chain_take": take, + "start_frame": start, + "end_frame": end, "head_guide": head_guide, "tail_guide": tail_guide, "keyframe": head_guide.frames[0], - "before": list(paths[: segment - 1]), - "after": [rest, *paths[segment + 1 :]], } -def _check_canvas(info, width, height, path): - for name, wanted in (("width", width), ("height", height)): - if wanted is None: - continue - own = info.get(name) - if own is not None and own != wanted: +def _check_canvas(size, width, height, path): + for name, wanted, own in (("width", width, size[0]), ("height", height, size[1])): + if wanted is not None and own != wanted: raise ValueError( f"{COMMAND}: '{name}' is {wanted}, but the chain was rendered at " f"{own} ({os.path.basename(path)}) - a repair renders on the " @@ -276,30 +448,196 @@ def _check_canvas(info, width, height, path): ) +def chain_repair_splice(chain_take, video, segment, start_frame, end_frame, keep_start): + """Task command: put a repair render's span into the take in place. + + Args: + chain_take: The take (`chain_repair_cut`'s `chain_take`) + video: The repair render + segment: The segment repaired, 1-based - which of the take's shots + the span opens + start_frame, end_frame: The take's frames the span replaces + keep_start: The render frame the span starts at (after the head guide) + + Returns: + An AudioVideo that is the take apart from frames + `[start_frame, end_frame)`: the take's own frames, frame count and + track outside them. Inside, the render's frames and - unless the + take's track is a match_audio one, kept whole - its audio, blended + from the take's track and back into it within the span over the + window the chain blended those seams with. Its shots are the take's, + with the span as one `(repaired)` shot. + """ + segment = whole_number(segment, "segment", SPLICE, required=True) + start = whole_number(start_frame, "start_frame", SPLICE, required=True) + end = whole_number(end_frame, "end_frame", SPLICE, required=True) + keep_start = whole_number(keep_start, "keep_start", SPLICE, required=True) + take = load_audio_video(validate_media_path(chain_take, what="chain_take")) + if isinstance(video, VideoFileReference): + video = load_audio_video(video.path) + elif isinstance(video, str): + video = load_audio_video(video) + frames = frames_as_pil_list(take) + rendered = frames_as_pil_list(video) + if not 0 < start < end <= len(frames): + raise ValueError( + f"{SPLICE}: frames {start}..{end} are not inside the take's {len(frames)}" + ) + if keep_start + end - start > len(rendered): + raise ValueError( + f"{SPLICE}: the render has {len(rendered)} frames, short of the " + f"{end - start} from frame {keep_start} the span needs" + ) + if rendered[0].size != frames[0].size: + raise ValueError( + f"{SPLICE}: the render's canvas {rendered[0].size} is not the " + f"take's {frames[0].size}" + ) + spliced = frames[:start] + rendered[keep_start : keep_start + end - start] + spliced += frames[end:] + fps = take.fps or getattr(video, "fps", None) + shots = take.shots + if not shots or len(shots) <= segment or shots[segment - 1]["start_frame"] != start: + shots = None + audio, blends, samples = _spliced_track( + take, video, fps, shots, segment, start, end, keep_start + ) + logger.info( + f"{SPLICE}: segment {segment} - take frames {start}..{end - 1} of " + f"{len(frames)} from the render's {keep_start}.." + f"{keep_start + end - start - 1}" + ) + return AudioVideo( + spliced, + audio, + take.sample_rate if audio is not None else None, + fps=fps, + shots=_spliced_shots(shots, segment, start, end, len(frames), blends, samples), + ) + + +def _blend_ms(shots, index): + """The window the chain blended the seam opening shot `index` with: its + realized crossfade, else the declick a held or butt-joined seam gets.""" + if shots is None: + return DECLICK_MS + return shots[index].get("crossfade_ms") or DECLICK_MS + + +def _spliced_track(take, video, fps, shots, segment, start, end, keep_start): + """(track, (in ms, out ms) or None, (first sample, end sample) or None).""" + if take.audio is None: + return None, None, None + if shots is not None and all(shot.get("start_sample") is None for shot in shots): + # match_audio: the caller's own track, laid under the take whole + return take.audio, None, None + if getattr(video, "audio", None) is None: + raise ValueError( + f"{SPLICE}: the render has no soundtrack to put under the span" + ) + rate = take.sample_rate + track = as_channels_samples(take.audio) + if shots is not None: + after = shots[segment] + first = int(shots[segment - 1]["start_sample"]) + last = int(after["start_sample"]) + frames_to_samples( + end - after["start_frame"], fps, rate + ) + else: + first = frames_to_samples(start, fps, rate) + last = frames_to_samples(end, fps, rate) + last = min(last, track.shape[1]) + own = as_channels_samples(video.audio) + if video.sample_rate != rate: + own = resample_waveform(own, video.sample_rate, rate) + offset = frames_to_samples(keep_start, getattr(video, "fps", None) or fps, rate) + span = numpy.zeros((track.shape[0], last - first), dtype=track.dtype) + piece = own[:, offset : offset + span.shape[1]] + span[:, : piece.shape[1]] = ( + piece if piece.shape[0] == span.shape[0] else piece.mean(axis=0, keepdims=True) + ) + blends = [] + for side, index in (("in", segment - 1), ("out", segment)): + window = min(int(_blend_ms(shots, index) / 1000.0 * rate), span.shape[1] // 2) + blends.append(round(window / rate * 1000, 1)) + if not window: + continue + fade_out, fade_in = equal_power_ramps(window) + if side == "in": + span[:, :window] = ( + track[:, first : first + window] * fade_out + span[:, :window] * fade_in + ) + else: + span[:, -window:] = ( + span[:, -window:] * fade_out + track[:, last - window : last] * fade_in + ) + audio = numpy.concatenate([track[:, :first], span, track[:, last:]], axis=1) + return audio, tuple(blends), (first, last) + + +def _spliced_shots(shots, segment, start, end, total, blends, samples): + """The take's shots with the span as one repaired shot, the shot after it + starting where the take resumes.""" + first, last = samples if samples else (None, None) + if shots is None: + return [ + shot_record("take", 0, start), + shot_record(f"segment {segment} (repaired)", start, end - start), + *([shot_record("take", end, total - end)] if end < total else []), + ] + seam = ("crossfade_ms", "trim_frames", "segment_file") + clean = [{k: v for k, v in shot.items() if k not in seam} for shot in shots] + repaired = shot_record( + f"{shots[segment - 1]['name']} (repaired)", + start, + end - start, + first, + None if first is None else last - first, + ) + result = [*clean[: segment - 1], repaired] + after = clean[segment] + after_end = after["start_frame"] + after["num_frames"] + if after_end > end: + resumed = {**after, "start_frame": end, "num_frames": after_end - end} + if first is not None: + resumed["start_sample"] = last + resumed["num_samples"] = after["start_sample"] + after["num_samples"] - last + result.append(resumed) + result.extend(clean[segment + 1 :]) + if blends: + repaired["crossfade_ms"] = blends[0] + if after_end > end: + result[segment]["crossfade_ms"] = blends[1] + return result + + +_WHOLE = ("segment", "num_frames", "head_frames", "tail_frames", "width", "height") + + @register_command( COMMAND, implementation="dw.tasks.chain_repair.chain_repair_cut", - domains={ - name: POSITIVE - for name in ( - "segment", - "num_frames", - "head_frames", - "tail_frames", - "width", - "height", - ) - }, - whole_numbers=( - "segment", - "num_frames", - "head_frames", - "tail_frames", - "width", - "height", - ), - media_arguments=("chain_output",), + domains={name: POSITIVE for name in _WHOLE}, + whole_numbers=_WHOLE, + media_arguments=("chain_take", "chain_output"), ) def _handle_chain_repair_cut(task, arguments, previous_pipelines): - """Cut a finished chain's segment files for an in-place repair of one""" + """Cut a finished chain's guides for an in-place repair of one segment""" return chain_repair_cut(**arguments) + + +@register_command( + SPLICE, + implementation="dw.tasks.chain_repair.chain_repair_splice", + domains={ + "segment": POSITIVE, + "start_frame": POSITIVE, + "end_frame": POSITIVE, + "keep_start": NON_NEGATIVE, + }, + whole_numbers=("segment", "start_frame", "end_frame", "keep_start"), + media_arguments=("chain_take",), +) +def _handle_chain_repair_splice(task, arguments, previous_pipelines): + """Put a repair render's span back into the chain's take in place""" + return chain_repair_splice(**arguments) diff --git a/dw/tasks/concat_videos.py b/dw/tasks/concat_videos.py index 4691ef7a7..ff14d8d00 100644 --- a/dw/tasks/concat_videos.py +++ b/dw/tasks/concat_videos.py @@ -120,9 +120,11 @@ def concat_videos( between shots - a chain segment repaired in place and spliced back between its neighbours (#1169). Each seam is recorded without `hard_cut`, so analyze_seams measures its frame jump as - it does a chain's own seams, and its audio is butt-joined with no - declick fade. A cut device beside it (`seam_fade_ms` above 0, - `audio_bleed_ms`) is refused. Default false: every seam is a cut + it does a chain's own seams, and its audio gets the default + few-ms declick (no `seam_fade_ms` recorded, so seam_hole still + measures it); `seam_fade_ms: 0` butt-joins it. A cut device + beside it (`seam_fade_ms` above 0, `audio_bleed_ms`) is refused. + Default false: every seam is a cut Returns: One AudioVideo; its audio is None when no input video carries any @@ -234,8 +236,9 @@ def _joined_tracks( def _continuous_seam_fade(continuous_joins, seam_fade_ms, audio_bleed_ms): - """The seam fade the joins take: 0 - a plain butt join - for continuous - joins, which refuse a cut's devices; the caller's own otherwise.""" + """The seam fade the joins take: the caller's own - None, the default + declick, unless asked - with a cut's devices refused for continuous + joins.""" if not isinstance(continuous_joins, bool): raise ValueError( f"concat_videos needs 'continuous_joins' as true or false, got " @@ -255,7 +258,7 @@ def _continuous_seam_fade(continuous_joins, seam_fade_ms, audio_bleed_ms): f"'audio_bleed_ms' ({audio_bleed_ms}) cannot be set beside it - a " "bleed covers a cut" ) - return 0 + return seam_fade_ms def _record_seam_fade(shot, applied): diff --git a/dw/workflow_schema.json b/dw/workflow_schema.json index c7fa11b57..16b0782bf 100644 --- a/dw/workflow_schema.json +++ b/dw/workflow_schema.json @@ -513,13 +513,19 @@ "minItems": 1 }, "save_segments": { - "description": "Write each completed segment to the output directory as a playable mp4 and free its frames, bounding memory to one segment - a crashed chain leaves the finished segments behind. The final video is streamed from the segment files. Requires PyAV and a frame rate.", - "type": "boolean", + "description": "Write each completed segment to the output directory as a playable mp4 and free its frames, bounding memory to one segment - a crashed chain leaves the finished segments behind. The final video is streamed from the segment files. Requires PyAV and a frame rate. Accepts a 'variable:' reference.", + "type": [ + "boolean", + "string" + ], "default": false }, "keep_segments": { - "description": "Leave the segment files in place after the final video is written (default: they are removed once it saves successfully). Only meaningful with save_segments.", - "type": "boolean", + "description": "Leave the segment files in place after the final video is written (default: they are removed once it saves successfully). Only meaningful with save_segments. Each kept file is named on the take's shots ('segment_file'), which is how repair-segment finds them. Accepts a 'variable:' reference.", + "type": [ + "boolean", + "string" + ], "default": false } }, diff --git a/tests/test_catalog_structure.py b/tests/test_catalog_structure.py index 674f35778..e15796581 100644 --- a/tests/test_catalog_structure.py +++ b/tests/test_catalog_structure.py @@ -337,6 +337,8 @@ def test_per_entry_problems_reports_a_mismatched_entries_count(): "prompt", "segment", }, + # a field of the chain block the repaired take was run with + "workflows/templates/minimax/repair-segment.json": {"keep_segments"}, # the pipeline's 'cache' block it writes inline, not the H3 templates' # variable of that name "workflows/templates/step-caching.json": {"cache"}, @@ -569,8 +571,10 @@ def test_no_stale_entry_in_the_allowlist(): # the take-list template keeps its own entry. Then to 13_450 (measured # 13_432) for `minimax/repair-segment` (#1169, task 14): the catalog's only # in-place repair of one chain segment, 25 variables because it carries the -# chain's own settings by argument. -COMPACT_BUDGET = 13_450 +# chain's own settings by argument. Then to 13_500 (measured 13_476) for the +# #1169 review fixes: repair-segment's 'chain_take', and the 'keep_segments' +# variable four H3 chain templates take so a chain can be repaired later. +COMPACT_BUDGET = 13_500 FILTERED_BUDGET = 1_500 diff --git a/tests/test_chain.py b/tests/test_chain.py index 053a8349c..1ae4724a9 100644 --- a/tests/test_chain.py +++ b/tests/test_chain.py @@ -638,6 +638,26 @@ def test_keep_segments_survives_cleanup(self, tmp_path): # cleaned - a cached Result stays retainable assert result.frames.cleaned is False + def test_kept_segment_files_are_named_on_the_take_shots(self, tmp_path): + # repair-segment (#1169) finds a take's segment files from its own + # shots rather than by globbing names a rerun's '-2' dedupe can mix + pipeline = self.make_pipeline(tmp_path) + (tmp_path / f"{pipeline.file_prefix}.0.segment-000.mp4").write_bytes(b"") + + result = run_chain(pipeline, self.chain(keep_segments=True), {}) + + names = [shot["segment_file"] for shot in result.shots] + assert names[0] == f"{pipeline.file_prefix}.0.segment-000-2.mp4" + assert names[1:] == [ + f"{pipeline.file_prefix}.0.segment-{index:03d}.mp4" for index in (1, 2) + ] + assert all((tmp_path / name).is_file() for name in names) + + def test_segment_files_cleaned_up_are_not_named(self, tmp_path): + pipeline = self.make_pipeline(tmp_path) + result = run_chain(pipeline, self.chain(), {}) + assert not any("segment_file" in shot for shot in result.shots) + def test_generated_audio_is_muxed_into_the_segment_files(self, tmp_path): import av diff --git a/tests/test_chain_repair.py b/tests/test_chain_repair.py index b70121dea..91a5e4eb0 100644 --- a/tests/test_chain_repair.py +++ b/tests/test_chain_repair.py @@ -8,10 +8,14 @@ from PIL import Image from dw.media_types import AudioVideo -from dw.tasks.chain_repair import chain_repair_cut, repair_geometry, segment_files -from dw.tasks.concat_videos import concat_videos +from dw.tasks.chain_repair import ( + chain_repair_cut, + chain_repair_splice, + repair_geometry, + segment_files, +) from dw.tasks.task import _COMMAND_REGISTRY -from dw.tasks.trim import trim_video +from dw.tasks.video_utils import load_audio_video from dw.task_domains import frames_to_samples # A 5-segment guide chain at the chained-segments template's defaults: 124 @@ -120,23 +124,29 @@ def test_two_segments_are_too_few(self): FPS = 24 RATE = 48000 +TAKE = "Wf-step.0-0.0.mp4" + + +def _frames(start, count, size): + """Frames whose red channel is their index on the take.""" + frames = numpy.zeros((count, size[1], size[0], 3), dtype=numpy.uint8) + frames[..., 0] = (numpy.arange(start, start + count) % 250)[:, None, None] + return frames -def write_chain(directory, lengths, base="Wf-step.0", size=(64, 32)): +def write_chain(directory, lengths, base="Wf-step.0", size=(64, 32), suffix=""): """Segment files as SegmentSpill names them, each frame's red channel its - index on the take and its audio a ramp of the take's sample index.""" + index on the take and its audio a level of its own.""" from dw.writers import encode_video import torch paths, start = [], 0 for index, count in enumerate(lengths): - frames = numpy.zeros((count, size[1], size[0], 3), dtype=numpy.uint8) - frames[..., 0] = (numpy.arange(start, start + count) % 250)[:, None, None] samples = frames_to_samples(count, FPS, RATE) audio = numpy.full((1, samples), 0.1 + 0.05 * index, dtype=numpy.float32) - path = os.path.join(directory, f"{base}.segment-{index:03d}.mp4") + path = os.path.join(directory, f"{base}.segment-{index:03d}{suffix}.mp4") encode_video( - torch.from_numpy(frames), + torch.from_numpy(_frames(start, count, size)), fps=FPS, output_path=path, audio=torch.from_numpy(audio), @@ -147,90 +157,347 @@ def write_chain(directory, lengths, base="Wf-step.0", size=(64, 32)): return paths +def write_run( + root, + lengths, + take_frames=None, + *, + match_audio=False, + crossfade_ms=None, + manifest=True, + suffix="", + size=(64, 32), +): + """A chain run as the engine leaves it: segment files and the take in + /final, and a manifest whose shots name the segment files. + + `take_frames` below the files' total is a duration_seconds take (cut on + replay); `match_audio` lays a 44.1 kHz track under it whole, whose shots + carry no sample side; `crossfade_ms` records the chain's blend on every + seam.""" + import json + + import torch + + from dw.writers import encode_video + + final = os.path.join(root, "run", "final") + os.makedirs(final, exist_ok=True) + paths = write_chain(final, lengths, size=size, suffix=suffix) + total = sum(lengths) if take_frames is None else take_frames + rate = 44100 if match_audio else RATE + noise = numpy.random.default_rng(7).uniform( + -0.5, 0.5, (1, frames_to_samples(total, FPS, rate)) + ) + take = os.path.join(final, TAKE) + encode_video( + torch.from_numpy(_frames(0, total, size)), + fps=FPS, + output_path=take, + audio=torch.from_numpy(noise.astype(numpy.float32)), + audio_sample_rate=rate, + ) + shots, start = [], 0 + for index, count in enumerate(lengths): + count = min(count, total - start) + shot = { + "name": f"segment {index + 1}", + "start_frame": start, + "num_frames": count, + "start_sample": None, + "num_samples": None, + "segment_file": os.path.basename(paths[index]), + } + if not match_audio: + first = frames_to_samples(start, FPS, rate) + shot["start_sample"] = first + shot["num_samples"] = frames_to_samples(start + count, FPS, rate) - first + if index and crossfade_ms: + shot["crossfade_ms"] = crossfade_ms + shot["trim_frames"] = 22 + shots.append(shot) + start += count + if manifest: + with open(os.path.join(root, "run", "manifest.json"), "w") as file: + json.dump( + { + "steps": [ + {"step": "step", "files": [f"final/{TAKE}"], "shots": shots} + ] + }, + file, + ) + return {"take": take, "segments": paths, "shots": shots} + + +def stand_in_render(cut, level=0.7): + """The render as H3 would return it: the head guide, new white frames, + the tail guide, over a track of its own.""" + new = [Image.new("RGB", cut["head_guide"].frames[0].size, (255, 255, 255))] * ( + cut["tail_frame"] - cut["keep_start"] + ) + frames = list(cut["head_guide"].frames) + new + list(cut["tail_guide"].frames) + samples = frames_to_samples(len(frames), FPS, RATE) + return AudioVideo( + frames, numpy.full((1, samples), level, dtype=numpy.float32), RATE, fps=FPS + ) + + +def repaired(run, segment=2, num_frames=107, tail_frames=22): + cut = chain_repair_cut( + chain_take=run["take"], + segment=segment, + num_frames=num_frames, + tail_frames=tail_frames, + ) + render = stand_in_render(cut) + spliced = chain_repair_splice( + chain_take=cut["chain_take"], + video=render, + segment=segment, + start_frame=cut["start_frame"], + end_frame=cut["end_frame"], + keep_start=cut["keep_start"], + ) + return cut, spliced, load_audio_video(run["take"]) + + +def red(frame): + return numpy.asarray(frame)[10, 10, 0] + + @pytest.fixture -def chain(tmp_path): - return write_chain(str(tmp_path), [40, 51, 51, 51]) +def run(tmp_path): + return write_run(str(tmp_path), [40, 51, 51, 51]) class TestSegmentFiles: - def test_finds_every_sibling_in_order(self, chain, tmp_path): - # Another chain's files, and an unrelated file, beside them - write_chain(str(tmp_path), [5], base="Wf-other.0") - (tmp_path / "Wf-step.0-0.0.mp4").write_bytes(b"") - assert segment_files(chain[2]) == chain - - def test_a_file_that_is_no_segment_is_refused(self, tmp_path): - path = tmp_path / "take.mp4" - path.write_bytes(b"") - with pytest.raises(ValueError, match="save_segments and keep_segments"): - segment_files(str(path)) - - def test_a_missing_segment_is_refused(self, chain): - os.remove(chain[1]) + def test_the_take_shots_name_the_files(self, tmp_path): + # A stale set from another run, under the names the regex matches, + # beside the run's own '-2' deduped files the take's shots name + made = write_run(str(tmp_path), [40, 51, 51, 51], suffix="-2") + write_chain(str(tmp_path / "run" / "final"), [5, 5, 5], size=(32, 16)) + cut = chain_repair_cut( + chain_take=made["take"], segment=2, num_frames=107, tail_frames=22 + ) + assert cut["segment_files"] == made["segments"] + assert cut["head_guide"].frames[0].size == (64, 32) + + def test_the_take_is_found_from_a_segment_file(self, run): + cut = chain_repair_cut( + chain_output=run["segments"][1], segment=2, num_frames=107, tail_frames=22 + ) + assert cut["chain_take"] == run["take"] + + def test_a_segment_file_of_another_take_is_refused(self, run, tmp_path): + other = write_chain(str(tmp_path), [5], base="Wf-other.0") + with pytest.raises(ValueError, match="not one of"): + chain_repair_cut( + chain_take=run["take"], + chain_output=other[0], + segment=2, + num_frames=107, + tail_frames=22, + ) + + def test_neither_input_is_refused(self): + with pytest.raises(ValueError, match="chain_take"): + chain_repair_cut(segment=2, num_frames=107) + + def test_without_a_manifest_a_segment_file_needs_the_take(self, tmp_path): + made = write_run(str(tmp_path), [40, 51, 51, 51], manifest=False) + with pytest.raises(ValueError, match="chain_take"): + chain_repair_cut( + chain_output=made["segments"][0], + segment=2, + num_frames=107, + tail_frames=22, + ) + + def test_without_a_manifest_the_siblings_are_found_by_name(self, tmp_path): + made = write_run(str(tmp_path), [40, 51, 51, 51], manifest=False) + cut = chain_repair_cut( + chain_take=made["take"], + chain_output=made["segments"][2], + segment=2, + num_frames=107, + tail_frames=22, + ) + assert cut["segment_files"] == made["segments"] + assert cut["start_frame"] == 40 and cut["end_frame"] == 40 + 51 + 12 + + def test_siblings_that_disagree_are_refused(self, tmp_path): + # Two runs' files mixed by the flat layout: one is another size + made = write_run(str(tmp_path), [40, 51, 51, 51], manifest=False) + (other,) = write_chain(str(tmp_path), [51], size=(32, 16)) + os.replace(other, made["segments"][3]) + with pytest.raises(ValueError, match="disagree"): + chain_repair_cut( + chain_take=made["take"], + chain_output=made["segments"][0], + segment=2, + num_frames=107, + tail_frames=22, + ) + + def test_a_missing_segment_is_refused(self, tmp_path): + made = write_run(str(tmp_path), [40, 51, 51, 51], manifest=False) + os.remove(made["segments"][1]) with pytest.raises(ValueError, match="segment-001 is missing"): - segment_files(chain[0]) + segment_files(made["segments"][0]) + + def test_the_neighbours_are_counted_by_their_decode(self, run, monkeypatch): + # The header is not read for the neighbours the cut decodes anyway + import dw.tasks.chain_repair as chain_repair + + monkeypatch.setattr( + chain_repair, "probe_metadata", lambda path: {"frame_count": 999} + ) + cut = chain_repair_cut( + chain_take=run["take"], segment=2, num_frames=107, tail_frames=22 + ) + assert cut["segment_frames"][0] == 40 and cut["segment_frames"][2] == 51 class TestCut: - def test_the_cut_and_the_splice(self, chain): + def test_the_cut(self, run): # 51-frame segments: head 22 + 51 + tail 22 = 95 -> 107, overrun 12 - cut = chain_repair_cut(chain[0], 2, 107, 22, 22, width=64, height=32) + cut = chain_repair_cut( + chain_take=run["take"], + segment=2, + num_frames=107, + tail_frames=22, + width=64, + height=32, + ) assert cut["segments"] == 4 - assert cut["segment_frames"] == [40, 51, 51, 51] assert cut["overrun"] == 12 and cut["tail_frame"] == 85 + assert cut["start_frame"] == 40 and cut["end_frame"] == 103 + assert cut["total_frames"] == 193 assert len(cut["head_guide"].frames) == 22 assert len(cut["tail_guide"].frames) == 22 assert cut["head_guide"].audio is not None assert cut["tail_guide"].audio is not None assert cut["keyframe"] is cut["head_guide"].frames[0] - assert cut["before"] == chain[:1] - assert len(cut["after"]) == 2 - assert len(cut["after"][0].frames) == 51 - 12 - assert cut["after"][1:] == chain[3:] - - # A stand-in render: the head guide, new frames, the tail guide - new = [Image.new("RGB", (64, 32), (255, 255, 255))] * cut["keep_frames"] - render_frames = ( - list(cut["head_guide"].frames) + new + list(cut["tail_guide"].frames) - ) - samples = frames_to_samples(len(render_frames), FPS, RATE) - render = AudioVideo( - render_frames, - numpy.zeros((1, samples), dtype=numpy.float32), - RATE, - fps=FPS, - ) - assert len(render.frames) == 107 - kept = trim_video(render, cut["keep_start"], cut["keep_frames"]) - before = concat_videos(cut["before"], fps=FPS, continuous_joins=True) - after = concat_videos(cut["after"], fps=FPS, continuous_joins=True) - take = concat_videos([before, kept, after], fps=FPS, continuous_joins=True) - assert len(take.frames) == sum(cut["segment_frames"]) - assert not any(shot.get("hard_cut") for shot in take.shots) - # The take resumes on segment 3's own frame 12 right after the repair - resume = 40 + cut["keep_frames"] - red = take.frames[resume].getpixel((10, 10))[0] - assert abs(red - (40 + 51 + 12)) <= 3 - - def test_a_canvas_other_than_the_chains_is_refused(self, chain): + # The head guide is segment 1's last 22 frames, the tail guide + # segment 3's from the overrun on + assert abs(int(red(cut["head_guide"].frames[0])) - 18) <= 3 + assert abs(int(red(cut["tail_guide"].frames[0])) - (91 + 12)) <= 3 + + def test_a_canvas_other_than_the_chains_is_refused(self, run): with pytest.raises(ValueError, match="chain's canvas"): - chain_repair_cut(chain[0], 2, 107, 22, 22, width=960, height=544) + chain_repair_cut( + chain_take=run["take"], + segment=2, + num_frames=107, + tail_frames=22, + width=960, + height=544, + ) - def test_the_last_segment_is_refused(self, chain): + def test_the_last_segment_is_refused(self, run): with pytest.raises(ValueError, match="one-sided repair"): - chain_repair_cut(chain[0], 4, 107, 22, 22) + chain_repair_cut( + chain_take=run["take"], segment=4, num_frames=107, tail_frames=22 + ) def test_registered(self): assert "chain_repair_cut" in _COMMAND_REGISTRY + assert "chain_repair_splice" in _COMMAND_REGISTRY + + +class TestSplice: + """Apart from the repaired span, the output IS the take.""" + + def test_the_take_outside_the_span(self, tmp_path): + made = write_run(str(tmp_path), [40, 51, 51, 51], crossfade_ms=75) + cut, spliced, take = repaired(made) + assert len(spliced.frames) == len(take.frames) == 193 + start, end = cut["start_frame"], cut["end_frame"] + for index in list(range(start)) + list(range(end, 193)): + assert numpy.array_equal( + numpy.asarray(spliced.frames[index]), numpy.asarray(take.frames[index]) + ), index + assert all(red(frame) == 255 for frame in spliced.frames[start:end]) + # Every other seam's crossfaded audio, bit for bit + s0 = made["shots"][1]["start_sample"] + s1 = made["shots"][2]["start_sample"] + frames_to_samples(12, FPS, RATE) + assert spliced.audio.shape == take.audio.shape + assert numpy.array_equal(spliced.audio[:, :s0], take.audio[:, :s0]) + assert numpy.array_equal(spliced.audio[:, s1:], take.audio[:, s1:]) + assert spliced.sample_rate == take.sample_rate + + def test_the_span_is_the_render_blended_as_the_chain_blended(self, tmp_path): + made = write_run(str(tmp_path), [40, 51, 51, 51], crossfade_ms=75) + cut, spliced, take = repaired(made) + s0 = made["shots"][1]["start_sample"] + s1 = made["shots"][2]["start_sample"] + frames_to_samples(12, FPS, RATE) + window = int(0.075 * RATE) + # In from the take's own track, out into it again, the chain's 75 ms + assert spliced.audio[0, s0] == pytest.approx(take.audio[0, s0], abs=1e-6) + assert numpy.allclose(spliced.audio[:, s0 + window : s1 - window], 0.7) + assert not numpy.allclose(spliced.audio[:, s1 - window // 2], 0.7) + assert spliced.shots[1]["crossfade_ms"] == 75.0 + assert spliced.shots[2]["crossfade_ms"] == 75.0 + + def test_a_guide_chain_seam_gets_the_declick(self, run): + cut, spliced, take = repaired(run) + assert spliced.shots[1]["crossfade_ms"] == 3.0 + assert spliced.shots[2]["crossfade_ms"] == 3.0 + + def test_a_duration_take_keeps_its_frame_count(self, tmp_path): + # duration_seconds: the spill holds 193 frames, the take 172 + for segment in (2, 3): + made = write_run(str(tmp_path / str(segment)), [40, 51, 51, 51], 172) + cut, spliced, take = repaired(made, segment=segment) + assert len(take.frames) == 172 + assert len(spliced.frames) == 172 + assert spliced.audio.shape == take.audio.shape + + def test_a_match_audio_take_keeps_its_source_track(self, tmp_path): + made = write_run(str(tmp_path), [40, 51, 51, 51], 180, match_audio=True) + cut, spliced, take = repaired(made) + assert take.sample_rate == 44100 + assert spliced.sample_rate == 44100 + assert numpy.array_equal(spliced.audio, take.audio) + assert len(spliced.frames) == 180 + + def test_the_repaired_seams_are_measured(self, run): + from dw.tasks.assess import analyze_seams + + cut, spliced, take = repaired(run) + names = [shot["name"] for shot in spliced.shots] + assert names == ["segment 1", "segment 2 (repaired)", "segment 3", "segment 4"] + assert spliced.shots[1]["start_frame"] == cut["start_frame"] + assert spliced.shots[2]["start_frame"] == cut["end_frame"] + assert sum(shot["num_frames"] for shot in spliced.shots) == 193 + assert not any(shot.get("hard_cut") for shot in spliced.shots) + assert all(not seam["hard_cut"] for seam in analyze_seams(spliced)["seams"]) + + def test_a_render_on_another_canvas_is_refused(self, run): + cut = chain_repair_cut( + chain_take=run["take"], segment=2, num_frames=107, tail_frames=22 + ) + render = stand_in_render(cut) + render.frames = [frame.resize((32, 16)) for frame in render.frames] + with pytest.raises(ValueError, match="canvas"): + chain_repair_splice( + chain_take=run["take"], + video=render, + segment=2, + start_frame=cut["start_frame"], + end_frame=cut["end_frame"], + keep_start=cut["keep_start"], + ) -TEMPLATE = ( +TEMPLATES = ( __import__("pathlib").Path(__file__).resolve().parent.parent / "workflows" / "templates" / "minimax" - / "repair-segment.json" ) +TEMPLATE = TEMPLATES / "repair-segment.json" def template_errors(arguments=None): @@ -243,6 +510,17 @@ def template_errors(arguments=None): return workflow.validation_errors(arguments) +def chain_blocks(definition): + if isinstance(definition, dict): + if isinstance(definition.get("chain"), dict): + yield definition["chain"] + for value in definition.values(): + yield from chain_blocks(value) + elif isinstance(definition, list): + for value in definition: + yield from chain_blocks(value) + + class TestTemplate: def test_validates_with_its_defaults(self): assert template_errors() == [] @@ -253,7 +531,7 @@ def test_validates_with_sample_arguments(self): { "segment": 3, "num_frames": 192, - "tail_frames": 56, + "tail_frames": 39, "prompt": "A glass figure turns towards the light.", "seed": 7, } @@ -265,23 +543,55 @@ def test_segment_1_is_refused_before_the_run(self): errors = template_errors({"segment": 1}) assert any("segment" in str(error) for error in errors), errors - def test_the_splice_is_continuous_and_the_guides_hold_audio(self): + @pytest.mark.parametrize( + "arguments", [{"head_frames": 39}, {"tail_frames": 56}, {"tail_frames": 5}] + ) + def test_holds_past_the_vram_fold_are_refused(self, arguments): + errors = template_errors(arguments) + assert any(next(iter(arguments)) in str(error) for error in errors), errors + + def test_the_vram_fold_is_an_upper_bound_on_every_allowed_render(self): + import json + + definition = json.loads(TEMPLATE.read_text()) + rate = definition["vram_estimate"]["bytes_per_voxel"] + constraints = definition["variable_constraints"] + head = constraints["head_frames"] + tail = constraints["tail_frames"] + assert head["min_frames"] == head["max_frames"] == 22 + assert (tail["min_frames"], tail["max_frames"]) == (22, 39) + frames = constraints["num_frames"] + for num_frames in range(frames["min_frames"], frames["max_frames"] + 1, 17): + # The t2va rate on the render and on both guides' frames + assert rate * num_frames >= 28.71 * (num_frames + 22 + 39), num_frames + assert "exact" not in definition["vram_estimate"]["reason"] + + def test_the_segment_bound_says_it_is_no_frame_count(self): import json - steps = {s["name"]: s for s in json.loads(TEMPLATE.read_text())["steps"]} + constraints = json.loads(TEMPLATE.read_text())["variable_constraints"] + assert "not a frame count" in constraints["segment"]["reason"] + + def test_the_steps_cut_render_and_splice_the_take(self): + import json + + definition = json.loads(TEMPLATE.read_text()) + steps = {s["name"]: s for s in definition["steps"]} + assert list(steps) == ["cut", "repair", "splice"] assert steps["cut"]["task"]["command"] == "chain_repair_cut" + assert steps["cut"]["task"]["arguments"]["chain_take"] == ( + "variable:chain_take" + ) guides = steps["repair"]["pipeline"]["arguments"]["guides"] assert [g["frame"] for g in guides] == [0, "previous_result:cut.tail_frame"] assert all(g["audio"] is True for g in guides) - for name in ("before", "after", "splice"): - arguments = steps[name]["task"]["arguments"] - assert steps[name]["task"]["command"] == "concat_videos" - assert arguments["continuous_joins"] is True - assert steps["splice"]["task"]["arguments"]["videos"] == [ - "previous_result:before", - "previous_result:keep", - "previous_result:after", - ] + splice = steps["splice"]["task"] + assert splice["command"] == "chain_repair_splice" + assert splice["arguments"]["chain_take"] == "previous_result:cut.chain_take" + assert splice["arguments"]["video"] == "previous_result:repair" + # No made-up output: path stands in for the caller's take + assert definition["variables"]["chain_take"] is None + assert definition["variables"]["chain_output"] is None def test_the_default_render_fits_the_default_chain(self): import json @@ -298,11 +608,30 @@ def test_the_default_render_fits_the_default_chain(self): assert variables["tail_frames"] > variables["head_frames"] assert variables["seed"] is None + @pytest.mark.parametrize( + "name", + [ + "chained-segments.json", + "chain-matched-to-audio.json", + "chain-matched-and-aligned.json", + "chain-video-continuity.json", + ], + ) + def test_the_h3_chains_can_keep_their_segments(self, name): + import json + + definition = json.loads((TEMPLATES / name).read_text()) + assert definition["variables"]["keep_segments"] is False + (chain,) = list(chain_blocks(definition)) + assert chain["keep_segments"] == "variable:keep_segments" + assert chain["save_segments"] in ("variable:keep_segments", True) + assert "repair-segment" in definition["description"] -def test_the_cut_feeds_the_template_steps_in_a_workflow(chain, tmp_path, monkeypatch): + +def test_the_cut_feeds_the_splice_in_a_workflow(run, tmp_path, monkeypatch): """The template's task steps as a workflow, with a stand-in for the H3 - render: a record's list and number fields reach trim_video and - concat_videos as one value each, and the splice keeps the take's length.""" + render: the record's fields reach the splice as one value each, and the + saved splice keeps the take's length.""" import av from dw.workflow import Workflow @@ -316,7 +645,7 @@ def test_the_cut_feeds_the_template_steps_in_a_workflow(chain, tmp_path, monkeyp "task": { "command": "chain_repair_cut", "arguments": { - "chain_output": chain[0], + "chain_take": run["take"], "segment": 2, "num_frames": 107, "tail_frames": 22, @@ -324,53 +653,29 @@ def test_the_cut_feeds_the_template_steps_in_a_workflow(chain, tmp_path, monkeyp }, }, { - # Stands in for the render: its kept span is the head guide's - # frames from overrun on (a span of the right shape) - "name": "keep", + # Stands in for the render: 107 frames of the take with audio + "name": "repair", "task": { "command": "trim_video", "arguments": { - "video": "previous_result:cut.head_guide", - "start_frame": "previous_result:cut.overrun", - "num_frames": 10, - "fps": 24, - }, - }, - }, - { - "name": "before", - "task": { - "command": "concat_videos", - "arguments": { - "videos": "previous_result:cut.before", - "fps": 24, - "continuous_joins": True, - }, - }, - }, - { - "name": "after", - "task": { - "command": "concat_videos", - "arguments": { - "videos": "previous_result:cut.after", + "video": run["take"], + "start_frame": 18, + "num_frames": 107, "fps": 24, - "continuous_joins": True, }, }, }, { "name": "splice", "task": { - "command": "concat_videos", + "command": "chain_repair_splice", "arguments": { - "videos": [ - "previous_result:before", - "previous_result:keep", - "previous_result:after", - ], - "fps": 24, - "continuous_joins": True, + "chain_take": "previous_result:cut.chain_take", + "video": "previous_result:repair", + "segment": 2, + "start_frame": "previous_result:cut.start_frame", + "end_frame": "previous_result:cut.end_frame", + "keep_start": "previous_result:cut.keep_start", }, }, "result": {"content_type": "video/mp4", "fps": 24}, @@ -383,5 +688,4 @@ def test_the_cut_feeds_the_template_steps_in_a_workflow(chain, tmp_path, monkeyp assert len(saved) == 1, saved with av.open(str(saved[0])) as container: frames = sum(1 for _ in container.decode(video=0)) - # segment 1 (40) + the 10-frame stand-in + segments 3 (51 - 12) and 4 (51) - assert frames == 40 + 10 + 39 + 51 + assert frames == 193 diff --git a/tests/test_concat_videos.py b/tests/test_concat_videos.py index 5a8055918..f2de8d373 100644 --- a/tests/test_concat_videos.py +++ b/tests/test_concat_videos.py @@ -1130,13 +1130,30 @@ def test_continuous_joins_record_no_hard_cut(self): assert [shot.get("hard_cut") for shot in result.shots] == [None] * 3 assert all("seam_fade_ms" not in shot for shot in result.shots) - def test_continuous_joins_butt_join_the_audio_untouched(self): + def test_continuous_joins_keep_the_default_declick(self): + result = concat_videos( + [ + audio_video(4, 0.1, sample_rate=8000), + audio_video(4, 0.2, sample_rate=8000), + ], + fps=4, + continuous_joins=True, + ) + # The few-ms declick, so the seam does not click - and no + # seam_fade_ms recorded, so seam_hole still measures the seam + assert result.audio[0, 7999] < 0.1 + assert result.audio[0, 8000] < 0.2 + assert numpy.allclose(result.audio[:, :7900], 0.1) + assert numpy.allclose(result.audio[:, 8100:], 0.2) + assert "seam_fade_ms" not in result.shots[1] + + def test_continuous_joins_take_a_zero_fade_as_a_butt_join(self): result = concat_videos( [audio_video(4, 0.1), audio_video(4, 0.2)], fps=4, continuous_joins=True, + seam_fade_ms=0, ) - # No declick ramp: every sample keeps its input's level assert numpy.allclose(result.audio[:, :100], 0.1) assert numpy.allclose(result.audio[:, 100:], 0.2) diff --git a/tests/test_shots.py b/tests/test_shots.py index c3ea569ff..f92780fc7 100644 --- a/tests/test_shots.py +++ b/tests/test_shots.py @@ -93,6 +93,8 @@ def audio_video(num_frames, level, fps=4, sample_rate=100): ("dw/pipeline_processors/chain.py", "inject"): ("none", 1), ("dw/tasks/face_track.py", "paste_face_track"): ("carries", 1), ("dw/tasks/trim.py", "trim_video"): ("trims", 1), + # the take's shots with the repaired span as one shot of its own + ("dw/tasks/chain_repair.py", "chain_repair_splice"): ("populates", 1), } _SHOTS_HELPER_BY_DECISION = { diff --git a/workflows/templates/minimax/README.md b/workflows/templates/minimax/README.md index d220fe184..db84a55dc 100644 --- a/workflows/templates/minimax/README.md +++ b/workflows/templates/minimax/README.md @@ -162,7 +162,7 @@ segments means half the compounding. | [chain-matched-to-audio.json](chain-matched-to-audio.json) | `match_audio`: a supplied track decides the length, and the final video is muxed against the original, seamless track | | [chain-video-continuity.json](chain-video-continuity.json) | `last_segment` continuity: the previous segment's tail rides along as a video reference, carrying motion and voice across the seam | | [chain-matched-and-aligned.json](chain-matched-and-aligned.json) | Everything together for a long soundtrack-driven take, with per-segment prompts and crash-safe `save_segments` | -| [repair-segment.json](repair-segment.json) | Re-rendering one interior segment of a finished chain (run with `save_segments` + `keep_segments`) between its neighbours, held to both by guides with their audio, and splicing it back with continuous joins | +| [repair-segment.json](repair-segment.json) | Re-rendering one interior segment of a finished chain (run with `keep_segments` on) between its neighbours, held to both by guides with their audio, and splicing it into the take itself, so everything outside the span stays the take's | ## Going long: cuts (digital shorts) diff --git a/workflows/templates/minimax/chain-matched-and-aligned.json b/workflows/templates/minimax/chain-matched-and-aligned.json index 73956613c..1c69952f8 100644 --- a/workflows/templates/minimax/chain-matched-and-aligned.json +++ b/workflows/templates/minimax/chain-matched-and-aligned.json @@ -1,6 +1,6 @@ { "id": "MiniMaxH3Ref2VAChainedAligned", - "description": "The chaining features used together, for a long clip driven by a supplied soundtrack. 'match_audio' sizes the chain to the audio while 'last_segment' continuity carries motion and framing across the seams, with 'carry_audio' off because the segments already take their audio from the supplied track rather than from each other. Also shows 'prompts', which lets a chain say something different on its opening segment than on its continuations, and 'save_segments', which writes each finished segment to disk so memory stays bounded to one segment and a crashed run keeps its progress.", + "description": "The chaining features used together, for a long clip driven by a supplied soundtrack. 'match_audio' sizes the chain to the audio while 'last_segment' continuity carries motion and framing across the seams, with 'carry_audio' off because the segments already take their audio from the supplied track rather than from each other. Also shows 'prompts', which lets a chain say something different on its opening segment than on its continuations, and 'save_segments', which writes each finished segment to disk so memory stays bounded to one segment and a crashed run keeps its progress. 'keep_segments' (default false) keeps the segment files this chain already saves beside the take, naming them on the take's shots - what 'repair-segment' needs to re-render one segment in place later.", "summary": "A long H3 clip sized to a supplied soundtrack, with video-tail continuity across every seam.", "seed": "variable:seed", "cost_drivers": [ @@ -25,6 +25,7 @@ "reason": "Ref2VA holds every reference's encoding resident beside the video latents, so the ceiling is a fixed base, a per-voxel term for the VAE decode's conv3d activation (width * height * num_frames), and a per-reference term. Classified against the 24 GB entry from the #479 field report (lem, RTX 3090 24 GB, int4 + 768p turbo LoRA, 9 steps, 2026-09-25/26) at 1344x768: 175 frames x 2 refs, 209 x 2 and 175 x 3 ran (projected 22.83, 23.77 and 23.83 GB); 209 x 3 OOM'd in reference encode needing a further 5.1 GB and 260 x 1 OOM'd at denoise step 1 needing a further 4.8 GB (projected 24.77 and 24.18 GB). The refused allocations are lower bounds, not peaks, so this is a classification with about 0.2 GB of margin either side, not a fitted curve. bytes_per_voxel is unchanged from #324's calibration (job 282172105da1); base_gb drops 0.3 GB now the references are counted separately" }, "variables": { + "keep_segments": false, "first_segment_prompt": "subject_definitions:\n is the cartoon astronaut in , whose face, hair, clothing and proportions define their appearance for the whole video.\n