workflows/ holds two trees, and which one a file is in says what it is for.
workflows/templates/ teaches a pattern. One file per capability - a shape
(image to video, a multi-shot cut sequence), a mechanism (shared components,
sub-workflows, pipeline_reference, typed references), or a reference
convention (prompt:, previous_result:). These are what to read and copy.
Where several checkpoints run the same pattern through the same pipeline class,
one template carries them all and its description spells out the per-checkpoint
argument sets. The templates/ltx2/ and templates/minimax/ subfolders each
hold a family whose members build on one baseline.
The full catalog is list_workflows over MCP, or GET /api/workflows and the web
UI's Workflows page, each entry with its shape and description. This guide and
docs/TASKS.md link only the templates that show a feature, so a template
named in neither is still in the catalog.
workflows/models/ records a hardware fact: the quantization, offloading and
component placement that make one checkpoint fit a real card. Each is runnable
and carries a configures naming the template it is an instance of:
{
"id": "flux-dev",
"description": "Text-to-image with FLUX.1 dev - the reference FLUX workflow.",
"configures": "templates/text-to-image",
"steps": [ ... ]
}Every workflow is a JSON file with an id, optional variables, and a list of steps.
The full structure is the JSON Schema in
dw/workflow_schema.json, which is what dw.validate
and the server check a workflow against. A running server also serves it at
GET /api/schema and shows it as a browsable tree on its Schema page
(see Server & Web UI).
{
"id": "my_workflow",
"variables": {
"prompt": "default prompt text",
"steps": 25
},
"steps": [ ... ]
}Variables define defaults that a caller overrides, from the command line or
as run_workflow's arguments:
python -m dw.run my_workflow.json prompt="a cat" steps=50A variable name matches ^[a-zA-Z_][a-zA-Z0-9_-]*$: a letter or underscore, then
letters, digits, underscores or hyphens.
Each step has a name and exactly one of four types:
Run a HuggingFace Diffusers model:
{
"name": "generate",
"pipeline": {
"configuration": { "component_type": "FluxPipeline" },
"from_pretrained_arguments": {
"model_name": "black-forest-labs/FLUX.1-dev",
"torch_dtype": "torch.bfloat16"
},
"arguments": {
"prompt": "variable:prompt",
"num_inference_steps": 25
}
},
"result": { "content_type": "image/jpeg" }
}A component_type can also name a community pipeline that dw ships in dw/community_pipelines/, by its
dotted path; list_pipelines lists them after the diffusers ones. RFInversionFluxPipeline
(dw.community_pipelines.pipeline_flux_rf_inversion.RFInversionFluxPipeline) inverts an image into FLUX
latents (below). LTX2RefinePipeline (dw.community_pipelines.pipeline_ltx2_refine.LTX2RefinePipeline) is
LTX2Pipeline plus a video argument: the clip is VAE-encoded at width x height and renoised at
noise_scale over sigmas, a refine at the clip's own size rather than refine-clip's 2x. num_frames
(8 * n + 1, no longer than the clip) defaults to the clip's length; video and latents are exclusive.
Re-run an already-loaded pipeline from an earlier step with a fresh set of arguments,
instead of loading the model again. This is how a two-pass technique like RF-Inversion
works: an invert step loads the pipeline, and a main step reuses it with the
inverted latents:
{
"name": "main",
"pipeline_reference": {
"reference_name": "invert",
"arguments": {
"prompt": "variable:prompt",
"inverted_latents": "previous_result:invert.inverted_latents",
"image_latents": "previous_result:invert.image_latents"
}
},
"result": { "content_type": "image/jpeg" }
}reference_name must name a step earlier in the same workflow that has a pipeline.
Two steps whose pipelines differ only in a LoRA's scale or alpha or a scheduler's
shift share one warm model, and a reference always runs at the referenced step's own
values; release_pipeline on either step frees the shared model, as it does for two
identical steps.
See workflows/templates/community-pipeline.json for a full example.
Run utility operations (image processing, QR codes, data gathering):
{
"name": "preprocess",
"task": {
"command": "canny",
"arguments": {
"image": { "location": "https://example.com/photo.jpg" }
}
},
"result": { "content_type": "image/jpeg" }
}A task can take inputs (a plain array) instead of arguments. Each array item becomes
its own iteration, the same way multiple previous_result values do. An item that is a
previous_result: reference becomes one iteration per result it names, and an object item
expands the way an arguments object would:
{
"name": "prompts",
"task": {
"command": "gather_inputs",
"inputs": ["a marmot on a bicycle", "a bug driving a cycle"]
}
}Invoke another workflow file:
{
"name": "expand",
"workflow": {
"path": "builtin:h3_context_ir.json",
"arguments": { "prompt": "variable:prompt" }
},
"result": { "content_type": "text/plain" }
}arguments become the child workflow's variables. How path resolves, where
the child's output is saved, and what validation checks are in
Composing a stored workflow.
Reference workflow variables with variable:name:
"prompt": "variable:prompt"A variable's declared value is both its default and its type: a value passed in is
converted to the type of the default, so declaring 25 and "25" are different
things. Declaring null makes the variable optional and untyped: it takes whatever
it is given and stays null when given nothing. That is how a workflow exposes an
argument a caller may pass. A caller can set only variables the workflow declares,
so an optional argument still has to be declared.
"variables": { "image": null }A value the library already declares does not need a variable at all - see Constant References.
Pass output from one step to another with previous_result:step_name:
{
"steps": [
{
"name": "preprocess",
"task": { "command": "canny", "arguments": { "image": { "location": "photo.jpg" } } }
},
{
"name": "generate",
"pipeline": {
"arguments": {
"control_image": "previous_result:preprocess",
"prompt": "a painting"
}
}
}
]
}A reference resolves anywhere in the arguments, including inside a list or nested
object, which is what lets a constructed object be
built from an earlier step. Several
previous_result references on one step multiply into a cartesian product (see
Several previous_result references multiply).
A step whose result is a dict (a task returning several named outputs, or a pipeline
step that returns something like inverted_latents) can be referenced property by
property with previous_result:step_name.property_name:
"inverted_latents": "previous_result:invert.inverted_latents"Images and videos load automatically for arguments named image/*_image and
video/*_video. Any other argument - mask, depth_map, a controlnet's second
conditioning image - can load the same way with an explicit form that says what the
media is instead of relying on its argument name:
"mask": { "media_type": "image", "location": "mask.png" }media_type is "image" or "video". location is a path relative to the workflow
file, or a URL, exactly like the plain image/video forms.
For an agent that has read the catalog (list_workflows), found nothing that
produces the shape it needs, and is about to write JSON. get_schema says what
is well-formed; this section says what the engine does with a well-formed
document, which is where a draft that validates still fails.
When exploring variations, pass note to run_workflow saying what each run
tests ("video_shift 5 against 3"), and call set_note to mark the keeper: the
note is shown with every file the run made. Each file carries a role
(final or intermediate) inferred from how a step's output is used - a file
a later run reads (output:, or an asset: kept from it) lists as
intermediate too; set result.role on a step to override it when the default
is wrong, and set "role": "intermediate" on a workflow whose deliverable is an
ingredient a later assembly consumes.
An argument value is a reference when it begins with one of these prefixes.
Each resolves before the step runs. variable: and previous_result: names
are checked statically, so a bad one is a validation error at the path it
sits at; a constant:, asset:, prompt: or output: name in the
definition body resolves only when the step runs, and one that is missing
fails the run - unless it arrives in the arguments passed to
validate_workflow, which checks an asset:, prompt: or output: there
for existence.
variable:-variable:nameis the workflow's ownvariablesentry, overridden by the caller'sarguments. A variable declarednullis optional and untyped. Schema validation runs before substitution, so a default must already be the JSON type the field expects:25, not"25". Once a workflow has avariablesblock, a reference to a variable it does not declare is a validation error (with no block, the string is passed through unresolved).previous_result:-previous_result:step_nameis the outputs of an earlier step, named by that step'sname. It iterates (see the cartesian rule below). A literal reference naming no earlier step is a validation error at its JSON path. A.fieldsuffix (previous_result:invert.inverted_latents) picks one field of a result that is a dict, or a data attribute of a result object.constant:-constant:module.path.NAMEis a value declared in Python, read by import rather than copied into JSON. Anything callable is refused.asset:-asset:nameis a file in the asset library, confined to it and never resolved relative to the workflow file.upload_assetreturns one of these names. It may be a bare string ("image": "asset:x.png"), a list element, or thelocationof a media dict ({"location": "asset:x.png"}, with or withoutmedia_type). A fileupload_assetcannot send (over 4MB ascontent=, or not readable from the MCP session's machine) goes straight toPOST /api/uploads, the route the web UI's file picker uses: the body is the raw bytes, up to 200MB, and the reply (201) carriesreference, theasset:name to use. Send the bearer token when the server requires one:The same route is the only way to hand the engine a file made on the client, such as a cut stitched locally: it has no run forcurl -H "Authorization: Bearer $DW_API_TOKEN" --data-binary @portrait.jpg \ "http://<host>:8765/api/uploads?filename=portrait.jpg&asset_name=cast/portrait.jpg&workspace=<ws>"output:orexport_jobto name.output:-output:<workflow identity>/<run id>/<file>is a file an earlier run wrote, confined to the output root.latestin the run-id position picks the newest run that holds that file;v<N>picks the run the gallery labelsv<N>(list_gallery'sversion), and only that run. A run id is not stable against pruning: to depend on a generated file, promote it withkeep_outputand reference theasset:name instead.prompt:-prompt:nameorprompt:folder/nameis a stored prompt'stext, rooted at the prompt library. That text may not itself begin with any of these prefixes. A template's default prompt is usually aprompt:reference to a caption written to the spec its model was trained on, so read the existing one before writing a prompt for a family:list_prompts(intended_model="ltx-2.5")for the shelf,get_prompt("ltx2/fox_dawn_choir")for a body. The listing leaves the bodies out by default and reports each one'stext_chars; asking for all of them at once is more than a client will accept.item:- only inside a step that carriesfor_each:item:is the entry the member was made for,item:fieldone field of an object entry, spliced in whole whatever its type. See "One step per entry" below.gather:-gather:shotis the result of every member of thefor_eachstepshot, in list order, as one list. Inside a list it splices into it.previous_result:shotnaming afor_eachstep is an error that says to usegather:.
A reference is always the whole value: "variable:base_prompt" resolves,
"variable:base_prompt, in fog" asks for a variable named base_prompt, in fog
and fails. Nothing is interpolated around a reference. When several steps share
a block of text (a character's description repeated in every shot), write it
once as a variable and assemble each step's prompt with a
compose_text task, whose parts are whole references
joined in order; the shot then reads "prompt": "previous_result:shot_1_prompt".
After an inline run worth keeping, get_job_workflow(job_id) returns the
realized workflow (the definition with that run's arguments, seed and prompts
pinned into it), and save_workflow gives it a name. export_job(job_id)
bundles the whole run (workflow, manifest, job row, input and output media) into
a directory on the server plus a zip URL.
Any key ending in _type or _dtype, or named dtype, has its string value
loaded as a Python object: a bare name ("FluxPipeline") from diffusers, a
dotted name ("torch.bfloat16", "sdnq.SDNQConfig") by full module path.
Wrapping a value in braces keeps it a plain string: "{nf4}" is the string
nf4. Getting this wrong fails at load time, after validation has already
passed, so a value meant as text under one of those keys must be braced. Two
keys are exempt and always stay strings: content_type and offload_type
(braces are accepted there too).
On a server without --trust-workflows, a dotted name must sit in one of the
packages dw/trust.py allows (diffusers, torch, transformers, sdnq,
torchao and the other quantization backends, dw), and must resolve to a
class defined there, or to a torch.dtype under a dtype key. Anything else is
refused at load.
presets lets one variable select a bundle of others - a template's quality
dial. The shape is {selector: {value: {variable: value}}}:
"variables": {"tier": "standard", "width": 1024, "num_inference_steps": 30},
"presets": {
"tier": {
"draft": {"width": 512, "num_inference_steps": 4},
"standard": {"width": 768, "num_inference_steps": 12}
}
}The chosen bundle (the caller's value for the selector, else its default) sets
every variable the caller did not pass; an explicit argument always wins, so
tier=draft num_inference_steps=30 runs 30 steps on the draft canvas. With no
arguments - or when a bad argument leaves them all unfolded - the default
bundle applies, so a run with no arguments and one passing the selector's
default are the same run, with the same fingerprint.
Everything that judges or prices a run sees the bundle's values: validation,
substitution, the estimate (the curated cost counts as measured at the
default bundle), the fingerprint, the VRAM projection, the host-RAM
projection, the observed-cost buckets, variable_constraints and the
argument warnings. A bundle value is rounded by a
snap: "up" rule, with the same warning a caller's value draws.
Validation checks every bundle, not only the chosen one, and reports each problem where it was written:
- a bundle key that is no declared variable, a bundle value that will not
coerce to its default's type or fails string validation, or one outside a
variable_constraintsrule, atpresets.<selector>.<value>.<variable>; - a selector that is not a declared variable, at
presets.<selector>; - a selector whose default is not a string (bundle names are JSON keys, so a
number can never match one) or names no bundle, at
variables.<selector>; - a caller value for the selector outside the bundle names, at
arguments.<selector>, naming the allowed ones.
A model's own rule about a value belongs in the workflow, not in engine code.
variable_constraints declares it per variable, in the same field names a chain
step's frame_snap uses:
"variable_constraints": {
"num_frames": {
"modulus": 17,
"remainder": 5,
"min_frames": 124,
"max_frames": 345,
"snap": "up",
"reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps"
}
}The value has to be modulus * n + remainder within min_frames to
max_frames. With snap: "up" an off-grid value is rounded to the next one
the rule accepts and the run says so - 130 becomes 141, reported as a
warning at validation time and again in the job's warnings; without snap
it is refused. The bounds are checked against the value the run will use, so
they hold for the rounded number: on the rule above 108 is accepted (it
becomes 124) and 346 is refused (it would become 362).
A string value takes min_length instead: the fewest characters it may have
once surrounding whitespace is dropped, a null counting as empty. Music 3
refuses an empty lyric only after the model has loaded, so its templates
declare "lyrics": {"min_length": 1, "reason": "..."}, and the reason
says what to send instead (a tag-only body for an instrumental).
"required": true marks a variable that has no default. Declare it null in
variables and set the flag: a run, or a validation given arguments, refuses
it while it is still null, with '<name>' is required and has no default and
the reason; a document check with no arguments does not judge the stored
null. Pair it with min_length: 1 so an empty string is refused too. It is for
a value a placeholder default would quietly fill with something the caller did
not choose: voice-clone, narration-long and narration-from-takes use it
for reference_audio, so a stray voice.wav asset can never be cloned by
omission. get_workflow reports it under constraints, the compact listing
shows required, and the web UI marks the field. A sub-workflow that hands the
variable down ("reference_audio": "variable:reference_audio") gets the
parent's value.
max_words_per_second with seconds_from warns, and never refuses, when a
string carries more words than its seconds hold at the workflow's measured
rate. The seconds come from the same list entry for an entry field, else from
the workflow's variables, and a [bracketed] tag is not a word. music-long
declares 0.73 words a second of each section's seconds, measured on the
long-form bench. It warns at validation and again in the job's warnings
(text_may_overrun).
The rule is checked at validation (POST /api/validate, validate_workflow and
the pre-queue check refuse a bad value at arguments.<name> or
variables.<name>), again at run time before anything loads (for a value the
static pass cannot see, such as one a parent workflow passed down), and is
reported in the catalog beside the default by list_workflows and
get_workflow(variables_only=true).
Where a template both declares a constraint and snaps a chain, the chain's
frame_snap names the constraint rather than repeating the numbers:
"frame_snap": "constraint:num_frames"A refused bound cannot depend on another variable (a maximum of fps * seconds);
only the word-rate warning reads a second value. A constraint key reaches a
top-level variable and, where a step consumes it as item:<name>, the same
field of a for_each list's entries (dialogue-short's per-shot num_frames).
The workflow object itself, step, task, workflow,
pipeline_reference, result, chain and a pipeline's configuration are
closed: a property the engine does not read is a validation error naming the
object and the key, not a warning. A component's own configuration is not
closed, so a misspelt key there is ignored.
There is no when, no retry, no select - if a draft reaches for one, the
shape it wants is a different arrangement of steps, not a flag. A mistyped
sedd or subfoldr is refused rather than silently ignored.
pipeline is closed to the same rule with one opening: any key whose value
is a component definition - an object carrying from_pretrained_arguments -
names a component to load, because diffusers grows component names faster
than the schema does (latent_upsampler, prompt_enhancer and processor
all appear that way in shipped templates). A pipeline key that is not one of
those is refused, which is what catches pipeline_type or model_name
written a level too high. from_pretrained_arguments stays open - it passes
its keys through to from_pretrained.
An argument that names a location is confined, untrusted (the default):
- a path must resolve inside the workflow's own directory, the asset
libraries, or the output root. An absolute path elsewhere is refused
whether or not it exists. The remedy is
upload_asset(orkeep_output) and anasset:reference - which is what those exist for. - a
globis confined the same way, and each match re-checked. - an
http(s)URL may not resolve to an address inside the deployment - loopback, link-local, private ranges. Its host must resolve, the body is capped at 1 GiB and a fetch at 10 minutes end to end; past that,upload_assetis the route. remote_text_encoder.urlis https-only, and only a HuggingFace host is sent this machine's token.model_namemust be a Hub repo id, or a path inside one of those roots.
All of it is reported by validate_workflow, before anything is queued, so
a draft that names a file the server may not read costs nothing to find out.
get_server_info's trust_workflows says which posture is in force.
A top-level host_ram_gb (a number, or variable: set per tier by a
presets.tier bundle) declares the host RAM a run needs. The server holds a
job while that, with a margin, does not fit beside the jobs already running
on the host - including another dw.serve's - and a larger observed peak wins
over it. Validate and submit refuse a run whose declared need exceeds the
host's total RAM, so the H3 best (int8) tier, which declares 69 GB, is
refused on a 64 GB machine.
A template can declare a vram_estimate, and a workflow over it that
projects past the card is refused before anything is queued. A workflow you
wrote declares none, so validate_workflow (and run_workflow's pre-queue
check) match each of its pipeline steps to the catalog by pipeline
identity - the step's component_type, from_pretrained_arguments.model_name
and from_pretrained_arguments.workflow - and project it against the
ceiling a catalog template declared for that same pipeline. Past it, the
answer stays valid: true and carries a warning of kind
vram_projection_inherited that names the source template and the heaviest
step (for a for_each, the heaviest member). It warns rather than refuses,
since your offload and quantization may differ from the template's. Cut
frames, size or references until it goes quiet, or declare your own
vram_estimate, which always wins and is judged exactly as a template's.
Under the ceiling, a workflow whose offload or quantization differs from the
template's (no offload where the template sets model, say) gets one
vram_config_differs warning naming both settings and the projection, since
real usage may be higher; over the ceiling the vram_projection_inherited
warning names that difference instead. A pipeline the catalog declares no
ceiling for gets no warning, which is not a clearance. H3's t2va and
ref2va are different identities with different ceilings.
The projection is base_gb, plus bytes_per_voxel times the product of
the voxel_variables, plus gb_per_reference per non-null reference, plus
bytes_per_guide_voxel times each guide's frames times the canvas (every
voxel variable but num_frames); an absent key adds nothing. A declared
estimate refuses on the guide term as on the rest; an inherited one only
warns. The guides counted are the step's own
guides with a non-null video, at their snapped length, and the one a
continuity: "guide" chain appends (guide_frames, 22 or 39). A guide
clip is probed at validate, at admission and at run; one that cannot be
probed before the run (a previous_result: clip) is charged at the step's
num_frames, and the message says it was charged at the worst case. H3's
t2va and fl2va templates declare the guide term, so a guided step is
judged by what its guides add rather than passing on the clip alone.
A server started without --trust-workflows refuses any
from_pretrained_arguments that sets trust_remote_code or
custom_pipeline, at load time, after validation has passed. Use a
pipeline diffusers ships: no bundled catalog entry carries either key, and a
workflow that does runs only on a server whose operator turned trust on,
which get_server_info reports as trust_workflows.
When one step carries two or more previous_result references, the engine runs
that step once for every combination, a cartesian product: four images and
three masks is twelve iterations, not three pairs. Past 10000 combinations the
run is refused.
So a pairing (shot i with speaker i, prompt i with portrait i) cannot
be expressed with two references on one step. Write one step per pair, each
referencing exactly the two things it pairs, gather the pairs upstream so each
is a single result, or use two for_each steps over one list, which pair by
entry (below).
A step that carries for_each runs once per entry of a list — a shot per
entry of shots — and the list is a variable the caller supplies, so a
six-shot episode is an argument rather than a different file.
{
"name": "shot",
"for_each": "variable:shots",
"pipeline": {
"arguments": {
"prompt": "item:prompt",
"references": "item:references"
}
}
}with
"shots": [
{ "name": "wide_open", "prompt": "the band walks on, wide",
"references": [{ "reference_type": "…", "from_previous_result": "draw_singer" }] },
{ "name": "closeup", "prompt": "closeup on the singer",
"references": [{ "reference_type": "…", "from_previous_result": "draw_singer" }] }
]and downstream
{ "name": "edit",
"task": { "command": "concat_videos", "arguments": { "videos": "gather:shot" } } }Before the run starts, the engine replaces the for_each step with one
ordinary step per entry, named shot@wide_open, shot@closeup: the entry's
name, or its index for an entry without one. Those are the names the
manifest, the job's events and the gallery show, and @ is reserved for them:
a hand-written step name may not contain it. An entry's name must be unique
in its list and match ^[a-zA-Z_][a-zA-Z0-9_-]*$. Give entries names: the step
cache keys on the member name, so a shot inserted into a named list leaves every
other shot cached, while in an indexed list every later shot regenerates.
item:field is the whole value of that field, so an entry can carry
anything a step argument can, including a references list whose length
differs by shot. Nothing is interpolated: "item:prompt" is the field,
"shot: item:prompt" is a literal string.
A field an entry may leave out is read with the object form
{"item": "singer", "default": "variable:singer_reference"} (the field's bare
name, not item:singer). An entry that carries singer gets its value; one
that does not gets the default, spliced in whole. An object with no default
is the plain required read. The object must have exactly those keys, and it
means something only inside a for_each step. music-video uses it so a shot
can condition on its own picture (singer) and the rest fall back to
singer_reference; the audio slice stays in the template. If every entry
supplies the field and the default's step is read nowhere else, that step is
elided like any unread one.
An entry may name another variable: "from_file": "variable:character_a_voice"
inside a references entry is that variable's value by the time the member
exists, so one variable sets a voice in every shot the character speaks in
and a caller who supplies the list still writes variable: for the parts the
template fixes. Those references are resolved before anything in the entry is
loaded, and an undeclared one is a validation error at the entry's path
(arguments.shots[2].references[1].from_file when the list is yours,
variables.shots[...] when it is the template's). A value may not reference
itself, directly or through another variable.
Two for_each steps over the same list are paired by key — the entry's
name, or its index for an entry without one: inside shot@closeup, a
reference to another for_each step slice over the same shots list
resolves to slice@closeup. That is how a shot reads the audio
slice cut for it when slicing and generating are two steps. It is the only
pairing the engine has: for_each runs over exactly one list, with no zip and
no loop index.
That pairing applies only where from_previous_result (or previous_result:)
sits directly in the for_each step's own arguments. Inside an item (an
entry field the step reads with item:, such as a references entry) nothing
pairs it, because the entry is data the caller wrote. Spell the member out:
{"reference_type": "…", "from_previous_result": "slice@closeup"}, not
"slice". Getting it wrong fails the run with an error naming the members that
exist.
Members of one list can't read each other. A literal reference to another
entry of the member's own list ("image": "previous_result:shot@accuse"
inside shot@deflect, in the template or in an entry, either spelling, with or
without a .property) is refused at validate. When one entry depends on
another (shot 2 starts from shot 1's last frame), use separate runs: generate
the first in its own run, with any derivation such as a get_last_frame step
after it, then pass that run's file to the next run as an output: reference in
the dependent entry. Another for_each step over the same list reading a
different key (trim@deflect reading shot@accuse) stays legal: every member
of shot has finished before trim starts.
Limits: a list has at most 32 entries, and an empty list is a validation
error. Validation realizes a constant: default before checking it, so a list
defaulted to a constant validates the way it will run. release_pipeline on a
for_each step releases after the last member. An entry key no step reads is
a validation warning at the entry's path.
Each entry is a full generation, so quote the cost first. The listing's lists
block names the fields an entry takes and the steps over it, and its cost
carries per_entry once one entry has been measured. validate_workflow with
the arguments you will run with expands your list (not the template's
default), resolves the variables your entries name, reports a duplicate name
or a missing field at the entry's path, and answers with a plan priced for
your list (the basis values are in "The loop", step 4).
Every error carries a path in the file you wrote, not in the expanded step
list: a bad reference inside a member is reported at the for_each step's
own path, with the member it failed in named in the message.
templates/minimax/dialogue-short and templates/minimax/music-video are
this shape: each takes one shots list, and get_workflow on either shows
the entry an item needs. templates/ltx2/restore-long runs two for_each
steps over one windows list - a window_video slice, then a restore of it
- and
gather:s the restores intojoin_windows; its list length is fixed by the source, andvalidate_workflownames the entries to add or drop.
-
validate_workflow- free and instant. It reports every schema error at once, each at its JSON path, plus warnings for argument names that are not in the real pipeline signature, and aprevious_result:(orfrom_previous_result) that names no earlier step. Pass theargumentsyou will run with: an undeclared name, a value that will not coerce to the declared type, and anasset:,prompt:oroutput:reference that names nothing in this workspace each come back atarguments.<name>. Without them the verdict is about the stored definition and its defaults;checked_argumentsin the answer says which it was. -
Fix everything reported, including the warnings: a typo against a real
__call__shows up only as a warning. Signature warnings are computed only for a schema-valid draft (while schema errors remain,warningsis[]), so validate again after fixing the errors. -
save_workflow- validates again on the way in and returns the catalog metadata the saved draft will carry. -
run_workflowwithacknowledged_cost, after telling the user what it costs; without it the call is refused. The figure to tell them is theplanon the validate answer:estimate.minuteswith itsbasis, and everydownloads_requiredentry as its own line item, since weights not on this box are minutes and gigabytes the estimate does not count.basisdecides how to quote it:observed- this box's own finished runs of this shape (the cold median overrunsof them, preferred over any curated figure): "about N minutes, measured over M runs".per_entry- a measured per-entry rate re-priced for your list.catalog- a measured total for a run whose lists are the ones it was measured with.derived- (acostentry'sfixed_minutes, the share that does not scale, is held out of the scaling) that total extrapolated over a list of another length, or scaled along the steps, canvas or frames you moved off its defaults (scalednames each axis): say it is an estimate.inherited- an inline workflow priced from this box's runs of the catalog template with the same pipeline (inherited_from): say it is an approximation, and relay the warning if offload, quantization or frame count differ.other_device- a figure from another accelerator: say so.unknown- no figure. A workflow you wrote carries nocost, butlist_workflows(include_models=true)finds themodels/entry that loads the same checkpoint, and its per-image figure times the number of images is the number to quote. Say "a few minutes" only when no entry with that pipeline has been measured.
gbon adownloads_requiredentry is null when the hub could not be asked, andsteps/list_entriessay how many members the list produced. Pass the plan back asacknowledged_cost={"fingerprint": plan.fingerprint, "minutes": plan.estimate.minutes, "downloads": [...]}: the server refuses with 409 if the run's shape changed since the quote, and the refusal carries the new plan.trueis for a plan that was null. -
wait_for_jobrather than a polling loop, with the job'splan.estimate(in seconds, plus a margin) astimeout_seconds, so one call can cover the whole run. The server caps a single call (the tool's description states the cap); the reply'stimeout_applied_secondsandtimeout_cappedsay what you got, andwaited_secondshow long it took. Call again onstill_running: true. Withouttimeout_secondsa call waits 20 seconds.A running job's
progresscarries the step; the phase (loading,generating,decoding,saving) with the model inphase_detail;seconds_in_phase;seconds_since_event, the time since the last progress event; anddenoise_step/denoise_total_steps, null until the denoise loop starts. Judge a slow run from a stuck one by whetherdenoise_stephas moved since a poll minutes ago, not by silence past a fixed threshold. A nulldenoise_stepundergeneratingis the pipeline's lead-in (encoding the prompt and every reference), which emits nothing and can run for many minutes with a video reference; gaps between denoise steps are uneven where a transformer block cache is configured. Both are normal, and the model family's skill carries the measured figures.denoise_total_stepsis the schedule the pipeline actually runs: MiniMax H3's scheduler counts sigma grid points including the terminal zero, so it runs N-1 model evaluations for N (9 reports 8, 20 reports 19). That is the vendor's convention, not a dropped step. -
get_output_imageto look at what was made, and say whether it answers the request. Nothing before this step establishes that it does.get_output_frameslooks at a video andget_output_audiolistens to a soundtrack. A text-only client reads levels and clipping fromget_gallery_metadatainstead (mediapeak and mean,findings;envelope=truefor where).To confirm the words a clip speaks (a text-only client can't consume the
AudioContentblockget_output_audioreturns), runtemplates/check-scriptagainst its script.validate_workflow(name="templates/check-script", arguments={"input_audio": "output:<name>", "lines": [...]})first: it takes an audio file or a video's muxed soundtrack, andlinesis the script in order, H3 markup allowed,{text, shot}to name a line's shot (plusstart_framewhen the cut repeats that shot's name),[]for a take that should be silent. Thenrun_workflow(..., acknowledged_cost={"fingerprint": ..., "minutes": ..., "downloads": [...]})bound to that plan withwait_seconds=60,get_output_texton the result, anddelete_output(job_id=...)for the scratch run. Itsfindingsare the places to listen: a dropped or changed line (line_mismatch, withheard), a markup word spoken aloud, speech in a shot meant to be silent, a last word clipped by the cut. Words Whisper invents over silence or music are underdiscarded, not findings. A line's similarity reads number words and digits as the same (Amendment Tenheard asAmendment 10scores 1.0; a hyphenated compound is split and then folded, sotwenty-oneis21), but a spelling variant (defence/defense) still costs it, and a dropped word or two costs a long line too little to show (Nordropped from a 13-word line scores 0.96): readheardtoo, not only the score. The plan comes backbasis: "unknown"withminutes: null; quote seconds, not minutes (a few seconds per clip). -
Getting the files to the user's machine.
download_outputandexport_jobwrite on the machine runningdw.serve, which over a remote--mcpendpoint is the GPU box. The last mile of every deliverable is theurleachlist_galleryentry carries (orexport_job'szip_url)./outputsand/exportsare served without the token (only/api/*and/mcpare gated), so the header below is harmless but not required:curl -H "Authorization: Bearer $DW_API_TOKEN" \ -o exports/still.png "http://<box>:8765/outputs/ltx2/Gyre/20260910-.../still.png"Put the result under
exports/in the session's working directory - it is the user's deliverable, not a temporary file.
"Four pictures of the same thing" is the commonest shape a request takes that the catalog does not name directly, and what "the same" means decides the workflow.
- The same style, different subjects or scenes. One prompt per picture,
the same
seedon the workflow, and a shared style phrase in every prompt. A shared seed does not make the pictures alike; it makes the run reproducible. Consistency here comes from the prompts. - The same object, differing in one stated way - four spoons identical
but for colour, one mug in four glazes, a product in each colourway.
Generate the object once, then run an image-edit pass per variant with
the base step's result as its
imageand an instruction that names only the change ("make the mug red"). Separate generations, seeded or not, draw a different object every time; an edit holds everything the instruction does not mention.templates/consistent-set.jsonis this shape. - The same character in different situations. A reference rather than an
edit: an identity-referencing pipeline or IP-Adapter conditioned on one
portrait, used by every picture (
templates/ip-adapter.json,templates/multi-image-reference.json, and for video the MiniMaxreference-to-videoanddialogue-shorttemplates). Theidentity-referencedtrait in the listing marks the workflows that take one.
A request to erase or remove an object and a request to put something new in its place are different operations, and only one of them takes a prompt.
- Erase or remove an object → the
remove_objecttask: prompt-free, it fills the hole from the surrounding scene, and suits small to medium objects (a cup, a sign, a watermark).templates/remove-object.jsontakes a mask you already have;templates/segment-and-remove.jsonmasks the object by name first. A large region, such as a person filling a third of the frame, comes back smeared. - Put something new inside a mask → a prompted inpainter,
models/z-image-inpaintortemplates/inpaint.json. The prompt names what to paint. Do not use one to erase: given a prompt for "empty background" it paints a new object or a patch that does not match (#946).
docs/TASKS.md Object Removal has the task's arguments and its mask rules.
A musical number: spoken shots, each carrying its own dialogue audio, then
shots lip-synced to one song, with the song entering under the last spoken
line. A concat_videos join is wrong here - it keeps each sung shot's own
audio, the separate slices it was generated against, where the number wants
the song unbroken. The task for it is
join_into_song (get_guide("tasks", section="join_into_song")), and the recipe is four parts:
- Pick
cue_seconds: the song time that lands on the first sung shot's frame 0. The song enters that long before the cut, under the last line;0starts it exactly at the cut. It may not be longer than the dialogue. - Generate the sung shots against
slice_audioslices of the song, astemplates/minimax/music-videodoes (itsslice,shotandtrimsteps; it sliceslead_framesearly and trims the run-up off). The first slice starts atcue_seconds, and each next one starts where the one before it ended, so the slices tile the song with no gap:start_secondsiscue_secondsplus the length of every sung shot before it (num_frames / fps- 24 fps for H3),duration_secondsits own length. The shots, not the slices, are what the join reads. join_into_songwith the dialogue shots, the sung shots, the unbroken song and the samecue_seconds. It measures the joined dialogue at run time and places the song socue_secondslands on the first sung frame, ducks the dialogue under it, and discards the sung shots' own audio. It does not set the output level.normalize_audioto -3 dBFS, thenpair_audio, as every template that muxes to video does - the encoder can add up to ~2 dB, and the written-peak warning reads the file.
The tail, with the shots from earlier runs (output: or asset:) or from
earlier steps of the same workflow (previous_result:, gather:):
{
"name": "joined",
"task": {
"command": "join_into_song",
"arguments": {
"dialogue": ["asset:scene/line-1.mp4", "asset:scene/line-2.mp4"],
"song_shots": ["asset:scene/sung-1.mp4", "asset:scene/sung-2.mp4"],
"song": "asset:scene/song.mp3",
"cue_seconds": 1.5
}
}
},
{
"name": "balanced",
"task": {
"command": "normalize_audio",
"arguments": { "audio": "previous_result:joined", "peak_dbfs": -3.0 }
}
},
{
"name": "number",
"task": {
"command": "pair_audio",
"arguments": {
"video": "previous_result:joined",
"audio": "previous_result:balanced",
"fit": "video"
}
},
"result": { "content_type": "video/mp4", "subfolder": "final" }
}joined and balanced save nothing and are read by later steps, so they run
and write no file of their own; number is the deliverable, and its shot map
(get_gallery_metadata's media.shots) names one shot per input. The frame
rate comes from the shots themselves - pass fps to the join only when none
of its inputs carries one.
A run writes everything into one directory, so a finished episode sits
beside the twenty scratch files that went into it. A step's result block
can name a subfolder of the run directory for its files:
"result": { "content_type": "video/mp4", "subfolder": "final" }The convention is two names: final for a step whose output the user will
be shown, intermediate for everything else. The engine treats no name
specially and applies no default (a step that names none writes to the run's
root), but the gallery, get_job and list_gallery all carry the value, so a
consumer can tell the deliverable from the scratch without knowing the
workflow. Mark every saving step of a multi-step workflow; a one-step workflow
needs nothing. Every shipped template marks each of its saving steps, with at
least one final.
The value is a relative path of any depth (shots/act-1), may be a
variable: or, inside a for_each step, an item: reference, and follows
the output: segment rule (each segment starts with a letter, digit or
underscore; .., a backslash and a leading . are refused), so every
subfolder written is one a later workflow can name:
output:dialogue-short/latest/final/episode.mp4. A bad value is a
validation error at its JSON path. file_base_name is a name, not a path: a
separator there is refused (see Result Configuration
for how it names files).
A step that saves nothing and that no later step reads does not run at all; see A step nothing reads does not run.
A step with a workflow block runs another workflow as one step of this one,
with arguments handed down as that workflow's variables. Its path is read
the way run_workflow's workflow_path is: a catalog name as list_workflows
reports it (templates/minimax/reference-to-video), with or without .json; a
path relative to the file that names it (../models/x.json); or
builtin:name.json for the packaged fragments in dw/workflows/. A name
resolves beside the referencing file first, then in this workspace's
workflows/, then in each read-only source the server lists, so a stored
template is composed by its catalog name rather than copied (a copy stops
tracking the original). A path that lands outside every source is refused.
Declare a result (with a content_type) on the composing step and the
composed output is saved there, once, under that step's name and subfolder;
the child's own last step does not write a second copy. Without one, the child
saves its own output. The child's other steps write into the same run
directory, prefixed with the composing step's name.
validate_workflow resolves the path, so a name that reaches nothing is an
error at steps[N].workflow.path before anything is queued; it also validates
the workflow named, refuses a composition cycle, and warns about an argument
the composed workflow declares no variable for.
The catalog derives each entry's shape — one of image, image-set,
image-edit, shot, sequence, audio, text, utility — and its traits
(has-audio, chained, image-conditioned, identity-referenced,
needs-input-media, composes-workflows) from the structure of the
definition, and its summary from the first sentence of description.
So write that first sentence to say what the workflow makes and what it needs supplied, in under 120 characters — "H3 video with audio between two supplied stills" — rather than what technique it demonstrates. A first sentence longer than that is truncated with an ellipsis in every listing.
Declare shape, traits or summary at the top level only when derivation
gets it wrong; the repo's catalog tests refuse a declaration that repeats the
derivation. cost is never derived: leave it absent until a run has been
measured. A cost entry is measured at the workflow's variable defaults, and a
run that moves num_inference_steps, width/height or num_frames (per
entry in a list, summed) is quoted scaled in proportion to them, so a figure
measured at another size or step count is re-priced to the defaults before it
is written (dialogue-short's 42 min at 1344x768 is 21.2 at its 960x544). An
entry's scales_with names further drivers its figure is proportional to: a
scalar variable, or a field of a list driver's entries summed over them
(Music 3's audio_duration, summed over music-long's sections). Its
fixed_minutes is the share that does not scale (model load, encode, decode),
held out of the scaling: a moved run is quoted fixed_minutes + (minutes - fixed_minutes) x factor, so a long chain is not charged its load once per
segment (minimax/chained-segments declares 10). Absent, the whole figure
scales. An entry carrying cost has been run to completion on the device it names; one
without has only been authored, and its first run is the one that finds out
what its description could not check.
cost_drivers names the variables that move the wall clock (a frame count, a
step count, a segment count, the list a for_each runs over), never a prompt
or a seed. The server buckets its own finished runs by those values and
reports the result as observed beside the curated cost, so a 345-frame run
never informs a 124-frame figure, and a list driver buckets on its length.
With no drivers, the figure falls back to runs that overrode nothing at all, so
a measured workflow with no drivers mostly answers "unknown". A driver the
estimate cannot scale along (anything but steps, width, height, frames and
the entry's scales_with, such as a weights dtype or a LoRA name) still splits the
buckets, a text value by its exact value. When a run moves one off its
default, the curated figure is kept but carries low_confidence: true, and
differs names the drivers. Each name must be a variable the workflow
declares (tests/test_observed_cost.py sweeps the catalog for that).
A cut joined from shots can succeed and still be wrong at a seam, and the
whole-file numbers get_gallery_metadata reports cannot see inside a join.
assess_output(name) (GET /api/gallery/{name}/assess) measures that
file on the server. It decodes the file once, runs every assessment probe
that applies to it (analyze_shots, analyze_seams, analyze_sync_drift),
and returns where to look. It queues nothing: the probes use only the CPU
and run in the server process, beside whatever job holds the GPU. name is
a gallery name, output: or asset:. The shot boundaries come from what
the file's run recorded: the run manifest for an output, and the sidecar
keep_output wrote for an asset.
A last shot's num_samples a few dozen samples off round(num_frames * sample_rate / fps) is expected, not a finding - see pair_audio in the tasks guide's
Video Processing section for why.
Procedure.
- After
wait_for_job, callassess_outputon the deliverable, which is the file underfinal/.get_gallery_metadatapoints at the tool whenevermedia.shotsis set. - Read
findings. When the list is empty, no rule crossed its threshold, which is a reason to listen less closely, not a pass. - Drill into each finding at the place its
atnames. For a seam, useget_output_frames(name, seams=[n])to see it andget_output_audio(name, start, duration)to hear the second around it. For a shot, look at that shot's span. Judge it against the request. - Fix what you confirmed (see the table below), then assess the new cut.
Pass probe="analyze_seams" (or either of the other two probe names) to
get that one probe's full body: every seam's or shot's measurements, not
just the ones that crossed a rule. detail=true adds every applicable
probe's full body under probes. An unknown probe is refused before
anything is read, and the error names the three probes. Each seam's full
record also carries look measures - saturation, lightness and contrast
steps, look_ratio and detail_ratio - that catch a colour or brightness
shift jump_ratio misses; no rule reads them (the tasks guide's
analyze_seams field table).
The answer.
| Field | What it holds |
|---|---|
findings |
Every rule a measurement crossed, each {rule, severity, at, value, threshold, says}. severity is warn or info. at names the shot or seam. |
rules_applied |
The rules that were checked, so an empty findings list says which checks came back clean. |
rules_skipped |
{probe, rule, reason} for each rule that could not be measured on this file. |
not_applicable |
{probe: why} for each probe that does not apply to this file. A still has no shots, seams or soundtrack. A mute file has no levels. A file with no recorded shots has no seams. |
shots_source |
Where the boundaries came from: manifest (the run's manifest, or an asset's sidecar), or none. |
The thresholds live in one table, dw/assessment_rules.py:
| Rule | Probe | Fires when |
|---|---|---|
shot_level_spread |
analyze_shots |
the shots' RMS levels span 6 dB or more |
seam_level_step |
analyze_seams |
the shots either side of a seam differ by more than 3 dB |
seam_click |
analyze_seams |
the join peaks more than 12 dB above the audio either side |
seam_hole |
analyze_seams |
the join's floor drops below -50 dBFS while both sides are voiced (above -30 dBFS) |
seam_frame_jump |
analyze_seams |
the picture changes more than 25x as much across the seam as inside either shot (info, and skipped at a shot marked hard_cut: true) |
shot_dead_air |
analyze_shots |
the shot holds a gap more than 0.4 seconds long below its dead-air floor (skipped inside a shot that is quiet throughout) |
sync_drift |
analyze_sync_drift |
by a shot's end, the audio sits more than 40 ms off the picture |
sync_length |
analyze_sync_drift |
the soundtrack and the picture differ in length by more than 40 ms |
A shots record whose start_frame/num_frames already reaches past the
file's own length is not measured against a threshold - it is clipped to the
file before any of the above run, and that clip is itself a shot_span_overrun
finding on all three probes (analyze_shots, analyze_seams,
analyze_sync_drift), with value naming how far past the end it reached.
validate_workflow catches the same mistake before the run for a shots
argument and an asset:/literal video whose length is knowable ahead of
time; it cannot for previous_result:/output: video not yet written, so
that case is left to the finding above.
Authority. A finding marks a place to look, not a verdict. Nothing in
the engine acts on one, and no run fails because of one. A finding you have
checked and accepted is simply left alone. A seam_frame_jump at a cut the
story wanted is the cut working, and a level step into a quieter scene can
be the scene. Tell the person what you confirmed, not what the probe
reported.
compare_outputs(names) (POST /api/gallery/compare)
lays 2-8 outputs (variations, reruns, the same shot with and without a speed
setting) in one grid, a row per output and a column per moment, every output
read at the same frame number, and measures each against a baseline (default
the first). Per non-baseline output it reports PSNR (dB, higher is closer;
identical when equal), SSIM (structure on luma, 1.0 identical) and mean
absolute difference (0-255, lower is closer), at source resolution, per column
and as mean/min/max. A shared crop is the region both shown and measured; one
box per name is display only and the metrics stay on the full frame; the reply's
region says which. Outputs of a different size get no numbers (never resized).
A bare PSNR means little alone; these bands, measured on full frames, are a
reading aid, not thresholds: about 44 dB is rerun noise (same settings,
nondeterministic kernels); 23-27 dB means the numerics changed (a different
cache, attention backend or quantization) but the shot is the same; about 19 dB
means it was re-staged (the composition changed). Look at the grid before
trusting a number.
compare_outputs(names, texture_bands=true) adds, per output,
the RMS energy of the picture's texture by wavelength over the measured region:
fine (2-8 px), mid (8-32 px) and coarse (32-128 px), in three channels,
luma, chroma_rg (R-G) and chroma_bg (B-G), on a 0-255 scale. A
least-squares plane (a lighting gradient) is removed, the region is windowed,
and the energy is summed over rings of the 2D spectrum. A band longer than the
box's short side is null. Each non-baseline cell also carries texture_ratio
(output over baseline, null where the baseline band is 0). With bands, one name
is allowed: it is the baseline, so there are no ratios; use several at
moments and read the numbers across. A crop must be one shared box of at
least 16 px a side; no crop is the full frame.
finesuggests grain, edge crispness and encoder texture.midsuggests blotches, stripes and coarsening.coarsesuggests large patches and shading.- It is a measurement, not a score.
midrising can be real detail. Every band falling can be blur or over-smoothing, which the eye often reads as "cleaner". - It means something only on a box with static framing and lighting. Pick a
static region with
crop; on moving content the numbers say nothing. - Everything is measured after encode, and dw chains write only MP4, so there
are no pre-encode frames. Encoder texture sits mostly in
fine. - To A/B, branch from one clean first segment, keep the seeds, prompts and references, change one variable at a time, and compare the same moments in the same box.
A recut reruns only the join over the shots the run
already made: each entry in videos is output: + the run's
intermediate/ shot file. That is cheap, and generates nothing new. A
regenerate is a new run, so quote its plan.estimate first.
| Finding | Fix | Kind |
|---|---|---|
shot_level_spread, seam_level_step |
match_levels: "rms" (with match_levels_dbfs for the target) on the concat_videos / dissolve_videos step |
recut |
seam_click |
a longer crossfade_ms on the join |
recut |
seam_hole |
audio_bleed_ms on the join, so the outgoing tail rings on across the seam |
recut |
seam_frame_jump |
a dissolve_videos join, or regenerate the incoming shot from the outgoing shot's last frame. If the cut was meant, leave it alone |
recut, or regenerate |
shot_dead_air |
find_loop_bed on the cut names a quiet stretch inside one shot and its gain; cut that bed with slice_audio, loop_audio it to the gap's length, and mix_audio it under the line rather than leaving the drop silent |
recut |
sync_drift |
regenerate the shot. Drift inside a shot is the model's, not the join's | regenerate |
sync_length |
rerun the mux through pair_audio with fit: "video", which cuts or pads the track to the picture |
recut |
"result": {
"content_type": "image/jpeg",
"save": true,
"file_base_name": "episode",
"subfolder": "final"
}Supported content types: image/jpeg, image/png, image/webp, image/gif, video/mp4, audio/wav, audio/flac, audio/mpeg (mp3), audio/ogg, audio/opus, audio/aiff, application/json, text/plain (plus the aliases audio/x-wav, audio/x-flac, audio/mp3, audio/vorbis). Any other audio/ or video/ type is a validation error, and so is a type a browser runs as a page (text/html, image/svg+xml, application/xml and the like) (dw/content_types.py).
A task command's implementation declares what it hands back - most answer an
artifact (a file result saves in one of the media content types above),
some (judge) answer a bare scalar that cannot be saved at all, and some
(the assessment probes in TASKS.md, find_loop_bed, and
attribute_voices) answer a json document - every measurement taken, in one dict. A step on a json command must set
content_type to application/json, which saves it as one document; a step on a scalar command
may not carry a result at all. Both are checked in validation, by the
command's own declared kind rather than a name match.
subfolder places the step's files in a subfolder of the run directory - see Saying which output is the deliverable above.
For video, "fps" is the rate the file is written at. It is rarely needed:
frames that know their own rate carry it - a video read from a file or an
asset:, a concat_videos/dissolve_videos join, an interpolation - and
the engine writes them at it. Frames that bring no rate (most generations)
fall back to 8, so a workflow that assembles from bare frames should say
what they run at. A declared fps always wins over the carried one and
warns when the two differ, which is how a deliberate slow motion is written.
For audio, add "sample_rate": 44100 when the waveform doesn't
already carry a rate of its own (a declared rate always wins). Setting embed_metadata: true
on an image result embeds the step's model name and arguments as generation metadata -
PNG info chunks for image/png, EXIF UserComment (via piexif) for image/jpeg and
image/webp.
A pipeline that generates a video with its own audio track (LTX-2, or a modular pipeline
whose output asks for both videos and audio) is muxed into one video/mp4 file
with PyAV. audio_sample_rate overrides the rate the pipeline itself reports, for the
rare case it needs correcting.
Audio is written through soundfile, so both lossless and compressed containers work:
"result": {
"content_type": "audio/mpeg",
"sample_rate": 44100,
"compression_level": 0.3
}subtype— encoding subtype, such as"PCM_24"for wav and flac. Defaults to the container's own default, which is"PCM_16"for wav and flac.compression_level— 0.0 to 1.0 for flac, mp3 and ogg. Higher means smaller files.bitrate_mode—"CONSTANT","AVERAGE"or"VARIABLE"for compressed formats.
audio/opus writes an Opus stream in an ogg container, and only encodes at sample rates
of 8000, 12000, 16000, 24000 or 48000.
Output files are saved as {output_dir}/{base_name}-{result_index}.{artifact_index}.{ext},
where base_name is {workflow_id}-{step_name}.{step_index} unless the step's result sets
file_base_name, which replaces it entirely. step_index is the step's position in the
workflow, result_index counts the argument-combination iterations the step ran (see
cartesian product, above), and artifact_index counts multiple artifacts within one result
(num_images_per_prompt > 1, or a dict result saved key by key); a JSON result is one
{base_name}-{result_index}.json. In a composed child the composing step's name leads
base_name. file_base_name is one name of letters, digits, _, ., @
and -, starting with a letter, digit or underscore, so an output: reference
can name the file it writes; anything else, a path separator included, is
refused at validation, naming the character; "file_base_name": "episode" in a final subfolder writes
final/episode-0.0.mp4. The derived name is what makes two steps' files distinct, so when
you replace it on more than one step in the same subfolder, give each a different name -
otherwise the second one gets a -2 counter.
A pipeline step's configuration is dw's own vocabulary rather than the model's, so it
is a closed set: a name the schema does not declare (a misspelt offload, say) fails
validation instead of being ignored. Model-side values have blocks of their own:
from_pretrained_arguments for the constructor, arguments for the call, and
configs for a modular pipeline's block configs. A component's configuration (under
transformer, vae and the like) is not closed.
Control how models use memory:
"configuration": {
"component_type": "FluxPipeline",
"offload": "model"
}"model"— Moves entire models between CPU and GPU. Good balance of speed and memory."sequential"— Moves individual layers. Slowest but uses least GPU memory (measured at 6x the per-step cost of model offload on FLUX dev and 3.4x on Z-Image; a streamed block-levelgroup_offloadof the transformer is the usual better answer - see RECIPES_24GB.md). On MPS it is downgraded to"model"with a warning: with unified memory there is no separate pool to keep small, so the per-layer copies cost speed and save nothing.exclude_from_cpu_offloadnames components the sweep should leave alone.- Omit for no offloading (fastest, requires enough VRAM).
For components the pipeline loads itself — which is all of a modular pipeline's — use
components, applied once the pipeline is loaded:
"configuration": {
"component_type": "ModularPipeline",
"components": {
"transformer": {
"group_offload": {
"offload_type": "block_level",
"num_blocks_per_group": 1,
"use_stream": true
}
},
"text_encoder.model": {
"group_offload": { "offload_type": "leaf_level", "use_stream": true }
},
"vae": { "device": "cuda", "residency": "on_demand" },
"audio_vae": { "device": "cuda" }
}
}group_offload— streams the component between system memory and the accelerator a block or a leaf module at a time, which is what fits a component larger than the device.offload_typeis required ("block_level"or"leaf_level");onload_devicedefaults to the pipeline's device andoffload_deviceto the CPU. Anything else in the block is passed through toapply_group_offloading, souse_stream,num_blocks_per_group,low_cpu_mem_usageandoffload_to_disk_pathwork as diffusers documents them.device— moves a component that is small enough to stay resident.residency—"resident"(the default) leaves the component on its device for the whole run;"on_demand"rests it in system memory and moves it to the device only while one of its own calls runs. See On-demand components.enable_tiling— tiled decode for a decoder not namedvae(LTX-2.5'sdiffusion_decoder, for example).attention_backend— a persistentset_attention_backendon one component, which a compiled component needs (the pipeline-levelattention_backendapplies per call). Every MiniMax-H3 template wires anattention_backendvariable here, on its H3 step'stransformerortransformer_ref, defaulting to"sparse"(dw's comfy-kitchen sparse attention, which needs no extra install), withsparse_topk0.15; passsage_hubornativeto override it. See ACCELERATION.md forsparseand Attention Backends forsage_huband what it needs.sparse_tau— the sparsity threshold for a component whoseattention_backendis"sparse"(number, minimum 0, default 0.5). A runtime setting: changing it reuses the loaded pipeline. See ACCELERATION.md.sparse_topk— top-k routing instead ofsparse_tau: keep this fraction of key tiles (strictly between 0 and 1). The H3 templates set 0.15; when set,sparse_tauis ignored, and a component that sets none routes bysparse_tau. A runtime setting, likesparse_tau.attn_processor_type— the attention processor the component runs, constructed with no arguments and handed toset_attn_processor. Theunetandtransformerblocks cover those two; this covers any other component that carries attention (LTX-2.5'sdiffusion_decoder, whose default processor is a portable fallback rather than the NATTEN path the decoder was built around).compile,truncate_layers,remove_modules— see ACCELERATION.md.- A dotted key reaches a module inside a component, for a component that holds the model rather than being one.
- A
componentsblock that group offloads anything, or marks anythingon_demand, already keeps the pipeline itself off the device - the components are placed individually, so moving the whole pipeline would load it in full before the hooks and wrappers exist. Nothing extra is needed for that.
preserve_device_placement covers the case that is left: a component loaded already
placed, which must not be moved afterwards. A device_map load or a quantization that
pins its tensors to one device is the usual reason.
"transformer": {
"configuration": {
"component_type": "FluxTransformer2DModel",
"preserve_device_placement": true
},
"from_pretrained_arguments": {
"model_name": "black-forest-labs/FLUX.1-dev",
"subfolder": "transformer",
"device_map": "cuda"
}
}Renamed: this setting was
do_not_send_to_device, which is no longer recognized. In a pipelineconfigurationthe old name is a validation error; in a component'sconfigurationit is ignored and the component is moved to the device anyway.
"residency": "on_demand" sits between the two placements above. A device component
holds VRAM for the whole run, wasted on a component used twice; group offloading
streams per submodule forward, so it restreams the model once per call of every leaf -
ruinous for a VAE, whose tiled decode calls its blocks once per tile. On-demand moves the
model as a whole around each call, so a tiling loop sits inside a single pair of
transfers.
"components": {
"vae": { "device": "cuda", "residency": "on_demand" },
"audio_vae": { "device": "cuda", "residency": "on_demand" }
}The component rests on the CPU and is moved to device around whichever of forward,
encode and decode it defines, then moved back and the freed VRAM released to the
driver. Nested calls are counted, so a decode that calls forward internally is moved
once, not twice.
- Use it for a component that is large but called a handful of times - a VAE that encodes references at the start and decodes the result at the end. Freeing it for the denoise loop is the whole point.
- Not for a component called every step. A denoising transformer would pay per-call transfers 20-50 times; group offloading is the tool for those.
- Cannot be combined with
group_offloadon the same component - a group offloaded module holds one group at a time and ignores whole-model moves, so the two cannot both own its placement. Configuring both is rejected at load. - Ignored when the component's device is the CPU, where there is nothing to move it off of.
On a 24GB card, templates/minimax/reference-to-video.json peaks at 18.9GiB of reserved VRAM with on-demand
VAEs against 23.2GiB resident, and the tighter resident fit costs 40 allocator retries -
cache flushes forced by a failed allocation - where the on-demand run has none. The
headroom is also what lets the chained variant run: its later segments carry an extra
reference and need ~1.9GiB more than the first.
The same holds for the frame-conditioned workflows. Generating 124 frames at 960x544 from a keyframe, with everything else held equal:
| VAE placement | peak reserved | allocator retries |
|---|---|---|
| resident | 22.71GiB | 22 |
| on-demand | 18.03GiB | 0 |
The resident run also logs a memory mapping failed with OOM warning per retry, with as
little as 3MB free while it tries to map 20MB. It completes - the allocator flushes its
cache and succeeds on the retry - but each one is a synchronising stall, and a run that
close to the limit fails outright on any workload that needs slightly more. Every
MiniMax H3 example uses on-demand VAEs for this reason.
Example: reference-to-video.json, image-to-video.json
Pipelines stay loaded for the whole run (and across runs, in the server's persistent worker) so repeated steps reuse them. When a workflow chains two large models that cannot both fit - generate with one, upscale with another - release the first once its step completes instead of configuring offload on everything:
{
"name": "generate",
"release_pipeline": true,
"pipeline": { ... }
}The step-level release_pipeline flag unloads the step's pipeline after its results are
saved. A later pipeline_reference to a released step is an error, and the process-wide
step cache will not retain it.
What keeps a pipeline warm: the server's worker keeps a pipeline loaded across runs for as
long as what it loads is unchanged. A LoRA's scale and alpha and a scheduler's (or
audio_scheduler's) shift are applied in place on the warm pipeline, so iterating on
them costs no reload; removing (nulling) an alpha or a shift reloads, so the checkpoint
default comes back. Changing a LoRA's model_name or weight_name, the quantization,
the placement or the scheduler type reloads it.
Task models - the checkpoints behind text_generation, segment, depth_estimator and
the rest - are cached separately from pipelines, so that a step running its task once per
result does not reload the same weights on every iteration. Nothing evicts that cache
during a run, which matters when a task loads a large model on the device ahead of a
generation step: a prompt-expanding language model would hold its weights for the whole
run. release_models clears it once the step completes:
{
"name": "expand_prompt",
"release_models": true,
"workflow": { "path": "builtin:h3_context_ir.json", "arguments": { ... } }
}The flag applies to any step type, and on a workflow step it fires once the whole
sub-workflow has finished. It clears every cached task model, not only this step's, and a
later step needing one of them reloads it.
Example: enhance-prompt.json
Before the first step executes, the engine drops any step whose result no later step
reads and which writes no file, and warns once per dropped step saying which and why.
That is how a template whose portraits can be supplied as asset: files skips the steps
that would have drawn them (#122). Elision follows from the definition, never from a
value produced during the run, so it is decided at validate time too.
Four things keep a step:
- it saves - a
resultwith acontent_type, andsavenotfalse."save": falseis how a step says it is scaffolding. - it is the last step - it is the run's answer, whatever it declares.
- something reads it -
previous_result:/from_previous_result(includingprevious_result:step.property), agather:(which is a list of those by the time this runs), apipeline_referencenaming it, or areused_componentsentry naming a component it shares. - a kept step reads it - elision is transitive, so dropping a step can drop the step it read in turn.
release_pipeline on an elided step moves onto the last surviving step before it when
that step loaded the same pipeline, and release_models moves unconditionally, so the
release still happens. The plan a
validate call answers with is computed after elision, so steps, downloads_required
and the cost it quotes are the work that will actually happen, and it lists what was
dropped under elided_steps; the run manifest records the same list.
If a step you expected to run is named in the warnings, the usual cause is a reference
to it spelled wrong somewhere later, or a step that was meant to declare a result.
"configuration": {
"vae": {
"enable_slicing": true,
"enable_tiling": true
}
}enable_slicing— Process VAE in slices to reduce memoryenable_tiling— Tile large images through the VAE
Attach one or more LoRAs to a pipeline with loras, a sibling of configuration:
"loras": [
{ "model_name": "XLabs-AI/flux-RealismLora", "adapter_name": "realism", "scale": 0.8 },
{ "model_name": "user/other-lora", "weight_name": "lora.safetensors", "subfolder": "loras" }
]model_name— the LoRA's hub repo, required.nullswitches the entry off: it is not loaded and the step runs without it. A template'sloraslist is fixed JSON, so this is how a caller drops its adapter - pass the variable behindmodel_nameasnull(MiniMax H3:lora_model_name). Validation and the run both warn (lora_disabled). A step-distillation (turbo) LoRA sets the step count and sigma shift the step runs at, so raise those to the base model's schedule with it or the step runs undertrained.weight_name/subfolder— pick a specific weights file within the repo.adapter_name— name passed toset_adapters(). Defaults to the LoRA's index in the list.scale— the adapter's weight, passed toset_adapters(). Defaults to1.0.
See workflows/templates/lora.json for a full example.
"ip_adapter": {
"model_name": "h94/IP-Adapter",
"weight_name": "ip-adapter_sdxl.bin",
"scale": 0.6
}model_name is required; weight_name, subfolder and scale are optional. The
adapter image itself is passed as a normal ip_adapter_image pipeline argument. See
workflows/templates/ip-adapter.json.
Two pipeline steps that load the same underlying component (a shared text encoder, for instance) can avoid loading it twice:
"configuration": { "component_type": "FluxPipeline", "shared_components": ["text_encoder"] }"configuration": { "component_type": "FluxPipeline", "reused_components": ["text_encoder"] }The step naming shared_components stores those components after it loads; a later step
naming the same names in reused_components gets them instead of loading its own copy.
The names must match exactly between the two steps. Either list can sit in the step's
configuration or beside it on the pipeline itself.
How the component reaches the second pipeline depends on what kind it is. A standard
pipeline takes it as a from_pretrained argument. A modular pipeline cannot — it is
built from the component specs in its own index — so it is registered with
update_components() before load_components() runs, which is also what keeps
load_components() from pulling a second copy: it only loads what is not already there.
That is what lets two MiniMax-H3 steps of different tasks (t2va and ref2va load
different transformer partitions) share the 14GB text encoder and the VAEs between them.
A reused component keeps the device placement the step that shared it gave it. Any
components entry naming one is skipped with a log line rather than applied a second
time — offloading hooks do not survive being installed twice, and the step that loaded
the component is the one that decided how it is placed.
Sharing outlives the pipeline that did it: a step can share a component and still set
release_pipeline, which frees everything else it loaded while the shared component
stays alive for the steps that reuse it.
"configuration": {
"component_type": "FluxPipeline",
"attention_backend": "flash_hub",
"enable_attention_slicing": true,
"no_generator": false
}enable_attention_slicing— process attention in slices to reduce memory at some cost to speed. Automatic on MPS unlessdisable_attention_slicingis set (UNet/ControlNet models only; worth disabling for SDXL on a Mac).attention_backend— selects a diffusers attention backend (e.g."flash_hub") for the duration of each pipeline call."sparse"is dw's own instead: comfy-kitchen sparse attention on a MiniMax-H3 transformer, installed for the call only, and dense with a warning where it cannot run. See ACCELERATION.md.sparse_tau— the sparsity threshold for a pipeline-levelattention_backend: "sparse"(number, minimum 0, default 0.5). See ACCELERATION.md.sparse_topk— top-k routing for a pipeline-level"sparse"instead ofsparse_tau(strictly between 0 and 1); unset unless the workflow sets it.prompt_weighting— enables A1111-style prompt weighting ((word:1.5),[word],((word))) and prompts over 77 tokens. Currently supports Flux pipelines. Mutually exclusive withremote_text_encoder.no_generator— settrueto skip creating atorch.Generatorfor pipelines that don't accept one.inversion— run the pipeline'sinvert()instead of the pipeline itself; the step returns the inverted/image latents for a later step to consume (see community-pipeline.json).generate— run the pipeline'sgenerate()instead, for components with a generation head (the step returnsgenerated_ids).
A cache block speeds up inference by skipping redundant computation:
"configuration": {
"cache": { "type": "first_block", "threshold": 0.05 }
}cache wraps diffusers' own cache hooks - type is one of first_block, faster,
mag, taylorseer or text_kv, each with its own tuning fields (threshold,
num_inference_steps, max_skip_steps, retention_ratio, cache_interval,
max_order, mag_ratios, calibrate, is_guidance_distilled — see dw/workflow_schema.json for which
fields apply to which type). See
workflows/templates/step-caching.json.
cache can also be a whole-object variable: reference, which must resolve to a cache
object or to null. A variable resolving to null drops the key: no cache, and the only
true "off", since an installed first_block hook changes the output even when it skips
nothing. The substituted object is checked against the cache shape at validate, so
{"type": "bogus"} or a bare "first_block" is refused, naming the field. Every
MiniMax-H3 template exposes a cache variable (default null) on its H3 step:
"arguments": {"cache": {"type": "first_block", "threshold": 0.05}}.
Device is auto-detected (CUDA > MPS > CPU). Dtype is set per-component:
"from_pretrained_arguments": {
"model_name": "black-forest-labs/FLUX.1-dev",
"torch_dtype": "torch.bfloat16"
}A step can name a device instead, in a pipeline configuration (which becomes the
default for that pipeline's components), in a component configuration, or in a task's
arguments. A device naming a backend the machine running the workflow does not have is
translated to the one it does, with a warning, so a workflow written on a CUDA box runs
on a Mac and back again:
"configuration": {
"component_type": "FluxPipeline",
"device": "cuda"
}Only the backend is translated. A device index survives when the backend matches, so
cuda:1 on a single-GPU CUDA box is an error; when the backend does not match, the index
is dropped and the warning says so. "device": "cpu" is never translated, since pinning
a step to the CPU is how a GPU-specific problem gets ruled out.
Modular pipelines (ModularPipeline and its subclasses) load their configuration and
their component weights separately, so from_pretrained_arguments only names the model
and load_components pulls the weights:
"configuration": {
"component_type": "MiniMaxMusic3ModularPipeline",
"load_components": { "dtype": "torch.bfloat16" },
"components_manager": { "enable_auto_cpu_offload": true }
}-
load_components— arguments forload_components(). Usedtypefor the component dtype andnamesto load only some of the components.quantization_configis keyed by component name, since a modular pipeline loads each component itself:"load_components": { "dtype": "torch.bfloat16", "quantization_config": { "transformer": { "configuration": { "config_type": "TorchAoConfig" }, "arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig", "modules_to_not_convert": ["proj_in", "proj_out"] } }, "language_model": { "configuration": { "config_type": "transformers.TorchAoConfig" }, "arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" } } } }
A component the map does not name loads unquantized. Note which
TorchAoConfigeach component takes: the diffusers one for its own models, the transformers one for a transformers model such as a conditioner. -
configs— values the pipeline's blocks declare and read while they run. They are neither components nor call arguments, which is why they have a block of their own:"configs": { "canvas_short_edge": 768, "reference_image_short_edge": 1024 }
The names are whatever the pipeline itself declares, so they differ per model rather than being a fixed list here — MiniMax-H3 declares
canvas_short_edge(768),canvas_max_pixels(1032192) andreference_image_short_edge(2048), the last being the resolution its image references are encoded at. A name the pipeline does not declare raises rather than passing quietly, since a dropped config reads as a setting that did nothing. -
remove_blocks— names of top-level blocks to pop off a modular pipeline's block graph once it is built (blocks.sub_blocks.pop). It is for a block that always runs and has no input to turn it off. FLUX 3 Klein'sprompt_upsampleris one: it rewrites every caption and needs an 18.8 GB component, so every Klein definition declares"remove_blocks": ["prompt_upsampler"]and the explicit upsample step is the only upsampler. A name the pipeline has no block for raises at load, listing the blocks it has (validate does not build the pipeline, so it cannot see this). -
components_manager— attaches aComponentsManager, which tracks the pipeline's components. Withenable_auto_cpu_offloadit keeps only the running components on the device and moves the rest to system memory, reservingmemory_reserve_margin(default"3GB") of free device memory. It requires a device that reports free memory (CUDA) and replacesoffload, which modular pipelines do not support.
A modular pipeline returns whatever its output argument asks for — one output by name,
or several of them together:
"arguments": {
"prompt": "variable:prompt",
"output": ["videos", "audio", "sampling_rate"]
}Asked for several, the outputs come back keyed by name. Video generated with its own
soundtrack is muxed into a single video/mp4 file, the same way a video pipeline's own
output is, and a later step can still reference any of the outputs by name.
Some repositories hold more than one task's weights. workflow names the task, which
prunes the pipeline to the blocks that task runs, so only the components it needs are
downloaded and loaded:
"from_pretrained_arguments": {
"model_name": "MiniMaxAI/MiniMax-H3",
"workflow": "t2va"
}A task is chosen by the arguments the step passes, so one workflow name can cover more
than one of them: MiniMax-H3's fl2va takes an image, a last_image, or both. Given
only a last_image it generates up to that frame, inventing everything that leads to
it — see workflows/templates/minimax/last-frame-only.json beside
workflows/templates/minimax/first-and-last-frame.json.
See workflows/templates/minimax/music.json and workflows/templates/minimax/video-with-audio.json for full examples.
A MiniMax-H3 step (t2va, fl2va or ref2va) takes a pipeline argument hold_audio:
an audio value, spelled asset:<file>, output:<file> or previous_result:<step>.audio
(an AudioTrack or AudioVideo). The video is generated to that soundtrack. The
track goes through H3's Ref2VA reference-audio encode, is written over the target audio
rows and held clean as conditioning through denoise, so the model draws picture that
fits audio it is not allowed to change:
"pipeline": {
"arguments": {
"prompt": "a drummer in a lit rehearsal room",
"num_frames": 125,
"hold_audio": "asset:track.wav"
},
"output": ["videos", "audio", "sampling_rate"]
}"audio" must be in output (as it already is in video-with-audio) for the held
track to be what the step returns. A track longer than the video is cropped, a shorter
one padded with silence. The step's audio is the caller's original waveform fitted to
the video's duration (num_frames / 24 s) at its own sample rate, not a VAE round trip.
previous_result:base.audio holds a track an earlier step generated, music for example.
run_workflow's arguments can set only a declared variable, so unless the template
has one, add "hold_audio": "variable:hold_audio" and a hold_audio variable, or edit
the argument. Validation refuses hold_audio on a step that is not H3, on a modular
pipeline loaded with a workflow other than t2va, fl2va or ref2va, on a path whose
extension is not .wav, .mp3, .flac or .ogg, and on any {"media_type": ...}
dict; a run refuses a pipeline with no hold blocks or a value that is not audio. Without
hold_audio the output is unchanged.
Hold plus the reference is the music-video default: the template passes each shot's slice
as a MiniMaxH3AudioReference and its hold_audio variable defaults to
previous_result:slice; set it to null for the reference alone. The match_audio chains
(chain-matched-to-audio, chain-matched-and-aligned) keep hold_audio null by default,
since a chain has no per-shot slice to name; set it to an audio file to hold. What was
measured:
- Reference plus hold keeps the audio in sync (#808,
music-video, 4 shots of 124 frames at two seeds,measure_syncagainst each source slice): 0.00 s offset on all eight shots, correlation 0.95 to 1.00. The reference alone put one shot 1.37 s late and another 0.18 s early, correlation 0.35 to 0.96. That measured the audio, not the mouth. - Hold with image-only references keeps the soundtrack exact but weakens the singing (#619, a 10 s sung track): the mouth was open at about 3 of 6 sung-word onsets against 6 of 6 with the audio reference alone, and read as speech to camera.
- On
fl2va, hold is the locked-frame lip-sync route (#985, a first frame plus a held 5.17 s song): frame 0 kept the picture's pixels and the mouth followed the song as in an audio-referenced Ref2VA shot, at about a fifth of the denoise time of Ref2VA[keyframe completion + audio reuse], which keeps only the framing.
Hold is also for picture that must fit audio exactly, such as motion cut to music.
refine_strength uses hold differently, to keep the base pass's own jointly generated
audio (see the refine section below).
A MiniMax-H3 t2va or fl2va step takes a pipeline argument guides: a list of
{"video": <reference>, "frame": <pixel frame>}, where the video is an asset:,
output: or previous_result: reference. Each guide clip is VAE-encoded on the
generation canvas and placed as condition rows, timed from the target frame it lands on,
that are never denoised, so the generated video is held to earlier footage. On fl2va
the image / last_image keyframes combine with guides. The clip is fitted to the
canvas (scaled to cover, centre-cropped). By default a guide holds video only and its own
audio is not used; "audio": true holds the guide's audio too (below).
"pipeline": {
"arguments": {
"prompt": "variable:prompt",
"num_frames": "variable:num_frames",
"width": "variable:width",
"height": "variable:height",
"num_inference_steps": "variable:num_inference_steps",
"guides": [
{"video": "asset:opening.mp4", "frame": 0},
{"video": "asset:closing.mp4", "frame": 85}
],
"output": ["videos", "audio", "sampling_rate"]
}
}This is the text_to_video_audio step of
workflows/templates/minimax/video-with-audio.json
with guides added. As with hold_audio, run_workflow's arguments can set only a
declared variable, so reference a guides variable or edit the argument.
| Rule | Value |
|---|---|
frame |
a multiple of 17 (a VAE chunk boundary): 0, 17, 34, ... |
| clip length | 1, 5 or 17m + 5 frames (22, 39, 56, ... 124); any other length is cut down to the longest such length (n < 5 gives 1, 5 <= n < 22 gives 5, else 17 * ((n - 5) // 17) + 5) with a warning |
| extent | frame + length must not run past num_frames (H3 rounds it up to 17n + 5; default 124); ending exactly at the end is fine |
| count | at most 4 guides per step. Each guide is also checked against the card: each guide frame adds attention rows, and the template's vram_estimate charges each guide's frames on the canvas; guides: [] is the same as none |
| where | t2va or fl2va on an H3 pipeline; not with references (ref2va), not on a non-H3 pipeline |
Validation refuses all of the above before the run (dw/guides.py); a previous_result:
guide is checked at run time, when its clip exists.
A guide with "audio": true holds its soundtrack as well as its frames:
{"video": "asset:opening.mp4", "frame": 0, "audio": true}. The guide video's audio over
the guide's span is encoded by the audio VAE (posterior mode, as ref2va reference audio
is) and placed as condition audio rows timed at the target's own audio time, then held,
never denoised. The output's audio under the guide reproduces the guide's audio, and the
model generates the rest to run on from it. Audio runs at 40 latents a second against
24 fps video, so a guide of n frames at frame k holds round((k + n) * 40 / 24) - floor(k * 40 / 24) audio latents: 37 for a 22-frame guide at frame 0, 65 (about 1.6 s)
for a 39-frame one. A guide that does not start at frame 0 has its audio padded to the
audio VAE's hop first, so its latents land at the right time. The clip is read as 24 fps
footage, as its frames are.
| Rule | Value |
|---|---|
audio |
true or false; absent is false. Any other value is refused |
| soundtrack | "audio": true on a guide whose video has no audio stream is refused - before the run for an asset:/output: file, at run time for a previous_result: clip |
with hold_audio |
allowed; a guide's audio rows come first, then the held track's |
A guide holds its span's look as well as its composition and motion: the frames under it come out close to a copy of the clip, whatever the prompt says about style. A full-length guide at frame 0 reproduces the take rather than restyling it (a 124-frame guide prompted as "a hand-painted watercolour animation" came back photoreal, #648). The prompt steers only the frames the guides leave free; restyling a whole take needs a LoRA trained on guides (#612).
With no guides (or guides: []) the step runs the stock layout and draws the same noise
from its seed as an unguided step. Repeat runs at one seed are bit-identical with the
default cudnn_deterministic: true and cudnn_benchmark: false (ACCELERATION.md,
TF32 and cuDNN). A server that turns benchmark on, or determinism off, can give two runs
that match frame for frame by eye but whose audio levels differ by a few dB.
A guide is held almost exactly, so the take freezes under it and leaves the guide's end
with a visible change of pace. guide_noise (an H3 t2va/fl2va argument, a number in
[0, 1), about 0.4 to start) loosens every video guide on the step: each guide frame gets
Gaussian noise of standard deviation amplitude x 255 on its 0-255 pixels, where the
amplitude falls on a cosine taper from guide_noise at the guide's first frame to 0 at
its last, so the frames nearest the new material are held clean. The noised frames are
encoded once, under the same encode seed as a clean guide; the noise draw is seeded per
guide (keyframe_encode_seed + the guide's index), so a seeded run repeats. The rows are
still held, never denoised, and the row plan, the refine schedule and guide audio
("audio": true) are unchanged. guide_noise: 0 (or absent) adds no noise, and the
latent is bit-identical to a step without it.
It is a step argument, so in a continuity: "guide" chain it rides to every segment and
loosens the carried guide too.
| Rule | Value |
|---|---|
| value | a number in [0, 1); anything else, or a value outside the range, is refused naming the range. Accepts a variable: reference |
| where | MiniMax-H3 t2va or fl2va, where guides run; refused on ref2va, on a step that passes references, and on a non-H3 step |
| no guides | a warning, not a refusal: a step with no guides and no continuity: "guide" chain has nothing to noise |
A chain on an H3 t2va or fl2va step can carry its seam with a guide instead of a
keyframe: "chain": {"segments": 3, "continuity": "guide", "guide_frames": 22}. Every
segment after the first gets the previous segment's last P = guide_frames frames as a
guide at frame 0, appended to the step's own guides, with "audio": true when
carry_audio is (the default), so motion and voice run on across the seam rather than
restarting from a still. The carried clip counts toward the four-guide limit, so a step
that writes its own guides may list at most three; validation says so before the run. On fl2va, image is set to the guide's first frame. The
next segment opens with a near-copy of those P frames, so P frames are trimmed from its
head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 is 328.
| Rule | Value |
|---|---|
guide_frames |
22 (default) or 39 - a whole-latent guide length; anything else is refused |
| where | MiniMax-H3 t2va or fl2va; refused on ref2va, with references, or on a non-H3 step - guides stay off ref2va |
carry_frames |
refused - the carry is always the last guide_frames frames |
guide_backoff |
0 (default) or a multiple of 17: the carry is the guide_frames frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs num_frames above P + back-off. Refused off guide continuity and with match_audio or duration_seconds |
keystone |
none (default) or segment_1: on fl2va, every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain. The guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Refused on last_frame (there the keyframe is the only carry), and any other value is refused naming the two |
keystone_slot |
with keystone: "segment_1", where segment 1's frame 0 sits in each later segment. first (or null, the default) is plain keystone, the keyframe above. last makes it the fl2va last_image instead and the keyframe is the guide's first frame again: on a static two-minute take it held the backdrop and the face with no reset pop, and the subject eases back to the opening pose at each segment's end (bench/records/structural.md, Keystone slot). last needs a step image and refuses a step last_image of its own. {"frame": 17j} is a one-frame guide at that frame, past the guide's frames and inside the segment (a match_audio or duration_seconds plan's shortened last segment included); its enter_below_sigma gate is refused unless DW_EXPERIMENTAL_REFINE_WINDOWS=1. The name experimental_keystone_slot is refused naming this one. templates/minimax/chained-segments sets last under shot: static_long; under last, keystone_images replaces segment 1's frame 0 with each segment's own picture (the next row). Experimental, refused unless DW_EXPERIMENTAL_REFINE_WINDOWS=1: latent writes segment 1's frame-0 latent (its own output latent; on a resume, its decoded frame 0 encoded again) over each later segment's guide latent 0 for the whole denoise, with segment 1's frame 0 as the keyframe too, so the two condition rows at time 0 agree; the guide's own latent 0 is put back after the loop, before decode. It combines with experimental_latent_carry, a held track and experimental_capstone |
keystone_images |
the moving keystone (docs/proposals/complete/narrated-long-take-complete.md, Amendment 1): with keystone: "segment_1" at keystone_slot: "last", one image per segment, in order, and segment k's fl2va last_image is entry k - 1 in place of segment 1's frame 0 - segment 1 included - so each segment ends on its own picture, such as a keyframe drawn along a walk, rather than being pulled back to the opening still. A list of image paths, asset:/output: references, URLs or previous_result: references (each its first image), or one previous_result:<step>.<key> naming a list of images: an image step's images, one per for_each entry, the first image of each (resolved like prompts and lines). With anchor_every, a later segment off the cadence gets no last image; on a probe (stop_after) and a resume_from, entry k stays segment k's. Refused, naming the key: without keystone: "segment_1" and keystone_slot: "last"; an entry that is not an image; off an H3 fl2va step; with a step last_image of its own; and fewer entries than the planned segments (the match_audio plan, segments or duration_seconds; the full plan's even for a probe, which is continued), naming both counts - at validate when the count can be told there, else before segment 1 renders. A held narration (match_audio with a step hold_audio) places them the same way, its guides carrying picture only. templates/minimax/narrated-take takes them as keyframes under shot: walk_keyframed |
anchor_every |
Experimental, refused unless DW_EXPERIMENTAL_REFINE_WINDOWS=1: the keystone and experimental_capstone apply only on later segments whose 0-based index i has i % anchor_every == 0 - 3 anchors segments 3, 6, 9 (by index) and leaves the rest a plain guide chain. A whole number from 1, default 1 (every later segment); refused on a chain with neither keystone: "segment_1" nor experimental_capstone |
carry_pick |
{"sharpest": N}, N from 1 to 8, with last_frame continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones; match_audio and duration_seconds are refused (the plan covers the length up front and the pick drops frames per seam, so the video would fall short - use segments); and when num_frames is known N must be less than num_frames minus trim_frames, or the pick could empty a segment |
look_hold |
true or {"pickup", "creep", "texture", "precompensate"} booleans, any model family: pixel corrections on each later segment's frames before its carry is taken (below). On a guide chain pickup and precompensate do not apply (the carried guide is a whole clip, left alone); an explicit true for either there is refused, and true alone runs creep and texture only |
trim_frames |
ignored (the trim is P), with a note in the job log |
crossfade_ms |
ignored when carry_audio is true: the guide held the audio, so the seam is joined with only a declick |
num_frames |
must exceed P, or the chain cannot progress |
guide_noise |
rides on the step's arguments to every segment, so segments 2+ noise the carried guide (above) |
latent_anchor works inside the H3 latent; look_hold works on the decoded frames, so it suits any
family. Each segment after the first is corrected before its carry is taken, so the
carried keyframe and the stitched video are the corrected frames (applied once, not
again to the stitched output). Segment 1 is the reference and is left as rendered.
"look_hold": true turns all four on; an object turns on the ones it names. The template's
look_hold variable is typed bool by its default, so arguments takes true or false
only: an object there is refused, naming the variable. The per-flag object form is reachable
only by editing the workflow (an inline copy of the template):
pickup: matches the segment's opening (frame 1; frame 0 is the keyframe copy) to the carried frame by a per-channel quantile map on YCbCr, carries the same map to the last frame and blends it per frame. Skipped on a flat frame, and on a jump too large to be drift (a cut).creep: when the last frame's contrast or saturation exceeds segment 1's by more than 0.03, scales both down by a factor running from 1 at the first frame to the needed gain at the last. Gains are at most 1: a darker or paler shot is untouched.texture: when either end's faint texture (the flat-area detail, not edges) exceeds the roughest of segment 1's first, middle and last frames by more than 0.03, subtracts up to half of it, easing in across the shot.precompensate: divides the carried keyframe's contrast and saturation by the boost measured so far (the running mean of the rendered opening's look over the fed keyframe's, each clamped to 1..1.25), so a model that boosts its own keyframe lands back on the look.
pickup and precompensate act on one carried keyframe, so they need continuity
last_frame; the carried frame of a guide chain is a whole guide clip and is left
alone. Absent or false, the chain is unchanged bit for bit. Concepts from
Smite79/MiniMax-H3-LongVideos (ATTRIBUTIONS).
A long H3 chain drifts in colour and contrast: each segment's latent statistics wander
a little from the last, and the carry passes the drift on. latent_anchor (an H3
t2va/fl2va argument) takes video frames - a path, an asset:/output: reference,
or a step's frames - VAE-encodes them on the render's canvas the way a guide is
encoded, and takes each latent channel's mean and standard deviation over every
position. After denoising and before decode, the generated video latent is moved onto
them, channel by channel: (x - mean_x) / std_x * std_a + mean_a. The audio is never
touched. By default it is a full match; latent_anchor_strength (a number in [0, 1],
default 1) blends it with the untouched latent, x + strength * (matched - x), so 0.5
moves each channel halfway and 0 changes nothing. Without latent_anchor the
step is unchanged, bit for bit, and its job log says nothing about an anchor. A call
that applies one logs a line naming its source, such as
segment 2/3: latent anchor applied - latent statistics from segment 1.
In a chain, "anchor": "segment_1" sets it for you: every segment after the first
gets segment 1's frames as its latent_anchor, so the whole chain is held to the
first segment's look. Segment 1 itself is not corrected. "anchor_strength": 0.5 sets
latent_anchor_strength on those segments, for a chain that should hold its look
without flattening a deliberate change in light.
| Rule | Value |
|---|---|
anchor |
none (default) or segment_1; anything else is refused, naming the values |
anchor_strength |
a number in [0, 1] (default 1, the full match); refused outside it, and refused without anchor: "segment_1". A step's latent_anchor_strength is the same number, and needs a latent_anchor |
segment_1 continuity |
last_frame or guide; refused with any other |
| where | MiniMax-H3 t2va or fl2va, where guides run; latent_anchor and segment_1 are refused on ref2va, on a step that passes references (that call runs ref2va), and on a non-H3 step |
segment_1 carry |
refused with segment_argument: "references" - the later segments would run ref2va |
| frames | cut to a whole-latent length (1, 5 or 17m + 5) |
refine_tiles |
refused beside latent_anchor, as the other per-call H3 arguments are |
workflows/templates/minimax/chained-segments.json takes continuity (default
guide, measured best for every shot type; last_frame is the uncommon case) and
guide_frames as variables. The ref2va chains (chain-video-continuity and the
match_audio chains) keep last_segment.
Once a 960x544 MiniMax-H3 take reads the way it should, upscale_h3_latents and
decode_h3_latents promote it to 1344x768 without denoising it again: only a small
3D-convolution network and a VAE decode run, not the transformer. With no refine pass the
result is sharper than the 544p take but cannot show detail the base pass never generated;
the refine section below adds one. Measured on a 124-frame crowd scene, it took 7.8 min
against a native 768p render's 12.7 min, and the faces came out soft and waxy where the
native render's were distinct. When faces matter, use the catalog's native 768p render,
templates/minimax/video-with-audio-768p.
Target width/height must be multiples of 16, each between 1x and 4x the base
latents' own size, and within the upscale cap of 1920x1088 (or portrait 1088x1920);
validation refuses a literal size off the grid or over the cap before the run. Above
1344x768 the plate is for a tiled refine (Refining above 768p in
tiles), and upscale-only it is soft. The latents
pass from base to up to decode in memory. A previous_result: property
on a modular step names a key of the dict its output list returns, spelled as the
pipeline spells it: base.latents, base.audio and base.sampling_rate (not
sample_rate, which the dict does not carry). A property no result carries is an
error, so a misspelt one fails the step. The base step carries no result, so the 544p
pass is not written.
The upscaler weights (LBH-123-AI/Minimax_h3_latent_Upscaler, read at a pinned
revision) and the H3 VAE download on first use. While the upscaler file is not cached, a
validate's plan.downloads_required lists it as a file entry ({repo, filename, revision, ...}); the VAE is covered by the pipeline step's MiniMaxAI/MiniMax-H3 entry.
{
"id": "H3LatentUpscalePreview",
"description": "Promote a MiniMax-H3 take from 960x544 to 1344x768 in latent space, decoded without refinement.",
"variables": {
"prompt": "prompt:minimax/fox_dawn_context_ir",
"num_frames": 124,
"num_inference_steps": 9,
"video_shift": 12.0,
"audio_shift": 3.0,
"weights_dtype": "{int4}",
"lora_scale": 1.0,
"lora_alpha": null,
"lora_model_name": "lightx2v/Minimax-h3-Turbo",
"lora_weight_name": "minimax_h3_fl2v_turbo_8step_v1.0_bf16.safetensors",
"lora_adapter_name": "turbo",
"seed": 42
},
"variable_constraints": {
"num_frames": {
"modulus": 17,
"remainder": 5,
"min_frames": 124,
"max_frames": 345,
"snap": "up",
"reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps"
}
},
"seed": "variable:seed",
"steps": [
{
"name": "base",
"pipeline": {
"configuration": {
"component_type": "ModularPipeline",
"pre_load_modules": [
"sdnq"
],
"load_components": {
"dtype": "torch.bfloat16",
"quantization_config": {
"transformer": {
"configuration": {
"config_type": "sdnq.SDNQConfig"
},
"arguments": {
"weights_dtype": "variable:weights_dtype",
"quantization_device": "cuda",
"return_device": "cpu",
"use_quantized_matmul": true,
"dequantize_fp32": false,
"modules_to_not_convert": [
"proj_in",
"audio_proj_in",
"context_embedder",
"time_embedder",
"time_proj",
"token_refiner",
"norm_out",
"proj_out",
"audio_proj_out"
]
}
},
"text_encoder": {
"configuration": {
"config_type": "sdnq.SDNQConfig"
},
"arguments": {
"weights_dtype": "variable:weights_dtype",
"quantization_device": "cuda",
"return_device": "cpu",
"dequantize_fp32": false,
"modules_to_not_convert": [
".model.visual",
"lm_head"
]
}
},
"vae": {
"configuration": {
"config_type": "sdnq.SDNQConfig"
},
"arguments": {
"weights_dtype": "{int8}",
"quant_conv": true,
"use_quantized_matmul_conv": true,
"quantization_device": "cuda",
"return_device": "cpu",
"dequantize_fp32": false
}
}
}
},
"components": {
"transformer": {
"group_offload": {
"offload_type": "block_level",
"num_blocks_per_group": 2,
"use_stream": true,
"record_stream": true,
"low_cpu_mem_usage": true
}
},
"text_encoder": {
"remove_modules": [
"lm_head"
]
},
"text_encoder.model": {
"truncate_layers": {
"language_model.layers": 51
},
"group_offload": {
"offload_type": "leaf_level"
}
},
"vae": {
"device": "cuda",
"residency": "on_demand"
},
"audio_vae": {
"device": "cuda",
"residency": "on_demand"
}
}
},
"from_pretrained_arguments": {
"model_name": "MiniMaxAI/MiniMax-H3",
"workflow": "t2va"
},
"loras": [
{
"model_name": "variable:lora_model_name",
"weight_name": "variable:lora_weight_name",
"adapter_name": "variable:lora_adapter_name",
"scale": "variable:lora_scale",
"alpha": "variable:lora_alpha"
}
],
"scheduler": {
"shift": "variable:video_shift"
},
"audio_scheduler": {
"shift": "variable:audio_shift"
},
"arguments": {
"prompt": "variable:prompt",
"num_frames": "variable:num_frames",
"width": 960,
"height": 544,
"num_inference_steps": "variable:num_inference_steps",
"output": [
"videos",
"audio",
"sampling_rate",
"latents"
]
}
}
},
{
"name": "up",
"task": {
"command": "upscale_h3_latents",
"arguments": {
"latents": "previous_result:base.latents",
"width": 1344,
"height": 768
}
}
},
{
"name": "decode",
"task": {
"command": "decode_h3_latents",
"arguments": {
"latents": "previous_result:up"
}
}
},
{
"name": "mux",
"task": {
"command": "pair_audio",
"arguments": {
"video": "previous_result:decode",
"audio": "previous_result:base.audio",
"sample_rate": "previous_result:base.sampling_rate",
"fit": "video"
}
},
"result": {
"content_type": "video/mp4",
"subfolder": "final"
}
}
]
}To keep the 544p take beside the promotion for comparison, add one more pair_audio
step with "video": "previous_result:base.videos" and the same audio/sample_rate
(and fit), saved with "result": {"content_type": "video/mp4", "subfolder": "intermediate"}. base.videos is the pipeline's batch - a list holding the one video -
and pair_audio unwraps a batch of one (a batch of several is refused, since one track
goes under one video). The H3 step's videos carry the model's 24 fps, so pair_audio
fits the track against 5.17 s and the file is written at 24 fps without being told.
Frames that carry no rate (a plain list from a task) need pair_audio's own fps
argument: fit measures the frames before the writer applies a declared rate, so
without it the track is padded to the 8 fps fallback's length, three times too long.
upscale_h3_latents followed by decode_h3_latents adds no detail. To let the
transformer add some, pass the upscaled latents to an H3 step as latents with
refine_strength. The step adds noise to the video latents up to sigma = refine_strength
and denoises from there down to 0, so the picture is kept and the fine detail is
regenerated at 1344x768. The refined video is decoded by the step itself, so there is no
decode_h3_latents step, and the step's videos is the deliverable.
Refine re-denoises the video only. The audio rows have to be held, so pass the base
pass's track as hold_audio (previous_result:base.audio) and the step's audio is that
track. This is a different use of hold from the one above: it keeps the audio the base pass
generated jointly with that video, rather than fitting picture to supplied audio. For lip
sync to supplied audio see the hold section above (#619, #795).
The workflow is the one above with decode and mux replaced by a refine step.
refine carries a copy of the base step's whole pipeline block - the same
configuration, from_pretrained_arguments, loras, scheduler and audio_scheduler,
so the loaded model is reused - with these arguments, and the step saves the result:
{
"name": "refine",
"pipeline": {
"arguments": {
"prompt": "variable:prompt",
"num_frames": "variable:num_frames",
"width": 1344,
"height": 768,
"num_inference_steps": 5,
"output": ["videos", "audio", "sampling_rate"],
"latents": "previous_result:up",
"refine_strength": 0.2,
"hold_audio": "previous_result:base.audio"
}
},
"result": { "content_type": "video/mp4", "fps": 24, "subfolder": "final" }
}num_inference_steps is the number of sigma points, ending at 0, spaced by the
scheduler's shift, so a refine runs num_inference_steps - 1 denoise evaluations. Here
5 points at shift 12 (the example's video_shift) and strength 0.2 are sigma 0.2, 0.157, 0.110, 0.058, 0: 4 evaluations. The time a refine takes scales with num_inference_steps. refine_strength
sets where it starts, not how many steps run: a higher strength moves the picture further
from the upscaled take, a lower one stays closer to it.
Validation refuses a refine_strength that is not a number between 0 and 1 (exclusive;
a regional edit also takes 1.0), a
refine with no latents or no hold_audio, a step that is not H3, and a
num_inference_steps below 2.
To refine a canvas above 1344x768, such as a 1920x1088 plate, see Refining above 768p in
tiles. To give the schedule's sigma points yourself rather
than spacing them by shift, see Explicit refine sigmas.
A recipe published with its own schedule cannot be reached by spacing points with
shift. refine_sigmas gives the schedule itself: the sigma points, from
refine_strength down to 0, used verbatim for both the video and audio schedules, with no
shift applied. NVlabs' Open-H3-Regenerate-2K recipe starts from
(1 - 0.909375)·up + 0.909375·noise and runs 3 Euler steps on four points:
"arguments": {
"latents": "previous_result:up",
"hold_audio": "previous_result:base.audio",
"refine_strength": 0.909375,
"num_inference_steps": 4,
"refine_sigmas": [0.909375, 0.857143, 0.666667, 0]
}refine_strength still sets the re-noise, and the list replaces only the spacing. The
step count is the number of points, so num_inference_steps is required beside the list
and must equal its length: four points are 3 denoise evaluations. Validation, and the
check before the call, refuse a refine_sigmas that:
- is not a list of numbers, or has fewer than 2 points;
- is not strictly decreasing;
- does not start at
refine_strength(within 1e-6); - does not end at exactly 0;
- has a length other than
num_inference_steps, or comes with nonum_inference_steps; - comes with no
refine_strength.
A step without refine_sigmas runs the shift-spaced schedule. refine_sigmas works with
refine_tiles too: every tile runs the same list.
To change part of a take and keep the rest, pass a video as source_video on an H3
t2va, fl2va or ref2va step. It is a path, asset:, output: or previous_result:
reference to a video, and it stands in for the latents of a refine: the step VAE-encodes
it over the generated rows and re-denoises from refine_strength, which is required and
in (0, 1]: 1.0 regenerates the masked region from pure noise, and about 0.9 replaces an
object in it (see Choosing refine_strength for a regional
edit). (Without a source_video the refine's range stays (0, 1).) With no
mask and a hold_audio, the whole source is refined under that soundtrack. With no mask
and no hold_audio, the call is not a refine at all: it is the audio pass (see Audio over
a finished picture).
{
"source_video": "asset:fox_dawn.mp4",
"mask_image": "asset:fox_mask.png",
"mask_grow": 1,
"refine_strength": 0.9,
"hold_audio": "asset:fox_dawn.mp4",
"prompt": "the fox holds a pine cone in its mouth, same light and motion"
}A mask_video (one frame per frame) or a mask_image (one mask for every frame) limits
what is regenerated. Pass one, not both. White regenerates and black is kept, binarized
at luma 128. The mask is max-pooled onto the transformer's patch grid, 16 px times the
patch size per cell, so any white pixel marks its whole cell, and mask_grow widens it by
that many cells (a whole number, 0 or more, default 1). In time the grain is the latent
frame: about 4 pixel frames each (3.4 - every 17 frames encode to 5 latent frames), times
the patch's time size, so a mask_video switching on or off mid-cell marks the whole cell. After every denoise step each
other row is written back as the source noised to that step's sigma, with the refine's own
noise, so at the end it is the source latent. The decode still round-trips it through the
VAE, so kept pixels are near-identical to the source, not bit-identical. A higher
refine_strength replaces what the mask covers, a lower one keeps its layout.
The useful range sits high. Measured on one take (a 960x544, 124-frame H3 shot of a woman
holding a cup, the mask over the middle of the frame around her and the cup, mask_grow 1), seed 1062, 5 turbo steps,
video_shift 6, with a prompt asking for a bright red raincoat:
refine_strength |
Inside the mask | Inside-mask SSIM vs the source |
|---|---|---|
| 0.85 or less | A light rework: re-detailed, the source's objects and colours stay | 0.86 at 0.8, 0.83 at 0.85 |
| 0.9 | A held object is replaced (the cup became a domed sundae cup); the same person, pose and clothing | 0.78 |
| 0.95 | Clothing is reworked too (a hooded navy jacket) and the object replaced; face, hair and pose mostly hold | 0.65 |
| 1.0 | A fresh generation: the prompt lands in full, but a different person and composition | 0.44 |
Outside the mask held at every strength (SSIM about 0.98 on a crop outside it). So: 0.9 to replace an object, 0.95 to rework clothing, 0.85 or less for a light rework. A large recolour of a big region, such as a dark top to a bright red coat, did not land below 1.0, and 1.0 gives up the region's identity: that is a fresh generation, not a regional edit.
Why the numbers sit high: refine_strength is the starting sigma σ₀ on the shifted
grid, the same as for any refine (and refine_tiles). The scheduler's shift maps the
unshifted position u to σ = shift·u / (1 + (shift − 1)·u), so the start point on the
unshifted schedule is u = s / (shift − (shift − 1)·s). At video_shift 6 that is
0.8 → 0.40, 0.85 → 0.49, 0.9 → 0.60 and 0.95 → 0.76: a strength of 0.8 re-noises less than
its number suggests. The bands above hold for those conditions only. Another
video_shift or step count moves them, the way the formula says: a lower shift brings u
closer to s, so the same change comes at a lower strength. A refine_sigmas
list is run as given, with no shift, so its points are the schedule itself.
The source and the mask must match the step's width x height, and a source_video or
mask_video its num_frames. A step that leaves width, height or num_frames out renders
the source's. An all-black mask is refused, as it would regenerate nothing. Validation reads
a source or mask that resolves already (a path, asset: or output:) and refuses a size,
frame count or empty mask there, the way it reads guide videos; a previous_result: is
checked at the call.
hold_audio is still required with refine_strength, since the refine holds the
soundtrack (a source_video with neither a mask nor hold_audio is the audio
pass instead). It is normally the source_video itself: a video path (.mp4, .mov, .mkv,
.webm, .avi) is held by its own soundtrack, so the edit keeps the take's audio. A video
with no audio stream is refused. An audio file (.wav, .mp3, .flac, .ogg) or an
earlier step's audio-carrying result works too.
Validation and the call both refuse:
source_videowithlatents, or on a chained step;- both mask kinds,
mask_videoandmask_image, on one step; - a mask with no
source_video, ormask_growwith no mask; source_videowith a mask orhold_audioand norefine_strength, or one outside (0, 1];source_videoon a pipeline that is not H3 (mask_imagethere is left alone, since inpaint pipelines own that name);- a
source_videowhose frame count (and so the step'snum_frames, inferred or given) is off the17n + 5grid: the pipeline would snap the count up and pad the source, and the mask would drift in time - trim or pad the clip to the grid first (the message names both neighbours). An inferredwidthandheightstill have to meet the family's canvas rules (multiples of 32), which the template's variables carry and an inferred size does not check; - any of it with
refine_tiles, which does not combine with a regional edit.
A source_video with no mask and no hold_audio is an audio pass: the picture is
finished and only the soundtrack is generated, from the prompt - its soundscape, its music
and its <d> lines. Every video row of the source is held as the exact clean latent at
every step, pinned clean in the row plan the way a reference row is, so no video row is
ever stepped. The audio runs the stock audio schedule from full noise, attending to the
real picture from the first step. The step returns the source's own frames, not a VAE
round trip of them, under the new track. A silent source is the usual case and is fine.
{
"source_video": "asset:silent_take.mp4",
"listen_scale": 1,
"prompt": "... overall_soundscape: ... <d>[English] Over here.</d> ...",
"num_inference_steps": 12,
"output": ["videos", "audio", "sampling_rate"]
}Run it without a distillation LoRA, on the dialogue schedule (num_inference_steps 12,
video_shift 12, audio_shift 3): the turbo LoRA is what makes H3 voices sound
digital (#984). listen_scale (1, 0.5 or 0.25, default 1) encodes the source at that
fraction of its canvas, area-resampled and snapped to 32 px, which is cheaper per step.
The scale keeps the source's aspect: the shorter side is snapped and the longer derived from
it. The audio's length still follows the source's duration, and the picture returned is the
full-size source. width, height and num_frames stay the source's, as for an edit.
The result keeps the source's own frame rate. A source at 24 fps is conditioned on as it
is. A source at another rate (30 fps, say) is conditioned on a copy retimed to H3's 24 fps
over the same duration, nearest frame, its count snapped up onto the 17n + 5 grid. The
step returns the source's own frames at the source's own rate, with the track trimmed to the
source's duration, so its own frame count need not be on the grid. The rate comes from the
file or the step result that supplied the video, or else from the length of its soundtrack.
A source with neither is refused.
Validation and the call refuse refine_strength, refine_sigmas or refine_tiles on an
audio pass (no video row is re-denoised; pass hold_audio for a whole-source refine
instead), a listen_scale other than those three, a listen_scale on a step that is not
an audio pass (on any pipeline, H3 or not), and an audio pass on a chained step or beside latents. The template
is templates/minimax/add-sound.
What it is for, by ear (Don, 2026-11-06, bench/records/audio.md): at listen_scale 1 a
line the source's picture was generated with lands on the mouth ("very good lip sync").
At 0.5 lip sync is still acceptable, about 23% faster: the fastest scale that holds it.
At 0.25 lip sync is lost, so use it only for a clip with no on-screen speech. It is not a
re-voice: the picture is the source's own pixels, so a new or regenerated line over an
on-screen speaker has no mouth drawn for it and plays as dubbed voice-over. Use it to
sound a silent clip, or to replace an off-screen or voice-over line.
H3's refine runs at 1344x768 at most, and upscale_h3_latents reaches 1920x1088 (or
1088x1920). An upscale-only plate above 768p is soft: upscale_h3_latents to 1920x1088
alone, decoded, is not a 1080p render, since the interpolation adds no detail. To put the
detail in, give the refine step refine_tiles, which refines a latents canvas bigger than
H3's own in overlapping tiles. It is a second argument to refine_strength, not a
replacement: the strength, the held audio and the step count mean what they did above.
{"refine_tiles": {"width": 1344, "height": 768, "overlap": 256}}The step plans the tiles on the 32 px grid: the fewest per axis whose overlap is at least
the requested one, spread evenly, the last flush to the edge. A 1920x1088 canvas at
1344x768 with overlap 256 is 2x2, tiles at x 0 and 576, y 0 and 320. The noise is drawn
once for the whole canvas from the step's generator and cropped per tile, so overlapping
tiles start from the same noise. Each tile is denoised through H3 without its decode block,
and the prompt is encoded once. The tiles are blended in latent space with a linear feather
across each overlap, the weights normalized, and the blended canvas is decoded once with VAE
spatial tiling. The step returns the full-canvas latents, the video and stage 1's held
audio, once. Its videos carry H3's 24 fps, so the pair_audio below writes them at
that rate without being told. The job's events show one log line per tile and one decoding phase.
It is slow. A 1920x1088 refine is roughly 4 tiles, each a 768p refine, plus a 1080p decode: about 12 minutes measured on an RTX 3090 for 124 frames (the recipe below, with its 544p base), against about 9.6 minutes for a native 768p take. The validate projection prices the denoise at the tile's size and the decode at the canvas's, so a tiled refine does not warn as if it denoised 1080p at once.
The workflow is the refine one above with four changes. The variables set the 768p turbo
LoRA and its schedule: "num_inference_steps": 5, "video_shift": 6.0 and
"lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors" (as
workflows/templates/minimax/video-with-audio-768p.json does, alpha left to the file);
base and refine both read them. up targets "width": 1920, "height": 1088. The
refine step's arguments drop width and height and add refine_tiles, and it saves
nothing itself. A mux step pairs the refined video with the held audio and saves it:
{
"name": "refine",
"pipeline": {
"arguments": {
"prompt": "variable:prompt",
"num_frames": "variable:num_frames",
"num_inference_steps": "variable:num_inference_steps",
"output": ["videos", "audio", "sampling_rate"],
"latents": "previous_result:up",
"refine_strength": 0.2,
"refine_tiles": { "width": 1344, "height": 768, "overlap": 256 },
"hold_audio": "previous_result:base.audio"
}
}
},
{
"name": "mux",
"task": {
"command": "pair_audio",
"arguments": {
"video": "previous_result:refine.videos",
"audio": "previous_result:base.audio",
"sample_rate": "previous_result:base.sampling_rate",
"fit": "video"
}
},
"result": { "content_type": "video/mp4", "fps": 24, "subfolder": "final" }
}The refine step carries no width or height: the canvas is the latents. Naming them is
allowed, and they must equal the canvas, which is the latent height x width x 16.
Validation and the run refuse, naming refine_tiles:
refine_tileswithoutrefine_strength, or without what refine needs:latents,hold_audio, an H3 step andnum_inference_stepsof at least 2;- a
width,heightoroverlapthat is not an integer multiple of 32; - a tile larger than 1344x768 (or 768x1344), or larger than the canvas;
- an
overlapbelow 192, or at or above half the tile's shorter side; - a reference image or video, a
<Picture N>, a keyframe orguideson the step; - an explicit
widthorheightthat does not match thelatentscanvas.
Experimental, off by default. experimental_refine_windows: {"frames": W, "overlap": O} (latent frames) on a refine_strength step refines the take in overlapping time
windows fused at every step. It needs DW_EXPERIMENTAL_REFINE_WINDOWS=1 in the server's
environment and a server restart, and a step that sets it without the flag is refused.
Only such a step may be longer than H3's 15 s. Every other H3 call keeps the ceiling.
A take over 15 s is three steps:
encode_h3_latentswithsourceset to the finished 24 fps take (output:orasset:). It reads and encodes one 17-frame VAE clip at a time, and the result is bit-identical to the whole-take encode. It returnslatents,num_frames(the take padded with its last frame up to the next 17m + 5) andsource_frames(the take's own count).- The H3 refine, with
latents: previous_result:encode.latents,num_frames: previous_result:encode.num_frames,refine_strength,experimental_refine_windowsandhold_audioset to the take's own soundtrack (hold_audioreads a video's track, so the take's.mp4can go in directly). The held track is encoded whole, and that encode has no ceiling of its own. Put"latents"in the step'soutput. decode_h3_latentswithlatents: previous_result:refine.latentsandnum_frames: previous_result:encode.source_frames, which trims the padding off so the take comes back at its own length.pair_audiothen puts back the take's own soundtrack, which issource_frameslong.
A source_video over 15 s is refused on a windowed step, and the refusal names
encode_h3_latents: that path would encode the whole take as one float32 tensor.
Video pipelines generate short clips - a chain block on a pipeline step runs the
pipeline once per segment and stitches the segments into one long video. The model
loads once; each segment's last frame is carried into the next segment as its
keyframe, the duplicated boundary frames are trimmed, and frames and audio are
joined into a single file:
"pipeline": {
"configuration": { "component_type": "LTX2ImageToVideoPipeline" },
"from_pretrained_arguments": { "model_name": "Lightricks/LTX-2.5-Diffusers" },
"chain": {
"segments": 3,
"trim_frames": 2,
"crossfade_ms": 80
},
"arguments": { "prompt": "variable:prompt", "image": "variable:image" }
}segments— how many times the pipeline runs. Total length is roughlysegments * num_frames, minustrim_framesper seam.match_audio— instead of a count, derive the length from the step'shold_audiotrack, or else the one audio reference in itsreferences. The audio is sliced into frame-aligned per-segment chunks, each segment is generated against its slice - a held track's slice is that segment'shold_audio, a reference's replaces the reference - and the final video is muxed with the original, unsliced track - so the soundtrack has no seams at all. Undercontinuity: "guide"a held track's slice already starts at the guide span, so the guide carries video only:carry_audiois forced off, and the job log says so once. Validate warns on a chain step withhold_audioand nomatch_audio(every segment holds the whole track from its start) and on a held chain whosenum_framesis not divisible by 3: H3's audio latents run at 40 a second against 24 fps, so the on-grid17n+5lengths are 141, 192, 243, 294 and 345. Requiresnum_frames(the per-segment length) and a frame rate, and a track: a step with neitherhold_audionor areferenceslist is refused. One ofsegments,match_audioorduration_secondsmust be given, andsegmentsbesidematch_audiois refused (besideduration_secondsit is ignored). Validation asks both rules of the substituted block, so avariable:value is caught before the model loads.duration_seconds— instead of a count, ask for the chain by its length: the plan coversround(duration_seconds * fps)frames (the same plannermatch_audiouses, so the last segment shrinks to what remains, snapped byframe_snap), and the stitched video and its generated audio are both trimmed to exactly that length, shot records included. Needsnum_framesin the step's arguments and a frame rate (chainfpsor aframe_rateargument). Refused withguide_backoffandcarry_pick, which drop frames the up-front plan cannot count. A cost estimate scales by the segment count it implies. It overridessegmentswhen both resolve (the run logs thatsegmentswas ignored), so the chained templates, whoseduration_secondsvariable defaults to null (a null variable drops the key), take justduration_seconds=20. It is refused withmatch_audio.continuity— how continuity carries across segments.last_frame(the default) extracts each segment's last frame and passes it to the next segment - single-frame conditioning, which carries pose and colour.last_segmentcarries the previous segment itself (frames and its generated soundtrack) into the next as a video reference, which also carries motion, camera, and voice across the seam; it requires asegment_argumentthat takes a references list.guide(MiniMax-H3t2va/fl2vaonly) lays the previous segment's lastguide_framesframes into the next as a held frame-0 guide (see H3: holding a clip withguides, Chaining with a guide).guide_frames— withguide, how many frames the guide carries: 22 (default) or 39.guide_backoff— withguide, a multiple of 17 (default 0): carry the frames that many before each segment's end and drop that tail at assembly. Not withmatch_audio.anchor—none(default) orsegment_1(MiniMax-H3t2va/fl2va, withlast_frameorguidecontinuity): every segment after the first is rescaled to segment 1's per-channel latent statistics before decode, holding the chain's colour and contrast (see Anchoring a chain's colour).keystone—none(default) orsegment_1(withguidecontinuity only): every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame (see Chaining with a guide).keystone_slot— withkeystone: "segment_1":first(default),last(segment 1's frame 0 as every later segment's fl2valast_image) or{"frame": 17j}, plus the experimentallatent(segment 1's frame-0 latent over each guide's latent 0) (see Chaining with a guide).keystone_images— withkeystone: "segment_1"atkeystone_slot: "last": one image per segment, segment k's fl2valast_imagein place of segment 1's frame 0 (the moving keystone; see Chaining with a guide).anchor_every— experimental: apply the keystone andexperimental_capstoneonly on every n-th later segment (0-based indexi % n == 0); default 1 (see Chaining with a guide).anchor_strength— withanchor: "segment_1", how far that match goes: 0 to 1, default 1 (the full match).carry_frames— withlast_segment, bound the carry to the last N frames of the segment (the audio is cut to the same span). Unset carries the whole segment.carry_audio— withlast_segmentorguide, whether the carried reference includes its soundtrack (defaulttrue).segment_argument— where the carried frame or reference lands:image(default) for image-to-video pipelines, orreferencesfor reference-conditioned modular pipelines, where it is appended alongside the workflow's own.trim_frames— image-to-video pipelines reproduce their keyframe as frame 0, so this many frames are dropped from the head of every segment after the first (default 1). The matching audio is used as crossfade material, so video and audio stay exactly in sync. It also bounds the crossfade window:trim_frames / fpsseconds (at 24 fps,trim_frames: 2allows the full default 75 ms fade).crossfade_ms— equal-power crossfade applied to generated audio at each seam (default 75). Not used withmatch_audio, which keeps the original track.fps— frame rate for the chain's audio math. Defaults to the pipeline'sframe_rateargument; pipelines with a fixed rate need it set (MiniMax H3: 24).frame_snap— the constraint the pipeline puts onnum_frames, used to snap the finalmatch_audiosegment to a valid length. MiniMax H3 accepts17n+5frames between 124 and 345:{ "modulus": 17, "remainder": 5, "min_frames": 124, "max_frames": 345 }. Where the workflow already declares that rule as avariable_constraintsentry, write"frame_snap": "constraint:num_frames"instead, so the numbers live in one place (What a variable is allowed to be). Aduration_secondsormatch_audiochain that declares noframe_snapderives one when the step'snum_framesis avariable:whose constraint has amodulus(the run logs that it did); when it cannot - a literalnum_frames, or no constraint - validation warns atchain.frame_snap, since the shortened last segment is otherwise asked for a count the pipeline may floor in silence.prompts— optional per-segment prompt list for narrative progression; segmentiusesprompts[min(i, len - 1)]. The whole list may be oneprevious_result:<step>.<key>naming a JSON list of strings (anarration_linesstep'sprompts); a result that is not a list of strings is refused at run time, naming the step and key.lines— optional list of strings, one expected spoken line per segment, exactly as many as segments (unlikeprompts, the last is not repeated; a wrong length or a non-string is refused). Read only by thewordscheck. Segmentk's line islines[k], else the<d>dialogue ofprompts[min(k, len - 1)], else of the step'sprompt. An empty string, or no<d>, means no line: the shot recordswords: nulland the segment is not re-rolled for it. Validate warns whenlinesis set but the words check is off. Likeprompts,linesmay be oneprevious_result:<step>.<key>naming a list of strings (anarration_linesstep'slines); validate accepts it unresolved and the run checks its length against the plan.check— check each segment after the first before it is kept (any model family):truefor the default ceilings, an object to set any of them, absent orfalsefor no check (the chain is unchanged). See Checking each segment below.retries— withcheckon, how many times a segment that fails is re-rendered, 0 to 3 (default 0). Above 0 withoutcheckis refused.save_segments— write each completed segment to the output directory as a playable mp4 and free its frames, bounding memory to roughly one segment regardless of chain length. The final video is streamed from the segment files at save time, and they are removed once it is written (keep_segments: trueretains them). A crashed chain leaves the finished segments behind - stitch them by hand by listing their paths in aconcat_videosstep (trim_frames: 0, the trim was already applied), or resume the chain from them (resume_from). Requires PyAV and a frame rate. The trade-off is one extra encode/decode cycle through h264 for the segment files. Both chain templates set it on.resume_from— resume a chain that died part way: anoutput:reference naming one segment file an earlier run of the same chain left. A chain that fails after spilling a segment says so in achain_resume_hintjob warning quoting the first file's reference (output:<workflow>/<run id>/<prefix>.0.segment-000.mp4); a failed run's files resolve by name, with or without its manifest. Rerun with the same settings and that reference: its finished segments become segments 1..k and only k+1..N render, which the estimate prices (N - k segments; a reference that does not resolve yet is priced for every segment and says so). That scales a figure the quote already has - the catalogcostor this server's history of the workflow at the same drivers; a workflow with neither (a template never run here, a differentsegments) staysbasis: unknownunderresume_fromas without it. A chain whose segment count the estimate cannot tell, or a reference keeping N or more segments, is not scaled anddiffersnames it. Each kept file must hold the frames the plan puts on the timeline for its segment, or the run is refused naming the segment and both counts - a resume never changes the plan. The files are copied, not moved, into the new run's spill, so the new take stands on its own andkeep_segmentsnames them on its shots (withresumed: true). The carry is read off the last kept file: its last frame (last_frame), or its lastguide_framesframes and their audio (guide). That first resumed seam's audio guide is the file's decoded AAC track, not a live carry, which the run'schain_resumedwarning ("Resumed k of N segments from ...") says.anchorandkeystoneread segment 1 from the first kept file; the kept segments are not checked again, and the first rendered one is checked against the last kept one as usual.look_holdreads its scene from the first kept file and is fed the last one's carry, but its learned boost starts again. Needssave_segments: trueand continuitylast_frameorguide; refused as a bare path. Under the experimentalexperimental_latent_carrythe first resumed seam's guide is the last kept file's decoded frames, encoded again (the carried latents are not on disk), which thechain_resumedwarning says (reencoded_guide: true); later seams carry latents again. The kept seams' audio is joined with the short declick fade the chain gives a seam with no head material, not thecrossfade_msblend: a kept file holds only its on-timeline audio, so the blend material is gone (the first rendered seam is crossfaded as usual). All kept files must carry audio or none may, and the rendered segments must agree with them, or the run is refused rather than laying a track shorter than the picture. Under aprevious_resultfan-out each iteration resumes from its own files: the reference names the iteration the hint quoted, and every iteration takes the<prefix>.<its iteration>.segment-NNN.mp4siblings beside it. An iteration with none there, or one that finished (a complete set it is not the named one of), renders in full, which achain_resumedwarning withkept: 0says.stop_after— render a probe: only the plan's first k segments (a whole number of at least 1; at or past the plan's count the whole take renders). Needssave_segmentsandkeep_segmentstrue. See Probe, then continue below.
Probe, then continue. stop_after: k leaves the plan the full take's - the same
segment count, num_frames and guide grid, so the same narration plans the same
segments - and renders segments 1..k only. The take is muxed with the match_audio
track cut to the rendered frames (a duration_seconds take's generated audio is cut
the same way), the run logs chain_stopped_after ("Chain stopped after k of N
segments"), and every shot carries planned_segments: N, so the manifest says the take
falls short. Watch the probe; to continue it, run the same chain again with
stop_after cleared and resume_from naming the probe's last segment file
(output:<workflow>/<run id>/<prefix>.0.segment-<k-1>.mp4, the segment_file of its
last shot): its k files become segments 1..k and only k+1..N render, through the resume
rules above. A stop_after still at or under k is refused before anything is copied,
since it would render nothing. The first resumed seam is not a live carry: its guide (or
keyframe) is decoded from the probe's last file - h264 frames, and for a guide chain
its AAC audio unless the step holds its track (hold_audio, which carries no guide
audio) - which the chain_resumed warning says. A probe's job history is left out of
this server's observed cost, as a resume's is. The estimate quotes a probe for the
segments it renders: k of N of the take's figure (segments k0+1..k of N for a probe
that resumes k0 kept ones), and for a match_audio chain, whose N comes from the track,
k - k0 times the per-segment figure; stop_after may be a digit string
(python -m dw.run ... stop_after=12).
The chain runs inside one iteration of the step, so it composes with
previous_result fan-out (three keyframes in, three chained videos out), and a
pipeline_reference step can carry its own chain. Seeds behave like a normal run:
the step's generator advances across segments, so one seed reproduces the whole
chain (a check re-roll runs on a generator of its own, below). Expect some visual drift across many segments with last_frame continuity -
it is single-frame conditioning; last_segment continuity exists for exactly that,
where the pipeline can take a video reference.
See workflows/templates/ltx2/chained-segments.json, workflows/templates/minimax/chained-segments.json, and workflows/templates/minimax/chain-matched-to-audio.json (audio-matched lip-sync of arbitrary length).
One bad segment of a finished H3 chain can be re-rendered in place instead of
re-rolling the chain, when the chain kept its segment files (save_segments and
keep_segments, which the H3 chain templates set from their keep_segments variable):
workflows/templates/minimax/repair-segment.json
renders segment k once with segment k-1's last frames as a guide at frame 0 and segment
k+1's frames as a guide ending at the render's end, both holding their audio on a chain's
own track, then splices the new frames into the take itself (chain_take). Outside the repaired span the
output is the take: its frame count, its picture and its soundtrack, every other seam's
blend included. The new frames also replace the first 12 frames of segment k+1 (the
17-frame grid's remainder, chain_repair_cut). The two
repaired seams are not cuts, so analyze_seams checks them as it checks a chain's own.
The last segment is repaired with the head guide only, the render ending at the take's
end; the first needs a one-sided repair this template does not make. The chain's
settings are passed to it as arguments - tier sets canvas, steps, weights, LoRA and
shifts together, and its narrated_standard and narrated_best are the narrated take's -
and the seed is new by default. For a match_audio take (a narrated take) the cut slices
the take's own track under the render's frames and the render holds it (hold_audio), so
the redrawn mouth follows the words heard; the guides then carry picture only, and the
master track stays muxed unchanged.
With check on, each segment is measured before anything of it reaches the stitched
video, its audio, its shot record, a save_segments file or the look-hold state. Segment
1 has no seam, so it is measured for audio_dbfs and words only:
| Measure | Against | Default ceiling | Fails when |
|---|---|---|---|
jump_ratio |
the previous segment: the largest frame change across the seam over the larger of the two segments' own typical change, floored at 2 grey levels (analyze_seams's measure) |
jump_ratio_max 4 |
above the ceiling |
look_ratio |
the previous segment: the largest saturation, lightness or contrast step across the seam over that measure's own rate inside either segment (analyze_seams's measure) |
look_ratio_max 5.5 |
above the ceiling |
texture_ratio |
segment 1: the mean fine-band (2-8 px) luma texture of 4 evenly spaced frames, whole frame | texture_ratio_max 2 |
above it or below its inverse (doubled or halved); null skips it |
cut_ratio |
the segment itself: its largest frame change over its own typical change, floored as jump_ratio is - a cut inside the segment, past the seam's windows, which the seam measures do not see |
cut_ratio_max 3.5 |
above the ceiling; null skips it |
audio_dbfs |
the segment itself: the RMS level (dBFS, to 0.1) of its generated audio over the kept span, past the head trim; segment 1 included | audio_dbfs_min -70 |
below the floor; null skips it. Skipped (no level) when the segment has no generated audio or the audio is held: match_audio or its hold_audio, or a step hold_audio argument. A guide chain carrying its audio is measured: the guide holds only the span the head trim drops |
words |
the segment's expected line (see lines): its kept audio transcribed (Whisper, on CPU) and compared with check_script's similarity; segment 1 included |
words_similarity_min null (off) |
below the minimum, a number above 0 and at most 1, or on any of the findings line_clipped_at_end, transcript_loop, tag_spoken, unscripted_speech at any similarity; the re-roll log and the chain_check_failed message name the findings and what was heard (words 1.0 with unscripted_speech, heard '...'). Off by default: bench/records/words.md found no clean separation of good takes from bad. chain_check.WORDS_SIMILARITY_MIN (0.80) is the ceiling that passed every good take seen, and catches only gross failures (a wrong or missing line, a garbled segment). Never measured when the chain holds its audio (match_audio, hold_audio), and skipped for a segment with no line |
face_drop |
segment 1: its median SFace similarity (analyze_face_drift's scoring) of 8 evenly spaced frames from frame 1 on to its own frame 0, minus this segment's median to that same frame 0; segment 1 itself is not measured |
face_drop_max null (record only) |
above the ceiling, a number above 0. Warn-only, like texture_ratio: recorded and warned, never re-rolled. null (unmeasured) when segment 1's frame 0 or this segment holds no face, or the face models cannot load |
A segment whose prompt asks for a cut inside it fails cut_ratio and re-rolls, so set
check: {"cut_ratio_max": null} to turn that measure off. A step that asks for silence
fails audio_dbfs the same way: set "audio_dbfs_min": null. The floor sits between the
-87.8 to -89.2 dBFS that dropout (silent) segments measured and the -57 dBFS of the
quietest real one. Unlike texture_ratio, a failed audio_dbfs re-rolls. The re-roll
draws fresh noise but is not guaranteed to bring sound back: on the one repro measured
(LTX-2.5 on mps), both seeds read about -89 dBFS and the re-roll cost about 3 minutes.
The chain_segment_silent warning reports the segment either way.
"check": true uses those ceilings; "check": {"jump_ratio_max": 20} overrides the ones
it names. face_drop is measured on every checked segment and recorded on its shot as
check ({face_drop}), but fails nothing until face_drop_max is set: no bench has yet
set it. A checked chain lists the YuNet and SFace weights it reads in the plan's
downloads_required. Lip sync is not a segment measure: on 70 real H3 dialogue segments
no threshold separated a segment's own audio from audio 0.5 s late or another segment's
(bench/records/narrated.md, Sync calibration), so judge sync by eye, and use
measure_lip_sync as a coarse indicator over a whole take.
lip_sync_min is refused as an unknown key. The jump and look ceilings are twice the
99th percentile of the bench's unchecked S1-S3 chains (bench/records/noise-floor.md);
the cut ceiling sits between the 2.98 most of 282 good bench segments read and the 4.16 of a cut 26 frames past a seam (bench/records/check.md); the texture ceiling is still provisional, because those chains' own texture ratio is the
drift it exists to catch. A static segment's own
motion is ~0, so the jump is divided by the floor rather than by ~0. Segment 1 has no seam
and is checked for audio_dbfs, when the floor is not null and the audio is
generated, and for words, when words_similarity_min is not null and the segment has a
line. (check_segment has a hard_cut exemption that skips every measure, audio
included, but run_chain does not pass it: a chain's own seams are never cuts, so every
later segment is checked.)
A re-roll renders a whole segment again, so re-rolls count in the step's timing segments.
A segment that fails is rendered again from the same carry and the same arguments, at a
new seed: attempt n (0 is the first) runs at the step's seed plus n x 1000003, on a
fresh generator, so the step's own generator - and every later segment - runs as it
would have. A workflow that names no seed draws one before the run and records it, so a
re-roll is repeatable from the manifest. The first attempt that passes is kept. When all
1 + retries fail, the chain keeps the one with the fewest failed measures, then the
lowest jump_ratio, and finishes - it never restructures the run. texture_ratio is
warn-only: texture drift is in the carry, so a fresh seed from the same carry inherits
it. face_drop is warn-only too, until a bench shows a fresh seed fixes it. A segment that fails only warn-only measures is kept as rendered, recorded in
check_failed and warned, and not re-rolled; one that also fails another measure re-rolls as above,
and choosing the kept attempt still counts every failed measure. The kept attempt's
frames and audio are the ones stitched, and the seam is crossfaded against them.
Each shot of a checked chain records attempts; a re-rolled one records the seed its
kept attempt ran at, and one that failed every attempt records check_failed (the
measures its kept attempt failed). The job gets one warning (kind: "chain_check_failed") per such segment, naming the measures, their values and the kept
attempt, and the job log names each re-roll. When words failed, the warning also carries
heard, similarity and findings (values.words is the similarity); when every
attempt failed words it adds every_attempt_failed: true and a note to repair the
segment with templates/minimax/repair-segment rather than retry more, since the cause
may be in the carry, which a fresh seed inherits. A shot of a words-checked chain also
records words (similarity, heard, findings, passed, seam_head_dropped, first_word_start) for
the kept attempt, or null with no line. A failed words on a segment with a head trim
(a guide chain's 22 guide frames) also transcribes the audio the trim cut off; if the
line's first heard word starts inside that span the findings carry line_clipped_at_head
(#1300): the opening words are cut from the delivered video and a fresh seed starts the
line at the same place, so move the line later in the prompt instead of re-rolling. A segment that passed
but whose kept audio opens past the line's first words (a carried segment, or one with a
head trim) carries a line_opening_missing finding (#1308: missing names the words,
at the first heard one) and fails words_opening without failing words, so it
re-rolls while retries remain; when every attempt carries it the job gets a
chain_check_failed warning naming the finding, with every_attempt_failed: true. For
a long monologue use a narrated take. Among attempts failing only words,
best_attempt keeps the higher similarity; a words-only failure ranks after any other
single-measure failure. Each shot also records audio_dbfs, the kept
attempt's level, when it is measurable. A kept segment whose generated audio still reads
under the floor (the re-rolls exhausted, or check off, the floor then fixed at -70)
gets a warning (kind: "chain_segment_silent") with the segment number, its
audio_dbfs and the floor; audio_dbfs_min: null silences it. best_attempt is
unchanged. A cost estimate quotes the worst case: a checked chain that generates its
audio (floor on, no match_audio, no step hold_audio; a guide chain carrying its
audio counts, its held span being the head trim) re-rolls segment 1 too, so N segments cost up to N(1 + retries) segment renders;
any other checked chain never re-rolls segment 1 and costs up to
1 + (N - 1)(1 + retries). The curated figure, measured without re-rolls, paid N. With words active the
estimate adds about 0.05 min (one Whisper CPU pass) per worst-case attempt,
(segments - kept) x (1 + retries) passes with segment 1 counted as checked, named in
the estimate's scaled text as "words check: N Whisper pass(es)".
A long take of one speaker held to a narration is two templates run in order.
templates/narration-retime makes the narration, retimed so each segment's
line fits its span. templates/minimax/narrated-take renders one H3 segment per
line as a match_audio guide chain that holds the narration as its soundtrack,
so the speech heard is the narration itself and the guide carries picture only.
The evidence is bench/records/narrated.md.
The audio grid. num_frames and guide_frames must be the same in both
templates (192 and 22 by default). The retime cuts the narration at segment
boundaries computed from them, and the take's chain lands its segments on the
same boundaries; if they differ, each segment starts in the wrong place in the
speech. Keep num_frames on H3's 3-frame audio grid (141, 192, 243, 294, 345)
or validation warns that the held slices drift. Listen to the retimed track
before spending GPU time on picture.
Probe, then continue. Validate with stop_after set to the number of
segments the chosen shot holds (4 for dialogue, up to 12 for standing);
the estimate prices only those. Run the probe and judge it. If it holds, run
again with resume_from set to the output: reference of the probe's last kept
segment and stop_after set to null, which renders only the segments after
it. A bad segment is re-rendered in place with templates/minimax/repair-segment.
face_drop. Each segment's shot record carries face_drop: how far the
face has drifted from segment 1's. It is recorded and never failed, because the
template's check leaves its ceiling null. Read it per segment next to a late
frame (get_output_frames). A step up is the face changing, and the first one an
eye called was 0.146 at 36 s on dialogue.
What was measured (12 segments, about 86 s, tier best; about 18-19 min per segment on dialogue and walk_keyframed; standing took 282 min for 12 segments (about 23.5, with two re-rolls), on an RTX 3090). The narration is held unbroken under every shot.
shot: dialogue(the default): the face is etched by about 30 s andface_dropsteps up at segment 6 (36 s). Good for a take of about 30 s.shot: standing(keystone: segment_1,keystone_slot: last, no keyframes): face and backdrop hold to 85 s,face_dropwithin 0.015 (seed 1 only). The speaker stands; each segment eases back to the opening frame.shot: walk_keyframedwithkeyframes: the face holds to 85 s (face_dropat most 0.041 on two seeds), with the cleanest seams, at the price of a hand-built keyframe per segment. Closed-mouth keyframes drop lip sync at segment ends, so each needs an open, mid-word mouth, and a keyframe face that differs from the still is followed and accumulates.- No arm cleared face, lip sync and motion together at 86 s, and nothing is
measured past 86 s.
experimental_latent_carrywithexperimental_capstoneand the latent keystone (anchor_every) stay experimental.
A piece longer than one clean take is split into passages, one narrated take per
passage (its own narration-retime, its own opening still or angle), joined as
cuts. That composition is not yet run end to end. The design is
docs/proposals/complete/narrated-long-take-complete.md.
Override the default scheduler:
"scheduler": {
"configuration": {
"scheduler_type": "DPMSolverMultistepScheduler"
},
"from_config_args": {
"use_karras_sigmas": true
}
}A scheduler block may also carry shift, the exponential sigma shift for
schedulers that take one (MiniMax H3's released checkpoint: 12.0 for video,
3.0 for audio). A pipeline that carries a second scheduler takes an
audio_scheduler block with the same shape - MiniMax H3 steps video and audio
latents down two schedules whose shifts are set independently.
Set a seed for reproducibility at workflow, step, or pipeline level - most specific wins:
a pipeline's own seed overrides its step's, which overrides the workflow's:
{
"id": "my_workflow",
"seed": 42,
"steps": [
{ "name": "step1", "seed": 123, "pipeline": { "seed": 7, ... } }
]
}Omit seed entirely to let the workflow draw a random one at run time. The seed a run
actually used - drawn or named - is recorded in its manifest.json, so a run you liked
can be reproduced after the fact.
Beside that manifest the run also writes workflow.json — the realized
workflow, meaning the one that actually ran. Every mutable input is pinned into
it: the caller's arguments folded into the variables defaults, the seed the
run used, each prompt: reference replaced by the stored text, and each
output:<identity>/latest/<file> (or /v<N>/) rewritten to the run id it resolved to.
asset:, constant:, previous_result: and builtin: are kept as written —
each already names something pinned by the asset library or by the manifest's
dw_version — and a sub-workflow named by local path is kept with its file's
SHA-256 recorded in the manifest. The manifest also lists which stored prompts
were inlined, since inlining loses the name.
The file is a valid workflow: python -m dw.run workflow.json, or its contents as
run_workflow's inline_workflow, runs it again. Its asset: and output: names
resolve against the server's or CLI's own libraries, not the run directory, so it
reproduces the run only where those libraries are the ones the original run used.
Writing it is best effort, like the manifest, and --output-layout flat writes no run
directory, so neither file.
A step that joins shots (concat_videos, dissolve_videos, or a pipeline
step with a chain) also records where each one landed, as shots on its
manifest entry (and on its step_end event):
{
"step": "cut",
"files": ["final/film.mp4"],
"subfolder": "final",
"shots": [
{"name": "shot@a", "start_frame": 0, "num_frames": 121, "start_sample": 0, "num_samples": 242267},
{"name": "shot@b", "start_frame": 121, "num_frames": 97, "start_sample": 242267, "num_samples": 194000}
]
}The shots partition the file's frames: the num_frames add up to the frame
count. The sample fields are measured off the track the join built, not
worked out from the frames. That means a shot whose track ran long shows it
here: the first shot above is 267 samples longer than 121 frames at 24 fps.
They are null when the video has no track, and for a chain that uses
match_audio. A shot is named shot@<key> when the step's videos entry was
a previous_result:shot@<key> reference, else by its input's position
(video N, a chain's segment N). A dissolve's shots after the first carry
overlap_frames, the head they share with the shot before. A step that wrote
several joined files marks each shot with its file.
The steps that keep the frames pass shots on. stabilize and the per-frame
tasks keep them as they are. interpolate_frames rescales them to the new
frame count and clears the samples. pair_audio measures the samples again
against the new track - every shot but the last is round(start_frame / fps * sample_rate), and the last one runs to the track's actual end - measured
again, once the file is written, against what it decodes to. So its
num_samples can be a few dozen samples off round(num_frames * sample_rate / fps): the AAC encode's trim, recorded in the job's event log
(not a dropped sample - a real
mismatch between the track and the video's length is its own warning,
audio_video_length_mismatch or audio_padded_to_video/audio_trimmed_to_video
with fit: "video"). Everything else drops them: an audio task's track,
say, or a video read back from a file. get_gallery_metadata reports the
recorded shots as media.shots, and get_output_frames(seams=true) uses them
when you pass no boundaries.
Any of the three levels accepts a variable: reference, which is how a seed becomes
settable per run without editing the file:
{
"variables": { "seed": 42 },
"seed": "variable:seed",
"steps": [ ... ]
}python -m dw.run workflows/models/z-image.json seed=1234Declare the variable with an integer default, as above: the value from the command line
arrives as a string and is converted to the declared type. A string that is not a
variable: reference is rejected by the schema.
The seed also reaches sub-workflows: a delegated workflow step runs the child under
the parent's seed unless the child names its own. Without that a child draws its own
random seed, and a workflow whose real generation happens inside a sub-workflow would
not reproduce from the seed it was given.
The step cache that lets a reproduced step skip re-running (see Runs in
Workspaces) is scoped to the output directory a run
writes into, which on dw.serve is the workspace's own outputs/. Two workspaces
holding the same prior run do not share a cache entry, so validate_workflow's
plan.cached_steps answers for the workspace the call is pinned to. Deleting a
workspace takes its cache entries with it.
The step cache is also per card and per workflow identity: the stored workflow's path,
or an inline workflow's id (workflow_identity in dw/worker_protocol.py). Switching
identity on a card empties that card's step cache (the worker's _cleanup_all), so:
- running any other workflow on the card in between starts the next run cold: A, then B,
then A runs the second A cold. On a multi-card server warm-identity routing
(
choose_slotindw/server/pool.py) mostly avoids it unless that card is busy; on a single card it happens every time. - a renamed copy, a new inline
id, or switching between a stored workflow and its inline copy also starts cold.
To reuse cached upstream steps, rerun the same workflow (same name, or the same inline
id), for example with rerun_job or changed arguments. Only arguments that reach
later steps leave the earlier steps cached: changing an argument the expensive first
step reads re-runs that step. To check before running, validate_workflow's
plan.cached_steps says how many steps would be reused.
- Keys ending in
_typeor_dtype, or nameddtype:"torch.bfloat16"becomestorch.bfloat16, and"sdnq.SDNQConfig"loads the class by import. - Braces keep a value a string:
"{nf4}"stays"nf4". content_typeandoffload_typealways stay strings.constant:values are read from Python rather than copied into the workflow (below).
The full rule, including what an untrusted server refuses, is in Types and escaping.
Some arguments have a value the library already declares: the sigma schedule a distilled
model was trained on, the negative prompt a model family ships. Reference it with
constant: and its dotted python name instead of copying it into the workflow:
"sigmas": "constant:diffusers.pipelines.ltx2.utils.DISTILLED_SIGMA_VALUES",
"negative_prompt": "constant:diffusers.pipelines.ltx2.utils.DEFAULT_NEGATIVE_PROMPT"The leading part of the name that imports is the module, and the rest is read from it - so a constant held in a config object is reachable too:
"prompt_max_new_tokens": "constant:diffusers.pipelines.ltx2.utils.GEMMA4_PROMPT_ENHANCEMENT_CONFIG.max_new_tokens"A reference resolves anywhere in a workflow's arguments, including in a variables
default, where it becomes the value a caller overrides - and its type, since a variable
is declared by its default:
"variables": { "negative_prompt": "constant:diffusers.pipelines.ltx2.utils.DEFAULT_NEGATIVE_PROMPT" }A constant is data: anything callable is refused (a type is named with a *_type key
instead). Mutable values are copied, so a pipeline that consumes its schedule in place
cannot edit the library's constant for the rest of the session. A constant that changes
upstream changes here, and one that is renamed or moved fails the run rather than
leaving a stale copy behind.
A prompt worth keeping is worth keeping once. Stored prompts live as JSON files in a
prompt library - the prompts/ folder by default - and a workflow argument written as
prompt: plus the file's name (without .json, optionally one folder deep) loads its
text at run time:
"prompt": "prompt:scenic_landscape",
"prompt": "prompt:minimax/fox_dawn_t2va"A prompt file holds the text plus the metadata the server's Prompts page shows:
{
"text": "A sweeping alpine valley at golden hour...",
"description": "General-purpose scenic landscape",
"intended_model": "z-image",
"negative_prompt": "blurry, low quality",
"tags": ["landscape", "golden-hour"]
}Only text is required, and it is what the reference resolves to. intended_model is
informational - the engine ignores it, but the library badges and filters by it, and
the server's prompt enhancer uses it to preselect a preset. One spelling per family:
list_prompts(intended_model=...) matches the whole value exactly, so minimax-music
beside minimax-music3 hides half a shelf, and tests/test_prompt_library.py sweeps
the repo's library for a variant.
The library's location is resolved in order (discover_library in dw/workspace.py):
the DW_PROMPT_DIR environment variable (which --prompt-dir on dw.serve sets); a
workspace named explicitly (--workspace, DW_WORKSPACE or the setting), whose
prompts/ it is; ./prompts in the working directory when it exists; the first
prompts/ folder found walking up from the workflow file's own directory; and last the
workspace's prompts/. dw.serve resolves it once at startup and pins it for every
job, so the Prompts page and prompt: resolution agree on one library. References are
rooted at that directory, not at the workflow file, so the same reference means the same
text from every workflow. A reference resolves anywhere in a workflow's arguments,
including a variables default, always to exactly one string: it never multiplies a
step's iterations. A prompt's text may not itself begin with a reference prefix such as
variable:.
A workflow's plain media paths resolve against the workflow file's own directory, which
means a workflow that reads anything has to keep that thing beside it. An asset:
reference is rooted at the asset library instead - the workspace's assets/ folder -
so a workflow and the media it reads do not have to live in the same place:
"image": "asset:iris.png",
"video": "asset:gyre/frames/web.mp4",
"references": [
{
"reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3ImageReference",
"from_file": "asset:subject.png"
}
]A reference names a file with its extension, at most four folders deep, and resolves to
that file's path, so it works under any argument that accepts a path: image, video,
a from_file, a list of any of them, or a task argument that names a file.
On dw.serve each job reads the asset library of the workspace it runs in. Elsewhere
the location is resolved in the prompt library's order, with DW_ASSET_DIR
(--asset-dir) and assets/ in place of DW_PROMPT_DIR and prompts/. See
Workspaces.
A reference can only name a file inside the library: .., an absolute path, or a
symlink pointing out of it are all refused. Browser uploads land in the library's
uploads/ folder and come back as asset:uploads/<name>, so a workflow saved after
an upload still resolves on the next run.
For multi-stage work (generate stills, then animate them in another run), an output:
reference names what an earlier run wrote:
"image": "output:ltx2/Gyre/latest/Gyre-still.0-0.0.png",
"audio": "output:ltx2/GyreScore/20260905-181530-a1b2c3d4/Gyre-score.10-0.0.wav"The name is a path under the output directory — the workflow's identity, the run, and
the file (see Runs). Writing latest where the run id goes
resolves to the newest run of that workflow that holds the file, so it keeps working
when the newest run failed part way or reused every step from the cache and wrote only
a manifest. Runs sort by their id, which starts with a UTC timestamp. v<N> in the same
position names the run whose version is N (the v4 the gallery labels its files with).
Unlike latest it picks exactly one run: v4 not holding the file is an error, not a
reason to try v3. latest and v<N> select a run only in the run-id position; a
workflow or file that happens to be called either is still named as itself.
Like asset:, a reference resolves to a path and then whatever loads paths loads it, so
it works under image, video, a from_file, or a list of them. The audio tasks take
a video file's path too and use the soundtrack muxed into it, which is how a finished
cut is scored in a later run without re-cutting it. It resolves against
the output directory the run was told to write to, and cannot leave it: .., an
absolute path, and a symlink pointing out are all refused.
To name an earlier step of the same run, use previous_result: instead — that passes
the value in memory rather than through the filesystem.
A generated file worth reusing repeatedly is better kept than referenced by the run
that made it: POST /api/assets/keep (the gallery's Keep as asset, or MCP's
keep_output) copies it into the workspace's asset library under a name you choose, and
from then on it is an asset: reference like any other — stable whatever happens to the
run directory it came from.
Some pipelines take arguments that are objects rather than plain media. An argument that
names a type and a from_file is constructed by that type's own from_file():
"references": [
{
"reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3ImageReference",
"from_file": "subject.png"
},
{
"reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3AudioReference",
"from_file": "voice.wav"
}
]Loading the media this way rather than as a plain image or video argument is what
brings its frame rate or sample rate along with it, which MiniMax-H3 resamples a
reference from. The file may be a path — relative to the workflow file, like all media a
workflow names — or a URL, and is validated like any other media. variable: references
work as the file location; previous_result: does not, since the object is built when
the workflow loads — use
from_previous_result for that. A dict that
merely contains a from_file key without a *_type key is not an object description
and is passed through untouched.
An entry in a list whose source is null is left out of that list. That is what
makes a reference optional: write it as an ordinary entry whose from_file (or
from_previous_result) is a variable, declare the variable null, and a run that is
given nothing for it generates exactly as it did before the reference existed — one
workflow serving both, instead of two spellings of the same steps. It applies to
from_file, from_previous_result and from_arguments alike. On its own rather than
in a list there is nothing to leave it out of, so a null source there is an error.
Any other key goes wherever the type can take it: to from_file() where its signature
names it, and onto the object it returns where it does not. That is what corrects a
decoded file, which is the only thing that knows what the container claimed:
{
"reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3VideoReference",
"from_file": "motion.mp4",
"fps": 30.0,
"audio": null
}fps overrides a rate the container got wrong — MiniMax-H3 resamples a reference onto
its own 24 fps, so a wrong rate is a request conditioned at the wrong speed — and
audio: null drops the decoded soundtrack, leaving a reference that conditions on
motion and camera alone. A name that is neither an argument of from_file() nor a field
of the object raises, with the fields it does have.
See workflows/templates/minimax/reference-to-video.json for a full example.
The same object can be built from what an earlier step generated, by naming the step instead of a file:
"references": [
{
"reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3ImageReference",
"from_previous_result": "draw_subject"
}
]from_file cannot do this — it names a file, and the object is built when the workflow
loads, before any step has run. from_previous_result waits: the description is checked
at load time and constructed once the step it names has produced its media, which is
what lets one workflow generate a subject and then condition on it without writing it
out and reading it back.
The media never touches the disk, so it arrives as the step produced it. Which field it
lands in comes from the type's own kind:
kind |
Built from |
|---|---|
image |
The generated image |
video |
The generated frames, and the soundtrack generated with them if there was one |
audio |
The generated soundtrack - or, for a step that produced audio alone (a music pipeline, a slice_audio task), the waveform itself. The rate travels with the waveform when the pipeline or task reports one (an AudioTrack - AudioLDM2, StableAudio, generate_speech); declare sample_rate beside from_previous_result only for a waveform from a task or file that carries none, and a declared rate always wins |
Any other key is a field of the object and wins over what the media carried —
"fps": 30.0 where the producing pipeline generated at a rate the consuming one does
not share, for instance. A step that produced several artifacts fans out the same way
every previous_result reference does: four images in, four videos out.
See workflows/templates/minimax/generated-subject-reference.json for a full example.
Not every type a pipeline takes knows how to open a file. LTX-2's keyframe conditions and IC-LoRA references are plain dataclasses holding frames the caller already loaded, plus the numbers that say what to do with them. Those are written as the arguments to construct the object with:
"conditions": [
{
"condition_type": "diffusers.pipelines.ltx2.pipeline_ltx2_condition.LTX2VideoCondition",
"from_arguments": {
"frames": { "media_type": "image", "location": "first.png" },
"index": 0,
"strength": 1.0
}
},
{
"condition_type": "diffusers.pipelines.ltx2.pipeline_ltx2_condition.LTX2VideoCondition",
"from_arguments": {
"frames": { "media_type": "image", "location": "last.png" },
"index": -1,
"strength": 1.0
}
}
]from_arguments holds every argument the type is constructed with - a key beside it
raises rather than being silently dropped, and so does an argument the type does not
take, naming the ones it does. The arguments inside are ordinary arguments: a
media reference loads there, a variable: reference resolves
there, and a previous_result: reference waits the way
from_previous_result does - the object is
constructed once the step it names has run.
Which of the three forms a type wants is decided by the type, not by preference:
| Form | For a type that |
|---|---|
from_file |
opens the media itself, bringing its frame or sample rate along (MiniMax-H3's references) |
from_previous_result |
declares a media kind, so a step's output lands in the right field on its own |
from_arguments |
is a plain record of fields - no from_file(), no kind (LTX-2's conditions and references) |
See workflows/templates/ltx2/keyframes.json for the file form and workflows/templates/ltx2/extend-clip.json for the one built from an earlier step.
A pipeline that generates video with a soundtrack returns the two paired, and the result muxes them into one file. A step that works on the frames alone - a latent upsampler, an interpolator - returns frames without it. Two tasks carry the pieces across:
video_framestakes a generated video and returns its frames as one(frames, height, width, channels)uint8 array - the 0-255 shape LTX-2's conditions want, and one artifact rather than one per frame.pair_audioputs a soundtrack back beside frames that lost it, so the step that saves them writes a single muxed mp4.
{
"name": "film",
"task": {
"command": "pair_audio",
"arguments": {
"video": "previous_result:edit",
"audio": "previous_result:balanced",
"sample_rate": "variable:sample_rate"
}
},
"result": { "content_type": "video/mp4", "fps": 24 }
}audio takes either a waveform or the earlier step whose video carried the soundtrack,
which brings its sample rate along; here it is an earlier step's waveform, so
sample_rate is given explicitly. The frames keep the rate they arrived
with - video given a file or an asset: carries that file's fps through
to the saved mp4 - so result.fps is only needed for frames that bring no
rate of their own. A mono track needs no preparation: an mp4 audio stream
takes stereo and nothing else, so saving duplicates the single channel into
two and emits a warning saying it did.
"fit": "video" makes the track fit the frames: it is cut to exactly the frames it is
laid over, or padded with silence and warned about when it is shorter. Use it for a
soundtrack under a cut whose length is an argument (a shots list), since nothing in a
workflow can multiply a list's length by a frame count (#142). Left unset, the track is
used as it is and a length mismatch is warned about.
Which shape a pipeline argument wants is the pipeline's business, and the two LTX-2
paths differ: a keyframe condition is mapped from 0-255, so it takes the video_frames
array, while an IC-LoRA reference goes through the video processor, which expects the
[0, 1] frames the pipeline returned - previous_result:step.frames hands those over
untouched.