Skip to content

Latest commit

 

History

History
3496 lines (3032 loc) · 193 KB

WORKFLOW_GUIDE.md

File metadata and controls

3496 lines (3032 loc) · 193 KB

Workflow Guide

How the catalog is organised

workflows/ holds two trees, and which one a file is in says what it is for.

workflows/templates/ teaches a pattern. One file per capability - a shape (image to video, a multi-shot cut sequence), a mechanism (shared components, sub-workflows, pipeline_reference, typed references), or a reference convention (prompt:, previous_result:). These are what to read and copy. Where several checkpoints run the same pattern through the same pipeline class, one template carries them all and its description spells out the per-checkpoint argument sets. The templates/ltx2/ and templates/minimax/ subfolders each hold a family whose members build on one baseline.

The full catalog is list_workflows over MCP, or GET /api/workflows and the web UI's Workflows page, each entry with its shape and description. This guide and docs/TASKS.md link only the templates that show a feature, so a template named in neither is still in the catalog.

workflows/models/ records a hardware fact: the quantization, offloading and component placement that make one checkpoint fit a real card. Each is runnable and carries a configures naming the template it is an instance of:

{
    "id": "flux-dev",
    "description": "Text-to-image with FLUX.1 dev - the reference FLUX workflow.",
    "configures": "templates/text-to-image",
    "steps": [ ... ]
}

Structure

Every workflow is a JSON file with an id, optional variables, and a list of steps. The full structure is the JSON Schema in dw/workflow_schema.json, which is what dw.validate and the server check a workflow against. A running server also serves it at GET /api/schema and shows it as a browsable tree on its Schema page (see Server & Web UI).

{
    "id": "my_workflow",
    "variables": {
        "prompt": "default prompt text",
        "steps": 25
    },
    "steps": [ ... ]
}

Variables define defaults that a caller overrides, from the command line or as run_workflow's arguments:

python -m dw.run my_workflow.json prompt="a cat" steps=50

A variable name matches ^[a-zA-Z_][a-zA-Z0-9_-]*$: a letter or underscore, then letters, digits, underscores or hyphens.

Step Types

Each step has a name and exactly one of four types:

Pipeline Steps

Run a HuggingFace Diffusers model:

{
    "name": "generate",
    "pipeline": {
        "configuration": { "component_type": "FluxPipeline" },
        "from_pretrained_arguments": {
            "model_name": "black-forest-labs/FLUX.1-dev",
            "torch_dtype": "torch.bfloat16"
        },
        "arguments": {
            "prompt": "variable:prompt",
            "num_inference_steps": 25
        }
    },
    "result": { "content_type": "image/jpeg" }
}

A component_type can also name a community pipeline that dw ships in dw/community_pipelines/, by its dotted path; list_pipelines lists them after the diffusers ones. RFInversionFluxPipeline (dw.community_pipelines.pipeline_flux_rf_inversion.RFInversionFluxPipeline) inverts an image into FLUX latents (below). LTX2RefinePipeline (dw.community_pipelines.pipeline_ltx2_refine.LTX2RefinePipeline) is LTX2Pipeline plus a video argument: the clip is VAE-encoded at width x height and renoised at noise_scale over sigmas, a refine at the clip's own size rather than refine-clip's 2x. num_frames (8 * n + 1, no longer than the clip) defaults to the clip's length; video and latents are exclusive.

Pipeline Reference Steps

Re-run an already-loaded pipeline from an earlier step with a fresh set of arguments, instead of loading the model again. This is how a two-pass technique like RF-Inversion works: an invert step loads the pipeline, and a main step reuses it with the inverted latents:

{
    "name": "main",
    "pipeline_reference": {
        "reference_name": "invert",
        "arguments": {
            "prompt": "variable:prompt",
            "inverted_latents": "previous_result:invert.inverted_latents",
            "image_latents": "previous_result:invert.image_latents"
        }
    },
    "result": { "content_type": "image/jpeg" }
}

reference_name must name a step earlier in the same workflow that has a pipeline. Two steps whose pipelines differ only in a LoRA's scale or alpha or a scheduler's shift share one warm model, and a reference always runs at the referenced step's own values; release_pipeline on either step frees the shared model, as it does for two identical steps. See workflows/templates/community-pipeline.json for a full example.

Task Steps

Run utility operations (image processing, QR codes, data gathering):

{
    "name": "preprocess",
    "task": {
        "command": "canny",
        "arguments": {
            "image": { "location": "https://example.com/photo.jpg" }
        }
    },
    "result": { "content_type": "image/jpeg" }
}

A task can take inputs (a plain array) instead of arguments. Each array item becomes its own iteration, the same way multiple previous_result values do. An item that is a previous_result: reference becomes one iteration per result it names, and an object item expands the way an arguments object would:

{
    "name": "prompts",
    "task": {
        "command": "gather_inputs",
        "inputs": ["a marmot on a bicycle", "a bug driving a cycle"]
    }
}

Workflow Steps

Invoke another workflow file:

{
    "name": "expand",
    "workflow": {
        "path": "builtin:h3_context_ir.json",
        "arguments": { "prompt": "variable:prompt" }
    },
    "result": { "content_type": "text/plain" }
}

arguments become the child workflow's variables. How path resolves, where the child's output is saved, and what validation checks are in Composing a stored workflow.

Cross-Step Data Flow

Variable References

Reference workflow variables with variable:name:

"prompt": "variable:prompt"

A variable's declared value is both its default and its type: a value passed in is converted to the type of the default, so declaring 25 and "25" are different things. Declaring null makes the variable optional and untyped: it takes whatever it is given and stays null when given nothing. That is how a workflow exposes an argument a caller may pass. A caller can set only variables the workflow declares, so an optional argument still has to be declared.

"variables": { "image": null }

A value the library already declares does not need a variable at all - see Constant References.

Previous Result References

Pass output from one step to another with previous_result:step_name:

{
    "steps": [
        {
            "name": "preprocess",
            "task": { "command": "canny", "arguments": { "image": { "location": "photo.jpg" } } }
        },
        {
            "name": "generate",
            "pipeline": {
                "arguments": {
                    "control_image": "previous_result:preprocess",
                    "prompt": "a painting"
                }
            }
        }
    ]
}

A reference resolves anywhere in the arguments, including inside a list or nested object, which is what lets a constructed object be built from an earlier step. Several previous_result references on one step multiply into a cartesian product (see Several previous_result references multiply).

A step whose result is a dict (a task returning several named outputs, or a pipeline step that returns something like inverted_latents) can be referenced property by property with previous_result:step_name.property_name:

"inverted_latents": "previous_result:invert.inverted_latents"

Media Arguments

Images and videos load automatically for arguments named image/*_image and video/*_video. Any other argument - mask, depth_map, a controlnet's second conditioning image - can load the same way with an explicit form that says what the media is instead of relying on its argument name:

"mask": { "media_type": "image", "location": "mask.png" }

media_type is "image" or "video". location is a path relative to the workflow file, or a URL, exactly like the plain image/video forms.

Authoring a workflow from an agent

For an agent that has read the catalog (list_workflows), found nothing that produces the shape it needs, and is about to write JSON. get_schema says what is well-formed; this section says what the engine does with a well-formed document, which is where a draft that validates still fails.

When exploring variations, pass note to run_workflow saying what each run tests ("video_shift 5 against 3"), and call set_note to mark the keeper: the note is shown with every file the run made. Each file carries a role (final or intermediate) inferred from how a step's output is used - a file a later run reads (output:, or an asset: kept from it) lists as intermediate too; set result.role on a step to override it when the default is wrong, and set "role": "intermediate" on a workflow whose deliverable is an ingredient a later assembly consumes.

References

An argument value is a reference when it begins with one of these prefixes. Each resolves before the step runs. variable: and previous_result: names are checked statically, so a bad one is a validation error at the path it sits at; a constant:, asset:, prompt: or output: name in the definition body resolves only when the step runs, and one that is missing fails the run - unless it arrives in the arguments passed to validate_workflow, which checks an asset:, prompt: or output: there for existence.

  • variable: - variable:name is the workflow's own variables entry, overridden by the caller's arguments. A variable declared null is optional and untyped. Schema validation runs before substitution, so a default must already be the JSON type the field expects: 25, not "25". Once a workflow has a variables block, a reference to a variable it does not declare is a validation error (with no block, the string is passed through unresolved).
  • previous_result: - previous_result:step_name is the outputs of an earlier step, named by that step's name. It iterates (see the cartesian rule below). A literal reference naming no earlier step is a validation error at its JSON path. A .field suffix (previous_result:invert.inverted_latents) picks one field of a result that is a dict, or a data attribute of a result object.
  • constant: - constant:module.path.NAME is a value declared in Python, read by import rather than copied into JSON. Anything callable is refused.
  • asset: - asset:name is a file in the asset library, confined to it and never resolved relative to the workflow file. upload_asset returns one of these names. It may be a bare string ("image": "asset:x.png"), a list element, or the location of a media dict ({"location": "asset:x.png"}, with or without media_type). A file upload_asset cannot send (over 4MB as content=, or not readable from the MCP session's machine) goes straight to POST /api/uploads, the route the web UI's file picker uses: the body is the raw bytes, up to 200MB, and the reply (201) carries reference, the asset: name to use. Send the bearer token when the server requires one:
    curl -H "Authorization: Bearer $DW_API_TOKEN" --data-binary @portrait.jpg \
      "http://<host>:8765/api/uploads?filename=portrait.jpg&asset_name=cast/portrait.jpg&workspace=<ws>"
    
    The same route is the only way to hand the engine a file made on the client, such as a cut stitched locally: it has no run for output: or export_job to name.
  • output: - output:<workflow identity>/<run id>/<file> is a file an earlier run wrote, confined to the output root. latest in the run-id position picks the newest run that holds that file; v<N> picks the run the gallery labels v<N> (list_gallery's version), and only that run. A run id is not stable against pruning: to depend on a generated file, promote it with keep_output and reference the asset: name instead.
  • prompt: - prompt:name or prompt:folder/name is a stored prompt's text, rooted at the prompt library. That text may not itself begin with any of these prefixes. A template's default prompt is usually a prompt: reference to a caption written to the spec its model was trained on, so read the existing one before writing a prompt for a family: list_prompts(intended_model="ltx-2.5") for the shelf, get_prompt("ltx2/fox_dawn_choir") for a body. The listing leaves the bodies out by default and reports each one's text_chars; asking for all of them at once is more than a client will accept.
  • item: - only inside a step that carries for_each: item: is the entry the member was made for, item:field one field of an object entry, spliced in whole whatever its type. See "One step per entry" below.
  • gather: - gather:shot is the result of every member of the for_each step shot, in list order, as one list. Inside a list it splices into it. previous_result:shot naming a for_each step is an error that says to use gather:.

A reference is always the whole value: "variable:base_prompt" resolves, "variable:base_prompt, in fog" asks for a variable named base_prompt, in fog and fails. Nothing is interpolated around a reference. When several steps share a block of text (a character's description repeated in every shot), write it once as a variable and assemble each step's prompt with a compose_text task, whose parts are whole references joined in order; the shot then reads "prompt": "previous_result:shot_1_prompt".

After an inline run worth keeping, get_job_workflow(job_id) returns the realized workflow (the definition with that run's arguments, seed and prompts pinned into it), and save_workflow gives it a name. export_job(job_id) bundles the whole run (workflow, manifest, job row, input and output media) into a directory on the server plus a zip URL.

Types and escaping

Any key ending in _type or _dtype, or named dtype, has its string value loaded as a Python object: a bare name ("FluxPipeline") from diffusers, a dotted name ("torch.bfloat16", "sdnq.SDNQConfig") by full module path. Wrapping a value in braces keeps it a plain string: "{nf4}" is the string nf4. Getting this wrong fails at load time, after validation has already passed, so a value meant as text under one of those keys must be braced. Two keys are exempt and always stay strings: content_type and offload_type (braces are accepted there too).

On a server without --trust-workflows, a dotted name must sit in one of the packages dw/trust.py allows (diffusers, torch, transformers, sdnq, torchao and the other quantization backends, dw), and must resolve to a class defined there, or to a torch.dtype under a dtype key. Anything else is refused at load.

Presets

presets lets one variable select a bundle of others - a template's quality dial. The shape is {selector: {value: {variable: value}}}:

"variables": {"tier": "standard", "width": 1024, "num_inference_steps": 30},
"presets": {
  "tier": {
    "draft":    {"width": 512,  "num_inference_steps": 4},
    "standard": {"width": 768,  "num_inference_steps": 12}
  }
}

The chosen bundle (the caller's value for the selector, else its default) sets every variable the caller did not pass; an explicit argument always wins, so tier=draft num_inference_steps=30 runs 30 steps on the draft canvas. With no arguments - or when a bad argument leaves them all unfolded - the default bundle applies, so a run with no arguments and one passing the selector's default are the same run, with the same fingerprint.

Everything that judges or prices a run sees the bundle's values: validation, substitution, the estimate (the curated cost counts as measured at the default bundle), the fingerprint, the VRAM projection, the host-RAM projection, the observed-cost buckets, variable_constraints and the argument warnings. A bundle value is rounded by a snap: "up" rule, with the same warning a caller's value draws.

Validation checks every bundle, not only the chosen one, and reports each problem where it was written:

  • a bundle key that is no declared variable, a bundle value that will not coerce to its default's type or fails string validation, or one outside a variable_constraints rule, at presets.<selector>.<value>.<variable>;
  • a selector that is not a declared variable, at presets.<selector>;
  • a selector whose default is not a string (bundle names are JSON keys, so a number can never match one) or names no bundle, at variables.<selector>;
  • a caller value for the selector outside the bundle names, at arguments.<selector>, naming the allowed ones.

What a variable is allowed to be

A model's own rule about a value belongs in the workflow, not in engine code. variable_constraints declares it per variable, in the same field names a chain step's frame_snap uses:

"variable_constraints": {
    "num_frames": {
        "modulus": 17,
        "remainder": 5,
        "min_frames": 124,
        "max_frames": 345,
        "snap": "up",
        "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps"
    }
}

The value has to be modulus * n + remainder within min_frames to max_frames. With snap: "up" an off-grid value is rounded to the next one the rule accepts and the run says so - 130 becomes 141, reported as a warning at validation time and again in the job's warnings; without snap it is refused. The bounds are checked against the value the run will use, so they hold for the rounded number: on the rule above 108 is accepted (it becomes 124) and 346 is refused (it would become 362).

A string value takes min_length instead: the fewest characters it may have once surrounding whitespace is dropped, a null counting as empty. Music 3 refuses an empty lyric only after the model has loaded, so its templates declare "lyrics": {"min_length": 1, "reason": "..."}, and the reason says what to send instead (a tag-only body for an instrumental).

"required": true marks a variable that has no default. Declare it null in variables and set the flag: a run, or a validation given arguments, refuses it while it is still null, with '<name>' is required and has no default and the reason; a document check with no arguments does not judge the stored null. Pair it with min_length: 1 so an empty string is refused too. It is for a value a placeholder default would quietly fill with something the caller did not choose: voice-clone, narration-long and narration-from-takes use it for reference_audio, so a stray voice.wav asset can never be cloned by omission. get_workflow reports it under constraints, the compact listing shows required, and the web UI marks the field. A sub-workflow that hands the variable down ("reference_audio": "variable:reference_audio") gets the parent's value.

max_words_per_second with seconds_from warns, and never refuses, when a string carries more words than its seconds hold at the workflow's measured rate. The seconds come from the same list entry for an entry field, else from the workflow's variables, and a [bracketed] tag is not a word. music-long declares 0.73 words a second of each section's seconds, measured on the long-form bench. It warns at validation and again in the job's warnings (text_may_overrun).

The rule is checked at validation (POST /api/validate, validate_workflow and the pre-queue check refuse a bad value at arguments.<name> or variables.<name>), again at run time before anything loads (for a value the static pass cannot see, such as one a parent workflow passed down), and is reported in the catalog beside the default by list_workflows and get_workflow(variables_only=true).

Where a template both declares a constraint and snaps a chain, the chain's frame_snap names the constraint rather than repeating the numbers:

"frame_snap": "constraint:num_frames"

A refused bound cannot depend on another variable (a maximum of fps * seconds); only the word-rate warning reads a second value. A constraint key reaches a top-level variable and, where a step consumes it as item:<name>, the same field of a for_each list's entries (dialogue-short's per-shot num_frames).

A workflow takes only the keys the engine reads

The workflow object itself, step, task, workflow, pipeline_reference, result, chain and a pipeline's configuration are closed: a property the engine does not read is a validation error naming the object and the key, not a warning. A component's own configuration is not closed, so a misspelt key there is ignored. There is no when, no retry, no select - if a draft reaches for one, the shape it wants is a different arrangement of steps, not a flag. A mistyped sedd or subfoldr is refused rather than silently ignored.

pipeline is closed to the same rule with one opening: any key whose value is a component definition - an object carrying from_pretrained_arguments - names a component to load, because diffusers grows component names faster than the schema does (latent_upsampler, prompt_enhancer and processor all appear that way in shipped templates). A pipeline key that is not one of those is refused, which is what catches pipeline_type or model_name written a level too high. from_pretrained_arguments stays open - it passes its keys through to from_pretrained.

Where a workflow may read and reach

An argument that names a location is confined, untrusted (the default):

  • a path must resolve inside the workflow's own directory, the asset libraries, or the output root. An absolute path elsewhere is refused whether or not it exists. The remedy is upload_asset (or keep_output) and an asset: reference - which is what those exist for.
  • a glob is confined the same way, and each match re-checked.
  • an http(s) URL may not resolve to an address inside the deployment - loopback, link-local, private ranges. Its host must resolve, the body is capped at 1 GiB and a fetch at 10 minutes end to end; past that, upload_asset is the route.
  • remote_text_encoder.url is https-only, and only a HuggingFace host is sent this machine's token.
  • model_name must be a Hub repo id, or a path inside one of those roots.

All of it is reported by validate_workflow, before anything is queued, so a draft that names a file the server may not read costs nothing to find out. get_server_info's trust_workflows says which posture is in force.

Declaring host RAM

A top-level host_ram_gb (a number, or variable: set per tier by a presets.tier bundle) declares the host RAM a run needs. The server holds a job while that, with a margin, does not fit beside the jobs already running on the host - including another dw.serve's - and a larger observed peak wins over it. Validate and submit refuse a run whose declared need exceeds the host's total RAM, so the H3 best (int8) tier, which declares 69 GB, is refused on a 64 GB machine.

A workflow you wrote inherits the catalog's VRAM ceiling

A template can declare a vram_estimate, and a workflow over it that projects past the card is refused before anything is queued. A workflow you wrote declares none, so validate_workflow (and run_workflow's pre-queue check) match each of its pipeline steps to the catalog by pipeline identity - the step's component_type, from_pretrained_arguments.model_name and from_pretrained_arguments.workflow - and project it against the ceiling a catalog template declared for that same pipeline. Past it, the answer stays valid: true and carries a warning of kind vram_projection_inherited that names the source template and the heaviest step (for a for_each, the heaviest member). It warns rather than refuses, since your offload and quantization may differ from the template's. Cut frames, size or references until it goes quiet, or declare your own vram_estimate, which always wins and is judged exactly as a template's. Under the ceiling, a workflow whose offload or quantization differs from the template's (no offload where the template sets model, say) gets one vram_config_differs warning naming both settings and the projection, since real usage may be higher; over the ceiling the vram_projection_inherited warning names that difference instead. A pipeline the catalog declares no ceiling for gets no warning, which is not a clearance. H3's t2va and ref2va are different identities with different ceilings.

The projection is base_gb, plus bytes_per_voxel times the product of the voxel_variables, plus gb_per_reference per non-null reference, plus bytes_per_guide_voxel times each guide's frames times the canvas (every voxel variable but num_frames); an absent key adds nothing. A declared estimate refuses on the guide term as on the rest; an inherited one only warns. The guides counted are the step's own guides with a non-null video, at their snapped length, and the one a continuity: "guide" chain appends (guide_frames, 22 or 39). A guide clip is probed at validate, at admission and at run; one that cannot be probed before the run (a previous_result: clip) is charged at the step's num_frames, and the message says it was charged at the worst case. H3's t2va and fl2va templates declare the guide term, so a guided step is judged by what its guides add rather than passing on the clip alone.

Remote code is refused by default

A server started without --trust-workflows refuses any from_pretrained_arguments that sets trust_remote_code or custom_pipeline, at load time, after validation has passed. Use a pipeline diffusers ships: no bundled catalog entry carries either key, and a workflow that does runs only on a server whose operator turned trust on, which get_server_info reports as trust_workflows.

Several previous_result references multiply

When one step carries two or more previous_result references, the engine runs that step once for every combination, a cartesian product: four images and three masks is twelve iterations, not three pairs. Past 10000 combinations the run is refused.

So a pairing (shot i with speaker i, prompt i with portrait i) cannot be expressed with two references on one step. Write one step per pair, each referencing exactly the two things it pairs, gather the pairs upstream so each is a single result, or use two for_each steps over one list, which pair by entry (below).

One step per entry: for_each

A step that carries for_each runs once per entry of a list — a shot per entry of shots — and the list is a variable the caller supplies, so a six-shot episode is an argument rather than a different file.

{
  "name": "shot",
  "for_each": "variable:shots",
  "pipeline": {
    "arguments": {
      "prompt": "item:prompt",
      "references": "item:references"
    }
  }
}

with

"shots": [
  { "name": "wide_open", "prompt": "the band walks on, wide",
    "references": [{ "reference_type": "…", "from_previous_result": "draw_singer" }] },
  { "name": "closeup", "prompt": "closeup on the singer",
    "references": [{ "reference_type": "…", "from_previous_result": "draw_singer" }] }
]

and downstream

{ "name": "edit",
  "task": { "command": "concat_videos", "arguments": { "videos": "gather:shot" } } }

Before the run starts, the engine replaces the for_each step with one ordinary step per entry, named shot@wide_open, shot@closeup: the entry's name, or its index for an entry without one. Those are the names the manifest, the job's events and the gallery show, and @ is reserved for them: a hand-written step name may not contain it. An entry's name must be unique in its list and match ^[a-zA-Z_][a-zA-Z0-9_-]*$. Give entries names: the step cache keys on the member name, so a shot inserted into a named list leaves every other shot cached, while in an indexed list every later shot regenerates.

item:field is the whole value of that field, so an entry can carry anything a step argument can, including a references list whose length differs by shot. Nothing is interpolated: "item:prompt" is the field, "shot: item:prompt" is a literal string.

A field an entry may leave out is read with the object form {"item": "singer", "default": "variable:singer_reference"} (the field's bare name, not item:singer). An entry that carries singer gets its value; one that does not gets the default, spliced in whole. An object with no default is the plain required read. The object must have exactly those keys, and it means something only inside a for_each step. music-video uses it so a shot can condition on its own picture (singer) and the rest fall back to singer_reference; the audio slice stays in the template. If every entry supplies the field and the default's step is read nowhere else, that step is elided like any unread one.

An entry may name another variable: "from_file": "variable:character_a_voice" inside a references entry is that variable's value by the time the member exists, so one variable sets a voice in every shot the character speaks in and a caller who supplies the list still writes variable: for the parts the template fixes. Those references are resolved before anything in the entry is loaded, and an undeclared one is a validation error at the entry's path (arguments.shots[2].references[1].from_file when the list is yours, variables.shots[...] when it is the template's). A value may not reference itself, directly or through another variable.

Two for_each steps over the same list are paired by key — the entry's name, or its index for an entry without one: inside shot@closeup, a reference to another for_each step slice over the same shots list resolves to slice@closeup. That is how a shot reads the audio slice cut for it when slicing and generating are two steps. It is the only pairing the engine has: for_each runs over exactly one list, with no zip and no loop index.

That pairing applies only where from_previous_result (or previous_result:) sits directly in the for_each step's own arguments. Inside an item (an entry field the step reads with item:, such as a references entry) nothing pairs it, because the entry is data the caller wrote. Spell the member out: {"reference_type": "…", "from_previous_result": "slice@closeup"}, not "slice". Getting it wrong fails the run with an error naming the members that exist.

Members of one list can't read each other. A literal reference to another entry of the member's own list ("image": "previous_result:shot@accuse" inside shot@deflect, in the template or in an entry, either spelling, with or without a .property) is refused at validate. When one entry depends on another (shot 2 starts from shot 1's last frame), use separate runs: generate the first in its own run, with any derivation such as a get_last_frame step after it, then pass that run's file to the next run as an output: reference in the dependent entry. Another for_each step over the same list reading a different key (trim@deflect reading shot@accuse) stays legal: every member of shot has finished before trim starts.

Limits: a list has at most 32 entries, and an empty list is a validation error. Validation realizes a constant: default before checking it, so a list defaulted to a constant validates the way it will run. release_pipeline on a for_each step releases after the last member. An entry key no step reads is a validation warning at the entry's path.

Each entry is a full generation, so quote the cost first. The listing's lists block names the fields an entry takes and the steps over it, and its cost carries per_entry once one entry has been measured. validate_workflow with the arguments you will run with expands your list (not the template's default), resolves the variables your entries name, reports a duplicate name or a missing field at the entry's path, and answers with a plan priced for your list (the basis values are in "The loop", step 4).

Every error carries a path in the file you wrote, not in the expanded step list: a bad reference inside a member is reported at the for_each step's own path, with the member it failed in named in the message.

templates/minimax/dialogue-short and templates/minimax/music-video are this shape: each takes one shots list, and get_workflow on either shows the entry an item needs. templates/ltx2/restore-long runs two for_each steps over one windows list - a window_video slice, then a restore of it

  • and gather:s the restores into join_windows; its list length is fixed by the source, and validate_workflow names the entries to add or drop.

The loop

  1. validate_workflow - free and instant. It reports every schema error at once, each at its JSON path, plus warnings for argument names that are not in the real pipeline signature, and a previous_result: (or from_previous_result) that names no earlier step. Pass the arguments you will run with: an undeclared name, a value that will not coerce to the declared type, and an asset:, prompt: or output: reference that names nothing in this workspace each come back at arguments.<name>. Without them the verdict is about the stored definition and its defaults; checked_arguments in the answer says which it was.

  2. Fix everything reported, including the warnings: a typo against a real __call__ shows up only as a warning. Signature warnings are computed only for a schema-valid draft (while schema errors remain, warnings is []), so validate again after fixing the errors.

  3. save_workflow - validates again on the way in and returns the catalog metadata the saved draft will carry.

  4. run_workflow with acknowledged_cost, after telling the user what it costs; without it the call is refused. The figure to tell them is the plan on the validate answer: estimate.minutes with its basis, and every downloads_required entry as its own line item, since weights not on this box are minutes and gigabytes the estimate does not count. basis decides how to quote it:

    • observed - this box's own finished runs of this shape (the cold median over runs of them, preferred over any curated figure): "about N minutes, measured over M runs".
    • per_entry - a measured per-entry rate re-priced for your list.
    • catalog - a measured total for a run whose lists are the ones it was measured with.
    • derived - (a cost entry's fixed_minutes, the share that does not scale, is held out of the scaling) that total extrapolated over a list of another length, or scaled along the steps, canvas or frames you moved off its defaults (scaled names each axis): say it is an estimate.
    • inherited - an inline workflow priced from this box's runs of the catalog template with the same pipeline (inherited_from): say it is an approximation, and relay the warning if offload, quantization or frame count differ.
    • other_device - a figure from another accelerator: say so.
    • unknown - no figure. A workflow you wrote carries no cost, but list_workflows(include_models=true) finds the models/ entry that loads the same checkpoint, and its per-image figure times the number of images is the number to quote. Say "a few minutes" only when no entry with that pipeline has been measured.

    gb on a downloads_required entry is null when the hub could not be asked, and steps/list_entries say how many members the list produced. Pass the plan back as acknowledged_cost={"fingerprint": plan.fingerprint, "minutes": plan.estimate.minutes, "downloads": [...]}: the server refuses with 409 if the run's shape changed since the quote, and the refusal carries the new plan. true is for a plan that was null.

  5. wait_for_job rather than a polling loop, with the job's plan.estimate (in seconds, plus a margin) as timeout_seconds, so one call can cover the whole run. The server caps a single call (the tool's description states the cap); the reply's timeout_applied_seconds and timeout_capped say what you got, and waited_seconds how long it took. Call again on still_running: true. Without timeout_seconds a call waits 20 seconds.

    A running job's progress carries the step; the phase (loading, generating, decoding, saving) with the model in phase_detail; seconds_in_phase; seconds_since_event, the time since the last progress event; and denoise_step/denoise_total_steps, null until the denoise loop starts. Judge a slow run from a stuck one by whether denoise_step has moved since a poll minutes ago, not by silence past a fixed threshold. A null denoise_step under generating is the pipeline's lead-in (encoding the prompt and every reference), which emits nothing and can run for many minutes with a video reference; gaps between denoise steps are uneven where a transformer block cache is configured. Both are normal, and the model family's skill carries the measured figures. denoise_total_steps is the schedule the pipeline actually runs: MiniMax H3's scheduler counts sigma grid points including the terminal zero, so it runs N-1 model evaluations for N (9 reports 8, 20 reports 19). That is the vendor's convention, not a dropped step.

  6. get_output_image to look at what was made, and say whether it answers the request. Nothing before this step establishes that it does. get_output_frames looks at a video and get_output_audio listens to a soundtrack. A text-only client reads levels and clipping from get_gallery_metadata instead (media peak and mean, findings; envelope=true for where).

    To confirm the words a clip speaks (a text-only client can't consume the AudioContent block get_output_audio returns), run templates/check-script against its script. validate_workflow(name="templates/check-script", arguments={"input_audio": "output:<name>", "lines": [...]}) first: it takes an audio file or a video's muxed soundtrack, and lines is the script in order, H3 markup allowed, {text, shot} to name a line's shot (plus start_frame when the cut repeats that shot's name), [] for a take that should be silent. Then run_workflow(..., acknowledged_cost={"fingerprint": ..., "minutes": ..., "downloads": [...]}) bound to that plan with wait_seconds=60, get_output_text on the result, and delete_output(job_id=...) for the scratch run. Its findings are the places to listen: a dropped or changed line (line_mismatch, with heard), a markup word spoken aloud, speech in a shot meant to be silent, a last word clipped by the cut. Words Whisper invents over silence or music are under discarded, not findings. A line's similarity reads number words and digits as the same (Amendment Ten heard as Amendment 10 scores 1.0; a hyphenated compound is split and then folded, so twenty-one is 21), but a spelling variant (defence/defense) still costs it, and a dropped word or two costs a long line too little to show (Nor dropped from a 13-word line scores 0.96): read heard too, not only the score. The plan comes back basis: "unknown" with minutes: null; quote seconds, not minutes (a few seconds per clip).

  7. Getting the files to the user's machine. download_output and export_job write on the machine running dw.serve, which over a remote --mcp endpoint is the GPU box. The last mile of every deliverable is the url each list_gallery entry carries (or export_job's zip_url). /outputs and /exports are served without the token (only /api/* and /mcp are gated), so the header below is harmless but not required:

    curl -H "Authorization: Bearer $DW_API_TOKEN" \
         -o exports/still.png "http://<box>:8765/outputs/ltx2/Gyre/20260910-.../still.png"
    

    Put the result under exports/ in the session's working directory - it is the user's deliverable, not a temporary file.

Keeping a set consistent

"Four pictures of the same thing" is the commonest shape a request takes that the catalog does not name directly, and what "the same" means decides the workflow.

  • The same style, different subjects or scenes. One prompt per picture, the same seed on the workflow, and a shared style phrase in every prompt. A shared seed does not make the pictures alike; it makes the run reproducible. Consistency here comes from the prompts.
  • The same object, differing in one stated way - four spoons identical but for colour, one mug in four glazes, a product in each colourway. Generate the object once, then run an image-edit pass per variant with the base step's result as its image and an instruction that names only the change ("make the mug red"). Separate generations, seeded or not, draw a different object every time; an edit holds everything the instruction does not mention. templates/consistent-set.json is this shape.
  • The same character in different situations. A reference rather than an edit: an identity-referencing pipeline or IP-Adapter conditioned on one portrait, used by every picture (templates/ip-adapter.json, templates/multi-image-reference.json, and for video the MiniMax reference-to-video and dialogue-short templates). The identity-referenced trait in the listing marks the workflows that take one.

Erasing versus repainting a masked region

A request to erase or remove an object and a request to put something new in its place are different operations, and only one of them takes a prompt.

  • Erase or remove an object → the remove_object task: prompt-free, it fills the hole from the surrounding scene, and suits small to medium objects (a cup, a sign, a watermark). templates/remove-object.json takes a mask you already have; templates/segment-and-remove.json masks the object by name first. A large region, such as a person filling a third of the frame, comes back smeared.
  • Put something new inside a mask → a prompted inpainter, models/z-image-inpaint or templates/inpaint.json. The prompt names what to paint. Do not use one to erase: given a prompt for "empty background" it paints a new object or a patch that does not match (#946).

docs/TASKS.md Object Removal has the task's arguments and its mask rules.

A spoken scene breaking into a song

A musical number: spoken shots, each carrying its own dialogue audio, then shots lip-synced to one song, with the song entering under the last spoken line. A concat_videos join is wrong here - it keeps each sung shot's own audio, the separate slices it was generated against, where the number wants the song unbroken. The task for it is join_into_song (get_guide("tasks", section="join_into_song")), and the recipe is four parts:

  1. Pick cue_seconds: the song time that lands on the first sung shot's frame 0. The song enters that long before the cut, under the last line; 0 starts it exactly at the cut. It may not be longer than the dialogue.
  2. Generate the sung shots against slice_audio slices of the song, as templates/minimax/music-video does (its slice, shot and trim steps; it slices lead_frames early and trims the run-up off). The first slice starts at cue_seconds, and each next one starts where the one before it ended, so the slices tile the song with no gap: start_seconds is cue_seconds plus the length of every sung shot before it (num_frames / fps - 24 fps for H3), duration_seconds its own length. The shots, not the slices, are what the join reads.
  3. join_into_song with the dialogue shots, the sung shots, the unbroken song and the same cue_seconds. It measures the joined dialogue at run time and places the song so cue_seconds lands on the first sung frame, ducks the dialogue under it, and discards the sung shots' own audio. It does not set the output level.
  4. normalize_audio to -3 dBFS, then pair_audio, as every template that muxes to video does - the encoder can add up to ~2 dB, and the written-peak warning reads the file.

The tail, with the shots from earlier runs (output: or asset:) or from earlier steps of the same workflow (previous_result:, gather:):

{
    "name": "joined",
    "task": {
        "command": "join_into_song",
        "arguments": {
            "dialogue": ["asset:scene/line-1.mp4", "asset:scene/line-2.mp4"],
            "song_shots": ["asset:scene/sung-1.mp4", "asset:scene/sung-2.mp4"],
            "song": "asset:scene/song.mp3",
            "cue_seconds": 1.5
        }
    }
},
{
    "name": "balanced",
    "task": {
        "command": "normalize_audio",
        "arguments": { "audio": "previous_result:joined", "peak_dbfs": -3.0 }
    }
},
{
    "name": "number",
    "task": {
        "command": "pair_audio",
        "arguments": {
            "video": "previous_result:joined",
            "audio": "previous_result:balanced",
            "fit": "video"
        }
    },
    "result": { "content_type": "video/mp4", "subfolder": "final" }
}

joined and balanced save nothing and are read by later steps, so they run and write no file of their own; number is the deliverable, and its shot map (get_gallery_metadata's media.shots) names one shot per input. The frame rate comes from the shots themselves - pass fps to the join only when none of its inputs carries one.

Saying which output is the deliverable

A run writes everything into one directory, so a finished episode sits beside the twenty scratch files that went into it. A step's result block can name a subfolder of the run directory for its files:

"result": { "content_type": "video/mp4", "subfolder": "final" }

The convention is two names: final for a step whose output the user will be shown, intermediate for everything else. The engine treats no name specially and applies no default (a step that names none writes to the run's root), but the gallery, get_job and list_gallery all carry the value, so a consumer can tell the deliverable from the scratch without knowing the workflow. Mark every saving step of a multi-step workflow; a one-step workflow needs nothing. Every shipped template marks each of its saving steps, with at least one final.

The value is a relative path of any depth (shots/act-1), may be a variable: or, inside a for_each step, an item: reference, and follows the output: segment rule (each segment starts with a letter, digit or underscore; .., a backslash and a leading . are refused), so every subfolder written is one a later workflow can name: output:dialogue-short/latest/final/episode.mp4. A bad value is a validation error at its JSON path. file_base_name is a name, not a path: a separator there is refused (see Result Configuration for how it names files).

A step that saves nothing and that no later step reads does not run at all; see A step nothing reads does not run.

Composing a stored workflow

A step with a workflow block runs another workflow as one step of this one, with arguments handed down as that workflow's variables. Its path is read the way run_workflow's workflow_path is: a catalog name as list_workflows reports it (templates/minimax/reference-to-video), with or without .json; a path relative to the file that names it (../models/x.json); or builtin:name.json for the packaged fragments in dw/workflows/. A name resolves beside the referencing file first, then in this workspace's workflows/, then in each read-only source the server lists, so a stored template is composed by its catalog name rather than copied (a copy stops tracking the original). A path that lands outside every source is refused.

Declare a result (with a content_type) on the composing step and the composed output is saved there, once, under that step's name and subfolder; the child's own last step does not write a second copy. Without one, the child saves its own output. The child's other steps write into the same run directory, prefixed with the composing step's name.

validate_workflow resolves the path, so a name that reaches nothing is an error at steps[N].workflow.path before anything is queued; it also validates the workflow named, refuses a composition cycle, and warns about an argument the composed workflow declares no variable for.

Being found next time

The catalog derives each entry's shape — one of image, image-set, image-edit, shot, sequence, audio, text, utility — and its traits (has-audio, chained, image-conditioned, identity-referenced, needs-input-media, composes-workflows) from the structure of the definition, and its summary from the first sentence of description.

So write that first sentence to say what the workflow makes and what it needs supplied, in under 120 characters — "H3 video with audio between two supplied stills" — rather than what technique it demonstrates. A first sentence longer than that is truncated with an ellipsis in every listing.

Declare shape, traits or summary at the top level only when derivation gets it wrong; the repo's catalog tests refuse a declaration that repeats the derivation. cost is never derived: leave it absent until a run has been measured. A cost entry is measured at the workflow's variable defaults, and a run that moves num_inference_steps, width/height or num_frames (per entry in a list, summed) is quoted scaled in proportion to them, so a figure measured at another size or step count is re-priced to the defaults before it is written (dialogue-short's 42 min at 1344x768 is 21.2 at its 960x544). An entry's scales_with names further drivers its figure is proportional to: a scalar variable, or a field of a list driver's entries summed over them (Music 3's audio_duration, summed over music-long's sections). Its fixed_minutes is the share that does not scale (model load, encode, decode), held out of the scaling: a moved run is quoted fixed_minutes + (minutes - fixed_minutes) x factor, so a long chain is not charged its load once per segment (minimax/chained-segments declares 10). Absent, the whole figure scales. An entry carrying cost has been run to completion on the device it names; one without has only been authored, and its first run is the one that finds out what its description could not check.

cost_drivers names the variables that move the wall clock (a frame count, a step count, a segment count, the list a for_each runs over), never a prompt or a seed. The server buckets its own finished runs by those values and reports the result as observed beside the curated cost, so a 345-frame run never informs a 124-frame figure, and a list driver buckets on its length. With no drivers, the figure falls back to runs that overrode nothing at all, so a measured workflow with no drivers mostly answers "unknown". A driver the estimate cannot scale along (anything but steps, width, height, frames and the entry's scales_with, such as a weights dtype or a LoRA name) still splits the buckets, a text value by its exact value. When a run moves one off its default, the curated figure is kept but carries low_confidence: true, and differs names the drivers. Each name must be a variable the workflow declares (tests/test_observed_cost.py sweeps the catalog for that).

Assessing a run's output

A cut joined from shots can succeed and still be wrong at a seam, and the whole-file numbers get_gallery_metadata reports cannot see inside a join. assess_output(name) (GET /api/gallery/{name}/assess) measures that file on the server. It decodes the file once, runs every assessment probe that applies to it (analyze_shots, analyze_seams, analyze_sync_drift), and returns where to look. It queues nothing: the probes use only the CPU and run in the server process, beside whatever job holds the GPU. name is a gallery name, output: or asset:. The shot boundaries come from what the file's run recorded: the run manifest for an output, and the sidecar keep_output wrote for an asset.

A last shot's num_samples a few dozen samples off round(num_frames * sample_rate / fps) is expected, not a finding - see pair_audio in the tasks guide's Video Processing section for why.

Procedure.

  1. After wait_for_job, call assess_output on the deliverable, which is the file under final/. get_gallery_metadata points at the tool whenever media.shots is set.
  2. Read findings. When the list is empty, no rule crossed its threshold, which is a reason to listen less closely, not a pass.
  3. Drill into each finding at the place its at names. For a seam, use get_output_frames(name, seams=[n]) to see it and get_output_audio(name, start, duration) to hear the second around it. For a shot, look at that shot's span. Judge it against the request.
  4. Fix what you confirmed (see the table below), then assess the new cut.

Pass probe="analyze_seams" (or either of the other two probe names) to get that one probe's full body: every seam's or shot's measurements, not just the ones that crossed a rule. detail=true adds every applicable probe's full body under probes. An unknown probe is refused before anything is read, and the error names the three probes. Each seam's full record also carries look measures - saturation, lightness and contrast steps, look_ratio and detail_ratio - that catch a colour or brightness shift jump_ratio misses; no rule reads them (the tasks guide's analyze_seams field table).

The answer.

Field What it holds
findings Every rule a measurement crossed, each {rule, severity, at, value, threshold, says}. severity is warn or info. at names the shot or seam.
rules_applied The rules that were checked, so an empty findings list says which checks came back clean.
rules_skipped {probe, rule, reason} for each rule that could not be measured on this file.
not_applicable {probe: why} for each probe that does not apply to this file. A still has no shots, seams or soundtrack. A mute file has no levels. A file with no recorded shots has no seams.
shots_source Where the boundaries came from: manifest (the run's manifest, or an asset's sidecar), or none.

The thresholds live in one table, dw/assessment_rules.py:

Rule Probe Fires when
shot_level_spread analyze_shots the shots' RMS levels span 6 dB or more
seam_level_step analyze_seams the shots either side of a seam differ by more than 3 dB
seam_click analyze_seams the join peaks more than 12 dB above the audio either side
seam_hole analyze_seams the join's floor drops below -50 dBFS while both sides are voiced (above -30 dBFS)
seam_frame_jump analyze_seams the picture changes more than 25x as much across the seam as inside either shot (info, and skipped at a shot marked hard_cut: true)
shot_dead_air analyze_shots the shot holds a gap more than 0.4 seconds long below its dead-air floor (skipped inside a shot that is quiet throughout)
sync_drift analyze_sync_drift by a shot's end, the audio sits more than 40 ms off the picture
sync_length analyze_sync_drift the soundtrack and the picture differ in length by more than 40 ms

A shots record whose start_frame/num_frames already reaches past the file's own length is not measured against a threshold - it is clipped to the file before any of the above run, and that clip is itself a shot_span_overrun finding on all three probes (analyze_shots, analyze_seams, analyze_sync_drift), with value naming how far past the end it reached. validate_workflow catches the same mistake before the run for a shots argument and an asset:/literal video whose length is knowable ahead of time; it cannot for previous_result:/output: video not yet written, so that case is left to the finding above.

Authority. A finding marks a place to look, not a verdict. Nothing in the engine acts on one, and no run fails because of one. A finding you have checked and accepted is simply left alone. A seam_frame_jump at a cut the story wanted is the cut working, and a level step into a quieter scene can be the scene. Tell the person what you confirmed, not what the probe reported.

Comparing outputs

compare_outputs(names) (POST /api/gallery/compare) lays 2-8 outputs (variations, reruns, the same shot with and without a speed setting) in one grid, a row per output and a column per moment, every output read at the same frame number, and measures each against a baseline (default the first). Per non-baseline output it reports PSNR (dB, higher is closer; identical when equal), SSIM (structure on luma, 1.0 identical) and mean absolute difference (0-255, lower is closer), at source resolution, per column and as mean/min/max. A shared crop is the region both shown and measured; one box per name is display only and the metrics stay on the full frame; the reply's region says which. Outputs of a different size get no numbers (never resized). A bare PSNR means little alone; these bands, measured on full frames, are a reading aid, not thresholds: about 44 dB is rerun noise (same settings, nondeterministic kernels); 23-27 dB means the numerics changed (a different cache, attention backend or quantization) but the shot is the same; about 19 dB means it was re-staged (the composition changed). Look at the grid before trusting a number.

Texture bands

compare_outputs(names, texture_bands=true) adds, per output, the RMS energy of the picture's texture by wavelength over the measured region: fine (2-8 px), mid (8-32 px) and coarse (32-128 px), in three channels, luma, chroma_rg (R-G) and chroma_bg (B-G), on a 0-255 scale. A least-squares plane (a lighting gradient) is removed, the region is windowed, and the energy is summed over rings of the 2D spectrum. A band longer than the box's short side is null. Each non-baseline cell also carries texture_ratio (output over baseline, null where the baseline band is 0). With bands, one name is allowed: it is the baseline, so there are no ratios; use several at moments and read the numbers across. A crop must be one shared box of at least 16 px a side; no crop is the full frame.

  • fine suggests grain, edge crispness and encoder texture. mid suggests blotches, stripes and coarsening. coarse suggests large patches and shading.
  • It is a measurement, not a score. mid rising can be real detail. Every band falling can be blur or over-smoothing, which the eye often reads as "cleaner".
  • It means something only on a box with static framing and lighting. Pick a static region with crop; on moving content the numbers say nothing.
  • Everything is measured after encode, and dw chains write only MP4, so there are no pre-encode frames. Encoder texture sits mostly in fine.
  • To A/B, branch from one clean first segment, keep the seeds, prompts and references, change one variable at a time, and compare the same moments in the same box.

Remediation

A recut reruns only the join over the shots the run already made: each entry in videos is output: + the run's intermediate/ shot file. That is cheap, and generates nothing new. A regenerate is a new run, so quote its plan.estimate first.

Finding Fix Kind
shot_level_spread, seam_level_step match_levels: "rms" (with match_levels_dbfs for the target) on the concat_videos / dissolve_videos step recut
seam_click a longer crossfade_ms on the join recut
seam_hole audio_bleed_ms on the join, so the outgoing tail rings on across the seam recut
seam_frame_jump a dissolve_videos join, or regenerate the incoming shot from the outgoing shot's last frame. If the cut was meant, leave it alone recut, or regenerate
shot_dead_air find_loop_bed on the cut names a quiet stretch inside one shot and its gain; cut that bed with slice_audio, loop_audio it to the gap's length, and mix_audio it under the line rather than leaving the drop silent recut
sync_drift regenerate the shot. Drift inside a shot is the model's, not the join's regenerate
sync_length rerun the mux through pair_audio with fit: "video", which cuts or pads the track to the picture recut

Result Configuration

"result": {
    "content_type": "image/jpeg",
    "save": true,
    "file_base_name": "episode",
    "subfolder": "final"
}

Supported content types: image/jpeg, image/png, image/webp, image/gif, video/mp4, audio/wav, audio/flac, audio/mpeg (mp3), audio/ogg, audio/opus, audio/aiff, application/json, text/plain (plus the aliases audio/x-wav, audio/x-flac, audio/mp3, audio/vorbis). Any other audio/ or video/ type is a validation error, and so is a type a browser runs as a page (text/html, image/svg+xml, application/xml and the like) (dw/content_types.py).

A task command's implementation declares what it hands back - most answer an artifact (a file result saves in one of the media content types above), some (judge) answer a bare scalar that cannot be saved at all, and some (the assessment probes in TASKS.md, find_loop_bed, and attribute_voices) answer a json document - every measurement taken, in one dict. A step on a json command must set content_type to application/json, which saves it as one document; a step on a scalar command may not carry a result at all. Both are checked in validation, by the command's own declared kind rather than a name match.

subfolder places the step's files in a subfolder of the run directory - see Saying which output is the deliverable above.

For video, "fps" is the rate the file is written at. It is rarely needed: frames that know their own rate carry it - a video read from a file or an asset:, a concat_videos/dissolve_videos join, an interpolation - and the engine writes them at it. Frames that bring no rate (most generations) fall back to 8, so a workflow that assembles from bare frames should say what they run at. A declared fps always wins over the carried one and warns when the two differ, which is how a deliberate slow motion is written. For audio, add "sample_rate": 44100 when the waveform doesn't already carry a rate of its own (a declared rate always wins). Setting embed_metadata: true on an image result embeds the step's model name and arguments as generation metadata - PNG info chunks for image/png, EXIF UserComment (via piexif) for image/jpeg and image/webp.

A pipeline that generates a video with its own audio track (LTX-2, or a modular pipeline whose output asks for both videos and audio) is muxed into one video/mp4 file with PyAV. audio_sample_rate overrides the rate the pipeline itself reports, for the rare case it needs correcting.

Audio Encoding

Audio is written through soundfile, so both lossless and compressed containers work:

"result": {
    "content_type": "audio/mpeg",
    "sample_rate": 44100,
    "compression_level": 0.3
}
  • subtype — encoding subtype, such as "PCM_24" for wav and flac. Defaults to the container's own default, which is "PCM_16" for wav and flac.
  • compression_level — 0.0 to 1.0 for flac, mp3 and ogg. Higher means smaller files.
  • bitrate_mode — "CONSTANT", "AVERAGE" or "VARIABLE" for compressed formats.

audio/opus writes an Opus stream in an ogg container, and only encodes at sample rates of 8000, 12000, 16000, 24000 or 48000.

Output files are saved as {output_dir}/{base_name}-{result_index}.{artifact_index}.{ext}, where base_name is {workflow_id}-{step_name}.{step_index} unless the step's result sets file_base_name, which replaces it entirely. step_index is the step's position in the workflow, result_index counts the argument-combination iterations the step ran (see cartesian product, above), and artifact_index counts multiple artifacts within one result (num_images_per_prompt > 1, or a dict result saved key by key); a JSON result is one {base_name}-{result_index}.json. In a composed child the composing step's name leads base_name. file_base_name is one name of letters, digits, _, ., @ and -, starting with a letter, digit or underscore, so an output: reference can name the file it writes; anything else, a path separator included, is refused at validation, naming the character; "file_base_name": "episode" in a final subfolder writes final/episode-0.0.mp4. The derived name is what makes two steps' files distinct, so when you replace it on more than one step in the same subfolder, give each a different name - otherwise the second one gets a -2 counter.

Pipeline Configuration

A pipeline step's configuration is dw's own vocabulary rather than the model's, so it is a closed set: a name the schema does not declare (a misspelt offload, say) fails validation instead of being ignored. Model-side values have blocks of their own: from_pretrained_arguments for the constructor, arguments for the call, and configs for a modular pipeline's block configs. A component's configuration (under transformer, vae and the like) is not closed.

Memory Offloading

Control how models use memory:

"configuration": {
    "component_type": "FluxPipeline",
    "offload": "model"
}
  • "model" — Moves entire models between CPU and GPU. Good balance of speed and memory.
  • "sequential" — Moves individual layers. Slowest but uses least GPU memory (measured at 6x the per-step cost of model offload on FLUX dev and 3.4x on Z-Image; a streamed block-level group_offload of the transformer is the usual better answer - see RECIPES_24GB.md). On MPS it is downgraded to "model" with a warning: with unified memory there is no separate pool to keep small, so the per-layer copies cost speed and save nothing. exclude_from_cpu_offload names components the sweep should leave alone.
  • Omit for no offloading (fastest, requires enough VRAM).

For components the pipeline loads itself — which is all of a modular pipeline's — use components, applied once the pipeline is loaded:

"configuration": {
    "component_type": "ModularPipeline",
    "components": {
        "transformer": {
            "group_offload": {
                "offload_type": "block_level",
                "num_blocks_per_group": 1,
                "use_stream": true
            }
        },
        "text_encoder.model": {
            "group_offload": { "offload_type": "leaf_level", "use_stream": true }
        },
        "vae": { "device": "cuda", "residency": "on_demand" },
        "audio_vae": { "device": "cuda" }
    }
}
  • group_offload — streams the component between system memory and the accelerator a block or a leaf module at a time, which is what fits a component larger than the device. offload_type is required ("block_level" or "leaf_level"); onload_device defaults to the pipeline's device and offload_device to the CPU. Anything else in the block is passed through to apply_group_offloading, so use_stream, num_blocks_per_group, low_cpu_mem_usage and offload_to_disk_path work as diffusers documents them.
  • device — moves a component that is small enough to stay resident.
  • residency — "resident" (the default) leaves the component on its device for the whole run; "on_demand" rests it in system memory and moves it to the device only while one of its own calls runs. See On-demand components.
  • enable_tiling — tiled decode for a decoder not named vae (LTX-2.5's diffusion_decoder, for example).
  • attention_backend — a persistent set_attention_backend on one component, which a compiled component needs (the pipeline-level attention_backend applies per call). Every MiniMax-H3 template wires an attention_backend variable here, on its H3 step's transformer or transformer_ref, defaulting to "sparse" (dw's comfy-kitchen sparse attention, which needs no extra install), with sparse_topk 0.15; pass sage_hub or native to override it. See ACCELERATION.md for sparse and Attention Backends for sage_hub and what it needs.
  • sparse_tau — the sparsity threshold for a component whose attention_backend is "sparse" (number, minimum 0, default 0.5). A runtime setting: changing it reuses the loaded pipeline. See ACCELERATION.md.
  • sparse_topk — top-k routing instead of sparse_tau: keep this fraction of key tiles (strictly between 0 and 1). The H3 templates set 0.15; when set, sparse_tau is ignored, and a component that sets none routes by sparse_tau. A runtime setting, like sparse_tau.
  • attn_processor_type — the attention processor the component runs, constructed with no arguments and handed to set_attn_processor. The unet and transformer blocks cover those two; this covers any other component that carries attention (LTX-2.5's diffusion_decoder, whose default processor is a portable fallback rather than the NATTEN path the decoder was built around).
  • compile, truncate_layers, remove_modules — see ACCELERATION.md.
  • A dotted key reaches a module inside a component, for a component that holds the model rather than being one.
  • A components block that group offloads anything, or marks anything on_demand, already keeps the pipeline itself off the device - the components are placed individually, so moving the whole pipeline would load it in full before the hooks and wrappers exist. Nothing extra is needed for that.

preserve_device_placement covers the case that is left: a component loaded already placed, which must not be moved afterwards. A device_map load or a quantization that pins its tensors to one device is the usual reason.

"transformer": {
    "configuration": {
        "component_type": "FluxTransformer2DModel",
        "preserve_device_placement": true
    },
    "from_pretrained_arguments": {
        "model_name": "black-forest-labs/FLUX.1-dev",
        "subfolder": "transformer",
        "device_map": "cuda"
    }
}

Renamed: this setting was do_not_send_to_device, which is no longer recognized. In a pipeline configuration the old name is a validation error; in a component's configuration it is ignored and the component is moved to the device anyway.

On-demand components

"residency": "on_demand" sits between the two placements above. A device component holds VRAM for the whole run, wasted on a component used twice; group offloading streams per submodule forward, so it restreams the model once per call of every leaf - ruinous for a VAE, whose tiled decode calls its blocks once per tile. On-demand moves the model as a whole around each call, so a tiling loop sits inside a single pair of transfers.

"components": {
    "vae": { "device": "cuda", "residency": "on_demand" },
    "audio_vae": { "device": "cuda", "residency": "on_demand" }
}

The component rests on the CPU and is moved to device around whichever of forward, encode and decode it defines, then moved back and the freed VRAM released to the driver. Nested calls are counted, so a decode that calls forward internally is moved once, not twice.

  • Use it for a component that is large but called a handful of times - a VAE that encodes references at the start and decodes the result at the end. Freeing it for the denoise loop is the whole point.
  • Not for a component called every step. A denoising transformer would pay per-call transfers 20-50 times; group offloading is the tool for those.
  • Cannot be combined with group_offload on the same component - a group offloaded module holds one group at a time and ignores whole-model moves, so the two cannot both own its placement. Configuring both is rejected at load.
  • Ignored when the component's device is the CPU, where there is nothing to move it off of.

On a 24GB card, templates/minimax/reference-to-video.json peaks at 18.9GiB of reserved VRAM with on-demand VAEs against 23.2GiB resident, and the tighter resident fit costs 40 allocator retries - cache flushes forced by a failed allocation - where the on-demand run has none. The headroom is also what lets the chained variant run: its later segments carry an extra reference and need ~1.9GiB more than the first.

The same holds for the frame-conditioned workflows. Generating 124 frames at 960x544 from a keyframe, with everything else held equal:

VAE placement peak reserved allocator retries
resident 22.71GiB 22
on-demand 18.03GiB 0

The resident run also logs a memory mapping failed with OOM warning per retry, with as little as 3MB free while it tries to map 20MB. It completes - the allocator flushes its cache and succeeds on the retry - but each one is a synchronising stall, and a run that close to the limit fails outright on any workload that needs slightly more. Every MiniMax H3 example uses on-demand VAEs for this reason.

Example: reference-to-video.json, image-to-video.json

Releasing a pipeline mid-workflow

Pipelines stay loaded for the whole run (and across runs, in the server's persistent worker) so repeated steps reuse them. When a workflow chains two large models that cannot both fit - generate with one, upscale with another - release the first once its step completes instead of configuring offload on everything:

{
    "name": "generate",
    "release_pipeline": true,
    "pipeline": { ... }
}

The step-level release_pipeline flag unloads the step's pipeline after its results are saved. A later pipeline_reference to a released step is an error, and the process-wide step cache will not retain it.

What keeps a pipeline warm: the server's worker keeps a pipeline loaded across runs for as long as what it loads is unchanged. A LoRA's scale and alpha and a scheduler's (or audio_scheduler's) shift are applied in place on the warm pipeline, so iterating on them costs no reload; removing (nulling) an alpha or a shift reloads, so the checkpoint default comes back. Changing a LoRA's model_name or weight_name, the quantization, the placement or the scheduler type reloads it.

Releasing task models mid-workflow

Task models - the checkpoints behind text_generation, segment, depth_estimator and the rest - are cached separately from pipelines, so that a step running its task once per result does not reload the same weights on every iteration. Nothing evicts that cache during a run, which matters when a task loads a large model on the device ahead of a generation step: a prompt-expanding language model would hold its weights for the whole run. release_models clears it once the step completes:

{
    "name": "expand_prompt",
    "release_models": true,
    "workflow": { "path": "builtin:h3_context_ir.json", "arguments": { ... } }
}

The flag applies to any step type, and on a workflow step it fires once the whole sub-workflow has finished. It clears every cached task model, not only this step's, and a later step needing one of them reloads it.

Example: enhance-prompt.json

A step nothing reads does not run

Before the first step executes, the engine drops any step whose result no later step reads and which writes no file, and warns once per dropped step saying which and why. That is how a template whose portraits can be supplied as asset: files skips the steps that would have drawn them (#122). Elision follows from the definition, never from a value produced during the run, so it is decided at validate time too.

Four things keep a step:

  • it saves - a result with a content_type, and save not false. "save": false is how a step says it is scaffolding.
  • it is the last step - it is the run's answer, whatever it declares.
  • something reads it - previous_result:/from_previous_result (including previous_result:step.property), a gather: (which is a list of those by the time this runs), a pipeline_reference naming it, or a reused_components entry naming a component it shares.
  • a kept step reads it - elision is transitive, so dropping a step can drop the step it read in turn.

release_pipeline on an elided step moves onto the last surviving step before it when that step loaded the same pipeline, and release_models moves unconditionally, so the release still happens. The plan a validate call answers with is computed after elision, so steps, downloads_required and the cost it quotes are the work that will actually happen, and it lists what was dropped under elided_steps; the run manifest records the same list.

If a step you expected to run is named in the warnings, the usual cause is a reference to it spelled wrong somewhere later, or a step that was meant to declare a result.

VAE Options

"configuration": {
    "vae": {
        "enable_slicing": true,
        "enable_tiling": true
    }
}
  • enable_slicing — Process VAE in slices to reduce memory
  • enable_tiling — Tile large images through the VAE

LoRAs

Attach one or more LoRAs to a pipeline with loras, a sibling of configuration:

"loras": [
    { "model_name": "XLabs-AI/flux-RealismLora", "adapter_name": "realism", "scale": 0.8 },
    { "model_name": "user/other-lora", "weight_name": "lora.safetensors", "subfolder": "loras" }
]
  • model_name — the LoRA's hub repo, required. null switches the entry off: it is not loaded and the step runs without it. A template's loras list is fixed JSON, so this is how a caller drops its adapter - pass the variable behind model_name as null (MiniMax H3: lora_model_name). Validation and the run both warn (lora_disabled). A step-distillation (turbo) LoRA sets the step count and sigma shift the step runs at, so raise those to the base model's schedule with it or the step runs undertrained.
  • weight_name / subfolder — pick a specific weights file within the repo.
  • adapter_name — name passed to set_adapters(). Defaults to the LoRA's index in the list.
  • scale — the adapter's weight, passed to set_adapters(). Defaults to 1.0.

See workflows/templates/lora.json for a full example.

IP-Adapter

"ip_adapter": {
    "model_name": "h94/IP-Adapter",
    "weight_name": "ip-adapter_sdxl.bin",
    "scale": 0.6
}

model_name is required; weight_name, subfolder and scale are optional. The adapter image itself is passed as a normal ip_adapter_image pipeline argument. See workflows/templates/ip-adapter.json.

Sharing Components Across Steps

Two pipeline steps that load the same underlying component (a shared text encoder, for instance) can avoid loading it twice:

"configuration": { "component_type": "FluxPipeline", "shared_components": ["text_encoder"] }
"configuration": { "component_type": "FluxPipeline", "reused_components": ["text_encoder"] }

The step naming shared_components stores those components after it loads; a later step naming the same names in reused_components gets them instead of loading its own copy. The names must match exactly between the two steps. Either list can sit in the step's configuration or beside it on the pipeline itself.

How the component reaches the second pipeline depends on what kind it is. A standard pipeline takes it as a from_pretrained argument. A modular pipeline cannot — it is built from the component specs in its own index — so it is registered with update_components() before load_components() runs, which is also what keeps load_components() from pulling a second copy: it only loads what is not already there. That is what lets two MiniMax-H3 steps of different tasks (t2va and ref2va load different transformer partitions) share the 14GB text encoder and the VAEs between them.

A reused component keeps the device placement the step that shared it gave it. Any components entry naming one is skipped with a log line rather than applied a second time — offloading hooks do not survive being installed twice, and the step that loaded the component is the one that decided how it is placed.

Sharing outlives the pipeline that did it: a step can share a component and still set release_pipeline, which frees everything else it loaded while the shared component stays alive for the steps that reuse it.

Attention and Performance

"configuration": {
    "component_type": "FluxPipeline",
    "attention_backend": "flash_hub",
    "enable_attention_slicing": true,
    "no_generator": false
}
  • enable_attention_slicing — process attention in slices to reduce memory at some cost to speed. Automatic on MPS unless disable_attention_slicing is set (UNet/ControlNet models only; worth disabling for SDXL on a Mac).
  • attention_backend — selects a diffusers attention backend (e.g. "flash_hub") for the duration of each pipeline call. "sparse" is dw's own instead: comfy-kitchen sparse attention on a MiniMax-H3 transformer, installed for the call only, and dense with a warning where it cannot run. See ACCELERATION.md.
  • sparse_tau — the sparsity threshold for a pipeline-level attention_backend: "sparse" (number, minimum 0, default 0.5). See ACCELERATION.md.
  • sparse_topk — top-k routing for a pipeline-level "sparse" instead of sparse_tau (strictly between 0 and 1); unset unless the workflow sets it.
  • prompt_weighting — enables A1111-style prompt weighting ((word:1.5), [word], ((word))) and prompts over 77 tokens. Currently supports Flux pipelines. Mutually exclusive with remote_text_encoder.
  • no_generator — set true to skip creating a torch.Generator for pipelines that don't accept one.
  • inversion — run the pipeline's invert() instead of the pipeline itself; the step returns the inverted/image latents for a later step to consume (see community-pipeline.json).
  • generate — run the pipeline's generate() instead, for components with a generation head (the step returns generated_ids).

Cache Acceleration

A cache block speeds up inference by skipping redundant computation:

"configuration": {
    "cache": { "type": "first_block", "threshold": 0.05 }
}

cache wraps diffusers' own cache hooks - type is one of first_block, faster, mag, taylorseer or text_kv, each with its own tuning fields (threshold, num_inference_steps, max_skip_steps, retention_ratio, cache_interval, max_order, mag_ratios, calibrate, is_guidance_distilled — see dw/workflow_schema.json for which fields apply to which type). See workflows/templates/step-caching.json.

cache can also be a whole-object variable: reference, which must resolve to a cache object or to null. A variable resolving to null drops the key: no cache, and the only true "off", since an installed first_block hook changes the output even when it skips nothing. The substituted object is checked against the cache shape at validate, so {"type": "bogus"} or a bare "first_block" is refused, naming the field. Every MiniMax-H3 template exposes a cache variable (default null) on its H3 step: "arguments": {"cache": {"type": "first_block", "threshold": 0.05}}.

Device and Dtype

Device is auto-detected (CUDA > MPS > CPU). Dtype is set per-component:

"from_pretrained_arguments": {
    "model_name": "black-forest-labs/FLUX.1-dev",
    "torch_dtype": "torch.bfloat16"
}

A step can name a device instead, in a pipeline configuration (which becomes the default for that pipeline's components), in a component configuration, or in a task's arguments. A device naming a backend the machine running the workflow does not have is translated to the one it does, with a warning, so a workflow written on a CUDA box runs on a Mac and back again:

"configuration": {
    "component_type": "FluxPipeline",
    "device": "cuda"
}

Only the backend is translated. A device index survives when the backend matches, so cuda:1 on a single-GPU CUDA box is an error; when the backend does not match, the index is dropped and the warning says so. "device": "cpu" is never translated, since pinning a step to the CPU is how a GPU-specific problem gets ruled out.

Modular Pipelines

Modular pipelines (ModularPipeline and its subclasses) load their configuration and their component weights separately, so from_pretrained_arguments only names the model and load_components pulls the weights:

"configuration": {
    "component_type": "MiniMaxMusic3ModularPipeline",
    "load_components": { "dtype": "torch.bfloat16" },
    "components_manager": { "enable_auto_cpu_offload": true }
}
  • load_components — arguments for load_components(). Use dtype for the component dtype and names to load only some of the components. quantization_config is keyed by component name, since a modular pipeline loads each component itself:

    "load_components": {
        "dtype": "torch.bfloat16",
        "quantization_config": {
            "transformer": {
                "configuration": { "config_type": "TorchAoConfig" },
                "arguments": {
                    "quant_type": "torchao.quantization.Int8WeightOnlyConfig",
                    "modules_to_not_convert": ["proj_in", "proj_out"]
                }
            },
            "language_model": {
                "configuration": { "config_type": "transformers.TorchAoConfig" },
                "arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" }
            }
        }
    }

    A component the map does not name loads unquantized. Note which TorchAoConfig each component takes: the diffusers one for its own models, the transformers one for a transformers model such as a conditioner.

  • configs — values the pipeline's blocks declare and read while they run. They are neither components nor call arguments, which is why they have a block of their own:

    "configs": {
        "canvas_short_edge": 768,
        "reference_image_short_edge": 1024
    }

    The names are whatever the pipeline itself declares, so they differ per model rather than being a fixed list here — MiniMax-H3 declares canvas_short_edge (768), canvas_max_pixels (1032192) and reference_image_short_edge (2048), the last being the resolution its image references are encoded at. A name the pipeline does not declare raises rather than passing quietly, since a dropped config reads as a setting that did nothing.

  • remove_blocks — names of top-level blocks to pop off a modular pipeline's block graph once it is built (blocks.sub_blocks.pop). It is for a block that always runs and has no input to turn it off. FLUX 3 Klein's prompt_upsampler is one: it rewrites every caption and needs an 18.8 GB component, so every Klein definition declares "remove_blocks": ["prompt_upsampler"] and the explicit upsample step is the only upsampler. A name the pipeline has no block for raises at load, listing the blocks it has (validate does not build the pipeline, so it cannot see this).

  • components_manager — attaches a ComponentsManager, which tracks the pipeline's components. With enable_auto_cpu_offload it keeps only the running components on the device and moves the rest to system memory, reserving memory_reserve_margin (default "3GB") of free device memory. It requires a device that reports free memory (CUDA) and replaces offload, which modular pipelines do not support.

A modular pipeline returns whatever its output argument asks for — one output by name, or several of them together:

"arguments": {
    "prompt": "variable:prompt",
    "output": ["videos", "audio", "sampling_rate"]
}

Asked for several, the outputs come back keyed by name. Video generated with its own soundtrack is muxed into a single video/mp4 file, the same way a video pipeline's own output is, and a later step can still reference any of the outputs by name.

Some repositories hold more than one task's weights. workflow names the task, which prunes the pipeline to the blocks that task runs, so only the components it needs are downloaded and loaded:

"from_pretrained_arguments": {
    "model_name": "MiniMaxAI/MiniMax-H3",
    "workflow": "t2va"
}

A task is chosen by the arguments the step passes, so one workflow name can cover more than one of them: MiniMax-H3's fl2va takes an image, a last_image, or both. Given only a last_image it generates up to that frame, inventing everything that leads to it — see workflows/templates/minimax/last-frame-only.json beside workflows/templates/minimax/first-and-last-frame.json.

See workflows/templates/minimax/music.json and workflows/templates/minimax/video-with-audio.json for full examples.

H3: generating to a held soundtrack

A MiniMax-H3 step (t2va, fl2va or ref2va) takes a pipeline argument hold_audio: an audio value, spelled asset:<file>, output:<file> or previous_result:<step>.audio (an AudioTrack or AudioVideo). The video is generated to that soundtrack. The track goes through H3's Ref2VA reference-audio encode, is written over the target audio rows and held clean as conditioning through denoise, so the model draws picture that fits audio it is not allowed to change:

"pipeline": {
    "arguments": {
        "prompt": "a drummer in a lit rehearsal room",
        "num_frames": 125,
        "hold_audio": "asset:track.wav"
    },
    "output": ["videos", "audio", "sampling_rate"]
}

"audio" must be in output (as it already is in video-with-audio) for the held track to be what the step returns. A track longer than the video is cropped, a shorter one padded with silence. The step's audio is the caller's original waveform fitted to the video's duration (num_frames / 24 s) at its own sample rate, not a VAE round trip. previous_result:base.audio holds a track an earlier step generated, music for example.

run_workflow's arguments can set only a declared variable, so unless the template has one, add "hold_audio": "variable:hold_audio" and a hold_audio variable, or edit the argument. Validation refuses hold_audio on a step that is not H3, on a modular pipeline loaded with a workflow other than t2va, fl2va or ref2va, on a path whose extension is not .wav, .mp3, .flac or .ogg, and on any {"media_type": ...} dict; a run refuses a pipeline with no hold blocks or a value that is not audio. Without hold_audio the output is unchanged.

Hold plus the reference is the music-video default: the template passes each shot's slice as a MiniMaxH3AudioReference and its hold_audio variable defaults to previous_result:slice; set it to null for the reference alone. The match_audio chains (chain-matched-to-audio, chain-matched-and-aligned) keep hold_audio null by default, since a chain has no per-shot slice to name; set it to an audio file to hold. What was measured:

  • Reference plus hold keeps the audio in sync (#808, music-video, 4 shots of 124 frames at two seeds, measure_sync against each source slice): 0.00 s offset on all eight shots, correlation 0.95 to 1.00. The reference alone put one shot 1.37 s late and another 0.18 s early, correlation 0.35 to 0.96. That measured the audio, not the mouth.
  • Hold with image-only references keeps the soundtrack exact but weakens the singing (#619, a 10 s sung track): the mouth was open at about 3 of 6 sung-word onsets against 6 of 6 with the audio reference alone, and read as speech to camera.
  • On fl2va, hold is the locked-frame lip-sync route (#985, a first frame plus a held 5.17 s song): frame 0 kept the picture's pixels and the mouth followed the song as in an audio-referenced Ref2VA shot, at about a fifth of the denoise time of Ref2VA [keyframe completion + audio reuse], which keeps only the framing.

Hold is also for picture that must fit audio exactly, such as motion cut to music. refine_strength uses hold differently, to keep the base pass's own jointly generated audio (see the refine section below).

H3: holding a clip with guides

A MiniMax-H3 t2va or fl2va step takes a pipeline argument guides: a list of {"video": <reference>, "frame": <pixel frame>}, where the video is an asset:, output: or previous_result: reference. Each guide clip is VAE-encoded on the generation canvas and placed as condition rows, timed from the target frame it lands on, that are never denoised, so the generated video is held to earlier footage. On fl2va the image / last_image keyframes combine with guides. The clip is fitted to the canvas (scaled to cover, centre-cropped). By default a guide holds video only and its own audio is not used; "audio": true holds the guide's audio too (below).

"pipeline": {
    "arguments": {
        "prompt": "variable:prompt",
        "num_frames": "variable:num_frames",
        "width": "variable:width",
        "height": "variable:height",
        "num_inference_steps": "variable:num_inference_steps",
        "guides": [
            {"video": "asset:opening.mp4", "frame": 0},
            {"video": "asset:closing.mp4", "frame": 85}
        ],
        "output": ["videos", "audio", "sampling_rate"]
    }
}

This is the text_to_video_audio step of workflows/templates/minimax/video-with-audio.json with guides added. As with hold_audio, run_workflow's arguments can set only a declared variable, so reference a guides variable or edit the argument.

Rule Value
frame a multiple of 17 (a VAE chunk boundary): 0, 17, 34, ...
clip length 1, 5 or 17m + 5 frames (22, 39, 56, ... 124); any other length is cut down to the longest such length (n < 5 gives 1, 5 <= n < 22 gives 5, else 17 * ((n - 5) // 17) + 5) with a warning
extent frame + length must not run past num_frames (H3 rounds it up to 17n + 5; default 124); ending exactly at the end is fine
count at most 4 guides per step. Each guide is also checked against the card: each guide frame adds attention rows, and the template's vram_estimate charges each guide's frames on the canvas; guides: [] is the same as none
where t2va or fl2va on an H3 pipeline; not with references (ref2va), not on a non-H3 pipeline

Validation refuses all of the above before the run (dw/guides.py); a previous_result: guide is checked at run time, when its clip exists.

Holding the guide's audio: "audio": true

A guide with "audio": true holds its soundtrack as well as its frames: {"video": "asset:opening.mp4", "frame": 0, "audio": true}. The guide video's audio over the guide's span is encoded by the audio VAE (posterior mode, as ref2va reference audio is) and placed as condition audio rows timed at the target's own audio time, then held, never denoised. The output's audio under the guide reproduces the guide's audio, and the model generates the rest to run on from it. Audio runs at 40 latents a second against 24 fps video, so a guide of n frames at frame k holds round((k + n) * 40 / 24) - floor(k * 40 / 24) audio latents: 37 for a 22-frame guide at frame 0, 65 (about 1.6 s) for a 39-frame one. A guide that does not start at frame 0 has its audio padded to the audio VAE's hop first, so its latents land at the right time. The clip is read as 24 fps footage, as its frames are.

Rule Value
audio true or false; absent is false. Any other value is refused
soundtrack "audio": true on a guide whose video has no audio stream is refused - before the run for an asset:/output: file, at run time for a previous_result: clip
with hold_audio allowed; a guide's audio rows come first, then the held track's

A guide holds its span's look as well as its composition and motion: the frames under it come out close to a copy of the clip, whatever the prompt says about style. A full-length guide at frame 0 reproduces the take rather than restyling it (a 124-frame guide prompted as "a hand-painted watercolour animation" came back photoreal, #648). The prompt steers only the frames the guides leave free; restyling a whole take needs a LoRA trained on guides (#612).

With no guides (or guides: []) the step runs the stock layout and draws the same noise from its seed as an unguided step. Repeat runs at one seed are bit-identical with the default cudnn_deterministic: true and cudnn_benchmark: false (ACCELERATION.md, TF32 and cuDNN). A server that turns benchmark on, or determinism off, can give two runs that match frame for frame by eye but whose audio levels differ by a few dB.

Loosening a guide's hold: guide_noise

A guide is held almost exactly, so the take freezes under it and leaves the guide's end with a visible change of pace. guide_noise (an H3 t2va/fl2va argument, a number in [0, 1), about 0.4 to start) loosens every video guide on the step: each guide frame gets Gaussian noise of standard deviation amplitude x 255 on its 0-255 pixels, where the amplitude falls on a cosine taper from guide_noise at the guide's first frame to 0 at its last, so the frames nearest the new material are held clean. The noised frames are encoded once, under the same encode seed as a clean guide; the noise draw is seeded per guide (keyframe_encode_seed + the guide's index), so a seeded run repeats. The rows are still held, never denoised, and the row plan, the refine schedule and guide audio ("audio": true) are unchanged. guide_noise: 0 (or absent) adds no noise, and the latent is bit-identical to a step without it.

It is a step argument, so in a continuity: "guide" chain it rides to every segment and loosens the carried guide too.

Rule Value
value a number in [0, 1); anything else, or a value outside the range, is refused naming the range. Accepts a variable: reference
where MiniMax-H3 t2va or fl2va, where guides run; refused on ref2va, on a step that passes references, and on a non-H3 step
no guides a warning, not a refusal: a step with no guides and no continuity: "guide" chain has nothing to noise

Chaining with a guide: continuity: "guide"

A chain on an H3 t2va or fl2va step can carry its seam with a guide instead of a keyframe: "chain": {"segments": 3, "continuity": "guide", "guide_frames": 22}. Every segment after the first gets the previous segment's last P = guide_frames frames as a guide at frame 0, appended to the step's own guides, with "audio": true when carry_audio is (the default), so motion and voice run on across the seam rather than restarting from a still. The carried clip counts toward the four-guide limit, so a step that writes its own guides may list at most three; validation says so before the run. On fl2va, image is set to the guide's first frame. The next segment opens with a near-copy of those P frames, so P frames are trimmed from its head: N segments of F frames give F + (N - 1)(F - P) frames - 3 x 124 at P = 22 is 328.

Rule Value
guide_frames 22 (default) or 39 - a whole-latent guide length; anything else is refused
where MiniMax-H3 t2va or fl2va; refused on ref2va, with references, or on a non-H3 step - guides stay off ref2va
carry_frames refused - the carry is always the last guide_frames frames
guide_backoff 0 (default) or a multiple of 17: the carry is the guide_frames frames that many before the segment's end (audio sliced the same span), and every segment but the last loses that tail at assembly, so the join lands inside the held copy and each segment's least-constrained tail is replaced by the next take. Frames drop by the back-off per seam (3 x 124 at P = 22 and 17 is 294). Needs num_frames above P + back-off. Refused off guide continuity and with match_audio or duration_seconds
keystone none (default) or segment_1: on fl2va, every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame, so each segment is pulled back toward the clean start of the chain. The guide stays the carried tail, and the keyframe region is trimmed at assembly, so segment 1's frame never reaches the output. Refused on last_frame (there the keyframe is the only carry), and any other value is refused naming the two
keystone_slot with keystone: "segment_1", where segment 1's frame 0 sits in each later segment. first (or null, the default) is plain keystone, the keyframe above. last makes it the fl2va last_image instead and the keyframe is the guide's first frame again: on a static two-minute take it held the backdrop and the face with no reset pop, and the subject eases back to the opening pose at each segment's end (bench/records/structural.md, Keystone slot). last needs a step image and refuses a step last_image of its own. {"frame": 17j} is a one-frame guide at that frame, past the guide's frames and inside the segment (a match_audio or duration_seconds plan's shortened last segment included); its enter_below_sigma gate is refused unless DW_EXPERIMENTAL_REFINE_WINDOWS=1. The name experimental_keystone_slot is refused naming this one. templates/minimax/chained-segments sets last under shot: static_long; under last, keystone_images replaces segment 1's frame 0 with each segment's own picture (the next row). Experimental, refused unless DW_EXPERIMENTAL_REFINE_WINDOWS=1: latent writes segment 1's frame-0 latent (its own output latent; on a resume, its decoded frame 0 encoded again) over each later segment's guide latent 0 for the whole denoise, with segment 1's frame 0 as the keyframe too, so the two condition rows at time 0 agree; the guide's own latent 0 is put back after the loop, before decode. It combines with experimental_latent_carry, a held track and experimental_capstone
keystone_images the moving keystone (docs/proposals/complete/narrated-long-take-complete.md, Amendment 1): with keystone: "segment_1" at keystone_slot: "last", one image per segment, in order, and segment k's fl2va last_image is entry k - 1 in place of segment 1's frame 0 - segment 1 included - so each segment ends on its own picture, such as a keyframe drawn along a walk, rather than being pulled back to the opening still. A list of image paths, asset:/output: references, URLs or previous_result: references (each its first image), or one previous_result:<step>.<key> naming a list of images: an image step's images, one per for_each entry, the first image of each (resolved like prompts and lines). With anchor_every, a later segment off the cadence gets no last image; on a probe (stop_after) and a resume_from, entry k stays segment k's. Refused, naming the key: without keystone: "segment_1" and keystone_slot: "last"; an entry that is not an image; off an H3 fl2va step; with a step last_image of its own; and fewer entries than the planned segments (the match_audio plan, segments or duration_seconds; the full plan's even for a probe, which is continued), naming both counts - at validate when the count can be told there, else before segment 1 renders. A held narration (match_audio with a step hold_audio) places them the same way, its guides carrying picture only. templates/minimax/narrated-take takes them as keyframes under shot: walk_keyframed
anchor_every Experimental, refused unless DW_EXPERIMENTAL_REFINE_WINDOWS=1: the keystone and experimental_capstone apply only on later segments whose 0-based index i has i % anchor_every == 0 - 3 anchors segments 3, 6, 9 (by index) and leaves the rest a plain guide chain. A whole number from 1, default 1 (every later segment); refused on a chain with neither keystone: "segment_1" nor experimental_capstone
carry_pick {"sharpest": N}, N from 1 to 8, with last_frame continuity on any model family: the next segment's keyframe is the sharpest of the segment's last N frames (variance of a 3x3 Laplacian on luma, ties to the latest) instead of the last, so a motion-blurred final frame does not pass its softness on. The segment is trimmed after the picked frame so motion continues from it (the last segment keeps its tail), and the job log names the pick per seam. Absent = the last frame; other continuities or other values are refused naming the allowed ones; match_audio and duration_seconds are refused (the plan covers the length up front and the pick drops frames per seam, so the video would fall short - use segments); and when num_frames is known N must be less than num_frames minus trim_frames, or the pick could empty a segment
look_hold true or {"pickup", "creep", "texture", "precompensate"} booleans, any model family: pixel corrections on each later segment's frames before its carry is taken (below). On a guide chain pickup and precompensate do not apply (the carried guide is a whole clip, left alone); an explicit true for either there is refused, and true alone runs creep and texture only
trim_frames ignored (the trim is P), with a note in the job log
crossfade_ms ignored when carry_audio is true: the guide held the audio, so the seam is joined with only a declick
num_frames must exceed P, or the chain cannot progress
guide_noise rides on the step's arguments to every segment, so segments 2+ noise the carried guide (above)

Holding a chain's look in pixels: look_hold

latent_anchor works inside the H3 latent; look_hold works on the decoded frames, so it suits any family. Each segment after the first is corrected before its carry is taken, so the carried keyframe and the stitched video are the corrected frames (applied once, not again to the stitched output). Segment 1 is the reference and is left as rendered. "look_hold": true turns all four on; an object turns on the ones it names. The template's look_hold variable is typed bool by its default, so arguments takes true or false only: an object there is refused, naming the variable. The per-flag object form is reachable only by editing the workflow (an inline copy of the template):

  • pickup: matches the segment's opening (frame 1; frame 0 is the keyframe copy) to the carried frame by a per-channel quantile map on YCbCr, carries the same map to the last frame and blends it per frame. Skipped on a flat frame, and on a jump too large to be drift (a cut).
  • creep: when the last frame's contrast or saturation exceeds segment 1's by more than 0.03, scales both down by a factor running from 1 at the first frame to the needed gain at the last. Gains are at most 1: a darker or paler shot is untouched.
  • texture: when either end's faint texture (the flat-area detail, not edges) exceeds the roughest of segment 1's first, middle and last frames by more than 0.03, subtracts up to half of it, easing in across the shot.
  • precompensate: divides the carried keyframe's contrast and saturation by the boost measured so far (the running mean of the rendered opening's look over the fed keyframe's, each clamped to 1..1.25), so a model that boosts its own keyframe lands back on the look.

pickup and precompensate act on one carried keyframe, so they need continuity last_frame; the carried frame of a guide chain is a whole guide clip and is left alone. Absent or false, the chain is unchanged bit for bit. Concepts from Smite79/MiniMax-H3-LongVideos (ATTRIBUTIONS).

Anchoring a chain's colour: latent_anchor and anchor: "segment_1"

A long H3 chain drifts in colour and contrast: each segment's latent statistics wander a little from the last, and the carry passes the drift on. latent_anchor (an H3 t2va/fl2va argument) takes video frames - a path, an asset:/output: reference, or a step's frames - VAE-encodes them on the render's canvas the way a guide is encoded, and takes each latent channel's mean and standard deviation over every position. After denoising and before decode, the generated video latent is moved onto them, channel by channel: (x - mean_x) / std_x * std_a + mean_a. The audio is never touched. By default it is a full match; latent_anchor_strength (a number in [0, 1], default 1) blends it with the untouched latent, x + strength * (matched - x), so 0.5 moves each channel halfway and 0 changes nothing. Without latent_anchor the step is unchanged, bit for bit, and its job log says nothing about an anchor. A call that applies one logs a line naming its source, such as segment 2/3: latent anchor applied - latent statistics from segment 1.

In a chain, "anchor": "segment_1" sets it for you: every segment after the first gets segment 1's frames as its latent_anchor, so the whole chain is held to the first segment's look. Segment 1 itself is not corrected. "anchor_strength": 0.5 sets latent_anchor_strength on those segments, for a chain that should hold its look without flattening a deliberate change in light.

Rule Value
anchor none (default) or segment_1; anything else is refused, naming the values
anchor_strength a number in [0, 1] (default 1, the full match); refused outside it, and refused without anchor: "segment_1". A step's latent_anchor_strength is the same number, and needs a latent_anchor
segment_1 continuity last_frame or guide; refused with any other
where MiniMax-H3 t2va or fl2va, where guides run; latent_anchor and segment_1 are refused on ref2va, on a step that passes references (that call runs ref2va), and on a non-H3 step
segment_1 carry refused with segment_argument: "references" - the later segments would run ref2va
frames cut to a whole-latent length (1, 5 or 17m + 5)
refine_tiles refused beside latent_anchor, as the other per-call H3 arguments are

workflows/templates/minimax/chained-segments.json takes continuity (default guide, measured best for every shot type; last_frame is the uncommon case) and guide_frames as variables. The ref2va chains (chain-video-continuity and the match_audio chains) keep last_segment.

Promoting an H3 take to 768p in latent space: upscale_h3_latents and decode_h3_latents

Once a 960x544 MiniMax-H3 take reads the way it should, upscale_h3_latents and decode_h3_latents promote it to 1344x768 without denoising it again: only a small 3D-convolution network and a VAE decode run, not the transformer. With no refine pass the result is sharper than the 544p take but cannot show detail the base pass never generated; the refine section below adds one. Measured on a 124-frame crowd scene, it took 7.8 min against a native 768p render's 12.7 min, and the faces came out soft and waxy where the native render's were distinct. When faces matter, use the catalog's native 768p render, templates/minimax/video-with-audio-768p.

Target width/height must be multiples of 16, each between 1x and 4x the base latents' own size, and within the upscale cap of 1920x1088 (or portrait 1088x1920); validation refuses a literal size off the grid or over the cap before the run. Above 1344x768 the plate is for a tiled refine (Refining above 768p in tiles), and upscale-only it is soft. The latents pass from base to up to decode in memory. A previous_result: property on a modular step names a key of the dict its output list returns, spelled as the pipeline spells it: base.latents, base.audio and base.sampling_rate (not sample_rate, which the dict does not carry). A property no result carries is an error, so a misspelt one fails the step. The base step carries no result, so the 544p pass is not written.

The upscaler weights (LBH-123-AI/Minimax_h3_latent_Upscaler, read at a pinned revision) and the H3 VAE download on first use. While the upscaler file is not cached, a validate's plan.downloads_required lists it as a file entry ({repo, filename, revision, ...}); the VAE is covered by the pipeline step's MiniMaxAI/MiniMax-H3 entry.

{
    "id": "H3LatentUpscalePreview",
    "description": "Promote a MiniMax-H3 take from 960x544 to 1344x768 in latent space, decoded without refinement.",
    "variables": {
        "prompt": "prompt:minimax/fox_dawn_context_ir",
        "num_frames": 124,
        "num_inference_steps": 9,
        "video_shift": 12.0,
        "audio_shift": 3.0,
        "weights_dtype": "{int4}",
        "lora_scale": 1.0,
        "lora_alpha": null,
        "lora_model_name": "lightx2v/Minimax-h3-Turbo",
        "lora_weight_name": "minimax_h3_fl2v_turbo_8step_v1.0_bf16.safetensors",
        "lora_adapter_name": "turbo",
        "seed": 42
    },
    "variable_constraints": {
        "num_frames": {
            "modulus": 17,
            "remainder": 5,
            "min_frames": 124,
            "max_frames": 345,
            "snap": "up",
            "reason": "the video VAE encodes 17 * n + 5 frames, and MiniMax-H3 generates between 5 and 15 seconds at 24 fps"
        }
    },
    "seed": "variable:seed",
    "steps": [
        {
            "name": "base",
            "pipeline": {
                "configuration": {
                    "component_type": "ModularPipeline",
                    "pre_load_modules": [
                        "sdnq"
                    ],
                    "load_components": {
                        "dtype": "torch.bfloat16",
                        "quantization_config": {
                            "transformer": {
                                "configuration": {
                                    "config_type": "sdnq.SDNQConfig"
                                },
                                "arguments": {
                                    "weights_dtype": "variable:weights_dtype",
                                    "quantization_device": "cuda",
                                    "return_device": "cpu",
                                    "use_quantized_matmul": true,
                                    "dequantize_fp32": false,
                                    "modules_to_not_convert": [
                                        "proj_in",
                                        "audio_proj_in",
                                        "context_embedder",
                                        "time_embedder",
                                        "time_proj",
                                        "token_refiner",
                                        "norm_out",
                                        "proj_out",
                                        "audio_proj_out"
                                    ]
                                }
                            },
                            "text_encoder": {
                                "configuration": {
                                    "config_type": "sdnq.SDNQConfig"
                                },
                                "arguments": {
                                    "weights_dtype": "variable:weights_dtype",
                                    "quantization_device": "cuda",
                                    "return_device": "cpu",
                                    "dequantize_fp32": false,
                                    "modules_to_not_convert": [
                                        ".model.visual",
                                        "lm_head"
                                    ]
                                }
                            },
                            "vae": {
                                "configuration": {
                                    "config_type": "sdnq.SDNQConfig"
                                },
                                "arguments": {
                                    "weights_dtype": "{int8}",
                                    "quant_conv": true,
                                    "use_quantized_matmul_conv": true,
                                    "quantization_device": "cuda",
                                    "return_device": "cpu",
                                    "dequantize_fp32": false
                                }
                            }
                        }
                    },
                    "components": {
                        "transformer": {
                            "group_offload": {
                                "offload_type": "block_level",
                                "num_blocks_per_group": 2,
                                "use_stream": true,
                                "record_stream": true,
                                "low_cpu_mem_usage": true
                            }
                        },
                        "text_encoder": {
                            "remove_modules": [
                                "lm_head"
                            ]
                        },
                        "text_encoder.model": {
                            "truncate_layers": {
                                "language_model.layers": 51
                            },
                            "group_offload": {
                                "offload_type": "leaf_level"
                            }
                        },
                        "vae": {
                            "device": "cuda",
                            "residency": "on_demand"
                        },
                        "audio_vae": {
                            "device": "cuda",
                            "residency": "on_demand"
                        }
                    }
                },
                "from_pretrained_arguments": {
                    "model_name": "MiniMaxAI/MiniMax-H3",
                    "workflow": "t2va"
                },
                "loras": [
                    {
                        "model_name": "variable:lora_model_name",
                        "weight_name": "variable:lora_weight_name",
                        "adapter_name": "variable:lora_adapter_name",
                        "scale": "variable:lora_scale",
                        "alpha": "variable:lora_alpha"
                    }
                ],
                "scheduler": {
                    "shift": "variable:video_shift"
                },
                "audio_scheduler": {
                    "shift": "variable:audio_shift"
                },
                "arguments": {
                    "prompt": "variable:prompt",
                    "num_frames": "variable:num_frames",
                    "width": 960,
                    "height": 544,
                    "num_inference_steps": "variable:num_inference_steps",
                    "output": [
                        "videos",
                        "audio",
                        "sampling_rate",
                        "latents"
                    ]
                }
            }
        },
        {
            "name": "up",
            "task": {
                "command": "upscale_h3_latents",
                "arguments": {
                    "latents": "previous_result:base.latents",
                    "width": 1344,
                    "height": 768
                }
            }
        },
        {
            "name": "decode",
            "task": {
                "command": "decode_h3_latents",
                "arguments": {
                    "latents": "previous_result:up"
                }
            }
        },
        {
            "name": "mux",
            "task": {
                "command": "pair_audio",
                "arguments": {
                    "video": "previous_result:decode",
                    "audio": "previous_result:base.audio",
                    "sample_rate": "previous_result:base.sampling_rate",
                    "fit": "video"
                }
            },
            "result": {
                "content_type": "video/mp4",
                "subfolder": "final"
            }
        }
    ]
}

To keep the 544p take beside the promotion for comparison, add one more pair_audio step with "video": "previous_result:base.videos" and the same audio/sample_rate (and fit), saved with "result": {"content_type": "video/mp4", "subfolder": "intermediate"}. base.videos is the pipeline's batch - a list holding the one video - and pair_audio unwraps a batch of one (a batch of several is refused, since one track goes under one video). The H3 step's videos carry the model's 24 fps, so pair_audio fits the track against 5.17 s and the file is written at 24 fps without being told. Frames that carry no rate (a plain list from a task) need pair_audio's own fps argument: fit measures the frames before the writer applies a declared rate, so without it the track is padded to the 8 fps fallback's length, three times too long.

Refining the upscaled latents: refine_strength

upscale_h3_latents followed by decode_h3_latents adds no detail. To let the transformer add some, pass the upscaled latents to an H3 step as latents with refine_strength. The step adds noise to the video latents up to sigma = refine_strength and denoises from there down to 0, so the picture is kept and the fine detail is regenerated at 1344x768. The refined video is decoded by the step itself, so there is no decode_h3_latents step, and the step's videos is the deliverable.

Refine re-denoises the video only. The audio rows have to be held, so pass the base pass's track as hold_audio (previous_result:base.audio) and the step's audio is that track. This is a different use of hold from the one above: it keeps the audio the base pass generated jointly with that video, rather than fitting picture to supplied audio. For lip sync to supplied audio see the hold section above (#619, #795).

The workflow is the one above with decode and mux replaced by a refine step. refine carries a copy of the base step's whole pipeline block - the same configuration, from_pretrained_arguments, loras, scheduler and audio_scheduler, so the loaded model is reused - with these arguments, and the step saves the result:

{
    "name": "refine",
    "pipeline": {
        "arguments": {
            "prompt": "variable:prompt",
            "num_frames": "variable:num_frames",
            "width": 1344,
            "height": 768,
            "num_inference_steps": 5,
            "output": ["videos", "audio", "sampling_rate"],
            "latents": "previous_result:up",
            "refine_strength": 0.2,
            "hold_audio": "previous_result:base.audio"
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24, "subfolder": "final" }
}

num_inference_steps is the number of sigma points, ending at 0, spaced by the scheduler's shift, so a refine runs num_inference_steps - 1 denoise evaluations. Here 5 points at shift 12 (the example's video_shift) and strength 0.2 are sigma 0.2, 0.157, 0.110, 0.058, 0: 4 evaluations. The time a refine takes scales with num_inference_steps. refine_strength sets where it starts, not how many steps run: a higher strength moves the picture further from the upscaled take, a lower one stays closer to it.

Validation refuses a refine_strength that is not a number between 0 and 1 (exclusive; a regional edit also takes 1.0), a refine with no latents or no hold_audio, a step that is not H3, and a num_inference_steps below 2.

To refine a canvas above 1344x768, such as a 1920x1088 plate, see Refining above 768p in tiles. To give the schedule's sigma points yourself rather than spacing them by shift, see Explicit refine sigmas.

Explicit refine sigmas

A recipe published with its own schedule cannot be reached by spacing points with shift. refine_sigmas gives the schedule itself: the sigma points, from refine_strength down to 0, used verbatim for both the video and audio schedules, with no shift applied. NVlabs' Open-H3-Regenerate-2K recipe starts from (1 - 0.909375)·up + 0.909375·noise and runs 3 Euler steps on four points:

"arguments": {
    "latents": "previous_result:up",
    "hold_audio": "previous_result:base.audio",
    "refine_strength": 0.909375,
    "num_inference_steps": 4,
    "refine_sigmas": [0.909375, 0.857143, 0.666667, 0]
}

refine_strength still sets the re-noise, and the list replaces only the spacing. The step count is the number of points, so num_inference_steps is required beside the list and must equal its length: four points are 3 denoise evaluations. Validation, and the check before the call, refuse a refine_sigmas that:

  • is not a list of numbers, or has fewer than 2 points;
  • is not strictly decreasing;
  • does not start at refine_strength (within 1e-6);
  • does not end at exactly 0;
  • has a length other than num_inference_steps, or comes with no num_inference_steps;
  • comes with no refine_strength.

A step without refine_sigmas runs the shift-spaced schedule. refine_sigmas works with refine_tiles too: every tile runs the same list.

H3: regional edit with source_video and a mask

To change part of a take and keep the rest, pass a video as source_video on an H3 t2va, fl2va or ref2va step. It is a path, asset:, output: or previous_result: reference to a video, and it stands in for the latents of a refine: the step VAE-encodes it over the generated rows and re-denoises from refine_strength, which is required and in (0, 1]: 1.0 regenerates the masked region from pure noise, and about 0.9 replaces an object in it (see Choosing refine_strength for a regional edit). (Without a source_video the refine's range stays (0, 1).) With no mask and a hold_audio, the whole source is refined under that soundtrack. With no mask and no hold_audio, the call is not a refine at all: it is the audio pass (see Audio over a finished picture).

{
    "source_video": "asset:fox_dawn.mp4",
    "mask_image": "asset:fox_mask.png",
    "mask_grow": 1,
    "refine_strength": 0.9,
    "hold_audio": "asset:fox_dawn.mp4",
    "prompt": "the fox holds a pine cone in its mouth, same light and motion"
}

A mask_video (one frame per frame) or a mask_image (one mask for every frame) limits what is regenerated. Pass one, not both. White regenerates and black is kept, binarized at luma 128. The mask is max-pooled onto the transformer's patch grid, 16 px times the patch size per cell, so any white pixel marks its whole cell, and mask_grow widens it by that many cells (a whole number, 0 or more, default 1). In time the grain is the latent frame: about 4 pixel frames each (3.4 - every 17 frames encode to 5 latent frames), times the patch's time size, so a mask_video switching on or off mid-cell marks the whole cell. After every denoise step each other row is written back as the source noised to that step's sigma, with the refine's own noise, so at the end it is the source latent. The decode still round-trips it through the VAE, so kept pixels are near-identical to the source, not bit-identical. A higher refine_strength replaces what the mask covers, a lower one keeps its layout.

Choosing refine_strength for a regional edit

The useful range sits high. Measured on one take (a 960x544, 124-frame H3 shot of a woman holding a cup, the mask over the middle of the frame around her and the cup, mask_grow 1), seed 1062, 5 turbo steps, video_shift 6, with a prompt asking for a bright red raincoat:

refine_strength Inside the mask Inside-mask SSIM vs the source
0.85 or less A light rework: re-detailed, the source's objects and colours stay 0.86 at 0.8, 0.83 at 0.85
0.9 A held object is replaced (the cup became a domed sundae cup); the same person, pose and clothing 0.78
0.95 Clothing is reworked too (a hooded navy jacket) and the object replaced; face, hair and pose mostly hold 0.65
1.0 A fresh generation: the prompt lands in full, but a different person and composition 0.44

Outside the mask held at every strength (SSIM about 0.98 on a crop outside it). So: 0.9 to replace an object, 0.95 to rework clothing, 0.85 or less for a light rework. A large recolour of a big region, such as a dark top to a bright red coat, did not land below 1.0, and 1.0 gives up the region's identity: that is a fresh generation, not a regional edit.

Why the numbers sit high: refine_strength is the starting sigma σ₀ on the shifted grid, the same as for any refine (and refine_tiles). The scheduler's shift maps the unshifted position u to σ = shift·u / (1 + (shift − 1)·u), so the start point on the unshifted schedule is u = s / (shift − (shift − 1)·s). At video_shift 6 that is 0.8 → 0.40, 0.85 → 0.49, 0.9 → 0.60 and 0.95 → 0.76: a strength of 0.8 re-noises less than its number suggests. The bands above hold for those conditions only. Another video_shift or step count moves them, the way the formula says: a lower shift brings u closer to s, so the same change comes at a lower strength. A refine_sigmas list is run as given, with no shift, so its points are the schedule itself.

The source and the mask must match the step's width x height, and a source_video or mask_video its num_frames. A step that leaves width, height or num_frames out renders the source's. An all-black mask is refused, as it would regenerate nothing. Validation reads a source or mask that resolves already (a path, asset: or output:) and refuses a size, frame count or empty mask there, the way it reads guide videos; a previous_result: is checked at the call.

hold_audio is still required with refine_strength, since the refine holds the soundtrack (a source_video with neither a mask nor hold_audio is the audio pass instead). It is normally the source_video itself: a video path (.mp4, .mov, .mkv, .webm, .avi) is held by its own soundtrack, so the edit keeps the take's audio. A video with no audio stream is refused. An audio file (.wav, .mp3, .flac, .ogg) or an earlier step's audio-carrying result works too.

Validation and the call both refuse:

  • source_video with latents, or on a chained step;
  • both mask kinds, mask_video and mask_image, on one step;
  • a mask with no source_video, or mask_grow with no mask;
  • source_video with a mask or hold_audio and no refine_strength, or one outside (0, 1];
  • source_video on a pipeline that is not H3 (mask_image there is left alone, since inpaint pipelines own that name);
  • a source_video whose frame count (and so the step's num_frames, inferred or given) is off the 17n + 5 grid: the pipeline would snap the count up and pad the source, and the mask would drift in time - trim or pad the clip to the grid first (the message names both neighbours). An inferred width and height still have to meet the family's canvas rules (multiples of 32), which the template's variables carry and an inferred size does not check;
  • any of it with refine_tiles, which does not combine with a regional edit.

Audio over a finished picture

A source_video with no mask and no hold_audio is an audio pass: the picture is finished and only the soundtrack is generated, from the prompt - its soundscape, its music and its <d> lines. Every video row of the source is held as the exact clean latent at every step, pinned clean in the row plan the way a reference row is, so no video row is ever stepped. The audio runs the stock audio schedule from full noise, attending to the real picture from the first step. The step returns the source's own frames, not a VAE round trip of them, under the new track. A silent source is the usual case and is fine.

{
    "source_video": "asset:silent_take.mp4",
    "listen_scale": 1,
    "prompt": "... overall_soundscape: ... <d>[English] Over here.</d> ...",
    "num_inference_steps": 12,
    "output": ["videos", "audio", "sampling_rate"]
}

Run it without a distillation LoRA, on the dialogue schedule (num_inference_steps 12, video_shift 12, audio_shift 3): the turbo LoRA is what makes H3 voices sound digital (#984). listen_scale (1, 0.5 or 0.25, default 1) encodes the source at that fraction of its canvas, area-resampled and snapped to 32 px, which is cheaper per step. The scale keeps the source's aspect: the shorter side is snapped and the longer derived from it. The audio's length still follows the source's duration, and the picture returned is the full-size source. width, height and num_frames stay the source's, as for an edit.

The result keeps the source's own frame rate. A source at 24 fps is conditioned on as it is. A source at another rate (30 fps, say) is conditioned on a copy retimed to H3's 24 fps over the same duration, nearest frame, its count snapped up onto the 17n + 5 grid. The step returns the source's own frames at the source's own rate, with the track trimmed to the source's duration, so its own frame count need not be on the grid. The rate comes from the file or the step result that supplied the video, or else from the length of its soundtrack. A source with neither is refused.

Validation and the call refuse refine_strength, refine_sigmas or refine_tiles on an audio pass (no video row is re-denoised; pass hold_audio for a whole-source refine instead), a listen_scale other than those three, a listen_scale on a step that is not an audio pass (on any pipeline, H3 or not), and an audio pass on a chained step or beside latents. The template is templates/minimax/add-sound.

What it is for, by ear (Don, 2026-11-06, bench/records/audio.md): at listen_scale 1 a line the source's picture was generated with lands on the mouth ("very good lip sync"). At 0.5 lip sync is still acceptable, about 23% faster: the fastest scale that holds it. At 0.25 lip sync is lost, so use it only for a clip with no on-screen speech. It is not a re-voice: the picture is the source's own pixels, so a new or regenerated line over an on-screen speaker has no mouth drawn for it and plays as dubbed voice-over. Use it to sound a silent clip, or to replace an off-screen or voice-over line.

Refining above 768p in tiles

H3's refine runs at 1344x768 at most, and upscale_h3_latents reaches 1920x1088 (or 1088x1920). An upscale-only plate above 768p is soft: upscale_h3_latents to 1920x1088 alone, decoded, is not a 1080p render, since the interpolation adds no detail. To put the detail in, give the refine step refine_tiles, which refines a latents canvas bigger than H3's own in overlapping tiles. It is a second argument to refine_strength, not a replacement: the strength, the held audio and the step count mean what they did above.

{"refine_tiles": {"width": 1344, "height": 768, "overlap": 256}}

The step plans the tiles on the 32 px grid: the fewest per axis whose overlap is at least the requested one, spread evenly, the last flush to the edge. A 1920x1088 canvas at 1344x768 with overlap 256 is 2x2, tiles at x 0 and 576, y 0 and 320. The noise is drawn once for the whole canvas from the step's generator and cropped per tile, so overlapping tiles start from the same noise. Each tile is denoised through H3 without its decode block, and the prompt is encoded once. The tiles are blended in latent space with a linear feather across each overlap, the weights normalized, and the blended canvas is decoded once with VAE spatial tiling. The step returns the full-canvas latents, the video and stage 1's held audio, once. Its videos carry H3's 24 fps, so the pair_audio below writes them at that rate without being told. The job's events show one log line per tile and one decoding phase.

It is slow. A 1920x1088 refine is roughly 4 tiles, each a 768p refine, plus a 1080p decode: about 12 minutes measured on an RTX 3090 for 124 frames (the recipe below, with its 544p base), against about 9.6 minutes for a native 768p take. The validate projection prices the denoise at the tile's size and the decode at the canvas's, so a tiled refine does not warn as if it denoised 1080p at once.

The workflow is the refine one above with four changes. The variables set the 768p turbo LoRA and its schedule: "num_inference_steps": 5, "video_shift": 6.0 and "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors" (as workflows/templates/minimax/video-with-audio-768p.json does, alpha left to the file); base and refine both read them. up targets "width": 1920, "height": 1088. The refine step's arguments drop width and height and add refine_tiles, and it saves nothing itself. A mux step pairs the refined video with the held audio and saves it:

{
    "name": "refine",
    "pipeline": {
        "arguments": {
            "prompt": "variable:prompt",
            "num_frames": "variable:num_frames",
            "num_inference_steps": "variable:num_inference_steps",
            "output": ["videos", "audio", "sampling_rate"],
            "latents": "previous_result:up",
            "refine_strength": 0.2,
            "refine_tiles": { "width": 1344, "height": 768, "overlap": 256 },
            "hold_audio": "previous_result:base.audio"
        }
    }
},
{
    "name": "mux",
    "task": {
        "command": "pair_audio",
        "arguments": {
            "video": "previous_result:refine.videos",
            "audio": "previous_result:base.audio",
            "sample_rate": "previous_result:base.sampling_rate",
            "fit": "video"
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24, "subfolder": "final" }
}

The refine step carries no width or height: the canvas is the latents. Naming them is allowed, and they must equal the canvas, which is the latent height x width x 16.

Validation and the run refuse, naming refine_tiles:

  • refine_tiles without refine_strength, or without what refine needs: latents, hold_audio, an H3 step and num_inference_steps of at least 2;
  • a width, height or overlap that is not an integer multiple of 32;
  • a tile larger than 1344x768 (or 768x1344), or larger than the canvas;
  • an overlap below 192, or at or above half the tile's shorter side;
  • a reference image or video, a <Picture N>, a keyframe or guides on the step;
  • an explicit width or height that does not match the latents canvas.

Experimental: refining a long take in time windows

Experimental, off by default. experimental_refine_windows: {"frames": W, "overlap": O} (latent frames) on a refine_strength step refines the take in overlapping time windows fused at every step. It needs DW_EXPERIMENTAL_REFINE_WINDOWS=1 in the server's environment and a server restart, and a step that sets it without the flag is refused. Only such a step may be longer than H3's 15 s. Every other H3 call keeps the ceiling.

A take over 15 s is three steps:

  1. encode_h3_latents with source set to the finished 24 fps take (output: or asset:). It reads and encodes one 17-frame VAE clip at a time, and the result is bit-identical to the whole-take encode. It returns latents, num_frames (the take padded with its last frame up to the next 17m + 5) and source_frames (the take's own count).
  2. The H3 refine, with latents: previous_result:encode.latents, num_frames: previous_result:encode.num_frames, refine_strength, experimental_refine_windows and hold_audio set to the take's own soundtrack (hold_audio reads a video's track, so the take's .mp4 can go in directly). The held track is encoded whole, and that encode has no ceiling of its own. Put "latents" in the step's output.
  3. decode_h3_latents with latents: previous_result:refine.latents and num_frames: previous_result:encode.source_frames, which trims the padding off so the take comes back at its own length. pair_audio then puts back the take's own soundtrack, which is source_frames long.

A source_video over 15 s is refused on a windowed step, and the refusal names encode_h3_latents: that path would encode the whole take as one float32 tensor.

Chained Video Generation

Video pipelines generate short clips - a chain block on a pipeline step runs the pipeline once per segment and stitches the segments into one long video. The model loads once; each segment's last frame is carried into the next segment as its keyframe, the duplicated boundary frames are trimmed, and frames and audio are joined into a single file:

"pipeline": {
    "configuration": { "component_type": "LTX2ImageToVideoPipeline" },
    "from_pretrained_arguments": { "model_name": "Lightricks/LTX-2.5-Diffusers" },
    "chain": {
        "segments": 3,
        "trim_frames": 2,
        "crossfade_ms": 80
    },
    "arguments": { "prompt": "variable:prompt", "image": "variable:image" }
}
  • segments — how many times the pipeline runs. Total length is roughly segments * num_frames, minus trim_frames per seam.
  • match_audio — instead of a count, derive the length from the step's hold_audio track, or else the one audio reference in its references. The audio is sliced into frame-aligned per-segment chunks, each segment is generated against its slice - a held track's slice is that segment's hold_audio, a reference's replaces the reference - and the final video is muxed with the original, unsliced track - so the soundtrack has no seams at all. Under continuity: "guide" a held track's slice already starts at the guide span, so the guide carries video only: carry_audio is forced off, and the job log says so once. Validate warns on a chain step with hold_audio and no match_audio (every segment holds the whole track from its start) and on a held chain whose num_frames is not divisible by 3: H3's audio latents run at 40 a second against 24 fps, so the on-grid 17n+5 lengths are 141, 192, 243, 294 and 345. Requires num_frames (the per-segment length) and a frame rate, and a track: a step with neither hold_audio nor a references list is refused. One of segments, match_audio or duration_seconds must be given, and segments beside match_audio is refused (beside duration_seconds it is ignored). Validation asks both rules of the substituted block, so a variable: value is caught before the model loads.
  • duration_seconds — instead of a count, ask for the chain by its length: the plan covers round(duration_seconds * fps) frames (the same planner match_audio uses, so the last segment shrinks to what remains, snapped by frame_snap), and the stitched video and its generated audio are both trimmed to exactly that length, shot records included. Needs num_frames in the step's arguments and a frame rate (chain fps or a frame_rate argument). Refused with guide_backoff and carry_pick, which drop frames the up-front plan cannot count. A cost estimate scales by the segment count it implies. It overrides segments when both resolve (the run logs that segments was ignored), so the chained templates, whose duration_seconds variable defaults to null (a null variable drops the key), take just duration_seconds=20. It is refused with match_audio.
  • continuity — how continuity carries across segments. last_frame (the default) extracts each segment's last frame and passes it to the next segment - single-frame conditioning, which carries pose and colour. last_segment carries the previous segment itself (frames and its generated soundtrack) into the next as a video reference, which also carries motion, camera, and voice across the seam; it requires a segment_argument that takes a references list. guide (MiniMax-H3 t2va/fl2va only) lays the previous segment's last guide_frames frames into the next as a held frame-0 guide (see H3: holding a clip with guides, Chaining with a guide).
  • guide_frames — with guide, how many frames the guide carries: 22 (default) or 39.
  • guide_backoff — with guide, a multiple of 17 (default 0): carry the frames that many before each segment's end and drop that tail at assembly. Not with match_audio.
  • anchor — none (default) or segment_1 (MiniMax-H3 t2va/fl2va, with last_frame or guide continuity): every segment after the first is rescaled to segment 1's per-channel latent statistics before decode, holding the chain's colour and contrast (see Anchoring a chain's colour).
  • keystone — none (default) or segment_1 (with guide continuity only): every later segment's keyframe is segment 1's frame 0 instead of the guide's first frame (see Chaining with a guide).
  • keystone_slot — with keystone: "segment_1": first (default), last (segment 1's frame 0 as every later segment's fl2va last_image) or {"frame": 17j}, plus the experimental latent (segment 1's frame-0 latent over each guide's latent 0) (see Chaining with a guide).
  • keystone_images — with keystone: "segment_1" at keystone_slot: "last": one image per segment, segment k's fl2va last_image in place of segment 1's frame 0 (the moving keystone; see Chaining with a guide).
  • anchor_every — experimental: apply the keystone and experimental_capstone only on every n-th later segment (0-based index i % n == 0); default 1 (see Chaining with a guide).
  • anchor_strength — with anchor: "segment_1", how far that match goes: 0 to 1, default 1 (the full match).
  • carry_frames — with last_segment, bound the carry to the last N frames of the segment (the audio is cut to the same span). Unset carries the whole segment.
  • carry_audio — with last_segment or guide, whether the carried reference includes its soundtrack (default true).
  • segment_argument — where the carried frame or reference lands: image (default) for image-to-video pipelines, or references for reference-conditioned modular pipelines, where it is appended alongside the workflow's own.
  • trim_frames — image-to-video pipelines reproduce their keyframe as frame 0, so this many frames are dropped from the head of every segment after the first (default 1). The matching audio is used as crossfade material, so video and audio stay exactly in sync. It also bounds the crossfade window: trim_frames / fps seconds (at 24 fps, trim_frames: 2 allows the full default 75 ms fade).
  • crossfade_ms — equal-power crossfade applied to generated audio at each seam (default 75). Not used with match_audio, which keeps the original track.
  • fps — frame rate for the chain's audio math. Defaults to the pipeline's frame_rate argument; pipelines with a fixed rate need it set (MiniMax H3: 24).
  • frame_snap — the constraint the pipeline puts on num_frames, used to snap the final match_audio segment to a valid length. MiniMax H3 accepts 17n+5 frames between 124 and 345: { "modulus": 17, "remainder": 5, "min_frames": 124, "max_frames": 345 }. Where the workflow already declares that rule as a variable_constraints entry, write "frame_snap": "constraint:num_frames" instead, so the numbers live in one place (What a variable is allowed to be). A duration_seconds or match_audio chain that declares no frame_snap derives one when the step's num_frames is a variable: whose constraint has a modulus (the run logs that it did); when it cannot - a literal num_frames, or no constraint - validation warns at chain.frame_snap, since the shortened last segment is otherwise asked for a count the pipeline may floor in silence.
  • prompts — optional per-segment prompt list for narrative progression; segment i uses prompts[min(i, len - 1)]. The whole list may be one previous_result:<step>.<key> naming a JSON list of strings (a narration_lines step's prompts); a result that is not a list of strings is refused at run time, naming the step and key.
  • lines — optional list of strings, one expected spoken line per segment, exactly as many as segments (unlike prompts, the last is not repeated; a wrong length or a non-string is refused). Read only by the words check. Segment k's line is lines[k], else the <d> dialogue of prompts[min(k, len - 1)], else of the step's prompt. An empty string, or no <d>, means no line: the shot records words: null and the segment is not re-rolled for it. Validate warns when lines is set but the words check is off. Like prompts, lines may be one previous_result:<step>.<key> naming a list of strings (a narration_lines step's lines); validate accepts it unresolved and the run checks its length against the plan.
  • check — check each segment after the first before it is kept (any model family): true for the default ceilings, an object to set any of them, absent or false for no check (the chain is unchanged). See Checking each segment below.
  • retries — with check on, how many times a segment that fails is re-rendered, 0 to 3 (default 0). Above 0 without check is refused.
  • save_segments — write each completed segment to the output directory as a playable mp4 and free its frames, bounding memory to roughly one segment regardless of chain length. The final video is streamed from the segment files at save time, and they are removed once it is written (keep_segments: true retains them). A crashed chain leaves the finished segments behind - stitch them by hand by listing their paths in a concat_videos step (trim_frames: 0, the trim was already applied), or resume the chain from them (resume_from). Requires PyAV and a frame rate. The trade-off is one extra encode/decode cycle through h264 for the segment files. Both chain templates set it on.
  • resume_from — resume a chain that died part way: an output: reference naming one segment file an earlier run of the same chain left. A chain that fails after spilling a segment says so in a chain_resume_hint job warning quoting the first file's reference (output:<workflow>/<run id>/<prefix>.0.segment-000.mp4); a failed run's files resolve by name, with or without its manifest. Rerun with the same settings and that reference: its finished segments become segments 1..k and only k+1..N render, which the estimate prices (N - k segments; a reference that does not resolve yet is priced for every segment and says so). That scales a figure the quote already has - the catalog cost or this server's history of the workflow at the same drivers; a workflow with neither (a template never run here, a different segments) stays basis: unknown under resume_from as without it. A chain whose segment count the estimate cannot tell, or a reference keeping N or more segments, is not scaled and differs names it. Each kept file must hold the frames the plan puts on the timeline for its segment, or the run is refused naming the segment and both counts - a resume never changes the plan. The files are copied, not moved, into the new run's spill, so the new take stands on its own and keep_segments names them on its shots (with resumed: true). The carry is read off the last kept file: its last frame (last_frame), or its last guide_frames frames and their audio (guide). That first resumed seam's audio guide is the file's decoded AAC track, not a live carry, which the run's chain_resumed warning ("Resumed k of N segments from ...") says. anchor and keystone read segment 1 from the first kept file; the kept segments are not checked again, and the first rendered one is checked against the last kept one as usual. look_hold reads its scene from the first kept file and is fed the last one's carry, but its learned boost starts again. Needs save_segments: true and continuity last_frame or guide; refused as a bare path. Under the experimental experimental_latent_carry the first resumed seam's guide is the last kept file's decoded frames, encoded again (the carried latents are not on disk), which the chain_resumed warning says (reencoded_guide: true); later seams carry latents again. The kept seams' audio is joined with the short declick fade the chain gives a seam with no head material, not the crossfade_ms blend: a kept file holds only its on-timeline audio, so the blend material is gone (the first rendered seam is crossfaded as usual). All kept files must carry audio or none may, and the rendered segments must agree with them, or the run is refused rather than laying a track shorter than the picture. Under a previous_result fan-out each iteration resumes from its own files: the reference names the iteration the hint quoted, and every iteration takes the <prefix>.<its iteration>.segment-NNN.mp4 siblings beside it. An iteration with none there, or one that finished (a complete set it is not the named one of), renders in full, which a chain_resumed warning with kept: 0 says.
  • stop_after — render a probe: only the plan's first k segments (a whole number of at least 1; at or past the plan's count the whole take renders). Needs save_segments and keep_segments true. See Probe, then continue below.

Probe, then continue. stop_after: k leaves the plan the full take's - the same segment count, num_frames and guide grid, so the same narration plans the same segments - and renders segments 1..k only. The take is muxed with the match_audio track cut to the rendered frames (a duration_seconds take's generated audio is cut the same way), the run logs chain_stopped_after ("Chain stopped after k of N segments"), and every shot carries planned_segments: N, so the manifest says the take falls short. Watch the probe; to continue it, run the same chain again with stop_after cleared and resume_from naming the probe's last segment file (output:<workflow>/<run id>/<prefix>.0.segment-<k-1>.mp4, the segment_file of its last shot): its k files become segments 1..k and only k+1..N render, through the resume rules above. A stop_after still at or under k is refused before anything is copied, since it would render nothing. The first resumed seam is not a live carry: its guide (or keyframe) is decoded from the probe's last file - h264 frames, and for a guide chain its AAC audio unless the step holds its track (hold_audio, which carries no guide audio) - which the chain_resumed warning says. A probe's job history is left out of this server's observed cost, as a resume's is. The estimate quotes a probe for the segments it renders: k of N of the take's figure (segments k0+1..k of N for a probe that resumes k0 kept ones), and for a match_audio chain, whose N comes from the track, k - k0 times the per-segment figure; stop_after may be a digit string (python -m dw.run ... stop_after=12).

The chain runs inside one iteration of the step, so it composes with previous_result fan-out (three keyframes in, three chained videos out), and a pipeline_reference step can carry its own chain. Seeds behave like a normal run: the step's generator advances across segments, so one seed reproduces the whole chain (a check re-roll runs on a generator of its own, below). Expect some visual drift across many segments with last_frame continuity - it is single-frame conditioning; last_segment continuity exists for exactly that, where the pipeline can take a video reference.

See workflows/templates/ltx2/chained-segments.json, workflows/templates/minimax/chained-segments.json, and workflows/templates/minimax/chain-matched-to-audio.json (audio-matched lip-sync of arbitrary length).

One bad segment of a finished H3 chain can be re-rendered in place instead of re-rolling the chain, when the chain kept its segment files (save_segments and keep_segments, which the H3 chain templates set from their keep_segments variable): workflows/templates/minimax/repair-segment.json renders segment k once with segment k-1's last frames as a guide at frame 0 and segment k+1's frames as a guide ending at the render's end, both holding their audio on a chain's own track, then splices the new frames into the take itself (chain_take). Outside the repaired span the output is the take: its frame count, its picture and its soundtrack, every other seam's blend included. The new frames also replace the first 12 frames of segment k+1 (the 17-frame grid's remainder, chain_repair_cut). The two repaired seams are not cuts, so analyze_seams checks them as it checks a chain's own. The last segment is repaired with the head guide only, the render ending at the take's end; the first needs a one-sided repair this template does not make. The chain's settings are passed to it as arguments - tier sets canvas, steps, weights, LoRA and shifts together, and its narrated_standard and narrated_best are the narrated take's - and the seed is new by default. For a match_audio take (a narrated take) the cut slices the take's own track under the render's frames and the render holds it (hold_audio), so the redrawn mouth follows the words heard; the guides then carry picture only, and the master track stays muxed unchanged.

Checking each segment: check and retries

With check on, each segment is measured before anything of it reaches the stitched video, its audio, its shot record, a save_segments file or the look-hold state. Segment 1 has no seam, so it is measured for audio_dbfs and words only:

Measure Against Default ceiling Fails when
jump_ratio the previous segment: the largest frame change across the seam over the larger of the two segments' own typical change, floored at 2 grey levels (analyze_seams's measure) jump_ratio_max 4 above the ceiling
look_ratio the previous segment: the largest saturation, lightness or contrast step across the seam over that measure's own rate inside either segment (analyze_seams's measure) look_ratio_max 5.5 above the ceiling
texture_ratio segment 1: the mean fine-band (2-8 px) luma texture of 4 evenly spaced frames, whole frame texture_ratio_max 2 above it or below its inverse (doubled or halved); null skips it
cut_ratio the segment itself: its largest frame change over its own typical change, floored as jump_ratio is - a cut inside the segment, past the seam's windows, which the seam measures do not see cut_ratio_max 3.5 above the ceiling; null skips it
audio_dbfs the segment itself: the RMS level (dBFS, to 0.1) of its generated audio over the kept span, past the head trim; segment 1 included audio_dbfs_min -70 below the floor; null skips it. Skipped (no level) when the segment has no generated audio or the audio is held: match_audio or its hold_audio, or a step hold_audio argument. A guide chain carrying its audio is measured: the guide holds only the span the head trim drops
words the segment's expected line (see lines): its kept audio transcribed (Whisper, on CPU) and compared with check_script's similarity; segment 1 included words_similarity_min null (off) below the minimum, a number above 0 and at most 1, or on any of the findings line_clipped_at_end, transcript_loop, tag_spoken, unscripted_speech at any similarity; the re-roll log and the chain_check_failed message name the findings and what was heard (words 1.0 with unscripted_speech, heard '...'). Off by default: bench/records/words.md found no clean separation of good takes from bad. chain_check.WORDS_SIMILARITY_MIN (0.80) is the ceiling that passed every good take seen, and catches only gross failures (a wrong or missing line, a garbled segment). Never measured when the chain holds its audio (match_audio, hold_audio), and skipped for a segment with no line
face_drop segment 1: its median SFace similarity (analyze_face_drift's scoring) of 8 evenly spaced frames from frame 1 on to its own frame 0, minus this segment's median to that same frame 0; segment 1 itself is not measured face_drop_max null (record only) above the ceiling, a number above 0. Warn-only, like texture_ratio: recorded and warned, never re-rolled. null (unmeasured) when segment 1's frame 0 or this segment holds no face, or the face models cannot load

A segment whose prompt asks for a cut inside it fails cut_ratio and re-rolls, so set check: {"cut_ratio_max": null} to turn that measure off. A step that asks for silence fails audio_dbfs the same way: set "audio_dbfs_min": null. The floor sits between the -87.8 to -89.2 dBFS that dropout (silent) segments measured and the -57 dBFS of the quietest real one. Unlike texture_ratio, a failed audio_dbfs re-rolls. The re-roll draws fresh noise but is not guaranteed to bring sound back: on the one repro measured (LTX-2.5 on mps), both seeds read about -89 dBFS and the re-roll cost about 3 minutes. The chain_segment_silent warning reports the segment either way.

"check": true uses those ceilings; "check": {"jump_ratio_max": 20} overrides the ones it names. face_drop is measured on every checked segment and recorded on its shot as check ({face_drop}), but fails nothing until face_drop_max is set: no bench has yet set it. A checked chain lists the YuNet and SFace weights it reads in the plan's downloads_required. Lip sync is not a segment measure: on 70 real H3 dialogue segments no threshold separated a segment's own audio from audio 0.5 s late or another segment's (bench/records/narrated.md, Sync calibration), so judge sync by eye, and use measure_lip_sync as a coarse indicator over a whole take. lip_sync_min is refused as an unknown key. The jump and look ceilings are twice the 99th percentile of the bench's unchecked S1-S3 chains (bench/records/noise-floor.md); the cut ceiling sits between the 2.98 most of 282 good bench segments read and the 4.16 of a cut 26 frames past a seam (bench/records/check.md); the texture ceiling is still provisional, because those chains' own texture ratio is the drift it exists to catch. A static segment's own motion is ~0, so the jump is divided by the floor rather than by ~0. Segment 1 has no seam and is checked for audio_dbfs, when the floor is not null and the audio is generated, and for words, when words_similarity_min is not null and the segment has a line. (check_segment has a hard_cut exemption that skips every measure, audio included, but run_chain does not pass it: a chain's own seams are never cuts, so every later segment is checked.) A re-roll renders a whole segment again, so re-rolls count in the step's timing segments.

A segment that fails is rendered again from the same carry and the same arguments, at a new seed: attempt n (0 is the first) runs at the step's seed plus n x 1000003, on a fresh generator, so the step's own generator - and every later segment - runs as it would have. A workflow that names no seed draws one before the run and records it, so a re-roll is repeatable from the manifest. The first attempt that passes is kept. When all 1 + retries fail, the chain keeps the one with the fewest failed measures, then the lowest jump_ratio, and finishes - it never restructures the run. texture_ratio is warn-only: texture drift is in the carry, so a fresh seed from the same carry inherits it. face_drop is warn-only too, until a bench shows a fresh seed fixes it. A segment that fails only warn-only measures is kept as rendered, recorded in check_failed and warned, and not re-rolled; one that also fails another measure re-rolls as above, and choosing the kept attempt still counts every failed measure. The kept attempt's frames and audio are the ones stitched, and the seam is crossfaded against them.

Each shot of a checked chain records attempts; a re-rolled one records the seed its kept attempt ran at, and one that failed every attempt records check_failed (the measures its kept attempt failed). The job gets one warning (kind: "chain_check_failed") per such segment, naming the measures, their values and the kept attempt, and the job log names each re-roll. When words failed, the warning also carries heard, similarity and findings (values.words is the similarity); when every attempt failed words it adds every_attempt_failed: true and a note to repair the segment with templates/minimax/repair-segment rather than retry more, since the cause may be in the carry, which a fresh seed inherits. A shot of a words-checked chain also records words (similarity, heard, findings, passed, seam_head_dropped, first_word_start) for the kept attempt, or null with no line. A failed words on a segment with a head trim (a guide chain's 22 guide frames) also transcribes the audio the trim cut off; if the line's first heard word starts inside that span the findings carry line_clipped_at_head (#1300): the opening words are cut from the delivered video and a fresh seed starts the line at the same place, so move the line later in the prompt instead of re-rolling. A segment that passed but whose kept audio opens past the line's first words (a carried segment, or one with a head trim) carries a line_opening_missing finding (#1308: missing names the words, at the first heard one) and fails words_opening without failing words, so it re-rolls while retries remain; when every attempt carries it the job gets a chain_check_failed warning naming the finding, with every_attempt_failed: true. For a long monologue use a narrated take. Among attempts failing only words, best_attempt keeps the higher similarity; a words-only failure ranks after any other single-measure failure. Each shot also records audio_dbfs, the kept attempt's level, when it is measurable. A kept segment whose generated audio still reads under the floor (the re-rolls exhausted, or check off, the floor then fixed at -70) gets a warning (kind: "chain_segment_silent") with the segment number, its audio_dbfs and the floor; audio_dbfs_min: null silences it. best_attempt is unchanged. A cost estimate quotes the worst case: a checked chain that generates its audio (floor on, no match_audio, no step hold_audio; a guide chain carrying its audio counts, its held span being the head trim) re-rolls segment 1 too, so N segments cost up to N(1 + retries) segment renders; any other checked chain never re-rolls segment 1 and costs up to 1 + (N - 1)(1 + retries). The curated figure, measured without re-rolls, paid N. With words active the estimate adds about 0.05 min (one Whisper CPU pass) per worst-case attempt, (segments - kept) x (1 + retries) passes with segment 1 counted as checked, named in the estimate's scaled text as "words check: N Whisper pass(es)".

A narrated long take

A long take of one speaker held to a narration is two templates run in order. templates/narration-retime makes the narration, retimed so each segment's line fits its span. templates/minimax/narrated-take renders one H3 segment per line as a match_audio guide chain that holds the narration as its soundtrack, so the speech heard is the narration itself and the guide carries picture only. The evidence is bench/records/narrated.md.

The audio grid. num_frames and guide_frames must be the same in both templates (192 and 22 by default). The retime cuts the narration at segment boundaries computed from them, and the take's chain lands its segments on the same boundaries; if they differ, each segment starts in the wrong place in the speech. Keep num_frames on H3's 3-frame audio grid (141, 192, 243, 294, 345) or validation warns that the held slices drift. Listen to the retimed track before spending GPU time on picture.

Probe, then continue. Validate with stop_after set to the number of segments the chosen shot holds (4 for dialogue, up to 12 for standing); the estimate prices only those. Run the probe and judge it. If it holds, run again with resume_from set to the output: reference of the probe's last kept segment and stop_after set to null, which renders only the segments after it. A bad segment is re-rendered in place with templates/minimax/repair-segment.

face_drop. Each segment's shot record carries face_drop: how far the face has drifted from segment 1's. It is recorded and never failed, because the template's check leaves its ceiling null. Read it per segment next to a late frame (get_output_frames). A step up is the face changing, and the first one an eye called was 0.146 at 36 s on dialogue.

What was measured (12 segments, about 86 s, tier best; about 18-19 min per segment on dialogue and walk_keyframed; standing took 282 min for 12 segments (about 23.5, with two re-rolls), on an RTX 3090). The narration is held unbroken under every shot.

  • shot: dialogue (the default): the face is etched by about 30 s and face_drop steps up at segment 6 (36 s). Good for a take of about 30 s.
  • shot: standing (keystone: segment_1, keystone_slot: last, no keyframes): face and backdrop hold to 85 s, face_drop within 0.015 (seed 1 only). The speaker stands; each segment eases back to the opening frame.
  • shot: walk_keyframed with keyframes: the face holds to 85 s (face_drop at most 0.041 on two seeds), with the cleanest seams, at the price of a hand-built keyframe per segment. Closed-mouth keyframes drop lip sync at segment ends, so each needs an open, mid-word mouth, and a keyframe face that differs from the still is followed and accumulates.
  • No arm cleared face, lip sync and motion together at 86 s, and nothing is measured past 86 s. experimental_latent_carry with experimental_capstone and the latent keystone (anchor_every) stay experimental.

A piece longer than one clean take is split into passages, one narrated take per passage (its own narration-retime, its own opening still or angle), joined as cuts. That composition is not yet run end to end. The design is docs/proposals/complete/narrated-long-take-complete.md.

Schedulers

Override the default scheduler:

"scheduler": {
    "configuration": {
        "scheduler_type": "DPMSolverMultistepScheduler"
    },
    "from_config_args": {
        "use_karras_sigmas": true
    }
}

A scheduler block may also carry shift, the exponential sigma shift for schedulers that take one (MiniMax H3's released checkpoint: 12.0 for video, 3.0 for audio). A pipeline that carries a second scheduler takes an audio_scheduler block with the same shape - MiniMax H3 steps video and audio latents down two schedules whose shifts are set independently.

Seeds

Set a seed for reproducibility at workflow, step, or pipeline level - most specific wins: a pipeline's own seed overrides its step's, which overrides the workflow's:

{
    "id": "my_workflow",
    "seed": 42,
    "steps": [
        { "name": "step1", "seed": 123, "pipeline": { "seed": 7, ... } }
    ]
}

Omit seed entirely to let the workflow draw a random one at run time. The seed a run actually used - drawn or named - is recorded in its manifest.json, so a run you liked can be reproduced after the fact.

Beside that manifest the run also writes workflow.json — the realized workflow, meaning the one that actually ran. Every mutable input is pinned into it: the caller's arguments folded into the variables defaults, the seed the run used, each prompt: reference replaced by the stored text, and each output:<identity>/latest/<file> (or /v<N>/) rewritten to the run id it resolved to. asset:, constant:, previous_result: and builtin: are kept as written — each already names something pinned by the asset library or by the manifest's dw_version — and a sub-workflow named by local path is kept with its file's SHA-256 recorded in the manifest. The manifest also lists which stored prompts were inlined, since inlining loses the name.

The file is a valid workflow: python -m dw.run workflow.json, or its contents as run_workflow's inline_workflow, runs it again. Its asset: and output: names resolve against the server's or CLI's own libraries, not the run directory, so it reproduces the run only where those libraries are the ones the original run used. Writing it is best effort, like the manifest, and --output-layout flat writes no run directory, so neither file.

A step that joins shots (concat_videos, dissolve_videos, or a pipeline step with a chain) also records where each one landed, as shots on its manifest entry (and on its step_end event):

{
  "step": "cut",
  "files": ["final/film.mp4"],
  "subfolder": "final",
  "shots": [
    {"name": "shot@a", "start_frame": 0,   "num_frames": 121, "start_sample": 0,      "num_samples": 242267},
    {"name": "shot@b", "start_frame": 121, "num_frames": 97,  "start_sample": 242267, "num_samples": 194000}
  ]
}

The shots partition the file's frames: the num_frames add up to the frame count. The sample fields are measured off the track the join built, not worked out from the frames. That means a shot whose track ran long shows it here: the first shot above is 267 samples longer than 121 frames at 24 fps. They are null when the video has no track, and for a chain that uses match_audio. A shot is named shot@<key> when the step's videos entry was a previous_result:shot@<key> reference, else by its input's position (video N, a chain's segment N). A dissolve's shots after the first carry overlap_frames, the head they share with the shot before. A step that wrote several joined files marks each shot with its file.

The steps that keep the frames pass shots on. stabilize and the per-frame tasks keep them as they are. interpolate_frames rescales them to the new frame count and clears the samples. pair_audio measures the samples again against the new track - every shot but the last is round(start_frame / fps * sample_rate), and the last one runs to the track's actual end - measured again, once the file is written, against what it decodes to. So its num_samples can be a few dozen samples off round(num_frames * sample_rate / fps): the AAC encode's trim, recorded in the job's event log (not a dropped sample - a real mismatch between the track and the video's length is its own warning, audio_video_length_mismatch or audio_padded_to_video/audio_trimmed_to_video with fit: "video"). Everything else drops them: an audio task's track, say, or a video read back from a file. get_gallery_metadata reports the recorded shots as media.shots, and get_output_frames(seams=true) uses them when you pass no boundaries.

Any of the three levels accepts a variable: reference, which is how a seed becomes settable per run without editing the file:

{
    "variables": { "seed": 42 },
    "seed": "variable:seed",
    "steps": [ ... ]
}
python -m dw.run workflows/models/z-image.json seed=1234

Declare the variable with an integer default, as above: the value from the command line arrives as a string and is converted to the declared type. A string that is not a variable: reference is rejected by the schema.

The seed also reaches sub-workflows: a delegated workflow step runs the child under the parent's seed unless the child names its own. Without that a child draws its own random seed, and a workflow whose real generation happens inside a sub-workflow would not reproduce from the seed it was given.

The step cache that lets a reproduced step skip re-running (see Runs in Workspaces) is scoped to the output directory a run writes into, which on dw.serve is the workspace's own outputs/. Two workspaces holding the same prior run do not share a cache entry, so validate_workflow's plan.cached_steps answers for the workspace the call is pinned to. Deleting a workspace takes its cache entries with it.

The step cache is also per card and per workflow identity: the stored workflow's path, or an inline workflow's id (workflow_identity in dw/worker_protocol.py). Switching identity on a card empties that card's step cache (the worker's _cleanup_all), so:

  • running any other workflow on the card in between starts the next run cold: A, then B, then A runs the second A cold. On a multi-card server warm-identity routing (choose_slot in dw/server/pool.py) mostly avoids it unless that card is busy; on a single card it happens every time.
  • a renamed copy, a new inline id, or switching between a stored workflow and its inline copy also starts cold.

To reuse cached upstream steps, rerun the same workflow (same name, or the same inline id), for example with rerun_job or changed arguments. Only arguments that reach later steps leave the earlier steps cached: changing an argument the expensive first step reads re-runs that step. To check before running, validate_workflow's plan.cached_steps says how many steps would be reused.

Type System

  • Keys ending in _type or _dtype, or named dtype: "torch.bfloat16" becomes torch.bfloat16, and "sdnq.SDNQConfig" loads the class by import.
  • Braces keep a value a string: "{nf4}" stays "nf4".
  • content_type and offload_type always stay strings.
  • constant: values are read from Python rather than copied into the workflow (below).

The full rule, including what an untrusted server refuses, is in Types and escaping.

Constant References

Some arguments have a value the library already declares: the sigma schedule a distilled model was trained on, the negative prompt a model family ships. Reference it with constant: and its dotted python name instead of copying it into the workflow:

"sigmas": "constant:diffusers.pipelines.ltx2.utils.DISTILLED_SIGMA_VALUES",
"negative_prompt": "constant:diffusers.pipelines.ltx2.utils.DEFAULT_NEGATIVE_PROMPT"

The leading part of the name that imports is the module, and the rest is read from it - so a constant held in a config object is reachable too:

"prompt_max_new_tokens": "constant:diffusers.pipelines.ltx2.utils.GEMMA4_PROMPT_ENHANCEMENT_CONFIG.max_new_tokens"

A reference resolves anywhere in a workflow's arguments, including in a variables default, where it becomes the value a caller overrides - and its type, since a variable is declared by its default:

"variables": { "negative_prompt": "constant:diffusers.pipelines.ltx2.utils.DEFAULT_NEGATIVE_PROMPT" }

A constant is data: anything callable is refused (a type is named with a *_type key instead). Mutable values are copied, so a pipeline that consumes its schedule in place cannot edit the library's constant for the rest of the session. A constant that changes upstream changes here, and one that is renamed or moved fails the run rather than leaving a stale copy behind.

Prompt References

A prompt worth keeping is worth keeping once. Stored prompts live as JSON files in a prompt library - the prompts/ folder by default - and a workflow argument written as prompt: plus the file's name (without .json, optionally one folder deep) loads its text at run time:

"prompt": "prompt:scenic_landscape",
"prompt": "prompt:minimax/fox_dawn_t2va"

A prompt file holds the text plus the metadata the server's Prompts page shows:

{
  "text": "A sweeping alpine valley at golden hour...",
  "description": "General-purpose scenic landscape",
  "intended_model": "z-image",
  "negative_prompt": "blurry, low quality",
  "tags": ["landscape", "golden-hour"]
}

Only text is required, and it is what the reference resolves to. intended_model is informational - the engine ignores it, but the library badges and filters by it, and the server's prompt enhancer uses it to preselect a preset. One spelling per family: list_prompts(intended_model=...) matches the whole value exactly, so minimax-music beside minimax-music3 hides half a shelf, and tests/test_prompt_library.py sweeps the repo's library for a variant.

The library's location is resolved in order (discover_library in dw/workspace.py): the DW_PROMPT_DIR environment variable (which --prompt-dir on dw.serve sets); a workspace named explicitly (--workspace, DW_WORKSPACE or the setting), whose prompts/ it is; ./prompts in the working directory when it exists; the first prompts/ folder found walking up from the workflow file's own directory; and last the workspace's prompts/. dw.serve resolves it once at startup and pins it for every job, so the Prompts page and prompt: resolution agree on one library. References are rooted at that directory, not at the workflow file, so the same reference means the same text from every workflow. A reference resolves anywhere in a workflow's arguments, including a variables default, always to exactly one string: it never multiplies a step's iterations. A prompt's text may not itself begin with a reference prefix such as variable:.

Asset References

A workflow's plain media paths resolve against the workflow file's own directory, which means a workflow that reads anything has to keep that thing beside it. An asset: reference is rooted at the asset library instead - the workspace's assets/ folder - so a workflow and the media it reads do not have to live in the same place:

"image": "asset:iris.png",
"video": "asset:gyre/frames/web.mp4",
"references": [
    {
        "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3ImageReference",
        "from_file": "asset:subject.png"
    }
]

A reference names a file with its extension, at most four folders deep, and resolves to that file's path, so it works under any argument that accepts a path: image, video, a from_file, a list of any of them, or a task argument that names a file.

On dw.serve each job reads the asset library of the workspace it runs in. Elsewhere the location is resolved in the prompt library's order, with DW_ASSET_DIR (--asset-dir) and assets/ in place of DW_PROMPT_DIR and prompts/. See Workspaces.

A reference can only name a file inside the library: .., an absolute path, or a symlink pointing out of it are all refused. Browser uploads land in the library's uploads/ folder and come back as asset:uploads/<name>, so a workflow saved after an upload still resolves on the next run.

Output References

For multi-stage work (generate stills, then animate them in another run), an output: reference names what an earlier run wrote:

"image": "output:ltx2/Gyre/latest/Gyre-still.0-0.0.png",
"audio": "output:ltx2/GyreScore/20260905-181530-a1b2c3d4/Gyre-score.10-0.0.wav"

The name is a path under the output directory — the workflow's identity, the run, and the file (see Runs). Writing latest where the run id goes resolves to the newest run of that workflow that holds the file, so it keeps working when the newest run failed part way or reused every step from the cache and wrote only a manifest. Runs sort by their id, which starts with a UTC timestamp. v<N> in the same position names the run whose version is N (the v4 the gallery labels its files with). Unlike latest it picks exactly one run: v4 not holding the file is an error, not a reason to try v3. latest and v<N> select a run only in the run-id position; a workflow or file that happens to be called either is still named as itself.

Like asset:, a reference resolves to a path and then whatever loads paths loads it, so it works under image, video, a from_file, or a list of them. The audio tasks take a video file's path too and use the soundtrack muxed into it, which is how a finished cut is scored in a later run without re-cutting it. It resolves against the output directory the run was told to write to, and cannot leave it: .., an absolute path, and a symlink pointing out are all refused.

To name an earlier step of the same run, use previous_result: instead — that passes the value in memory rather than through the filesystem.

A generated file worth reusing repeatedly is better kept than referenced by the run that made it: POST /api/assets/keep (the gallery's Keep as asset, or MCP's keep_output) copies it into the workspace's asset library under a name you choose, and from then on it is an asset: reference like any other — stable whatever happens to the run directory it came from.

Objects Built From a File

Some pipelines take arguments that are objects rather than plain media. An argument that names a type and a from_file is constructed by that type's own from_file():

"references": [
    {
        "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3ImageReference",
        "from_file": "subject.png"
    },
    {
        "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3AudioReference",
        "from_file": "voice.wav"
    }
]

Loading the media this way rather than as a plain image or video argument is what brings its frame rate or sample rate along with it, which MiniMax-H3 resamples a reference from. The file may be a path — relative to the workflow file, like all media a workflow names — or a URL, and is validated like any other media. variable: references work as the file location; previous_result: does not, since the object is built when the workflow loads — use from_previous_result for that. A dict that merely contains a from_file key without a *_type key is not an object description and is passed through untouched.

An entry in a list whose source is null is left out of that list. That is what makes a reference optional: write it as an ordinary entry whose from_file (or from_previous_result) is a variable, declare the variable null, and a run that is given nothing for it generates exactly as it did before the reference existed — one workflow serving both, instead of two spellings of the same steps. It applies to from_file, from_previous_result and from_arguments alike. On its own rather than in a list there is nothing to leave it out of, so a null source there is an error.

Any other key goes wherever the type can take it: to from_file() where its signature names it, and onto the object it returns where it does not. That is what corrects a decoded file, which is the only thing that knows what the container claimed:

{
    "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3VideoReference",
    "from_file": "motion.mp4",
    "fps": 30.0,
    "audio": null
}

fps overrides a rate the container got wrong — MiniMax-H3 resamples a reference onto its own 24 fps, so a wrong rate is a request conditioned at the wrong speed — and audio: null drops the decoded soundtrack, leaving a reference that conditions on motion and camera alone. A name that is neither an argument of from_file() nor a field of the object raises, with the fields it does have.

See workflows/templates/minimax/reference-to-video.json for a full example.

Objects Built From an Earlier Step

The same object can be built from what an earlier step generated, by naming the step instead of a file:

"references": [
    {
        "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3ImageReference",
        "from_previous_result": "draw_subject"
    }
]

from_file cannot do this — it names a file, and the object is built when the workflow loads, before any step has run. from_previous_result waits: the description is checked at load time and constructed once the step it names has produced its media, which is what lets one workflow generate a subject and then condition on it without writing it out and reading it back.

The media never touches the disk, so it arrives as the step produced it. Which field it lands in comes from the type's own kind:

kind Built from
image The generated image
video The generated frames, and the soundtrack generated with them if there was one
audio The generated soundtrack - or, for a step that produced audio alone (a music pipeline, a slice_audio task), the waveform itself. The rate travels with the waveform when the pipeline or task reports one (an AudioTrack - AudioLDM2, StableAudio, generate_speech); declare sample_rate beside from_previous_result only for a waveform from a task or file that carries none, and a declared rate always wins

Any other key is a field of the object and wins over what the media carried — "fps": 30.0 where the producing pipeline generated at a rate the consuming one does not share, for instance. A step that produced several artifacts fans out the same way every previous_result reference does: four images in, four videos out.

See workflows/templates/minimax/generated-subject-reference.json for a full example.

Objects Built From Named Arguments

Not every type a pipeline takes knows how to open a file. LTX-2's keyframe conditions and IC-LoRA references are plain dataclasses holding frames the caller already loaded, plus the numbers that say what to do with them. Those are written as the arguments to construct the object with:

"conditions": [
    {
        "condition_type": "diffusers.pipelines.ltx2.pipeline_ltx2_condition.LTX2VideoCondition",
        "from_arguments": {
            "frames": { "media_type": "image", "location": "first.png" },
            "index": 0,
            "strength": 1.0
        }
    },
    {
        "condition_type": "diffusers.pipelines.ltx2.pipeline_ltx2_condition.LTX2VideoCondition",
        "from_arguments": {
            "frames": { "media_type": "image", "location": "last.png" },
            "index": -1,
            "strength": 1.0
        }
    }
]

from_arguments holds every argument the type is constructed with - a key beside it raises rather than being silently dropped, and so does an argument the type does not take, naming the ones it does. The arguments inside are ordinary arguments: a media reference loads there, a variable: reference resolves there, and a previous_result: reference waits the way from_previous_result does - the object is constructed once the step it names has run.

Which of the three forms a type wants is decided by the type, not by preference:

Form For a type that
from_file opens the media itself, bringing its frame or sample rate along (MiniMax-H3's references)
from_previous_result declares a media kind, so a step's output lands in the right field on its own
from_arguments is a plain record of fields - no from_file(), no kind (LTX-2's conditions and references)

See workflows/templates/ltx2/keyframes.json for the file form and workflows/templates/ltx2/extend-clip.json for the one built from an earlier step.

Frames Across a Step Boundary

A pipeline that generates video with a soundtrack returns the two paired, and the result muxes them into one file. A step that works on the frames alone - a latent upsampler, an interpolator - returns frames without it. Two tasks carry the pieces across:

  • video_frames takes a generated video and returns its frames as one (frames, height, width, channels) uint8 array - the 0-255 shape LTX-2's conditions want, and one artifact rather than one per frame.
  • pair_audio puts a soundtrack back beside frames that lost it, so the step that saves them writes a single muxed mp4.
{
    "name": "film",
    "task": {
        "command": "pair_audio",
        "arguments": {
            "video": "previous_result:edit",
            "audio": "previous_result:balanced",
            "sample_rate": "variable:sample_rate"
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

audio takes either a waveform or the earlier step whose video carried the soundtrack, which brings its sample rate along; here it is an earlier step's waveform, so sample_rate is given explicitly. The frames keep the rate they arrived with - video given a file or an asset: carries that file's fps through to the saved mp4 - so result.fps is only needed for frames that bring no rate of their own. A mono track needs no preparation: an mp4 audio stream takes stereo and nothing else, so saving duplicates the single channel into two and emits a warning saying it did.

"fit": "video" makes the track fit the frames: it is cut to exactly the frames it is laid over, or padded with silence and warned about when it is shorter. Use it for a soundtrack under a cut whose length is an argument (a shots list), since nothing in a workflow can multiply a list's length by a frame count (#142). Left unset, the track is used as it is and a length mismatch is warned about.

Which shape a pipeline argument wants is the pipeline's business, and the two LTX-2 paths differ: a keyframe condition is mapped from 0-255, so it takes the video_frames array, while an IC-LoRA reference goes through the video processor, which expects the [0, 1] frames the pipeline returned - previous_result:step.frames hands those over untouched.

Example: workflows/templates/assemble-and-score.json