Skip to content

feat(producer): write public render provenance sidecar next to outputs - #3758

Open
mvanhorn wants to merge 2 commits into
heygen-com:mainfrom
mvanhorn:feat/render-provenance-sidecar
Open

feat(producer): write public render provenance sidecar next to outputs#3758
mvanhorn wants to merge 2 commits into
heygen-com:mainfrom
mvanhorn:feat/render-provenance-sidecar

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

What

Every successful render now writes a portable JSON receipt next to the output artifact: out.mp4 gets out.mp4.hf-render.json. The sidecar records tool versions (producer / node / ffmpeg), the entry HTML sha256, the compiled-composition hash, @font-face families, a variables hash (count + sha256 of canonical JSON — never the raw values), format / exact-rational fps / resolution / duration / total frames, encoder facts (codec, preset, pixel format), the committed artifact's sha256 + size, per-stage wall-clock timings, worker count, HDR flag, outcome, and capture-readiness warning codes.

  • Default ON. --no-provenance (or --provenance false) disables; --provenance <path> relocates the sidecar.
  • Batch mode: each row writes its own <row output>.hf-render.json (verified: distinct variables hashes per row). A fixed custom path with --batch is rejected up front.
  • Docker mode: the disable flag is forwarded into the container; the default sidecar lands next to the output in the mounted output directory. A custom host path with --docker is rejected up front (not container-visible).
  • JSON Schema published at packages/core/schemas/hf-render-sidecar.json ($id https://hyperframes.heygen.com/schema/hf-render-sidecar.json); every emitted sidecar in testing validates against it (ajv, draft 2020-12).
  • Docs: docs/reference/render-provenance.mdx, registered under the "Rendering paths" nav group.

Why

Agents and CI need a machine-readable receipt beside the MP4 answering "what rendered this, from what inputs, and how" without re-running anything — reproducibility checks, cache keys, and support triage all want it. Today the closest thing (runtimeEnvSnapshot, perfSummary) is internal-only, and the engine's embedded container tags carry only renderer name + version. The sidecar complements those tags: full receipt, separate file, deliberately excluding secrets (variable values are hashed, no env dump, no host paths/usernames).

How

  • packages/producer/src/services/render/provenanceSidecar.ts — sidecar builder + emitter. Wired into renderOrchestrator.ts immediately after artifactTransaction.commit() (the actual commit site; there is no stages/commitArtifact.ts in the codebase). A sidecar write failure logs a warning and never un-completes a render whose artifact already committed.
  • Reuses existing producer facts at the commit site: the compiled HTML (fonts scan), the observability composition hash, the encoder preset, perfStages, workerCount, and job warnings. Producer version prefers the bundler-injected __PRODUCER_VERSION__ define (the CLI inlines the producer via noExternal, where the package-walking reader can't resolve) with the filesystem walker as the unbundled fallback; ffmpeg version is probed once via the engine's binary resolution (honors HYPERFRAMES_FFMPEG_PATH) and cached.
  • Threading: --provenance flag → render plan (parseProvenanceArg, tri-state) → RenderOptionscreateRenderRequest (validated: false or non-empty string) → RenderConfig.provenance → orchestrator. dockerRunArgs forwards --no-provenance.
  • png-sequence directory outputs skip artifact sha256/size; encoder is null for png-sequence/gif.

Test plan

  • Unit tests added/updated — provenanceSidecar.test.ts (19 tests: path resolution incl. directory outputs, font extraction, canonical variables hashing, receipt shape, secrets-not-embedded, emit integration incl. write-failure and unreadable-entry tolerance) and render.provenance.test.ts (12 tests: flag parsing, plan threading, batch/docker conflicts, docker arg forwarding). Full CLI suite: 3021 passed. Producer unit lane: green except audioPadTrim.integration.test.ts, which fails identically on unmodified main in this environment (host-ffmpeg loudness variance — pre-existing, unrelated).
  • Manual testing performed — real end-to-end renders with the branch CLI: single render (sidecar written, output.sha256 matches sha256sum of the MP4), --no-provenance (no sidecar), --provenance <path> (relocated), --batch with 2 rows (one sidecar per row, distinct variables hashes), sidecars validated against the JSON Schema with ajv. bun run build, tsc --noEmit (producer + cli), oxlint/oxfmt, and the full lefthook pre-commit suite (incl. the fallow gate) all pass.
  • Documentation updated (if applicable) — docs/reference/render-provenance.mdx + docs/docs.json.

Walkthrough (HyperFrames-rendered)

HyperFrames walkthrough

MP4: https://raw.githubusercontent.com/mvanhorn/hyperframes/walkthrough-assets/render-provenance-walkthrough.mp4

The demo was storyboarded first (script, beat sheet, and seam vector ledger — render-provenance-ce-plan.md), built as a HyperFrames composition (source) that passed hyperframes lint and hyperframes check (0 errors, WCAG AA contrast), and rendered with:

node packages/cli/bin/hyperframes.mjs render /tmp/provenance-walkthrough --output walkthrough.mp4 --fps 30

Self-proof: that render exercised this PR's feature on itself. Its own receipt, render-provenance-walkthrough.mp4.hf-render.json, records

"sha256": "6af87ebf6a630a417e64db98c6d6e05f50f3fa99dcc5063e3757ae3d4e8b8de1"

which is exactly sha256sum of the published MP4 above (verified after download from the public URL).

AI disclosure

This PR — feature code, tests, schema, docs, the walkthrough storyboard, composition, and render — was authored end-to-end by an AI agent (Claude Fable 5, running as a Cursor Cloud Agent), with all repository gates (build, typecheck, lint/format, lefthook, targeted test suites) executed and passing before submission.

Every successful render now writes <output>.hf-render.json after the
artifact commits: tool versions (producer/node/ffmpeg), entry sha256,
compiled-composition hash, @font-face families, variables hash (never
raw values), format/fps/resolution/duration, encoder facts, output
sha256+size, stage timings, worker count, HDR flag, and warning codes.

- Default ON; --no-provenance (or --provenance false) disables,
  --provenance <path> relocates. Batch rows each write their own
  sidecar; --docker forwards the disable flag into the container.
- JSON Schema published at packages/core/schemas/hf-render-sidecar.json.
- Docs at docs/reference/render-provenance.mdx.
- Sidecar write failures log a warning and never un-complete a render.
sidecarPath: string,
sidecar: RenderProvenanceSidecar,
): Promise<void> {
await writeFile(sidecarPath, `${JSON.stringify(sidecar, null, 2)}\n`, "utf-8");
Comment thread packages/producer/src/services/render/provenanceSidecar.ts Fixed
…+ rename

Addresses the CodeQL insecure-temporary-file finding: the receipt is now
written to a sibling .tmp file in the sidecar's own destination directory
(never the shared, world-writable os.tmpdir()), fsynced, then renamed over
the final path. Readers can only observe a complete receipt and there is
no cross-device or symlink-swap window between write and publish. Failed
renames clean up the temp file. Test asserts no .tmp residue remains.
@mvanhorn

mvanhorn commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a fix for the CodeQL "Insecure temporary file" finding (d5426f1):

writeRenderProvenanceSidecar no longer writes the receipt directly to its final path. It now:

  1. writes the JSON to a sibling sidecarPath + '.tmp' file in the same directory as the destination — never the shared, world-writable os.tmpdir()
  2. fsyncs the file handle before closing
  3. renames the temp file over the final path (atomic on the same filesystem, so no cross-device window), cleaning up the temp file if the rename fails

Readers can only ever observe a complete receipt, and there is no window where a world-writable location holds the file between write and publish.

Added a test asserting the published sidecar parses and no .tmp residue remains. Full provenanceSidecar suite (20 tests) and the CLI render.provenance suite (12 tests) pass; changed files pass oxlint/oxfmt.

sidecar: RenderProvenanceSidecar,
): Promise<void> {
const tmpPath = `${sidecarPath}.tmp`;
const handle = await open(tmpPath, "w", 0o644);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants