feat(producer): write public render provenance sidecar next to outputs - #3758
Open
mvanhorn wants to merge 2 commits into
Open
feat(producer): write public render provenance sidecar next to outputs#3758mvanhorn wants to merge 2 commits into
mvanhorn wants to merge 2 commits into
Conversation
Every successful render now writes <output>.hf-render.json after the artifact commits: tool versions (producer/node/ffmpeg), entry sha256, compiled-composition hash, @font-face families, variables hash (never raw values), format/fps/resolution/duration, encoder facts, output sha256+size, stage timings, worker count, HDR flag, and warning codes. - Default ON; --no-provenance (or --provenance false) disables, --provenance <path> relocates. Batch rows each write their own sidecar; --docker forwards the disable flag into the container. - JSON Schema published at packages/core/schemas/hf-render-sidecar.json. - Docs at docs/reference/render-provenance.mdx. - Sidecar write failures log a warning and never un-complete a render.
| sidecarPath: string, | ||
| sidecar: RenderProvenanceSidecar, | ||
| ): Promise<void> { | ||
| await writeFile(sidecarPath, `${JSON.stringify(sidecar, null, 2)}\n`, "utf-8"); |
…+ rename Addresses the CodeQL insecure-temporary-file finding: the receipt is now written to a sibling .tmp file in the sidecar's own destination directory (never the shared, world-writable os.tmpdir()), fsynced, then renamed over the final path. Readers can only observe a complete receipt and there is no cross-device or symlink-swap window between write and publish. Failed renames clean up the temp file. Test asserts no .tmp residue remains.
Contributor
Author
|
Pushed a fix for the CodeQL "Insecure temporary file" finding (d5426f1):
Readers can only ever observe a complete receipt, and there is no window where a world-writable location holds the file between write and publish. Added a test asserting the published sidecar parses and no |
| sidecar: RenderProvenanceSidecar, | ||
| ): Promise<void> { | ||
| const tmpPath = `${sidecarPath}.tmp`; | ||
| const handle = await open(tmpPath, "w", 0o644); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every successful render now writes a portable JSON receipt next to the output artifact:
out.mp4getsout.mp4.hf-render.json. The sidecar records tool versions (producer / node / ffmpeg), the entry HTML sha256, the compiled-composition hash,@font-facefamilies, a variables hash (count + sha256 of canonical JSON — never the raw values), format / exact-rational fps / resolution / duration / total frames, encoder facts (codec, preset, pixel format), the committed artifact's sha256 + size, per-stage wall-clock timings, worker count, HDR flag, outcome, and capture-readiness warning codes.--no-provenance(or--provenance false) disables;--provenance <path>relocates the sidecar.<row output>.hf-render.json(verified: distinct variables hashes per row). A fixed custom path with--batchis rejected up front.--dockeris rejected up front (not container-visible).packages/core/schemas/hf-render-sidecar.json($idhttps://hyperframes.heygen.com/schema/hf-render-sidecar.json); every emitted sidecar in testing validates against it (ajv, draft 2020-12).docs/reference/render-provenance.mdx, registered under the "Rendering paths" nav group.Why
Agents and CI need a machine-readable receipt beside the MP4 answering "what rendered this, from what inputs, and how" without re-running anything — reproducibility checks, cache keys, and support triage all want it. Today the closest thing (
runtimeEnvSnapshot,perfSummary) is internal-only, and the engine's embedded container tags carry only renderer name + version. The sidecar complements those tags: full receipt, separate file, deliberately excluding secrets (variable values are hashed, no env dump, no host paths/usernames).How
packages/producer/src/services/render/provenanceSidecar.ts— sidecar builder + emitter. Wired intorenderOrchestrator.tsimmediately afterartifactTransaction.commit()(the actual commit site; there is nostages/commitArtifact.tsin the codebase). A sidecar write failure logs a warning and never un-completes a render whose artifact already committed.perfStages,workerCount, and job warnings. Producer version prefers the bundler-injected__PRODUCER_VERSION__define (the CLI inlines the producer vianoExternal, where the package-walking reader can't resolve) with the filesystem walker as the unbundled fallback; ffmpeg version is probed once via the engine's binary resolution (honorsHYPERFRAMES_FFMPEG_PATH) and cached.--provenanceflag → render plan (parseProvenanceArg, tri-state) →RenderOptions→createRenderRequest(validated:falseor non-empty string) →RenderConfig.provenance→ orchestrator.dockerRunArgsforwards--no-provenance.png-sequencedirectory outputs skip artifact sha256/size;encoderisnullforpng-sequence/gif.Test plan
provenanceSidecar.test.ts(19 tests: path resolution incl. directory outputs, font extraction, canonical variables hashing, receipt shape, secrets-not-embedded, emit integration incl. write-failure and unreadable-entry tolerance) andrender.provenance.test.ts(12 tests: flag parsing, plan threading, batch/docker conflicts, docker arg forwarding). Full CLI suite: 3021 passed. Producer unit lane: green exceptaudioPadTrim.integration.test.ts, which fails identically on unmodifiedmainin this environment (host-ffmpeg loudness variance — pre-existing, unrelated).output.sha256matchessha256sumof the MP4),--no-provenance(no sidecar),--provenance <path>(relocated),--batchwith 2 rows (one sidecar per row, distinct variables hashes), sidecars validated against the JSON Schema with ajv.bun run build,tsc --noEmit(producer + cli), oxlint/oxfmt, and the full lefthook pre-commit suite (incl. the fallow gate) all pass.docs/reference/render-provenance.mdx+docs/docs.json.Walkthrough (HyperFrames-rendered)
MP4: https://raw.githubusercontent.com/mvanhorn/hyperframes/walkthrough-assets/render-provenance-walkthrough.mp4
The demo was storyboarded first (script, beat sheet, and seam vector ledger —
render-provenance-ce-plan.md), built as a HyperFrames composition (source) that passedhyperframes lintandhyperframes check(0 errors, WCAG AA contrast), and rendered with:Self-proof: that render exercised this PR's feature on itself. Its own receipt,
render-provenance-walkthrough.mp4.hf-render.json, recordswhich is exactly
sha256sumof the published MP4 above (verified after download from the public URL).AI disclosure
This PR — feature code, tests, schema, docs, the walkthrough storyboard, composition, and render — was authored end-to-end by an AI agent (Claude Fable 5, running as a Cursor Cloud Agent), with all repository gates (build, typecheck, lint/format, lefthook, targeted test suites) executed and passing before submission.