perf: render init templates in the engine, and fix two module names that could not load

[eunomie]

Trace-driven pass over the three surfaces this module owns: dagger module init python, dagger generate, and what the SDK contributes to dagger call. Baseline, method, raw runs and the ideas that were measured and rejected are all in hack/designs/2026-08-14-python-sdk-performance.md.

What changed

Init renders templates in the engine instead of a Go toolchain. renderedTemplate pulled golang:1.25-alpine, compiled helpers/render-template from source, and ran it — to substitute three variables across a handful of small text files. It now walks the template with glob, expands {{ .Var }} in paths and contents, and carries non-.tmpl files over with withFile. helpers/render-template/ is deleted.

One Go base image instead of two. The helpers pinned golang:1.25-alpine while the polyfill dependency's Go runtime pulls golang:1.26-alpine, so a cold engine pulled both. The remaining helper (helpers/pyproject, used by config get/set and flag-bearing init) now builds on the image the workspace already has.

Two latent bugs fixed, both found while measuring:

• dagger module init python app --python-version 3.13 produced a module that could not be loaded at all — ModuleLoadError: name 'App' is not defined. The default template's create() returns the class it is declared in, and deferred annotations only became the default in 3.14 (PEP 649). Fixed with from __future__ import annotations; verified by initializing, generating and calling at 3.12, 3.13 and 3.14.
• dagger module init python s3-bucket — any name containing a digit — was also broken on main. strcase.ToSnake renders src/s_3_bucket/, which is not what uv_build derives from name = "s3-bucket", so the module failed to load with call constructor: exit code: 1. The new renderer produces src/s3_bucket/ and the module loads and calls.

Honest numbers

Local engine v1.0.0-beta.9; the host is shared and noisy, so arms were interleaved and every run is recorded in the doc.

	before	after
cold-engine withInitModule	12.4 / 12.0 / 12.7 s	11.2 / 11.3 / 11.1 s
warm-engine withInitModule	1.8 / 1.8 / 2.0 / 2.0 s	1.9 / 1.8 / 1.8 / 1.8 s

So: ~1.2s (≈10%) off a cold init, and nothing measurable when warm — the go build layer was already cached in the warm case. The saving is smaller than the removed work suggests because the helper's build overlapped the polyfill Go build that remains. Expect little or no CI wall-clock change; an earlier draft of the doc predicted a large CI win from check durations and that reasoning was wrong, which the doc now says.

The durable wins are as much qualitative as numeric: the default scaffolding path no longer needs a Go toolchain or a Docker Hub round-trip for a Go image, and one of two helper binaries is gone.

Rendered output was diffed against the deleted Go helper's for my-module, my_module, myModule, HTTPServer and simple across the default, empty and legacy templates — 15 combinations, byte-identical. Digit-bearing names deliberately differ, because strcase's answer was the broken one.

Measured and deliberately not shipped

• Pinning the polyfill dependency by commit to remove a per-call git ls-remote. It does not work: NewGitRepository calls backend.Remote() unconditionally, and the ref already carries the SHA via refPin. The fix belongs in the engine; dropped rather than shipped as a no-op.
• use-uv = false measures −12% per call, reproduced across two independent benchmark batches. Not adopted: every in-container micro-benchmark is identical between the arms, so the ~200ms/process lives somewhere nobody could isolate, and it trades away uv's install speed on a path that was not benchmarked. Recorded for the runtime owners instead.
• base-image = "python:3.14-alpine" is worth knowing about in the other direction: +24% per call, because musl CPython walks the same import chain in 607ms instead of 453ms.

Where the call-path time actually goes

A warm dagger call is ~3.4s, of which ~2.1s is two exec.processRun spans in the Python runtime container. Instrumenting inside that container splits one ~1050ms process as: ~450ms import dagger (300ms of it the generated client/gen.py), 253ms first-connect handshake, ~200ms container setup/teardown, ~60ms recompiling the vendored SDK to bytecode that is then thrown away, 38ms telemetry init — and ~8ms of actual interpreter start, 0.8% of the total.

None of that is fixable here; it lives in dagger/dagger's sdk/python/runtime. The three cost centres are written up in the doc for whoever picks them up upstream.

Testing

All 35 workspace checks pass locally (13 e2e, 22 sdk-sdk). The e2e additions were written to protect this rewrite specifically: exact-file-set assertions (which caught a stray file during development), a naming matrix including the digit and HTTPServer cases, an assertion that pyproject.toml is actually expanded — .ModuleName appears nowhere else, so a regression there previously passed every check — and an assertion that the __future__ import stays in the template.

The doc records what was accepted and not fixed: template symlinks are no longer rejected, empty template directories are dropped, and the template language is narrowed from Go text/template to {{ .Var }} substitution only.