Repository navigation
Reduce compiler report allocation and reuse C generation work - #1649
Open
flip-engineer wants to merge 9 commits into
Open
flip-engineer wants to merge 9 commits into
flip-engineer wants to merge 9 commits into
Conversation
This was referenced Oct 17, 2026
Author
|
The foreign refresh path needs one additional correctness check before merge. Please add a regression that introduces |
added 4 commits
October 16, 2026 23:39
try_refresh: decline raw substitution when the changed fragment carries CID()/FID() call syntax (effect_srcs/c_ids rewrites it on generation). emit_c_cached: refuse cache dirs that are not owner-private real directories with clean ancestors (zero cache I/O, rebuild instead). Fixture: correct MAIN_TS to bend2/main.ts; add I-step (CID decline + parity) and J-step (open-mode/symlink refusal + parity). 39/39 green.
J asserts empty event logs (not merely no hit), byte-unchanged hostile dirs, outputs anchored to a separately built trusted-cache binary, plus an ancestor-symlink refusal case. Authority walks every ancestor (real dir, never symlink; ownership/privacy at leaf); rename/mount races remain documented residual, digests gate served bytes. Zero artifact I/O claim corrected: the check itself only creates-if-missing and lstats. 45/45 green.
Map the reviewed memo-scope change (baton/cgen-fixedpoint-impl-20261017
fa582b5) onto upstream bend2/comp.ts. OPENS/USES/SPINES are pure over the
frozen book, so memo_gc no longer drops them at every definition; a
pre-opened let's key is fresh and definition-lived, so it is tracked in
OPEN_LET and evicted at each definition boundary instead.
Probe lifetime is part of the mapping. This compiler truncates PROBES to
1 per definition (compile_book), which makes probe indices definition-
local while retained openings hold probe objects and USES values are keyed
by probe index. Porting the retention alone crashes immediately
("undefined is not an object (evaluating 'b.n')" from bind_of when
probe_of resolves a retained Var's index against the truncated table), so
the per-definition truncation is removed and probes live for the compile,
as file_new already resets them. Retention is then sound: an opening's
probes keep their identities and indices for the whole compile.
Bounded evidence (shared host, load 22-26, a full Main generation resident;
paired runs, medians of three, +/-10%):
- 85 upstream programs byte-identical (demos, tests/io, tools/bend-lint),
12,263,641 C bytes in both modes.
- app_slash_boss_3d byte-identical: 486,784 -> 508,240 KiB peak (+4.4%),
5.83 -> 5.27 s.
- fp_deep_big 722,392 -> 763,284 KiB (+5.7%), fp_list_big 245,172 ->
250,132 KiB (+2.0%), fp_clo neutral; all C byte-identical.
- JS backend byte-identical on five inputs.
- Cache: the compiler identity in cache.ts COMPILER_FILES changes, so the
key changes (6ed054f6... at comp.ts 069aa862... vs 5b55985b... at comp.ts
1bf8f3dd...); an old entry is not reused, the new one stores and then
hits (compile -> hit in log.jsonl).
Limits: bounded fixtures only, no Main-scale claim in either direction;
probe memory is now proportionally larger at Main scale, which this
evidence does not measure.
a430d53 evicts pre-opened let keys at definition boundaries, but the Set is module-global and file_new replaces FL and its memo maps per compile. A sequential compile in one process could therefore hold the previous book's last definition until the next memo_gc ran, and a compile that failed before any memo_gc would hold it indefinitely. Declare OPEN_LET with the other compile-scoped caches and clear it in file_new beside PROBES.length = 1, the point where the old memo object is replaced. memo_gc keeps evicting per definition. Qualified in one process: three books compiled twice each and then again after an interleaved second book all produced identical C, matching the fresh-process CLI hashes (fp_list_big 5592efe9..., fp_clo 2039d187..., io_hello_world d4e19ab9...). Bounded re-check after the change, paired base vs mapped, fresh CGEN_CACHE_DIR per compile: 85/85 upstream programs byte-identical, 12,263,641 C bytes both modes; cache identity moves from key 6ed054f6... (comp.ts 069aa862...) to key c7159990... (comp.ts 5fa43bdc...), storing fresh and hitting on the next run, same artifact 5592efe9... Peak RSS stays inside the shared-host noise band and is not a claim: boss_3d medians were +4.4% in the first round and +9.2% in the second, fp_deep +5.7% then -7.2%, fp_list +2.0% then -1.9%. Elapsed improved in both rounds (boss_3d 5.94 -> 5.18 s this round). No Main-scale result in either direction, and this is not the fork's 35% RSS improvement.
flip-engineer
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change reduces allocation while collecting unsafe and foreign-effect references. The higher-term traversal uses the compiler's existing fresh-variable operation and preserves the dependent-reference report.
The normal C and native build paths reuse generated C when the compiler, source closure, foreign inputs, and relevant environment match the stored identity. Foreign-only refresh validates the receipt, manifest structure, and artifact digest before applying one exact source substitution. Invalid or incomplete cache entries trigger ordinary generation. Diagnostic requests execute the compiler to preserve diagnostics.
A changed foreign fragment containing compiler-owned CID/FID transforms triggers generation. Cache authority checks require a private, user-owned directory and reject symlinks throughout its ancestor chain. The manifest and artifact digests validate each reused result.
Structural opening, use, and spine memo entries survive definition boundaries within a compiled book. Definition-specific let entries are evicted at those boundaries, and the set of those entries is cleared when a new book starts. Probe indices remain valid across the book, so this change removes the per-definition probe-table reset. Peak memory for large books still needs measurement.
Remote validation with pinned Bun 1.3.14:
These measurements cover bounded fixtures. Full repository validation, Main-scale memory behavior, and a full Main cache-hit measurement remain pending.