Skip to content

Reduce the work of keeping LastCode current with upstream nightlies #720

Open

Description

@lastobelus

Tracking issue for the council-reviewed plan to reduce the recurring work of keeping LastCode current with upstream nightlies. The owner approved it on 2026-11-14. GPT-6.1 Sol (xhigh) and Claude Opus 5.5 each rated the final draft high with no blocking objections.

Each workstream lands as its own PR referencing this issue. Stage B reports come back here for the owner's decisions.

What this plan is

The fork edits 1,036 upstream files in place and upstream lands about 198 commits a day. Every one
of the last 45 nightlies conflicted. No mechanism makes that cheap; only the owner's decisions about
what the fork carries change it.

The plan has three parts:

  • Stage A fixes what the evidence already supports. It is committed.
  • Stage B produces two things the owner needs in order to decide: what each capability costs to
    carry, and whether a simpler replay shape works end to end.
  • Stage C is those two decisions, each made by the owner.

One rule holds throughout: inconclusive means no change. An experiment that runs out of budget,
or whose costs cannot be measured, goes back to the owner, who may fund more or drop it.

What v10 removed. Drafts v2 to v9 also contained three smaller levers: moving fork tests into
fork-owned files, stepping around upstream sweeps, and extracting seams in hot source files. Five
review passes tried to design a pilot and an adoption check that could show each one saves work.
Every design failed for the same reason: with a dozen agent runs and two or three recorded nights,
a modest saving cannot be told from noise, a rare expensive repair path cannot be ruled out, and
in production there is no measurement of what the same night would have cost without the change.
Both reviewers expected those experiments to end inconclusive after about 14 days of agent work.
That is not evidence the levers cannot help; it is evidence they cannot be shown to help this way.
They are deferred, with what would reopen them, near the end of this plan.

What "work" means

# Measure Baseline
C1 Conflict encounters per checkpoint: files and hunks the resolver actually meets, summed over replay steps Single-step median 48 files, 93 hunks. Real replay on the five exactly simulated nights: 612 file encounters against 495 single-step
C2 Files changed before publication that had no textual conflict Median 19. Not every one is known to be a necessary repair
C3 Promoted checkpoint commits that fail CI on lastcode/main 12 failures since the gate changed on 2026-10-23, in 23 or 24 runs
C4 Pull-request CI runs that fail the carry step 82 of 309 failed runs, on 61 branches, since 2026-10-03
C5 Effort, as defined in the next section Not measured
C6 Lines the fork changes inside upstream-owned files 182,418

Effort accounting

This applies to A2, to B1b's acceptance test and to the review after a replay-shape change.

  • Unit. Agent turns and tokens, and a gate must pass on both. Nobody picks the favourable one
    after seeing results. Owner interruptions and elapsed time to an installable build are
    must-not-worsen conditions.
  • Grain. Effort is attributed at the grain an agent actually works at: per replay stop, per
    repair phase, per pull-request carry failure, per revision cycle. Not per hunk.
  • Ranges. Each cost taken from records is reported as the minimum and maximum observed, or a
    single observation plus and minus 30%, and used at the end unfavourable to the change.
  • Unattributable costs. If a cost cannot be attributed from records, the gate that needs it is
    inconclusive.
  • Payback. One-time effort, plus recurring effort the change adds (operation, validation,
    review, upkeep of its tooling), against recurring effort it removes, over a horizon. The horizon
    is eight weeks unless the owner sets another, and it is fixed before the prototype starts.

Stage A: committed

A1. The same three checks on every publication path

Add vp run knip:check and the two generate --check steps from .github/workflows/ci.yml:119-126
to FULL_STEPS in scripts/lastcode-local-ci.ts and to the smoke gate, as one shared list. This
closes those three omissions; the smoke gate still does not run every full-CI test.

  • Steps run in list order, so list the unused-code step before the desktop build. Do not add a
    dependsOn between them: a step whose prerequisite failed is blocked
    (scripts/lib/lastcode-ci-collection.ts:26-39), which would hide a build failure on the same
    night the unused-code check fails.
  • Eight of the twelve failures since 2026-10-23 failed on the unused-code check alone. They would
    be intercepted before promotion if they reproduce in the prepublication environment. Count the
    benefit as revision cycles avoided as well as fewer C3 failures.
  • The other four failed in tests or typecheck that passed local full CI, one of them alongside the
    unused-code check. Classify each as inherited from upstream, fork regression or infrastructure
    before setting any target for them.
  • Follow-up decision, on two weeks of residual data: whether to hold promotion until GitHub CI is
    green on the exact candidate. That needs a CI route for a candidate ref. Measure the delay first.

A2. Effort baseline (budget: four days)

From the maintenance thread and the service run records, attribute effort under the contract above.

  • Sample 1: the last ten checkpoint episodes. Per episode: attempts, turns, tokens where
    available, owner interruptions, and elapsed time split into active agent work, queue and build
    waits, and waits for authorization. Per replay stop and repair phase: what kind it was (regroup
    encounter, generated-file conflict, test-file conflict, other textual conflict, semantic repair,
    unused-code repair, infrastructure failure).
  • Sample 2: ten pull requests that failed the carry step, split into missing trailer, regroup
    conflict, restack forced by the checkpoint rewrite, and ordinary stacked-PR or other causes, with
    the turns each cost. Removing the check is not credited with removing the underlying restack or
    conflict work, and the last category earns no credit from any replay shape.
  • Sample 3: tooling upkeep. Turns spent on pull requests that changed checkpoint or carry
    tooling, per week.
  • Also record, as the input for the deferred levers: the share of episode effort that went to
    test-file conflicts, and the effort of the four recorded sweep episodes (3238, 3457, 3655, 4630)
    against ordinary nights.
  • If records lack turns or tokens: say so per category. Anything that depends on that category
    is then inconclusive.
  • Afterwards, add to the run record only the fields that proved useful, and always turns and
    tokens per episode: the review after a replay-shape change needs them. No telemetry system.

A3. Regenerate two derived files

  • pnpm-lock.yaml: once the final tree's manifests are resolved, start from upstream's lockfile and
    regenerate. Check that fork-only package versions, shared-package overrides and patches are
    preserved.
  • agentContextBudget.json: no generator exists; its values come only from the failing assertion
    at AgentContextBudget.test.ts:160-181. Add one.
  • Both run as a step on the completed tree after conflicts are resolved, so they do not depend on
    the replay shape or on intermediate trees being runnable.
  • The budget workflow runs only on pull requests, so checkpoints need no exemption from its label
    rule. Build-repair pull requests still need the label.
  • Verified so far: 36 single-step conflict events on these two paths over 45 nights, 0.8 per
    checkpoint; excluding them moves the median from 48 to 47. The operational saving is not yet
    demonstrated; the expected value is fewer tokens on the lockfile.

A4. Report delta growth

  • Pull-request checks print two numbers, computed as endpoint comparisons over a fixed set of
    upstream-owned paths at a pinned upstream commit: what this PR adds and deletes there, and the
    whole fork's total. It reports; it does not fail. C6 is recorded weekly.
  • Guidance, narrow: new files are the default where practical; call-site edits and fixes to
    existing behavior in upstream files remain fine; tests for fork-owned modules go in fork-owned
    test files. This is model-facing text, so it follows evals/README.md.
  • No immediate effect on C1. It makes regrowth visible; the fork added about 392k lines in three
    months.

Stage B: two inputs for the owner

Order: B4 step 1 at once, with A1 and A2. Then B1a. Then B1b.

B4. Capability cost ledger, top ten (budget: three days)

  • Step 1, immediately: preserve the attribution data before anything can delete it. Snapshot the
    objects reachable from refs/lastcode/carry-sources/pr-* (about 390 refs on origin and
    growing), the Carry-Contribution trailers, and the logs of the 82 carry-step runs, which GitHub
    starts dropping in early January.
  • Step 2: attribute in-place lines and conflict load to capabilities per PR. Shared and unknown
    costs are allowed. Include tooling and build-ci (132k added lines) and each capability's
    dependencies, so a drop is actionable.
  • Step 3: for fixes with an upstream lifecycle, record per fix whether upstream has merged it.
  • Deliverable: ten decisions for the owner, each marked keep, drop or propose upstream, with
    its measured cost.
  • Acceptance: the owner can act on at least five of the ten without further analysis.
  • A proposal to upstream earns no saving until the downstream patch is actually retired.
  • Dropping a capability needs no pilot. It removes the patches and dependencies that are actually
    retired, so the sync cost attributed to that capability alone goes with them. Cost in files it
    shares with other capabilities stays, and the removal itself costs a one-time change and its
    validation. What it gives up is the capability, which is the owner's call.

B1. Replay shape

Two candidates against today's six group commits:

Shape History Needs Removes
(b) One aggregate carry commit Rewritten daily, as today A new bootstrap Carry-Group trailers, regroup conflicts, repeated resolutions of one file, the PR carry step, retained source refs
(d) True merge of each nightly Stable, with upstream ancestry Merge commits allowed on lastcode/main for the sync path, or a bypass for the service (today: squash only, required_linear_history) Everything in (b), plus force-push and the restacks the daily rewrite forces

Fixed points for both: the main write lock stays; exact upstream identity stays on the tag;
non-fast-forward protection stays. (d) keeps upstream ancestry, so the ancestry proofs
(scripts/lastcode-local-ci.ts:1119, :1133; scripts/lastcode-checkpoint.ts:2300) and
scripts/check-changed.ts:165 keep working. (d) keeps per-line attribution for later PRs only.
Ordinary stacked-PR and content-conflict work remains under both.

B1a. Inventory and choice of shape (budget: two days)

  1. Migration inventory per shape: scripts deleted, kept and changed; the guarded merge; repository
    rules; tag metadata; revisions; service recovery and retry when an ordinary merge lands during a
    sync; rollback, including what returning to six groups would cost.
  2. Encounter comparison extended to at least five nights that folded PR merges.
  3. Output: which shape to prototype, and a budget estimate for B1b for the owner to approve.

B1b. Lifecycle prototype of the chosen shape (budget: eight days, or as approved after B1a)

Run the shape against a scratch remote with the same rulesets over two consecutive nightlies. The
guarded merge and gate scripts select it through LASTCODE_GITHUB_REPOSITORY; the checkpoint
script selects it through --push-remote and --upstream-remote. The cases: publish and tag; the
checkpoint CI stamp and build validation that bind commit, tag and upstream identity; a recovery
that is retried; an ordinary PR merged during the sync; a revision; an upstream-only contribution
branch.

Acceptance, all required:

  • Every lifecycle case completes with the tag and build contract unchanged for consumers.
  • Payback under the effort contract. The recurring savings counted are the ones the shape removes
    by construction: the fold and its regroup conflicts; repeated resolutions of one file across
    groups; the sample-2 categories the carry mechanism itself causes (missing trailer, regroup
    conflict) at their measured cost; and restacks forced by the rewrite, for shape (d) only.
    Ordinary stacked-PR work is never credited. The one-time cost is the larger of twice the effort
    the prototype actually used and B1a's migration estimate, since the prototype does not cover the
    bootstrap, rule changes or documentation. Recurring costs the shape adds, in tooling, operation
    and resolution, are taken from the prototype.
  • Reported alongside, not as a gate: the production service replays the same two nightlies with
    six groups, so the prototype's resolution effort per night is set next to the service's for those
    nights. Two nights cannot decide anything, but it is the one same-night check that a single large
    step is not harder to resolve.

Net deletion of tooling is recorded as supporting evidence. It is not required on its own.

Stage C: two owner decisions

  • Replay shape. If B1b passes, the owner decides whether to migrate, including any repository
    rule change for shape (d).
  • Capabilities. From B4's ledger, the owner marks each of the ten keep, drop or propose
    upstream.

Review after a replay-shape change. A dated tracking issue is opened when it lands, naming who
runs the review. Eight weeks later:

  • What the shape removed by construction is confirmed gone: no fold, no carry step, no regroup
    conflicts.
  • What it added is counted from records: new failure modes, and turns spent on its tooling, against
    sample 3.
  • Checkpoint effort is compared before and after, both in total and per single-step conflict
    encounter. That count can be computed mechanically for every night under any shape. It is an
    imperfect adjustment: equal counts can hide different semantic repairs and failures, so it does
    not control for difficulty. This is a before-and-after comparison, not a paired one, and it is
    reported as such.
  • The result goes to the owner. A replay shape cannot be cheaply undone, so the plan does not
    promise a revert; it promises the owner the measured result.

What the evidence supports today

Change Supported statement Strength
A1 Eight of twelve recent post-promotion failures were in the omitted unused-code check Job names; not reproduced locally
A3 36 single-step conflict events on the two paths; median 48 to 47 without them Counted; saving not demonstrated
B1, single-step shape 19.1% fewer file encounters (612 to 495) and 9.1% fewer hunks (1,199 to 1,090); the carry step itself disappears Exact on five nights with no folded PRs; effort unknown; underlying restack work not shown to disappear
B4 Unknown until the owner decides

What is committed is stage A, whose expected effect is fewer broken promotions and a small saving
on two generated files. The replay shape and the capability decisions are where larger savings can
come from, and both are conditional.

Deferred, with what would reopen them

Lever What is known Why deferred
Fork tests in fork-owned files Test files are 34% of all file-conflicts; addition-only test conflicts, the eligible ones, are at most 12.7% of all file-conflicts, and that count is an upper bound Moved cases cannot conflict, but they can fail instead, so the saving has to be shown in effort
Stepping around upstream sweeps Four sweep commits touched files in 536 conflicts; the two declared classes cover two or three nights in 45 Stepping repeats encounters; the saving, if any, is in judgement, which only effort shows
Seams in hot source files The ten files v1 named are 7.6% of file-conflicts Real engineering with behavior risk, for a small share

A small pilot cannot show a modest effort saving for any of these: see "What v10 removed". What
would reopen one:

  • A2 shows that test-file conflicts, or sweep nights, take a share of episode effort, not only
    of counts, that the owner judges worth pursuing. The plan sets no threshold.
  • The owner then commissions a randomized rollout for that lever: eligible suites (or occurrences)
    assigned at random to changed and unchanged groups, each handled as its own recorded agent task,
    and compared on the same nights. Randomizing gives a real control on the same night and removes
    the dependence on a historical baseline. It does not by itself give enough occurrences to see a
    rare expensive path; the design would have to size for that. It is not designed here.

A4's narrow guidance still applies in the meantime, so new fork tests for fork-owned modules stop
adding to the first lever's problem.

Deliberately not in the plan

  • A hand-curated patch series for the whole fork. 586 files are shared between only six groups,
    and 61 branches in six weeks failed the carry step. Targeted patches for fixes with an upstream
    lifecycle are a possible B4 outcome.
  • More or finer carry groups. More reorder conflicts.
  • Squash-sync (a never-rewritten linear branch with upstream squashed in). It removes upstream
    ancestry, which breaks the build contract's ancestry proofs and check:changed, and a later real
    merge of upstream in any PR branch would silently reverse fork decisions.
  • rerere. 51 repeated hunks of 6,368 across nights. Revisit after B1.
  • Changing cadence. The owner wants nightlies, and larger batches mean larger repairs.

Decisions needed from the owner

  1. Approve stage A and the stage B budgets: A2 four days, B4 three, B1a two, B1b eight. That is 17
    days of agent work, of which only stage A is expected to pay back without further evidence.
  2. After B1a: which replay shape to prototype, the B1b budget and payback horizon, and whether
    repository rules may change for shape (d).
  3. After B1b: whether to migrate.
  4. After B4: keep, drop or propose upstream for each of the ten capabilities.

Activity

  1. lastobelus commented on Nov 15, 2026

    @lastobelus
    OwnerAuthor

    B1a done; owner decision recorded (2026-11-14).

    • B1a's migration inventory recommends prototyping shape (d), a true merge of each nightly into a stable lastcode/main. Its largest risks: the real repo would need a narrowly scoped merge path for the sync service while keeping protection for ordinary writers; publishing safely when a PR lands during a sync; and rollback to six groups needing a new bootstrap.
    • B1a's six-group encounter comparison is a proxy, because the real compile needs installed dependencies. B1b will measure it exactly.
    • The owner approved B1b on shape (d), capped at eight agent-days, on a private scratch repository with the same rulesets. Nothing changes on this repo until the owner decides after the prototype.
    • The payback horizon is fixed at six months. The one-time cost is the larger of twice the prototype's measured effort or the 8 to 12 agent-day migration estimate.
  2. 40 remaining items

  3. lastobelus commented on Nov 16, 2026

    @lastobelus
    OwnerAuthor

    Progress (2026-11-15 PT)

    New follow-ups (not started):

    1. Merges during checkpoint recovery. Recovery re-absorbs current lastcode/main before each service pass. For checkpoint 4723 that happened three times (test: move fork-only assertions into fork-owned test suites #729/refactor(web): drop the fork's keyboard-scrolling fix now that upstream ships it #731, a tooling change, refactor(threads): retire native subagent promotion #733), at about 15–25 minutes each, and refactor(threads): retire native subagent promotion #733's absorption also broke preparation by duplicating a dependency pin. Consider holding merges while a recovery is running, or letting recovery leave later merges to the next nightly.
    2. The recovery gate misses failures that Full CI catches. 4723's second service pass failed on 8 tests and a format error that the recovery gate had passed, costing a full one-hour service cycle. Its third pass failed on 2 more server tests.
    3. The PR watch may not wake when the same head's check passes again. On fix(ci): verify source CI when a PR's title or body is edited #736 the watch expired without waking when the metadata run's CI Gate succeeded; the result had to be checked directly.
  4. lastobelus commented on Nov 16, 2026

    @lastobelus
    OwnerAuthor

    #325 vs upstream pingdotgg#16941: reconciled in PR #740; open, blocked by existing CI failures.

    Upstream https://redirect.github.com/pingdotgg/t3code/pull/16941, merge 8f947a313b29c30c1deb9960ea0450482cb6e19f, is already an ancestor of both origin/main and the LastCode base 6cd814fceafb68cfd26d475e2a1f038b88f2c534.

    PR #740 restores upstream's numeric queue-deadline API and removes the duplicate signal-aware overload. The broader native-rendering, background-subscription, broker-isolation and page/recording-cancellation behavior stays because upstream's dispatch check does not cover it. Cancellation remains at the existing native-operation seam. Four server files changed: 45 lines added, 63 removed.

    Validation: changed-file checks passed, including 354 tests and affected typechecks; exact-head Quick CI passed. One local Terra xhigh review approved, and one exact-head GitHub Codex review found no issues. Zero unresolved review threads.

    Merge blocker: CI failed twice on head 592c73d4ec80387a2d38566dfbc1f1e8ba2a73ce. The server-browser test worker exhausts its JavaScript heap, and the dev-runner process fixture fails to start. Both failures already occur on the unchanged base in https://github.com/lastobelus/lastCode/actions/runs/43871192455; PR attempts are at https://github.com/lastobelus/lastCode/actions/runs/43924831969. No guard was bypassed and no merge occurred.

    Next: diagnose the existing server-test worker memory exhaustion and dev-runner process-fixture startup failures, then obtain green CI before guarded merge. This is incomplete until #740 merges. #720 remains open because it also tracks other audit decisions.

  5. lastobelus commented on Nov 16, 2026

    @lastobelus
    OwnerAuthor

    #325 vs upstream pingdotgg#16941: reconciled and merged in #740 (006e9d3b50a50df3d21ed3061e9185b186845b1b). This item is resolved.

    Upstream https://redirect.github.com/pingdotgg/t3code/pull/16941, merge 8f947a313b29c30c1deb9960ea0450482cb6e19f, was already in both upstream and the LastCode base. #740 restores its numeric queue-deadline API and removes our duplicate signal-aware overload. Cancellation stays at the existing native-operation boundary.

    Retained beyond upstream: native rendering leases and measured viewport; root browser subscriptions across thread switches and epoch fencing; broker request retirement without disconnecting unrelated tabs; stalled page/CDP cancellation and recording cleanup. Upstream's pre-dispatch deadline does not cover those behaviors, so the broader carry is narrowed rather than retired.

    The PR also fixes the two existing CI failures that blocked delivery: the browser dialog fixture now blocks only its own expression instead of intercepting title watches, and the dev-runner shutdown fixture stubs Cargo instead of compiling the resource monitor during startup. Both fixture repairs replay with their owning incubator group; the production reconciliation stays in upstream-bugfixes. Each source commit has one Carry-Group trailer.

    Proof: 407 focused tests, affected typechecks, changed-file checks, exact-head Quick CI, and exact same-base carry compilation passed. One local Terra xhigh review approved; three GitHub Codex rounds found no issues, including the final head. Zero unresolved review threads. PR CI and the merged commit's main push CI passed: https://github.com/lastobelus/lastCode/actions/runs/43933075336.

    No checkpoint, package build, installation or restart was requested. No follow-up remains for this item. #720 stays open for its other audit decisions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions