Skip to content

heavy-install: link t3-capacity to the kit's copy; self-check warns on another (#919) - #922

Merged
scuttletoby merged 5 commits into
mainfrom
fork/heavy-t3-capacity-link
Nov 22, 2026
Merged

scuttletoby merged 5 commits into
mainfrom
fork/heavy-t3-capacity-link

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

What and why

t3-capacity now has one source: the heavy kit's own bin/t3-capacity (fork part of #919, steps 1 and 3).

  • heavy-install links /usr/local/bin/t3-capacity to current/bin/t3-capacity, the same way it links heavy and heavy-install. Installing a newer kit now changes what t3-capacity does too, so the two copies can no longer drift apart.
  • A host's own plain copy is replaced only when you ask (--replace-t3-capacity), as the issue's plan says ("unless told"). Without the option, the installer leaves the file alone, prints a note, repeats a WARNING in its closing installed release line, and writes no installed.tsv row for it. A link there that the kit did not make is treated the same way without the option (left alone, noted, no row). It is opt-in because a host's installer and CI may read or compare that exact file (see the consumer list below). Replacing it without being asked would break them. Once the link is in place, later runs keep it without the option.
  • The old plain copy is kept in /usr/local/libexec/heavy-kept/t3-capacity.<sha256[:16]>, stored by hash. uninstall puts it back and then removes that folder. It is kept outside /usr/local/libexec/heavy and /etc/heavy, because every released installer's uninstall (ci/v6 to ci/v22) deletes those two folders whole; and it is not in replaced/, because those installers restore the newest file of replaced/ as /usr/local/bin/heavy.
  • Refused before anything is written: with --replace-t3-capacity, a t3-capacity that is a link to somewhere else (remove it by hand: there is no file to keep); with or without the option, a folder there.
  • heavy --self-check warns when the t3-capacity found on PATH is not this kit's: either it is missing, or a different copy comes first on PATH (for example a user's own wrapper). The warning does not change the exit code.
  • README: the Files table, the Install, Stops half-way and Rollback text, and the self-check row.

The release needs no new file: bin/t3-capacity was already in the release's source files. The #920 release test passes unchanged (ok release holds what the heavy kit's installer reads).

Why the kit's reading differs from an old copy

There are three things a caller can reach today, and they measure different things:

  • Kit (heavy facts): available= is the box's available memory less the hold of the reservation files (~/.config/t3-capacity/reserve.d). reserved= is the room that admitted work may still grow into: each running T3 agent counted as a writer (768 MB start cost minus what it uses now), machine-file holds, plus that same reservation hold again. So the hold is inside both numbers: do not compute available − reserved, that subtracts the hold twice. free= is the kit's own answer and is right. The answer is OK only when free covers the request and the level is ok.
  • The old plain copy (the host's installed file, run directly): reads no reservation files and does not count agents. Its reserved= is only the growth room of other busy budgets (the heavy slice, fork runners, the CI-light slice), 0MB when they are idle. Instead it checks box memory pressure (PSI), a soft cap, and load above 1.5× the core count.
  • A user's own wrapper in front of it on PATH (one host has one): reads the reservation files, prints reservations: …(available below is after these), lowers available= by their hold, and then runs the plain copy. It still does not count agents.

Readings on one machine, each row pair at the same moment, with one 1536 MB reservation holding all 1536 MB. The kit is this branch's bin/t3-capacity run from the source folder (A to C: from a test-prefix install), next to the plain copy run directly, and in E also the wrapper:

run need command available reserved free answer
A 3072 MB kit 7900 MB 6763 MB 1392 MB NOT OK: level severe
A 3072 MB plain copy 9438 MB 0 MB (not shown) NOT OK: box memory pressure
B 1536 MB kit 6076 MB 6327 MB −22 MB NOT OK: level severe
B 1536 MB plain copy 7619 MB 0 MB (not shown) NOT OK: load above 1.5× cores, box pressure
C 1536 MB kit 7811 MB 5909 MB 2123 MB NOT OK: level severe
C 1536 MB plain copy 9326 MB 0 MB (not shown) NOT OK: load above 1.5× cores, box pressure
D 1536 MB kit 5780 MB 7507 MB −1503 MB NOT OK: 1503 MB short (level ok)
D 1536 MB plain copy 7316 MB 0 MB (not shown) OK
E 1536 MB kit 4358 MB 7561 MB −2969 MB NOT OK: short, level elevated
E 1536 MB wrapper 4343 MB 0 MB (not shown) NOT OK: box soft cap
E 1536 MB plain copy 5857 MB 0 MB (not shown) NOT OK: box soft cap

What the readings show:

  • The kit's available= is about 1.5 GB below the plain copy's in every run (1538, 1543, 1515, 1536, 1499 MB): that is the reservation hold, not a different source. Against the wrapper, which also takes the hold off, it is the same within the time between the two reads (E: 4358 vs 4343 MB).
  • Only run D shows a different answer: the plain copy said OK and the kit said NOT OK, because the kit counts the room 12 running agents may still grow into (reserved= less the hold: about 6 GB). The wrapper was not read at that moment. In A, B, C and E both said NOT OK, for different reasons.

Kit line from D:

available=5780MB swapfree=2094MB load=5.65/12cores agents=12 jobs=0 waiting=0 reserved=7507MB need=1536MB free=-1503MB level=ok tier=full
NOT OK: 1536 MB (-1503 MB free, 1536 MB needed, level ok)

Plain copy line from D (first line shortened):

available=7316MB swapfree=2094MB load=5.65/12cores java=0 agent-sessions=12 heavy-slice=2247MB/6144MB fork-slice=51MB/6144MB reserved=0MB need=1536MB box=heavy.slice ...
OK

Does any number get worse for agents? Yes, it gets stricter (run D): with many agents running, the kit holds back their unused room, so a request the old copy allowed can be refused. In return, the kit does not refuse on load alone, and it does not read the old copy's box pressure or soft-cap checks, T3_CAPACITY_BOX, or the CI-light slice. The first line also loses the old copy's extra fields (java=, agent-sessions=, heavy-slice=, fork-slice=, box…=). The answer stays exit 0 = OK / exit 1 = NOT OK. A separate double count inside the kit (an agent working under a reserved folder is held both as an agent and by the reservation) is #923, not this PR.

Consumer part of #919 (step 2, not in this PR)

What a consumer whose installer writes its own /usr/local/bin/t3-capacity must change in its own repo:

  1. Its installer: stop writing t3-capacity to /usr/local/bin. Instead, run heavy-install --replace-t3-capacity once, which keeps the old file in /usr/local/libexec/heavy-kept/ and links the kit's, or check that /usr/local/bin/t3-capacity resolves into /usr/local/libexec/heavy/current/. Never cp, install or mv onto the path:
    • cp writes through the link into the release folder;
    • an install/mv of a renamed copy puts a plain copy back.
  2. CI steps that grep the installed file (for a LIGHT= slice line) must read that fact elsewhere, for example from the kit's config or the machine file, because the kit's script has no such line.
  3. A build-slot helper that compares the installed copy with the repo copy (and gives one slot until they match) must compare against the kit release instead, or stop comparing. Otherwise it stays on one slot forever.
  4. Callers that set T3_CAPACITY_BOX, or rely on the CI-light box or load checks, lose those checks. Move the holds into the machine file's heavy.capacity.holds, or keep a separate check in the consumer.
  5. Readers of the extra first-line fields listed above must switch to heavy facts or the kit's fields.
  6. Steps that write the consumer's own copy next to a helper (not to /usr/local/bin) are not affected.

State answers

  • Runs again: the link already points into current, so nothing changes. A test snapshots the whole prefix before and after a second run, with and without the option, and with a foreign link left alone.
  • Stops half-way: every install step was stopped and then run again (test_every_stop_half_way_then_run_again, with an old t3-capacity and the option). The old command keeps working until the links step, when it is swapped in one rename (a temp link renamed over the path). After the rerun there is exactly one kept copy and the link. Another test makes the t3-capacity swap itself fail: the old command still runs, and the next run finishes the job.
  • Old or empty state:
    • nothing there: the link is made;
    • our link (also a broken one left by an older uninstall): kept;
    • a plain file without the option: left alone, with a note and a WARNING in the closing line;
    • a plain file with the option: kept in heavy-kept/ and linked;
    • a link elsewhere without the option: left alone, with the same note and warning, no installed.tsv row;
    • a link elsewhere with the option, or a folder: refused before anything is written.
  • Rollback:
    • uninstall puts the newest kept copy back only if the path is empty, then removes heavy-kept/ (it stays, with a note, if something else is at the path);
    • rolling current back to a release from before this PR leaves the link pointing at that release's bin/t3-capacity, which exists in every release;
    • an older installer's uninstall (every release up to ci/v22) deletes /usr/local/libexec/heavy and /etc/heavy whole, so the kept copy lives outside both and survives. That uninstall does not know the t3-capacity link and leaves it broken (gates then fail with 127); running this release's sbin/heavy-install uninstall from its source folder removes the broken link and puts the copy back (README, Rollback). Tested with ci/v22's own installer, taken from git: install with this branch and the option, ci/v22's install and uninstall on the same prefix, then this uninstall restores the original bytes.

Tests

First round, without the change (main's kit plus that round's test file): Ran 22 tests ... FAILED (failures=16, errors=1). Every new test failed, and so did every step of the stop-half-way test. The 15 tests that already existed passed.

Fix round 1 (review comment on 975cdef), each new check run against 975cdef with this round's test file:

  • kept copy survives an older uninstall (test_older_installers_install_and_uninstall_keep_the_hosts_t3_capacity, runs ci/v22's own heavy-install from git archive; it skips with a message in a checkout without the tag, such as CI's shallow clone): at 975cdef (folder name only aliased) it fails with FileNotFoundError: .../usr/local/libexec/heavy/replaced-capacity right after ci/v22's uninstall, i.e. the copy was deleted. Now it passes, and this release's uninstall puts the original bytes back.
  • same deletes without git (test_kept_t3_capacity_is_outside_every_folder_an_uninstall_deletes, runs everywhere, CI too): at 975cdef AssertionError: True is not false : /usr/local/libexec/heavy (the kept folder is inside it).
  • foreign link without the option is left alone (test_foreign_t3_capacity_link_is_left_alone_without_the_option): at 975cdef 1 != 0 : ... /usr/local/bin/t3-capacity exists and is not the kit's link.
  • closing-line warning (added to test_old_plain_t3_capacity_is_left_alone_without_the_option): at 975cdef "WARNING: /usr/local/bin/t3-capacity is still the host's own, not the kit's (--replace-t3-capacity switches it)" not found in 'heavy-install: installed release aaaa…: slice heavy.slice, grace 30s'.
  • The refusal test now covers a foreign link with the option and a folder with and without it.

Tests in fork-kit/heavy/tests/install.test.py for this PR:

  • fresh install links t3-capacity to the release
  • a plain old copy is left alone without the option, with a note and the closing-line warning
  • with the option, a plain old copy is kept and linked, a rerun without the option changes nothing, and uninstall puts it back and removes heavy-kept/
  • the kept copy is outside every folder an uninstall deletes; after those deletes this uninstall restores it
  • ci/v22's install and uninstall over this state keep the copy; this uninstall restores it
  • the t3-capacity swap fails: the old command still works, and running again fixes it
  • a foreign link without the option is left alone (note, warning, no row, rerun changes nothing, uninstall leaves it)
  • a foreign link with the option, or a folder, is refused before anything is written
  • self-check warns when t3-capacity on PATH is not the kit's
  • the stop-half-way test also covers t3-capacity

At the new head, all through heavy:

  • install.test.py 25 OK (the ci/v22 test ran, not skipped)
  • heavy.test.py 68 OK (55 skipped on this machine, as before)
  • .github/scripts/fork-ci-release.test.py: all passed, 86 ok
  • seam check (fork-seam-check.cjs --config-ref origin/main origin/main HEAD): exit 0
  • vp fmt --check fork-kit/heavy/README.md: clean
  • First round only (files not changed since): facts 57 OK, capacity_rule 12 OK, queue 70 OK, measure 47 OK, sweep 82 OK (1 skipped)

General vs project

  • General (this PR): the kit links its own t3-capacity, keeps any old copy outside the folders any uninstall deletes, replaces a host's own copy only when told, and the self-check warns when a different copy is on PATH. Nothing here names a project, path or machine. The kit's install folder, the option and the backup folder are the same for every user.
  • Project: a consumer's own installer and CI (the list above) change in that consumer's repo. No project facts were added to the fork.

Part of #919. This PR is the fork part (steps 1 and 3). Step 2, the consumer part, is the list above.

🤖 Generated with Claude Code

scuttletoby and others added 2 commits November 22, 2026 09:18
…n another (#919)

heavy-install now also links /usr/local/bin/t3-capacity to
current/bin/t3-capacity, so the command has one source: the release in
use. A plain old file there (a host's own copy) is kept in
/usr/local/libexec/heavy/replaced-capacity/ (not replaced/: an older
installer's uninstall restores the newest file of replaced/ as heavy) and
stays in place until its link replaces it in one rename; uninstall puts it
back. A link elsewhere or a folder there is refused before anything is
written. heavy --self-check warns when the t3-capacity found on PATH is
not this kit's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-replace-t3-capacity

A host's installer may write its own t3-capacity to /usr/local/bin and its
scripts may read or compare that file, so the installer now leaves such a
plain file alone (with a note, and no installed.tsv row for it) unless it is
run with --replace-t3-capacity. With nothing there, or the kit's link, it
links as before; once linked, later runs keep the link without the option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: code · reviewer: lens agent (Opus) · read at: 975cdef
covered: the whole diff (sbin/heavy-install, bin/heavy capacity_on_path, README, tests/install.test.py), #919 and its comment, ci/v22's heavy-install (install and uninstall), bin/t3-capacity, lib/heavy_facts.py (reservations, pieces, answer) and lib/capacity_rule.py (_read_facts, decision)
checks: ran through heavy at this head: tests/install.test.py 22 OK, tests/heavy.test.py 68 OK (55 skipped here, so the self-check warning test path was read, not run), .github/scripts/fork-ci-release.test.py 86 ok; read ci/v22 sbin/heavy-install install() and uninstall() against this state (not run: this lens may run only the listed tests); compared the live output of a host's old plain copy
state: runs again: no change (the link is ours; checked by the snapshot test); stops half-way: the old file is copied to its folder first and the link renamed over it in one step, a rerun finishes (tested at every step); old/empty state: nothing there or a dangling link to current/bin/t3-capacity is linked, a plain file is left alone (or kept and linked with the option), a foreign link or a folder refuses the whole install before writing; upgrades: the link names current/bin/t3-capacity, so it follows every new release; rollback to an older installer: its install leaves the link (its release has bin/t3-capacity) and replaced-capacity/, but its uninstall deletes replaced-capacity/ (major below)

blocker: none
major: an older installer's uninstall loses the host's own t3-capacity. REPLACED_CAPACITY is inside /usr/local/libexec/heavy (sbin/heavy-install:39), and ci/v22's uninstall() ends with shutil.rmtree of /usr/local/libexec/heavy and /etc/heavy without restoring anything from replaced-capacity/ and without removing the t3-capacity link (it only knows the heavy and heavy-install links). So after --replace-t3-capacity, a rollback with the older installer followed by its uninstall (the path the README's Rollback bullet allows, README.md:717) deletes the only kept copy and leaves /usr/local/bin/t3-capacity a dangling link: every t3-capacity N && cmd gate then fails with 127 until someone reinstalls, and the host's copy is gone for good. The PR body says the opposite ("an older installer's uninstall ... never touches replaced-capacity/"), and no test runs an older installer against this one's state. Fix: keep the copy outside both folders an older uninstall removes (for example /usr/local/libexec/heavy-kept/t3-capacity.<sha>, checked as a folder in check_old_state), add a test that installs with the option, then runs main's heavy-install install and uninstall on the same prefix and checks the copy survives; and say in the README that an older uninstall leaves a dangling t3-capacity link (run this release's uninstall first, or reinstall)
minor: a link at /usr/local/bin/t3-capacity that is not exactly current/bin/t3-capacity refuses the whole install even without --replace-t3-capacity (sbin/heavy-install:260-269), while a plain file there is left alone. Until this PR the README told hosts to wire the command in themselves, so a host that linked it (to a version folder, to current by another spelling, or to its own script) can no longer upgrade heavy at all until someone removes the link by hand. Suggest: without the option, treat a non-kit link like a plain file (leave it, note it, no row); with the option, replace a link without keeping it (nothing to keep) or keep refusing
minor: the push to switch is weak, so hosts can stay on the old copy for good. Opt-in is the right call (#919 says "unless told", and consumer CI reads that exact file), but the only signals are one note: line in the install output and a WARNING: from heavy --self-check that keeps exit 0, and nothing runs heavy --self-check on a schedule. Suggest: repeat it in the closing installed release ... line ("t3-capacity: the host's own copy, not the kit's"), and track consumer step 2 in its own issue so the switch has an owner
minor: PR body, "Why the kit's reading differs": the old consumer copy does read reservation files (on a host running it now its output starts reservations: <name>:1536MB(using 0,holding 1536) (available below is after these)), so "does not count agents or reservation files at all" is wrong for reservations, and the lower kit available= is not "a different source" alone: bin/t3-capacity:51 prints available less the whole hold, while reserved= is owed plus that same hold (lib/heavy_facts.py:491). A reader who computes available − reserved subtracts the hold twice; free= itself is right. In all three runs both commands said NOT OK, the kit for level severe, so the table shows no request the kit refuses and the old copy admits; the "stricter" claim stands from the code, not from these readings
minor: double count in the kit (not in this diff, but this PR is what moves hosts onto it): an agent scope whose working folder is under a reserve.d folder is a writer piece owing 768 MB less its use (lib/capacity_rule.py:345-346), and the same reservation still holds its MB less the use of every process in the folder (lib/heavy_facts.py:162-197). A project reservation meant to cover that project's agents, servers and builds then holds the agents' unused room a second time, up to the smaller of their owed room and the reservation's hold (with one idle 300 MB agent and a 1536 MB reservation: 468 + 1236 = 1704 MB held, against 1536 meant); a heavy job started in the folder is held twice the same way while it starts. Other callers are then refused by up to the reservation's size too early; the caller's own folder is excluded, so the project's own agents are not. Not a merge stop because the switch is opt-in, but it should get its own issue and a fix (a reservation's hold less the owed room of the pieces working in its folder) before a host with reservations runs --replace-t3-capacity

…tall deletes

- The kept copy moves to /usr/local/libexec/heavy-kept/: every released uninstall
  (ci/v6 to ci/v22) deletes /usr/local/libexec/heavy and /etc/heavy whole, so a copy
  kept inside them was lost. This release's uninstall puts it back (also after an
  older uninstall left the t3-capacity link broken) and then removes the folder.
- A link at /usr/local/bin/t3-capacity that the kit did not make is left alone with
  a note when --replace-t3-capacity is not given (refused with it, as before).
- The closing install line repeats that the host still uses its own t3-capacity.
- Tests: an older release's install and uninstall (ci/v22, from git) over this
  state; the same deletes without git; a foreign link; the closing-line warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: code r2 · reviewer: lens agent (Opus) · read at: ca97789
verdict: not safe to merge
covered: the fix diff 975cdef..ca97789 (sbin/heavy-install, README, tests/install.test.py) and the whole PR diff again (bin/heavy capacity_on_path); the uninstall() of every released installer (ci/v6 to ci/v22); which releases ship bin/t3-capacity; bin/t3-capacity and lib/heavy_facts.py free/reserved/hold (for the PR body numbers); this head's CI jobs
checks: through heavy at this head: tests/install.test.py 25 OK (the ci/v22 test ran here, not skipped), tests/heavy.test.py 68 OK (55 skipped here), .github/scripts/fork-ci-release.test.py 86 ok; this head's test file against 975cdef's installer: FAILED (failures=3, errors=13), so the new tests fail on the round-1 code; recomputed the PR body table (kit available is 1538/1543/1515/1536/1499 MB below the plain copy; D reserved less the hold is 5971 MB; free = available - reserved + hold - margin, so free is not available - reserved, as the body says)
state: runs again: a second --replace-t3-capacity after the copy is kept finds the kit's link, keep_old skips a link and switch_link does nothing (snapshot test); a foreign link without the option is left alone each run (snapshot test). Stops half-way: a crash after keep_old leaves the kept copy and the plain file; a rerun with the option finds the same hash and only links (tested), a rerun without it leaves the file (note and WARNING), and uninstall then removes heavy-kept/ because the file there equals the kept copy. Old state: an older installer's install over this state leaves the link to current/bin/t3-capacity (ci/v21 and ci/v22 ship that file), its uninstall deletes only /usr/local/libexec/heavy and /etc/heavy (checked in all 17 released installers), so heavy-kept/ survives, and this release's uninstall from its source folder removes the broken link, puts the bytes back and removes heavy-kept/ (tested with ci/v22's own code). heavy-kept/ from a crashed run: a stale t3-capacity..tmp is overwritten by the next keep_old and ignored by uninstall. Empty state: nothing there is linked; uninstall with nothing kept removes only the link
round 1, fixed: the kept copy is now /usr/local/libexec/heavy-kept/, outside both folders every released uninstall deletes, and two tests cover it (one runs ci/v22's installer)
round 1, fixed: a link elsewhere without the option is left alone with a note, the closing WARNING and no installed.tsv row; with the option it is refused before anything is written (tested)
round 1, fixed: the closing installed release line now carries the WARNING while the host's own t3-capacity is in place (tested)
round 1, fixed: the PR body now says the wrapper reads reservations, explains the hold inside both available= and reserved=, and adds run D where the plain copy says OK and the kit says NOT OK; the numbers check out
blocker: none
major: this head's CI is red because of the new test: "Fork kit tests 2" fails with install.test.py skipped Install.test_older_installers_install_and_uninstall_keep_the_hosts_t3_capacity, which it does not declare, since CI's shallow clone has no ci/v22 tag and the job refuses an undeclared skip; fork-ci-result is failure. Fix: add a whole-line # fork-ci: may skip Install.test_older_installers_install_and_uninstall_keep_the_hosts_t3_capacity near the top of fork-kit/heavy/tests/install.test.py (as .github/scripts/fork-fleet.test.py:7 does), or fetch that tag in the job so the test runs; the no-git test test_kept_t3_capacity_is_outside_every_folder_an_uninstall_deletes still covers the deletes in CI
minor: uninstall unlinks the t3-capacity link (sbin/heavy-install:441-443) and only then copies the kept file back (sbin/heavy-install:449-453), so for that moment t3-capacity is missing, and a crash between leaves it missing (gates fail 127) until uninstall runs again. Renaming the restored copy over the link in one step would close it. Follow-up
minor: the uninstall branch that leaves heavy-kept/ in place because a different file is at the path (sbin/heavy-install:454-459) has no test, and a heavy-kept/ that holds only a crashed .tmp is never removed (no backups, so the branch does not run). Follow-up
minor: rolling back to ci/v20 or older (no bin/t3-capacity in those releases) leaves the link broken right away; the README Rollback text says to uninstall first, which is enough. A note in the closing line or self-check when current/bin/t3-capacity is missing would make it visible. Follow-up
minor: the PR body ends with "Fixes #919" while consumer step 2 stays open, so merging closes the issue that tracks step 2. Use "Part of #919", or open its own issue for step 2 first (as round 1 suggested). Follow-up

…checkout

CI's clone has no ci/v22 tag, so the test skips there; fork-ci refuses an
undeclared skip (round-2 code lens on #922).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

Round-2 review handled:

This was the second lens round, so the PR merges once CI is green.

🤖 Generated with Claude Code

…SKIP

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby
scuttletoby merged commit f2ad5ed into main Nov 22, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant