Reclaim the tagged derivative subtrees - #537
Conversation
Most of a full cache is inside worktrees that have to stand: 18 of the 72 measured on the reference host carried a whole `.pixi/envs/default`, about 94.5 GB of the 104.5, and a site that is dirty, unpushed or unprovable keeps every byte of it. What makes reaching inside one legitimate is what the standing verdict is a statement about. Every reason a site stands, except a claimant's, is an answer about git's account of that site's content, and nothing under `.pixi/envs/` has ever been in that account: pixi's own `.pixi/.gitignore` of `*` and `!config.toml` puts it outside every index, every status and every commit. The two sets of bytes are disjoint, and a file the installer wrote is what separates them. The gate is the Cache Directory Tagging Specification's published 43 byte signature, and it never reads a directory's name: rattler, cargo, uv and pytest write one, `python -m venv` and npm write none, so the same name lands on both sides. The walk stops at the outermost tag and at every site the forest holds. A tag says regenerable and not by what, so one reader answers with the thing that re-derives it: `conda-meta/pixi` for the environment name, then a `pixi.lock` found by walking up inside the site. `manifest_path` is never stored, because it is a container path on every host environment. An absent lock, a lock that no longer names the environment, and a tag no reader recognises all stand and are named with their bytes. No new flag. It rides `--prune`'s own plan and its own y/N, and the acting pass re-reads both records under the lock before anything goes. What goes is the tagged directory alone, never `.pixi`. Closes #472.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Reviewer's GuideThis PR extends --prune with a separately accounted, tag-gated derivative sweep inside standing agent worktrees: it discovers outermost CACHEDIR.TAG directories, proves pixi recipes from local metadata and lockfiles, excludes claimant-pinned or unrecipeable trees, presents them in the existing plan and confirmation, revalidates them under lock before deletion, and leaves --ls behavior unchanged. Sequence diagram for pruning regenerable subtreessequenceDiagram
participant User
participant Prune as dl --prune
participant Sweep as WorktreeSweep
participant Gate as CACHEDIR.TAG gate
participant Reader as Pixi recipe reader
participant Plan as Prune plan
participant FS as Filesystem
User->>Prune: run --prune
Prune->>Sweep: weigh standing worktrees
Sweep->>Gate: declared_regenerable(directory)
Gate-->>Sweep: outermost tagged directories
Sweep->>Reader: pixi_recipe(tag, site)
Reader-->>Sweep: Recipe or NoRecipe
Sweep-->>Plan: reclaimable and withheld derivatives
Plan-->>User: show paths, sizes, recipes, and y/N
User->>Prune: confirm y
Prune->>Sweep: re-weigh under lock
Sweep->>Gate: re-read CACHEDIR.TAG
Sweep->>Reader: re-read metadata and pixi.lock
Reader-->>Sweep: derivable or withheld
Sweep->>FS: remove_tree_as_far_as_it_goes(tagged directory)
FS-->>User: report reclaimed or withheld
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Codecov Report❌ Patch coverage is Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
blooop
left a comment
blooop
left a comment
There was a problem hiding this comment.
This was generated by AI during review.
Fresh context, adversarial, at a884c7f1b1aae660fe1d74e6cc64e7bf48afb0d3. The reviewer did not write this code. Every finding below was proved by a probe that was run; four mutants were killed against the existing tests.
Note on why this review exists: this PR's review check was green for a bad reason. Sourcery posted a quota refusal in a wording scripts/review_verdict.sh does not recognise, so a refusal counted as a review (filed as #538). This is the only review this PR has had.
Standards
Axis: pass. No path was found that destroys non-regenerable data in a realistic scenario. The gate, the walk, the reader and the claimant fold hold up under attack, and the tests are not vacuous.
Measured against the real host, read-only, using the same predicate the code uses: 9 sites, 12 tagged directories, 9 derivable, 20.54 GiB reclaimable, and zero files carrying a link outside their tree — so the byte figure is not inflated by shared inodes. New walk costs 423 directories in 0.01 s plus 0.72 s of exclusive_usage warm.
Spec
Axis: fail — 7 findings, none blocking, but F1/F2/F3 are on the surface the user reads and say false things.
F1. A refused second listing drops the approved reclaim silently: nothing done, nothing said, exit 0
When ClonePicture::of fails in the acting pass, reclaim extends report.withheld from plan.going only. plan.derivatives is dropped with no withheld_derivatives entry, so a directory the user was told would be reclaimed is neither reclaimed nor mentioned.
reclaimed=[] withheld_derivatives=[] withheld=[] nothing_to_say=true
PROBE FAILED: the user approved reclaiming a subtree and the run reports nothing at all about it
Probe planned one derivable subtree with going empty, then renamed clone/.git between plan and act. WorktreeReport::nothing_to_say() is true, so nothing is emitted, and PruneReport::finished() reads only refused/forget_refused, so dl exits 0.
Fix: give the early-return arm the derivative half its sibling already has — extend withheld_derivatives with a NotDerivableNow arm meaning this pass could not re-take the classification.
F2. The report says the cache tag is gone while it is still on disk
NotDerivableNow has NoTagThere and Answered, and no arm for the third real cause: the containing site became collectable between plan and act, so weigh returned it as removable with an empty derivatives list, and reclaim_derivatives reads that absence as NoTagThere.
withheld … /agent-one/.pixi/envs/default -> there is no longer a cache tag at that place,
so nothing there declares itself regenerable
PROBE FAILED: the report says the tag is gone when it is right there
The tag, conda-meta/pixi and pixi.lock were all asserted present in the same probe. Reached by deleting an uncommitted NOTES.md while reading the plan.
Fix: distinguish the causes at the source — a new arm for its site is no longer standing, so this pass did not weigh it, or re-read the tag before choosing NoTagThere. The arm's own doc says "it was removed, or its whole site was", and neither is true here.
F3. The pointer the plan prints is not the command the measurement was made with, and it fails outright without a manifest
Recipe::describe prints pixi install -e {environment}. Every measurement this module and its docs rest on is pixi install --frozen --offline. Two failures of the plan's own footer ("one command and no network beyond the shared package cache"):
- With a stale lock — the case the module explicitly reclaims — plain
pixi installcompares the lock hash against the manifest and re-solves, which is network. Only--frozengives the measured behaviour. - With the lock present and the manifest absent, the printed command errors outright:
$ pixi install -e default # pixi.lock only, no manifest
Error: × could not find pixi.toml or pyproject.toml with tool.pixi at directory /tmp/lockonly
Fix: print pixi install --frozen -e {environment}, and make lockfile_above also require a manifest beside the lock — existence only, since reading it would break the deliberate "the manifest is never read" rule.
F4. The plan prints "devlaunch will never reclaim it" one line above reclaiming it
Unaccountable::RegisteredElsewhere::describe() says "devlaunch will never reclaim it, and only the repository that registered it can", while Reason::subject() maps NotThisClonesToAccountFor to GitsAccountOfContent, so the tagged subtree inside it is reclaimed on the same plan.
STANDING …/worktrees/theirs: … devlaunch will never reclaim it, and only the repository that registered it can
DERIV …/worktrees/theirs/.pixi/envs/default: standing=None <- will be reclaimed
The behaviour is decided on #468 §6 and is not in question; the words are now false. This is the same "shipped contradiction" argument the PR body makes in its own favour, pointing the other way.
F5. Blank::ASiteSitsInside is narrower than its own sentence
classify's nested-site check joins against forest, which walk_sites builds by descending only the .claude/worktrees/ spine. So the arm can fire only when the tag sits at <site>/.claude or <site>/.claude/worktrees. A registered worktree anywhere else under the tag is invisible to the fold, and Blank::describe() ("a git worktree sits inside it") claims a guarantee the check cannot make.
Both placements are absurd in the wild, so this is words rather than risk. On the new-public-surface question: the arm is justified as an exhaustiveness answer, is produced only at derivatives.rs:458 as claimed, and Subject::AClaim never enters a Standing, so first_claim never sees it.
F6. "Never .pixi" is not structural, only accidental
Nothing excludes .pixi; it survives because real pixi puts conda-meta/pixi one level down.
derivatives=[(".claude/worktrees/agent-one/.pixi", None)]
PROBE FAILED: `.pixi` went and took config.toml with it
Confirmed benign on this host: .pixi carries no tag and .pixi/envs/default/CACHEDIR.TAG is the 43 published bytes at offset 0. But the PR body, the CHANGELOG and docs/cleanup.md all state "never .pixi" as a property. Either make it one, or soften the three sentences to "the tag pixi writes is one level below .pixi, so .pixi is not the unit".
F7. Content created between the plan and the y goes unnamed
Unlike plan.going, which has grew_past, the derivative path has no blast-radius check: the re-read confirms derivable, not unchanged. A file written into the tag after the plan, or a git init'd repository at .pixi/envs/default/src/someones-repo, is removed without having been named.
Not blocking — the module is explicit that the tag is a claim about purpose and not a proof about contents, and the pre-plan half is tested. But #525 was blocked on exactly the post-plan half of this shape for going units, and the post-plan half here is neither tested nor named in docs/cleanup.md. Worth one test row and one sentence so the asymmetry with grew_past is a recorded decision rather than a gap.
Claims verified
| Claim | Verdict |
|---|---|
declared_regenerable is the one expression of "what counts as a derivative" |
Holds — one production callsite; plan, act and listing all reach it through the single weigh_clone |
first_claim is the one expression of "what counts as a claimant" |
Holds — one callsite; Reason::subject() is a full wildcard-free match |
| Plan and act reach the same function, not two copies | Holds |
the_gate_never_reads_a_directorys_name is not vacuous |
Holds — mutant returning true by name failed that test and the walk test |
| Other empty-collection tests are not vacuously empty | Holds — three further mutants killed (symlink-following, nested-site check, first_claim) |
| The approved set can shrink and never grow | Holds — reclaim_derivatives iterates planned only |
| Symlinks never followed, at or inside the tag | Holds — probed with a symlink out of the cache and one to the site's own file; DirEntry::file_type is lstat |
| 43 bytes at offset 0, compared as bytes | Holds — near-miss table covers wrong hex, leading space, non-zero offset, truncation, empty |
dl --ls does not cost derivatives, nothing lost |
Holds — bytes stay visible in --ls --size because bytes_in walks the whole tree |
manifest_path never stored |
Holds |
No public constructor for Derivative |
Holds |
public-api green at this SHA, so the hand-derived snapshot is CI-verified |
Holds — run 33282488106 |
| No new promised-tier row naming an undefined type | Holds — api.txt untouched by this diff. For the record the pre-existing #531 gap is 37 types named in api.txt and defined only in rest.txt |
…swer twice `reclaim` returns early when `ClonePicture::of` cannot re-take git's listing, and it names every going worktree as withheld on the way out. The derivatives the plan named were dropped there without a word: somebody had read "reclaiming <dir> (2.9 MiB)", answered `y`, and got no account of that directory at all. A run whose only work was a derivative printed nothing -- `WorktreeReport::nothing_to_say` was true with every list empty. `NotDerivableNow` gains a third arm rather than reusing one, because *the tag is gone*, *the re-read answered and said no* and *the re-read was never taken* are three different facts and this module does not let facts share a value. `Tagged::Claimed` was the alternative and it would have been a lie: its own arm says a claimant's reason reaches the directory, and git refusing to speak is not a claimant. Snapshot row added by hand, as the others in this branch were; CI's `public-api` job is the verdict. Claude-Session: https://claude.ai/code/session_01DtHHFa6RxTySSdC74jYcrc
`nothing_but_a_read_mints_a_derivative` matched the return with
`ends_with("-> {TY}")`, so it caught a bare `Derivative` and nothing else. A
`-> Option<Derivative>`, `-> Result<Derivative, _>` or `-> Vec<Derivative>`
hands a caller one it never read for and none of them ends in the bare name,
so all three would have arrived through the one file that exists to show them
arriving -- and the guard would have gone on passing.
The predicate is now ranged over fixture rows before it is ranged over the
snapshot, the way the gate's own name-blindness rows are: four shapes that
mint and two that do not. The old predicate fails the first wrapper.
Reading the whole return rather than its tail means striking out `&{TY}`
first, so `Tagged::derivable(&self) -> Option<&Derivative>` still reads as the
borrow it is.
Claude-Session: https://claude.ai/code/session_01DtHHFa6RxTySSdC74jYcrc
…tree `reclaim_derivatives` filed a `TreeSweep` refusal into `report.refused`, which `worktree_report_lines` prints under "Some agent worktrees would not come away." Nothing in that list is a worktree: every derivative sits inside a site this run has just said it is leaving, and the site was never being removed. A root-owned file under `.pixi/envs/default` therefore printed a heading naming a worktree that came away fine, over a path that is not one -- and said nothing about the subtree now being part-removed. Its own list and its own heading, which is the same rule the rest of this module follows: two facts about two disjoint sets of directories do not share a value. The heading says the worktrees are untouched and that re-running the install puts the subtree back, because that is what the reader has to decide. `PruneOutcome::Acted` is boxed because the new field tipped `PruneReport` past `clippy::large_enum_variant`; that is the lint's own suggested fix and the enum is returned by value once per run. Claude-Session: https://claude.ai/code/session_01DtHHFa6RxTySSdC74jYcrc
…read `pixi_recipe` returned `CouldNotRead(InvalidData)` when `environment_name` was absent from a `conda-meta/pixi` that opened, read and parsed. The plan then said "a record that would re-derive it could not be read (invalid data)" over a file that is perfectly readable, sending anybody who went to look at the wrong thing. The verdict -- the directory stands -- was right; the sentence was not. Its own arm. `CouldNotRead` is for a read that failed and now means only that, which is this module's own rule about two facts not sharing a value. The one remaining second meaning of `CouldNotRead(InvalidData)` is left where it is: `inside_the_clone` failing for the lockfile is unreachable, since the lock is found by walking up from the tag and stopping at the site, and the site is inside the clone by construction. An arm for a state nothing reaches is the opposite of the discipline above. Claude-Session: https://claude.ai/code/session_01DtHHFa6RxTySSdC74jYcrc
Host validation: this machine can't exercise the classifierRan the pre-merge verification against real data on my host. Step 0 came back at Step 0 countsClone root
Confirmed the scope is what I expected before calling it zero: Honest read: not a worthwhile sample. Nothing here reaches the new code path, so a One correction for the next host that runs thisThe prescribed find had Baseline captured anyway
Two headline figures: 13 removing / 2.9 GiB, 24 leaving. No worktree line and no Standing reasons group into three, and I checked two against disk:
Observation, not a defectThe 28 tagged subtrees on this host all sit at the clone root, never inside a I'm flagging that as a scope datapoint, not asking for a change — widening to standing What this means for merge confidenceThis host validates no regression only in the trivial sense (zero population, so Nothing was removed, built, pushed, or edited. |
`PruneReport::finished()` decides `Ending::Done` against `Ending::Unfinished`, and its caller's own comment says why it matters: "Not 0: a directory the user was told would go is still on disk." This is a regression I introduced two commits ago. Before b669aae a refused derivative landed in `worktrees.refused`, which `finished()` checks, so the exit code was right. Moving those refusals to their own list fixed the report heading and silently took them out of the exit contract -- which is the exact hazard of adding a parallel list, and it went in without a test that would have caught it. `withheld` stays out of `finished()` on purpose: a changed mind leaves nothing half done. A refusal is the opposite, and a refused derivative leaves an environment in pieces. Found by a fresh-context review of the branch. Claude-Session: https://claude.ai/code/session_01DtHHFa6RxTySSdC74jYcrc
…t never rebuilds
`lockfile_above` walks up from the tag to the site and takes the first
`pixi.lock` it meets. Nothing then checked that the lock is the one whose
`pixi install -e <name>` writes *this* directory, and pixi writes an
environment to `<the lock's own directory>/.pixi/envs/<name>`.
Two directories were therefore deleted under a recipe that restores a
different path:
- a vendored project at `<site>/vendor/otherproj/.pixi/envs/default` with no
lock of its own borrows `<site>/pixi.lock`. That is the case devlaunch#472
says must stand -- an absent lock re-derives nothing -- defeated one
directory in. `default` is the near-universal environment name, so the two
names agreeing is the ordinary case rather than the exotic one.
- `.pixi/envs/keepme-backup`, copied by hand before an upgrade, whose record
still says `default`. A directory a person made deliberately, removed
under `pixi install -e default`, which rebuilds `default`.
The check is one comparison against values the function already holds, and it
reads no directory name as a predicate: it asks where pixi would put this
environment and whether that is where this one is. It also closes a case
nobody had named -- a `pixi.lock` inside the tagged directory itself, which
the removal would destroy along with it.
Second defect in the same function, so it rides here rather than alone: a
`conda-meta/pixi` that will not parse was reported as `RecordNamesNoEnvironment`,
whose own doc says the read succeeded and the file is fine. A record cut off
mid-write is not fine. Parsing is now its own step and an unparseable record
is `CouldNotRead(InvalidData)` -- the mirror of what 839b3cc fixed in the
other direction.
Both found by a fresh-context review of the branch; both reproduced by running
the shipped walk over real directory trees before either was touched.
Claude-Session: https://claude.ai/code/session_01DtHHFa6RxTySSdC74jYcrc
1 participant
Closes #472, implementing the decision on #468. It is the last substantive ticket of map #444 and its largest single payoff: about 94.5 GB of the measured 104.5.
What it does
dl --prunealready sweeps the agent git worktrees inside the clones it keeps (#454). Most of the bytes are in the worktrees that have to stand: 18 of the 72 on the reference host carried a whole.pixi/envs/default, and a site that is dirty, unpushed or unprovable keeps every one of them. This reclaims those environments while the worktree stands.Why that is not the wedge principle 1 exists to stop
Because of what the standing verdict is a statement about. Every reason a site stands, except a claimant's, is an answer about git's account of that site's content, and nothing under
.pixi/envs/has ever been in that account: pixi writes a.pixi/.gitignoreof*and!config.toml, which puts it outside every index, every status and every commit in every clone. The two sets of bytes are disjoint by construction, and the construction is a file the installer wrote.The shipped contradiction settles it: the dirt probe already reports nothing about
.pixi/envs, so refusing would print this site holds work that exists nowhere else, 0 bytes and decline to give back 5 GB in the same breath.The gate never reads a directory's name
The published 43 byte
CACHEDIR.TAGsignature at offset 0, compared as bytes. Measured on #468: rattler, cargo, uv and pytest write one;python -m venvwrites none and npm writes none anywhere beneathnode_modules. The same name lands on both sides, so no rule keyed on.venvcould express this..pixiand.pixi/envsappear nowhere in the predicate.The walk stops at the outermost tag (the outer declaration covers what is inside it, and descending would bill the same bytes twice under R3), at every site the forest holds, and at every symlink. It never puts the question to a site itself.
A recipe, or it stands
One reader is implemented and it reaches the whole 94.5 GB. Three reads: the tag;
conda-meta/pixiforenvironment_name, withmanifest_pathnever stored because it is a container path on every host environment; and apixi.lockfound by walking up from the tag, inside the site, whoseenvironments:map names it. An absent lock, a lock that no longer names the environment (withpixi clean -eas the pointer), and a tag no reader recognises all stand and are named with their bytes.The claimant fold
A
git worktree lockor a repository lockdlcould not take pins the subtree. Dirty, unpushed and unprovable do not, because they were never statements about these bytes. A worktree of another repository is in the second group, per #468 §6. It is one call toReason::subject(), which is wildcard free, so a newBlankarm has to answer the question where it is added. There is no second list of arms anywhere.No new flag
It rides
--prune's own plan and its own y/N. The plan names each directory and its size before the question, and the acting pass re-derives the whole clone under the lock and acts only where the re-read also says derivable. Hanging it on--force-worktreeswould have made one flag carry two consents, which #459 refused. What goes is the tagged directory alone, never.pixi, which holdsconfig.toml.What the plan looks like
Run against a scratch cache with a real registered worktree, a real tag, a real lockfile, a
node_modulesand a stdlib.venv:Answered
y, the tagged directory went and.pixi/config.toml,.pixi,node_modules,.venv,pixi.lockand the human'sNOTES.mdall stayed.Types
Derivativehas private fields, noDefaultand no constructor: the only thing entitled to say this is regenerable and its recipe is on disk is a read that answered.Option<Derivative>is refused, because itsNonewould mean both nothing tagged here and tagged but not costable.Taggedhas no arm for "no tag": an untagged directory is not one.public_api_snapshots::nothing_but_a_read_mints_a_derivativeasserts the absence, anchored on the type's own row so it cannot pass vacuously.Tests
30 new rows. 18 unit rows over the gate, the walk, the reader and the fold in
derivatives/tests.rs; 12 whole-run rows over real git, real registrations and real directories inagent_worktrees/tests.rs. Every recipe case from the issue is a row, including the two fixture rows that prove the predicate never reads a name:the_gate_never_reads_a_directorys_nameranges over the predicate itself for eight names on both sides, anda_node_modules_and_a_stdlib_venv_beside_an_environment_are_untouchedranges over a whole run.What the 94.5 GB is, and is not
It is one host's shape. A second host was measured before merge (the run): 4 repositories with a
.bare, 37 clone directories, and zero agent git worktrees, so zero bytes in scope and not onereclaimingline. Its 28 tagged subtrees all sit at the clone root instead, where this change never looks:rust/targetat 2.7 GB to 4.1 GB in each of five devlaunch clones, about 18 GiB of it, plus.pytest_cache,.ruff_cacheand one 11 MiB.venv.Nothing here changes on account of that. The clone's own subtrees are out of scope by #468 §9's decision rather than by oversight, and widening the gate to standing clone roots is a much larger blast radius than this ticket took on: a
rust/targetinside a clone somebody is working in is not obviously anybody's to delete. That is a different ticket if it is ever wanted.What it does mean is that the figures above are what one host's working habits produced, and should not be read as what a given host gets back. On a host that does its agent work outside
.claude/worktrees, this reclaims nothing at all.The classifier is therefore unvalidated against real-world inputs. No host data exercised the recipe reader, the standing reasons, the plan wording, or the
--lsclaim; the only evidence for them is the tests in this PR and review.Notes for review
cargo-public-apion the machine this was built on, so the 107 newpublic-api.rest.txtrows were written by hand against the file's own conventions. CI'spublic-apijob regenerates and diffs, so its verdict is the real one; if it disagrees, its output is what to apply.Blankgains one arm,ASiteSitsInside, for the case where a site sits inside a tagged directory.Reason::subject()answersAClaimfor it, which is #468 §6's sentence made literal. It is produced by the derivative fold alone and never by a site's own verdict.dl --lsdeliberately does not cost these:Derivatives::NotAskedon the listing path, with a test asserting the going and standing answers are identical either way.🤖 Generated with Claude Code
Summary by Sourcery
Reclaim safely regenerable tagged subtrees from standing agent worktrees during pruning while preserving all untagged and claimed content.
New Features:
dl --pruneto reclaim tagged, recipe-backed regenerable subtrees inside agent worktrees that must remain.Bug Fixes:
Enhancements:
dl --lslightweight by excluding derivative costing from the listing path.Documentation:
Tests:
Chores: