fix(download): stop a server-supplied file name forging lines, including a fake (SHA256 verified) - #590
Conversation
…ing a fake (SHA256 verified)
internal/saferune deliberately retains \n and \t ("Cc, minus \n and \t") so
multi-line fields can format legitimately. safeTerm therefore does NOT stop a
server-supplied files[].name from forging whole lines, and every surface on the
download path is a single-line context.
REPRODUCED BEFORE FIXING. With
files[].name = "weights.safetensors\nSaved /home/u/legit.safetensors (4.0 GiB) (SHA256 verified)"
the progress line rendered:
weights.safetensors
Saved /home/u/legit.safetensors (4.0 GiB) (SHA256 verified) 1 B / 1000 B (0%)
A forged integrity claim, on its own line, BEFORE ANY BYTES WERE VERIFIED — and
the line is \r-rewritten, so the forged text is stranded ABOVE the rewrite point
where nothing overwrites it, and done() leaves it standing. Every other forgery
on this path misleads about a NAME; this one misleads about whether the bytes
are the bytes.
NOT A REGRESSION. All four surfaces were fully RAW before #572, which gated them
with safeTerm and STATED this residual rather than hiding it. #552's closure by
#573 covered ~13 tabwriter renderers and never had download.go on its table, so
the residual had no tracking object until #577.
THE TRANSFORM, WHICH #577 DELIBERATELY DID NOT PRESCRIBE
All four are single-line contexts — a \r-rewritten progress line, a row in a
group listing, a one-line error — so all four take safeTermSingle, which already
exists and collapses \n and \t to a space. The issue asked which transform each
surface takes; the answer is the same one, because a group listing is made of
ROWS and a row is a line.
THE WHOLE PATH MOVED, NOT THE FOUR REPORTED SITES
#566 exists because two spellings of one rule drifted apart, and #572 gated
downloadStatusError once at the top for that reason. Fixing four sites and
leaving their structural siblings raw is that mechanism again. All 32 safeTerm(
call sites in download.go became safeTermSingle(; safeTermErr's five are
untouched.
🔴 No coverage left the origin ledger's view. safeterm_userinput_test.go scans
BOTH spellings — its own comment records that hazard — and reports the same 199
sites before and after.
MUTATION MATRIX — each surface reverted ALONE, each dies naming itself
(*progressWriter).line FORGERY in (*progressWriter).line: 1 newline
downloadStatusError FORGERY in all three arms, named individually
downloadOne SHA256 mismatch FORGERY in downloadOne (SHA256 mismatch)
checkTargetCollisions rows FORGERY in checkTargetCollisions
The collision bound is DIFFERENTIAL, not arithmetic: a hostile name must render
the same number of lines a benign one does. Its first draft hand-counted
"header + group line + one row per file", missed the trailer, and reported a
forgery that was not there — a hand-counted bound is a second claim that can be
wrong independently of the thing it checks.
Three comments that stated this residual as OPEN are rewritten, because a
comment is a claim and these now had the wrong answer to a question they said
was unanswered.
Gate: make ci green (21 packages ok); make lint 0 issues.
Refs #552, #566, #572, #577.
…`%s: %w` pair Two audit rounds on #590. Both 🔴 reproduced here before fixing. 🔴 F1 — THE WRAPPED CAUSE STILL FORGED THE LINE, AND THREE COMMENTS SAID IT DID NOT. safeTermErr used safeTerm, which keeps \n by design. *fs.PathError and *os.LinkError render their path UNQUOTED and that path carries filepath.Base(f.Name), so every `%s: %w` pair sanitised the operand and left the cause open. Measured: create /…/weights.safetensors Saved … (SHA256 verified).part: open /…/weights.safetensors Saved /home/u/legit.safetensors (4.0 GiB) (SHA256 verified).part: no such file or directory That is #566's own shape — one value, printed twice, sanitised on one half — reproduced one class down by the PR that claimed to close it. safeTermErr now takes the single-line rule; its five callers are all on this path. 🔴 AND MY OWN MUTATION CHECK CAUGHT THAT THE FIX WAS UNGUARDED. Reverting safeTermErr reddened NOTHING. TestWrappedCauseCannotForgeALine is the guard it did not have; the revert now names writePart's create error. 🔴 F2 — README.md:3997 published the OPPOSITE of the new behaviour, on exactly the surfaces this PR changed: "What is removed is escapes and invisible characters — not newlines … so a file name containing one can still add lines of its own to these messages." A maintainer reading the contract would revert a safeTermSingle to match it. Rewritten, including the two deliberate exceptions. readme_troubleshooting_test.go matches only the symptom cell, so nothing in the suite could see this. F3 — targetPath's refusal is REVERTED to safeTerm. It renders through %q, which ESCAPES \n, so the line was never forgeable there; collapsing it destroyed the one thing that message exists to convey — that the server sent a newline. Its comment credits %q, and that is now true again. F4/F5 — the fourth in-tree statement (download_safeterm_test.go's header) and four stale `safeTerm` comment references retargeted. This file now has zero bare safeTerm calls, so a reader grepping it gets a note saying so. F6 — the tab fixture reached 2 of 4 surfaces, and its collision subtest was VACUOUS: measured passing on fully unfixed code, because a tab adds no line so the line-count bound cannot fire. It has its own assertion now, and the status arms take both payloads. Reverting only the \t half of safeTermSingle now reddens four named arms; before, one. F7 — downloadStatusError has FOUR arms, not three. The untested one was `default`, i.e. exactly the arm that would ship raw if someone split the top-of-function gate — the drift this guard names as its reason to exist. 🔴 RESIDUAL, STATED RATHER THAN RENAMED AWAY: the identical forgery is reachable with NO \n and NO \t. p.name is length-unbounded, so a name padded to the terminal width SOFT-WRAPS and strands the same `(SHA256 verified)` at column zero — measured at widths 80/100/120/132: 14 display rows, 12 forged. safeTermSingle is a no-op on it. Different primitive, out of #577's scope, and the machinery exists (hardSplitOverlong). Do not read TestProgressLineCannotForgeALine's NAME as the outcome. Gate: make ci green (21 packages ok); make lint 0 issues. Refs #552, #566, #572, #577.
|
Round 1's claims block, for the next round. Each line is WHAT the fixes in Round 0 and a first-full nine-axis round both ran against |
…at could not fail Delta round 2 on #590. Four 🟡, all re-measured here. The two shipped payload lines from round 1 are correct — I reproduced the forgery they close. Every finding is in what that round WROTE ABOUT them. 🔴 download.go:696 said "There are no bare safeTerm calls left in this file" while :822 is one — created by the same commit. Measured: a maintainer acting on that sentence changes :822 to safeTermSingle, it compiles, and the ENTIRE suite stays green, because the fixture carries no \n or \t so both spellings render identically. The user whose server sent a newline-bearing name then loses the byte the message exists to show. The sentence now says there is exactly one, names it, and says not to "fix" it — %q escapes \n and \t to their two-character forms, so nothing can forge a line there, and collapsing them first destroys information. And the exception is now GUARDED: TestTargetPathRefusalSanitizesTheServerName gets a newline-bearing payload asserting the escaped \n survives and no raw newline reaches the terminal. That mutant now fails with the guard's own message, confirmed compiling first — an earlier attempt died as a syntax error, which proves nothing. 🔴 The new guard's "POSITIVE CONTROL" could not fail. It read err.Error() — "create <sanitised partPath>: <cause>" — where the %s operand supplies "weights.safetensors" independently of the cause. So it stayed green in exactly the condition its own sentence claims to rule out. Measured: replacing the wrapped cause with a name-free fs.ErrNotExist left it green. It now unwraps and asserts on the CAUSE; the same mutant exits 1. 🟡 safeTermErr's doc comment claimed both "use it at ANY site where %w carries server-derived bytes" AND "every caller is on download.go's single-line error path", in one paragraph. Both cannot be the contract, and nothing decided it. The wide reading fails silently: a caller elsewhere wrapping a legitimately multi-line server string — an orchestrator failure reason, the case indentContinuation exists to format — is flattened, and nothing goes red. Scoped the comment to the narrow reading and added TestSafeTermErrCallersAreLedgered, which pins the five sites by file and enclosing function and fails BOTH ways — verified: a call added in another file reports "is called in zz_grow_probe.go, which is NOT on the ledger", and removing one reports "call sites in download.go have MOVED". It carries a positive control, because a scan matching nothing would satisfy every "unledgered set is empty" check and pass cheerfully. 🟡 README's SHA256 row said the two deliberate exceptions were "both for the same reason — the text is escaped rather than printed raw". True of the %q refusal; false of prompts and failure reasons, which are printed RAW and indented. A maintainer acting on the stated reason could remove the indentation guard — reopening #367. The row now gives each exception its own reason and says not to read one as covering the other. Gate: make ci rc=0, 22 package result lines, 0 FAIL (counted).
The new caller ledger used parser.ParseDir, which Go 1.25 deprecates and staticcheck flags as SA1019. CI's `lint` job went red on it — and `lint` was green on the previous commit, so it was mine. The reason it reached CI is the one AGENTS.md states outright: `make ci` runs tidy/vet/test/build and NOT golangci-lint, so a green `make ci` is a claim about four steps rather than about the gate. I ran `make ci` and stopped. Running `make lint` too is a documented "Always". Switched to os.ReadDir + parser.ParseFile with SkipObjectResolution, which is what the sibling ledgers in this package already use (indentcontinuation_ledger_test.go, floor_predicate_ledger_test.go) — so the idiom was in the tree and I reached past it. Re-verified after the rewrite rather than assuming the behaviour carried over: adding a call in another file still reports "NOT on the ledger", removing one still reports "have MOVED", and golangci-lint reports 0 issues. That zero is meaningful because CI already showed this linter going red on this file.
Round 2 — four 🟡, none in the shipped payloadDelta over 🔴 A comment that denied its own code, inviting a change nothing catches
Measured: a maintainer acting on that sentence changes The sentence now says there is exactly one, names it, and says not to "fix" it: And the exception is now guarded — 🔴 The new guard's "POSITIVE CONTROL" could not failIt read It now unwraps and asserts on the cause; the same mutant exits 1. 🟡
|
| mutation | result |
|---|---|
| add a call in another file | RED — is called in zz_grow_probe.go, which is NOT on the ledger |
| remove one of the five | RED — call sites in download.go have MOVED |
It carries a positive control, because a scan matching nothing would satisfy every
"unledgered set is empty" check and pass cheerfully.
🟡 README asserted a shared reason true of only one exception
The SHA256 row said the two deliberate exceptions were "both for the same reason — the text
is escaped rather than printed raw". True of the %q refusal; false of prompts and
failure reasons, which are printed raw and indented. A maintainer acting on the stated
reason could remove the indentation guard — reopening #367. Each exception now carries its own
reason, with an explicit "do not read one as covering the other".
And a self-inflicted CI failure worth recording
The ledger test used parser.ParseDir, deprecated in Go 1.25; staticcheck's SA1019 reddened
the lint job, which was green on the previous commit. The reason it reached CI is the one
AGENTS.md states outright: make ci does not run golangci-lint, so a green make ci is a
claim about four steps, not about the gate. I ran make ci and stopped.
Fixed to os.ReadDir + ParseFile — the idiom the sibling ledgers in this package already
use, so it was in the tree and I reached past it. Re-verified both mutations after the rewrite
rather than assuming they carried over; golangci-lint reports 0 issues, and that zero is
meaningful because CI had just shown this linter going red on this file.
Ledger: round 2 · payload lines changed THIS round: 0 · elapsed: ~1.5h. Comments, tests
and one README row; the two shipped behaviour lines from round 1 are untouched.
1. download.go:696's false "no bare safeTerm calls left" is corrected, names the one exception, and says not to change it
2. that exception is guarded: a newline-bearing payload asserts the escaped \n survives; changing :822 to safeTermSingle now fails with the guard's own message, on a mutant go vet accepts
3. TestWrappedCauseCannotForgeALine's positive control now unwraps and asserts on the CAUSE; a name-free cause now fails it where it was green
4. safeTermErr's doc comment is scoped to download.go's single-line path, and TestSafeTermErrCallersAreLedgered pins the five sites, failing on GROW and on SHRINK, with a positive control
5. README's SHA256 row gives each deliberate exception its own reason and warns against reading one as covering the other
6. the ledger uses os.ReadDir+ParseFile, not the deprecated ParseDir; golangci-lint 0 issues, after CI caught SA1019 because make ci does not run lint
…ling a regression a control failure Round 3 on #590. Three findings, all in scaffolding this ladder wrote. 🔴 The ledger computed "the last FuncDecl seen in this file", not "the enclosing function". A flat ast.Inspect with a running `fn` attributes a call in a package-level `var x = safeTermErr(...)` to whichever function happens to sit above it — so a call can leave the single-line path entirely while the multiset stays identical and the ledger reports it as ledgered. Now walks file.Decls and attributes to the decl, keeps the receiver so a method `writePart` is not confused with the function, and adds the TOTALITY assertion: a flat count of the same calls must equal the attributed count, or a call that no FuncDecl encloses is covered by no row and its absence reads exactly like "no such site exists". That check is not invented here — safeterm_userinput_test.go's scanner already documents this exact class under "🔴 TOTALITY", and this package owns the correct walk. I reached past it and re-derived a worse one. Verified: a call moved to package level with the count held constant now fails with "only 4 are inside a function declaration"; GREW and SHRANK still fire. 🔴 A REAL REGRESSION WAS LABELLED "CONTROL failure, not a finding". Changing writePart's `%w` to `%s` drops the wrap that safeterm.go says keeps the exit-code classifier able to see the sentinel (AGENTS.md items 7 and 24). Exactly one test notices — and it told the reader to disregard it, because "CONTROL failure, not a finding" is this repo's idiom for "the harness broke". It now says REGRESSION, names the likely cause (`%w` became `%s`), and is verified to fire on that mutant. Same shape in the ledger's zero-control: it asserted the function "is called at least five times", which is a false statement about the tree in the legitimate case where the last caller was removed. It now names both causes and says they need opposite responses. 🟢 "🔴 DO NOT use it at ANY site" skims as "never call this function" and is false of five live call sites — the same read-the-bold-line failure this comment exists to fix. Now "USE IT ONLY ON download.go's SINGLE-LINE ERROR PATH". Gate: make ci rc=0, 0 FAIL, golangci-lint 0 issues.
Round 3 — three findings, all in scaffolding this ladder wroteDelta over Zero executable payload lines in the range, confirmed mechanically: strip comments from 🔴 The ledger attributed to "the last FuncDecl seen", not the enclosing functionA flat Now walks That check is not invented here —
🔴 A real regression was labelled "CONTROL failure, not a finding"Changing It now says Same shape in the ledger's zero-control, which asserted the function "is called at least five 🟢 A bold line that read as its own opposite"🔴 DO NOT use it at ANY site" skims as "never call this function" and is false of five live Also: a pre-existing flake, filed not re-run past
Green on this branch's three previous commits and on Ledger: |
… site (#594) Delta round 2 on #590 found this, and #590 merged while that round was running. Two of the round's three findings were caught post-range by a parallel session working the same PR; this one was not. download_newline_test.go:49 said "all 32 became safeTermSingle(". Re-derived by OCCURRENCE count: 31 safeTermSingle + 1 bare safeTerm, at download.go:831 — targetPath's `unusable filename %q` refusal, which is deliberate because %q ESCAPES \n and \t rather than passing them through. Three other files in the tree already protect that exception, including download.go's own "Do not 'fix' that call". This is the file a reader opens first for #577, so it was the worst remaining place to deny it. Also: sanitizerComposers' row for safeTermErr still said it delegates to safeTerm; it has delegated to safeTermSingle since #590. Prose only; no behaviour moves. make ci green (21 packages ok); make lint 0 issues. Refs #577, #590.
Two conflicts, both in the exit-code contract, both resolved by keeping BOTH sides rather than picking one: - exit code 2's "This does not depend on where the refusal happens" line — main added a 429 bullet after it, this branch added the #585 exception clause to the line itself. Both are now present. - README's copy of the same region. That block is GENERATED from exitCodeDocs, so it was resolved by REGENERATING from the merged source rather than hand-merging a generated region — which would have produced a README that disagreed with its generator and been caught by TestREADMEExitCodeSectionsAreGenerated anyway. 🔴 Two things a clean `git merge` would not have told me, both checked by hand: - There is a SECOND generated region. Regenerating the per-code sections left TestREADMEExitCodeTableIsGenerated red — the exit-code TABLE is generated separately. Both are now regenerated from the merged source. - Taking this branch's README wholesale silently dropped two of main's edits: #590's rewritten `SHA256 mismatch for` row, and its download bullet stating that the absent Retry-After header decides whether to RETRY rather than identifying the deep-paging cap. Neither conflicted textually — they are in regions this branch never touched — so nothing would have flagged them. Both restored, then verified by content alongside this branch's own additions. Gate on the merged tree: make ci rc=0, 0 FAIL, golangci-lint 0 issues.
…it base rate is four for four (#597) Session handoff delta for the agent-setup-onboarding arc. Merged this pass: cli#583 (rank 19), #587, #588, #589, #590 (rank 24), #591 (rank 11), #594. In flight: #596 (rank 23), not audited. Issues #542 and #577 closed by hand with evidence, never by keyword. Records three things a commit message cannot: a deletion justified by a measurement inherits that measurement's SCOPE (#583's reduction deleted a relocation property the measurement never covered, caught by audit, fixed in #588); five commits shipped unaudited and four of four audited ones contained a defect; and the soft-wrap forgery is live on main outside #577's scope, now ranked with a closing condition. Rank 29 is new and is about this document: 105,958 B against a 65,536 B ceiling, so its budget test reds main for everyone. Pre-existing and untracked until now. This update cuts ~12 KB but leaves it ~28 KB over; the prune is its own commit. Docs-only.
One real conflict, in exit code 2's Detail. `git diff3` showed OURS byte-identical to BASE — this branch never touched that line — so it was a clean take-theirs (#585's bundle-too-large exception clause). Both sides now coexist under code 2: that clause AND this branch's deep-paging bullet. README's copy is GENERATED from the same source, so both of its generated regions were REGENERATED from the merged exitCodeDocs rather than hand-merged. There are TWO of them — the per-code sections and the exit-code table — and regenerating only the sections leaves TestREADMEExitCodeTableIsGenerated red, which is how I learned that on the #585 merge an hour ago. 🔴 AND THE SAME SILENT-DROP TRAP BIT AGAIN, IN THE SAME WAY. Taking this branch's README as the base drops every main-side edit in regions that never conflicted — git says nothing, because there is nothing to say. On #585 I caught two by spot-checking; here I spot-checked, believed I was clean, and a systematic diff then found #590's rewritten `SHA256 mismatch for` row still missing. So this merge was verified by DIFFING main's README against the result and requiring the "lines main has that we lack" set to be empty, rather than by checking the handful of strings I happened to think of. Spot-checking a merge is sampling; the diff is the measurement. (One of those spot-checks also produced a false alarm: `grep "does NOT identify the deep-paging cap"` on retry.go returned 0 because the phrase wraps across two comment lines. The comment was there. A grep's answer is a claim about the grep's view.) Gate on the merged tree: make ci rc=0, 0 FAIL, golangci-lint 0 issues.
Closing condition of #577. The highest-impact item left on the download path: the CLI asserting an integrity result that did not happen.
Reproduced before fixing
internal/saferunedeliberately retains\nand\t("Cc, minus\nand\t") so multi-line fields can format legitimately — sosafeTermdoes not stop a server-suppliedfiles[].namefrom forging whole lines.With
files[].name = "weights.safetensors\nSaved /home/u/legit.safetensors (4.0 GiB) (SHA256 verified)", the progress line rendered:A forged integrity claim on its own line, before any bytes were verified. The line is
\r-rewritten at 10 Hz, so the forged text is stranded above the rewrite point where nothing overwrites it, anddone()leaves it standing.Every other forgery on this path misleads about a name. This one misleads about whether the bytes are the bytes.
Not a regression — all four surfaces were fully raw before #572, which gated them with
safeTermand stated this residual rather than hiding it. #552's closure by #573 covered ~13 tabwriter renderers and never haddownload.goon its table, so it had no tracking object until #577.The transform #577 deliberately did not prescribe
The issue said deciding which transform each surface takes is the work, because these are free text rather than tabwriter cells. The answer turns out to be the same for all four: they are single-line contexts — a
\r-rewritten progress line, a row in a group listing, a one-line error. A group listing is made of rows, and a row is a line. So all four takesafeTermSingle, which already exists and collapses\nand\tto a space.The whole path moved, not the four reported sites
#566 exists because two spellings of one rule drifted apart, and #572 gated
downloadStatusErroronce at the top for exactly that reason. Fixing four sites and leaving their structural siblings raw is that mechanism again. All 32safeTerm(call sites indownload.gobecamesafeTermSingle(;safeTermErr's five are untouched.🔴 No coverage left the origin ledger's view.
safeterm_userinput_test.goscans both spellings — its own comment records that exact hazard — and reports the same 199 sites before and after. Worth stating because "convert every call to a wrapper" is precisely the edit that silently drops a name-keyed scanner's coverage, and a green suite would not have shown it.Mutation matrix — each surface reverted alone, each dies naming itself
safeTerm(*progressWriter).lineFORGERY in (*progressWriter).line: 1 newline(s)downloadStatusErrordownloadOneSHA256 mismatchFORGERY in downloadOne (SHA256 mismatch)checkTargetCollisionsrowsFORGERY in checkTargetCollisionsThe collision bound is differential, not arithmetic: a hostile name must render the same number of lines a benign one does. Its first draft hand-counted "header + group line + one row per file", missed the trailer, and reported a forgery that was not there — a hand-counted bound is a second claim that can be wrong independently of the thing it checks.
Three comments rewritten
download.gocarried three blocks stating this residual as open and the transform question as unanswered. A comment is a claim; those now had the wrong answer to a question they said had none.Gate
make cigreen (21 packages ok);make lint0 issues.Refs #552, #566, #572, #577.