Repository navigation
[feature] sp dedup — content-hash duplicate detection ("you already scratched this") #28
Copy link
Copy link
Closed
Labels
enhancementNew feature / backlog itemNew feature / backlog item
Description
Pitch
Throwaway files breed duplicates: you paste the same log three times, sp new the same snippet across two projects, or an agent re-captures identical API output on every run. Over time the store fills with byte-identical (or near-identical) scratches that all age out separately and waste footprint. sp dedup gives the store a duplicate report based on content hashing, and optionally collapses exact dupes — a genuinely new hygiene primitive distinct from sp doctor (which reconciles index-vs-disk drift, not content equality) and sp reap (which is TTL-based).
Behavior
sp dedup(read-only default): hash every live scratch's content (e.g. SHA-256), group by digest, and report clusters of identical scratches — showing which is the oldest "canonical" copy and how many bytes the redundant copies cost. Colorized report +--jsonfor scripting.sp dedup --collapse(opt-in, still non-destructive): move the redundant copies to the morgue (never hard-delete), keeping the oldest/pinned member as canonical. Consistent with scratchpatch's ironclad "destructive actions are always two-step, morgue-first" rule.- Optional stretch:
--nearto also flag near-duplicates (normalized whitespace / trivial diffs), reported separately and never auto-collapsed.
Why it fits
- Reuses
internal/storefor content access + the existing move-to-morgue path; no new destructive machinery. - Mirrors the layering of
sp doctor/sp reap: pure grouping logic instore, presentation inrender, thininternal/cliwiring,--jsonvia the existing render/json pattern. - Pairs naturally with [feature] sp new --stdin / quick-capture from a pipe (headless scratch creation) #27 (headless capture) — agents that re-pipe identical output get automatic "already have this" hygiene.
Acceptance criteria
-
sp dedupproduces a read-only report of exact-duplicate clusters among live scratches (canonical = oldest; count + wasted bytes per cluster; total reclaimable). Clean "no duplicates" headline when the store is unique. -
--jsonoutput mode (arrays nevernull, color/personality-free, same contract assp ls --json/sp doctor --json). -
sp dedup --collapsemoves redundant copies to the morgue (never hard-deletes), preserving the canonical member; respectssp pinif/when present (pinned copies are never collapsed). - Default is strictly read-only;
--collapserequires the explicit flag and reports exactly what it moved. - Tombstone-flavored wording in the human report; zero personality/color leakage into
--json. - Handles edge cases: single scratch, empty store, unreadable/missing content file (surface, don't crash), and identical content with different names/tags (still a dupe by content).
- Tests: exact-dupe grouping, canonical selection, wasted-byte math,
--collapsemorgue-move (with a read-only guarantee for the default path), and--jsonshape.go build/go vet/go test ./.../gofmtclean. - README documents
sp dedup(+ the morgue-first--collapsebehavior).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature / backlog itemNew feature / backlog item
Activity
rwrife commentedon Jul 11, 2026
Implemented in #30 —
sp dedup(read-only clusters +--json) with morgue-first--collapse. Store/render/CLI tests all green.Add sp dedup: content-hash duplicate detection (#28) (#30)
rwrife commentedon Jul 12, 2026
Merged via #30 (squash).
sp dedupis live: read-only content-hash duplicate report by default, morgue-first--collapse, stable--jsoncontract. Closing out.