Skip to content

[feature] sp dedup — content-hash duplicate detection ("you already scratched this") #28

Closed

Description

@rwrife

Pitch

Throwaway files breed duplicates: you paste the same log three times, sp new the same snippet across two projects, or an agent re-captures identical API output on every run. Over time the store fills with byte-identical (or near-identical) scratches that all age out separately and waste footprint. sp dedup gives the store a duplicate report based on content hashing, and optionally collapses exact dupes — a genuinely new hygiene primitive distinct from sp doctor (which reconciles index-vs-disk drift, not content equality) and sp reap (which is TTL-based).

Behavior

  • sp dedup (read-only default): hash every live scratch's content (e.g. SHA-256), group by digest, and report clusters of identical scratches — showing which is the oldest "canonical" copy and how many bytes the redundant copies cost. Colorized report + --json for scripting.
  • sp dedup --collapse (opt-in, still non-destructive): move the redundant copies to the morgue (never hard-delete), keeping the oldest/pinned member as canonical. Consistent with scratchpatch's ironclad "destructive actions are always two-step, morgue-first" rule.
  • Optional stretch: --near to also flag near-duplicates (normalized whitespace / trivial diffs), reported separately and never auto-collapsed.

Why it fits

  • Reuses internal/store for content access + the existing move-to-morgue path; no new destructive machinery.
  • Mirrors the layering of sp doctor/sp reap: pure grouping logic in store, presentation in render, thin internal/cli wiring, --json via the existing render/json pattern.
  • Pairs naturally with [feature] sp new --stdin / quick-capture from a pipe (headless scratch creation) #27 (headless capture) — agents that re-pipe identical output get automatic "already have this" hygiene.

Acceptance criteria

  • sp dedup produces a read-only report of exact-duplicate clusters among live scratches (canonical = oldest; count + wasted bytes per cluster; total reclaimable). Clean "no duplicates" headline when the store is unique.
  • --json output mode (arrays never null, color/personality-free, same contract as sp ls --json/sp doctor --json).
  • sp dedup --collapse moves redundant copies to the morgue (never hard-deletes), preserving the canonical member; respects sp pin if/when present (pinned copies are never collapsed).
  • Default is strictly read-only; --collapse requires the explicit flag and reports exactly what it moved.
  • Tombstone-flavored wording in the human report; zero personality/color leakage into --json.
  • Handles edge cases: single scratch, empty store, unreadable/missing content file (surface, don't crash), and identical content with different names/tags (still a dupe by content).
  • Tests: exact-dupe grouping, canonical selection, wasted-byte math, --collapse morgue-move (with a read-only guarantee for the default path), and --json shape. go build/go vet/go test ./.../gofmt clean.
  • README documents sp dedup (+ the morgue-first --collapse behavior).

Activity

  1. rwrife commented on Jul 11, 2026

    @rwrife
    OwnerAuthor

    Implemented in #30 — sp dedup (read-only clusters + --json) with morgue-first --collapse. Store/render/CLI tests all green.

  2. rwrife commented on Jul 12, 2026

    @rwrife
    OwnerAuthor

    Merged via #30 (squash). sp dedup is live: read-only content-hash duplicate report by default, morgue-first --collapse, stable --json contract. Closing out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature / backlog item

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions