Skip to content

NU-005 Watcher and Landing Zone: scheduled scan, hash dedupe #19

Closed

Description

@BasuruK

Part of #30.

Watch the configured directory, land originals, and dedupe by content hash.

Build

  • Recursive scan on a schedule. Not inotify — this is self-hosted, so the watched directory
    may be a bind-mounted volume, an SMB share, or an NFS mount, where events are best-effort (even
    OrbStack's virtiofs mount, faster than Docker Desktop's old gRPC-FUSE, is not a guarantee every
    fs event propagates). Polling works everywhere. And a startup/crash reconciliation scan is needed
    regardless, so events would be pure addition on top of a scan you must write anyway.
  • mtime-stability guard decoupled from scan interval. Stability window 30s, scan interval 24h.
    PRD FR-1 ties them together ("stable for one scan interval"); at a 24h interval that means a
    file must sit unmodified for 24h before it is even read, then wait up to 24h more — roughly
    48h worst case from drop to review queue.
  • mtime finds candidates; content hash decides identity. touch on an unchanged file must
    not cost an LLM pass or a human review. Same content hash under a different filename is also
    a no-op — insert a bare row keyed by content_hash first (ON CONFLICT DO NOTHING,
    RETURNING); only the arrival whose insert actually returns a row goes on to call NU-003's
    parse_header with its own filename and UPDATE the metadata in. A losing concurrent arrival's
    insert returns nothing and never parses — no discarded parse result, no window where two
    different filenames could produce two different stored dates for the same hash. A failed parse
    isn't a dead end: the row just sits at landed and retries through the ordinary
    claim/attempt_count loop (Spec: Tracer Bullet 01 — document to cited answer #30's Worker claim / lease and Attempts), same as any other failed
    transition — no later arrival needs to (or can, thanks to ON CONFLICT DO NOTHING) retrigger it.
  • RustFS object write happens before the documents insert, not after — a crash between them
    leaves an unreferenced object and no row (harmless; next scan/upload just no-ops the object write
    and inserts the row), instead of a row pointing at an object that was never confirmed written.
  • Insert into documents (entry='directory'), state landed.

Definition of done

  • New file appears in documents within one scan.
  • A file mid-write is not read until stable.
  • Unreadable / zero-byte / non-UTF-8 files are logged and do not block the rest of the scan.
  • Re-scan of an unchanged file performs no work at all (A5).

Closes

  • A5 — scan-skip half. NU-003 (deterministic core) closes the hash/identity half: that story
    proves content_hash/plan_delta correctly classify "unchanged"; this story proves a re-scan
    actually consults that classification before doing anything else.
  • The directory half of A12. See NU-003 (identity half), NU-004 (storage half), NU-012
    (upload half).

Activity

  1. added
    wayfinder:taskWayfinding child ticket: implementation task
    ready-for-agentFully specified, ready for an AFK agent
    on Aug 17, 2026
  2. changed the title [-]Watcher and Landing Zone: scheduled scan, hash dedupe[/-] [+]NU-004 Watcher and Landing Zone: scheduled scan, hash dedupe[/+] on Aug 17, 2026
  3. changed the title [-]NU-004 Watcher and Landing Zone: scheduled scan, hash dedupe[/-] [+]NU-005 Watcher and Landing Zone: scheduled scan, hash dedupe[/+] on Aug 17, 2026
  4. self-assigned this
    on Sep 6, 2026
  5. linked a pull request that will close this issueFeature/basuruk/nu 005 watcher and landing zone #40on Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ready-for-agentFully specified, ready for an AFK agentwayfinder:taskWayfinding child ticket: implementation task

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions