Repository navigation
NU-005 Watcher and Landing Zone: scheduled scan, hash dedupe #19
Copy link
Copy link
Closed
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agentwayfinder:taskWayfinding child ticket: implementation taskWayfinding child ticket: implementation task
Description
Part of #30.
Watch the configured directory, land originals, and dedupe by content hash.
Build
- Recursive scan on a schedule. Not
inotify— this is self-hosted, so the watched directory
may be a bind-mounted volume, an SMB share, or an NFS mount, where events are best-effort (even
OrbStack's virtiofs mount, faster than Docker Desktop's old gRPC-FUSE, is not a guarantee every
fs event propagates). Polling works everywhere. And a startup/crash reconciliation scan is needed
regardless, so events would be pure addition on top of a scan you must write anyway. - mtime-stability guard decoupled from scan interval. Stability window 30s, scan interval 24h.
PRD FR-1 ties them together ("stable for one scan interval"); at a 24h interval that means a
file must sit unmodified for 24h before it is even read, then wait up to 24h more — roughly
48h worst case from drop to review queue. - mtime finds candidates; content hash decides identity.
touchon an unchanged file must
not cost an LLM pass or a human review. Same content hash under a different filename is also
a no-op — insert a bare row keyed bycontent_hashfirst (ON CONFLICT DO NOTHING,
RETURNING); only the arrival whose insert actually returns a row goes on to call NU-003's
parse_headerwith its own filename andUPDATEthe metadata in. A losing concurrent arrival's
insert returns nothing and never parses — no discarded parse result, no window where two
different filenames could produce two different stored dates for the same hash. A failed parse
isn't a dead end: the row just sits atlandedand retries through the ordinary
claim/attempt_countloop (Spec: Tracer Bullet 01 — document to cited answer #30's Worker claim / lease and Attempts), same as any other failed
transition — no later arrival needs to (or can, thanks toON CONFLICT DO NOTHING) retrigger it. - RustFS object write happens before the
documentsinsert, not after — a crash between them
leaves an unreferenced object and no row (harmless; next scan/upload just no-ops the object write
and inserts the row), instead of a row pointing at an object that was never confirmed written. - Insert into
documents(entry='directory'), statelanded.
Definition of done
- New file appears in
documentswithin one scan. - A file mid-write is not read until stable.
- Unreadable / zero-byte / non-UTF-8 files are logged and do not block the rest of the scan.
- Re-scan of an unchanged file performs no work at all (A5).
Closes
- A5 — scan-skip half. NU-003 (deterministic core) closes the hash/identity half: that story
provescontent_hash/plan_deltacorrectly classify "unchanged"; this story proves a re-scan
actually consults that classification before doing anything else. - The directory half of A12. See NU-003 (identity half), NU-004 (storage half), NU-012
(upload half).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agentwayfinder:taskWayfinding child ticket: implementation taskWayfinding child ticket: implementation task
Activity
[-]Watcher and Landing Zone: scheduled scan, hash dedupe[/-][+]NU-004 Watcher and Landing Zone: scheduled scan, hash dedupe[/+][-]NU-004 Watcher and Landing Zone: scheduled scan, hash dedupe[/-][+]NU-005 Watcher and Landing Zone: scheduled scan, hash dedupe[/+]