Skip to content

NU-004 RustFS object storage: content-addressed originals #18

Closed

Description

@BasuruK

Part of #30.

Store original document bytes in RustFS, content-addressed, via fsspec/s3fs.

Build

  • put(data: bytes) -> str writes to {sha256[0:2]}/{sha256} and returns the key. Idempotent /
    conditional (write-only-if-absent): a second put of an already-stored key leaves the existing
    object untouched, required for A12 and for safe retries after a crashed write.
  • Read back and re-hash before acking the write. Content-addressing makes integrity
    verifiable for free, and this catches a write failure while a user can still retry. This is a
    write-time check, not durability — it gives zero protection against loss after a successful
    ack (bit rot, a RustFS bug, disk failure). See "Why read-back matters more here" below.
  • get(key) -> bytes.
  • Backend configured as a URI so MinIO/S3 are drop-in.

Why the hash and not a UUID

Content hash is already the document identity. Keying objects by UUID would mean the same
document arriving by two paths stores twice, breaking dedupe and making A5/A12 meaningless.

Why read-back matters more here than usual

RustFS is Beta — its own README marks distributed mode, lifecycle management and KMS as
"Under Testing". Directory-ingested files keep a second copy in the watched directory.
Uploaded files do not. For those, RustFS holds the only copy. Read-back-and-re-hash catches a
failed write at ack time; it does nothing for loss after a successful ack. That window is an
accepted risk for this slice, not one this check closes.

Because storage goes through fsspec, swapping to MinIO or S3 is an endpoint change if Beta bites.

Definition of done

Round-trip test against the compose RustFS: put → get → bytes identical, key matches the hash,
and a corrupted-write simulation is detected rather than acked.

Closes

A12 (storage half).

Activity

  1. added
    wayfinder:taskWayfinding child ticket: implementation task
    ready-for-agentFully specified, ready for an AFK agent
    on Aug 17, 2026
  2. changed the title [-]RustFS object storage: content-addressed originals[/-] [+]NU-003 RustFS object storage: content-addressed originals[/+] on Aug 17, 2026
  3. changed the title [-]NU-003 RustFS object storage: content-addressed originals[/-] [+]NU-004 RustFS object storage: content-addressed originals[/+] on Aug 17, 2026
  4. self-assigned this
    on Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ready-for-agentFully specified, ready for an AFK agentwayfinder:taskWayfinding child ticket: implementation task

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions