docs(swift-sdk): describe partial reads, every-type hydration and the file budget

content/docs/(overview)/devices.mdx

@@ -13,13 +13,13 @@ A device chooses the **slice** of the instance it keeps: a list of types, a tier
 
 Reads come from the copy, with no connection needed. A file's bytes are there once the device has fetched them. The core also [reads saved views](/guides/read-a-view), including layout order, groups, counts and sums. **A view's item or group read needs the whole query held** and is refused with a reason if the copy cannot answer it completely.
 
-**While a device fills its copy, it answers direct item reads from what has arrived.** A list or search without a saved view returns the items the copy holds so far and marks the answer `partial`; once the copy is whole, the same read is marked `complete`. A read by ID answers for an item that has arrived, and is refused `hydration_incomplete` for one that hasn't. Until the copy is whole, edge reads are refused `hydration_incomplete`, because an edge's two ends can arrive on different pages. A write to an item that has arrived, or a new item, is queued against the version the copy holds and sent once the copy is whole. This includes a file added on its own or attached to an item that has arrived: the device keeps its bytes and uploads them with the rest of the queue. A write to an item that hasn't arrived, or a file attached to one, is refused `hydration_incomplete`, because there is no version to base it on, and a drain is refused `hydration_incomplete` until the copy is whole. **Edge writes are refused `hydration_incomplete` until the copy is whole**, because a page still to arrive can bring the instance's edge in place of the one a write was made against. An edit that changes an item's body is still queued, and the device reads the body's links and embeds and makes their edges once the server answers the edit. The device's status reports how many items the fill has brought so far and, where the instance counted them, how many the slice held when the fill began, so an app can show how far the fill has come. Compare those two figures, not the count of items the copy holds, which also includes pinned items and creates that are still queued. The Swift SDK answers these partial reads once it adopts the core's new binding; until then, its reads throw during a fill.
+**While a device fills its copy, it answers direct item reads from what has arrived.** A list or search without a saved view returns the items the copy holds so far and marks the answer `partial`; once the copy is whole, the same read is marked `complete`. A read by ID answers for an item that has arrived, and is refused `hydration_incomplete` for one that hasn't. Until the copy is whole, edge reads are refused `hydration_incomplete`, because an edge's two ends can arrive on different pages. A write to an item that has arrived, or a new item, is queued against the version the copy holds and sent once the copy is whole. This includes a file added on its own or attached to an item that has arrived: the device keeps its bytes and uploads them with the rest of the queue. A write to an item that hasn't arrived, or a file attached to one, is refused `hydration_incomplete`, because there is no version to base it on, and a drain is refused `hydration_incomplete` until the copy is whole. **Edge writes are refused `hydration_incomplete` until the copy is whole**, because a page still to arrive can bring the instance's edge in place of the one a write was made against. An edit that changes an item's body is still queued, and the device reads the body's links and embeds and makes their edges once the server answers the edit. The device's status reports how many items the fill has brought so far and, where the instance counted them, how many the slice held when the fill began, so an app can show how far the fill has come. Compare those two figures, not the count of items the copy holds, which also includes pinned items and creates that are still queued. The Swift SDK returns these answers as `Rows` with a `completeness`.
 
 **A fill that stops partway resumes from the page it reached.** The device saves its place with each page, so a fill that was stopped, lost its connection or ended with the app keeps the items it has, and the next fill reads only the pages left. It then applies what changed on the instance while it was stopped, so the copy is as whole as one filled in a single run. The next fill **starts again from the first page** when resuming couldn't give a whole copy: the slice is different, another instance answers at the address, the credential reads other items, the instance's types changed, the event log no longer reaches back to where the fill began or ends before it, as after a restore, or the server refuses the page the device saved. The fill's report says whether it resumed, and if it started again, why. A fill that completed is never resumed: the next one reads every page.
 
-A device keeps the files it fetched within a **file budget**, so they don't fill the disk. By default the budget is a tenth of the volume, never under 2 GiB, and small enough to leave a twentieth of the volume free, but **never below 256 MiB**. If the device can't read the volume's size, the default is 2 GiB, and its status says so. If it can't read its own cache folder, its status gives the bytes held as unknown, with the reason, and it removes nothing until it can. When the files kept pass the budget, the device removes the ones read longest ago, and fetches them again when they are asked for. It never removes the bytes of a queued upload, or bytes a caller in the same app is still waiting for. Once a path is answered, a later trim can remove its file, so open it straight away. A person can set a byte count instead, or choose to keep every file, with `marfa device blobs budget`, which reports the bytes removed, and an app built on the core's Node or Swift binding can do the same.
+A device keeps the files it fetched within a **file budget**, so they don't fill the disk. By default the budget is a tenth of the volume, never under 2 GiB, and small enough to leave a twentieth of the volume free, but **never below 256 MiB**. If the device can't read the volume's size, the default is 2 GiB, and its status says so. If it can't read its own cache folder, its status gives the bytes held as unknown, with the reason, and it removes nothing until it can. When the files kept pass the budget, the device removes the ones read longest ago, and fetches them again when they are asked for. It never removes the bytes of a queued upload, or bytes a caller in the same app is still waiting for. Once a path is answered, a later trim can remove its file, so open it straight away. A person can set a byte count instead, or choose to keep every file, with `marfa device blobs budget`, which reports the bytes removed, and an app built on the core's Node binding or the Swift SDK can do the same.
 
-**Files attached to an item the copy holds stay on the device.** A file item with an `attached-to` edge to an item in the copy is an attachment. The device never removes its bytes to fit the budget, and they don't count against it, so the default budget counts only the other files' bytes as ones a trim can free. After the copy fills, and after each catch-up, an app asks the device to fetch the attachments it doesn't hold yet, smallest first: `marfa device blobs fetch-kept`, or `fetchKept` in the core's Node or Swift binding. The fetch resumes any attachment a fetch left part way. When the server asks it to wait a minute or less, it waits and asks again; it stops at the first file the volume has no room for, when the server can't be reached, or when the server asks for a longer wait. It goes on past a file the server doesn't hold, and past any other file it can't fetch or read, so one bad file never keeps the rest off the device.
+**Files attached to an item the copy holds stay on the device.** A file item with an `attached-to` edge to an item in the copy is an attachment. The device never removes its bytes to fit the budget, and they don't count against it, so the default budget counts only the other files' bytes as ones a trim can free. After the copy fills, and after each catch-up, an app asks the device to fetch the attachments it doesn't hold yet, smallest first: `marfa device blobs fetch-kept`, or `fetchKept` in the core's Node binding or the Swift SDK. The fetch resumes any attachment a fetch left part way. When the server asks it to wait a minute or less, it waits and asks again; it stops at the first file the volume has no room for, when the server can't be reached, or when the server asks for a longer wait. It goes on past a file the server doesn't hold, and past any other file it can't fetch or read, so one bad file never keeps the rest off the device.
 
 The device's status counts the attachments it holds and their bytes, and the ones it doesn't hold, by reason: not fetched yet, no room, the server unreachable, the server's rate limit, absent on the server, failed, or not in the copy. A reason comes from any fetch of the file, a read included, and stays until the bytes arrive, so an app can say which files are on the device and why the others aren't. Once a file's last edge to an item in the copy goes, or that item goes to the bin or leaves the copy, the file is cached like any other. **From the start of a fill until one completes, the device keeps the bytes of every file that was an attachment when the fill began or has become one since**, because the pages that make a file an attachment may not have arrived again. It still removes other files to fit the budget, and its status gives the attachments as unknown. Once a fill completes, a file that is no longer an attachment is cached like any other.
 

content/docs/(reference)/swift-sdk.mdx

@@ -47,7 +47,7 @@ let copy = try await WorkingCopy.open(store: storeURL, server: Server(url: serve
 
 | Call | What it does |
 | --- | --- |
-| `WorkingCopy.open(store:server:)` | Opens the file. Without a `Server`, the copy works offline only. |
+| `WorkingCopy.open(store:server:fileBudget:)` | Opens the file. Without a `Server`, the copy works offline only. A `fileBudget` sets the [file budget](#keep-files-on-the-device) as the copy opens, and a reader refuses it with `readingHandle`. |
 | `WorkingCopy.openReader(store:)` | Opens a store another process writes, to read only. It refuses a path where no store exists. |
 | `handle` | `.writer` or `.reader`. A reader refuses writes, hydration and catch-up with `readingHandle`. |
 | `useKey(_:)` | Gives the open copy a new credential. Held `changes()` streams go on. |
@@ -81,13 +81,45 @@ Your app decides when the copy talks to the instance:
 
 | Call | What it does |
 | --- | --- |
-| `status()` | A `Status`: server origin, instance ID, slice, pinned IDs, event cursor, `hydration`, counts and `catalogVersion`. |
-| `hydrate(types:tier:edgeTypes:)` | Replaces the copy with every item of the types you name at a `SliceTier` (`.library`, `.feed` or `.all`), with their tags and outgoing edges. The queue survives. |
+| `status()` | A `Status`: server origin, instance ID, slice, pinned IDs, event cursor, `hydration`, `items` and `edges` held, the fill's `itemsListed` and `itemsExpected`, and `catalogVersion`. |
+| `hydrate(types:tier:edgeTypes:)` | Replaces the copy with every item of the types you name at a `SliceTier` (`.library`, `.feed` or `.all`), with their tags and outgoing edges. The queue survives. `["*"]` names every type except `system.*` ones. |
 | `catchUp()` | Applies what changed since the copy's cursor. |
 | `pin(_:)`, `unpin(_:)` | Hold an item whatever the slice says, or let it go unless queued writes still need it. |
 | `search(_:filters:limit:)` | Full-text search over indexed fields and tags, best match first. |
 
-`Status.hydration` is `.never`, `.inProgress`, `.complete` or `.expired`. Before the first hydration, reads work and writes queue. **While a hydration is in progress, or once the copy has expired, reads and writes throw `MarfaError.hydrationIncomplete`.**
+`Status.hydration` is `.never`, `.inProgress`, `.complete`, `.expired` or `.stale`. Before the first hydration, reads work and writes queue. `.stale` means the server's event log let the copy's cursor go; see [Refill a stale copy](#refill-a-stale-copy).
+
+### Hydrate every type
+
+`hydrate(types: ["*"], tier:)` brings in every type except the `system.*` ones. Name `system.*` types beside `"*"` to add them. Naming any other type beside `"*"` throws `invalid`, because `"*"` already holds it. The `HydrateReport` says how many pages the fill read, in `pagesTotal`, and whether it `resumed` an earlier fill or `restarted` because the earlier one couldn't continue.
+
+While a fill runs, `status()` gives `itemsListed`, the items the fill has brought so far, and `itemsExpected`, how many the slice held when the fill began, where the instance counted them. **Compare those two for progress**, not `items`, which also includes pinned items and queued creates.
+
+### Read while the copy fills
+
+While a hydration is in progress, a list or a search answers the items that have arrived and says so. `items.list(_:sort:)`, `search(_:filters:limit:)` and the saved-view reads return `Rows`, a collection of `rows` with a `completeness`:
+
+| `Completeness` | Meaning |
+| --- | --- |
+| `.complete` | The copy holds everything the read asked for. |
+| `.partial` | The fill hasn't finished, so more rows may arrive. |
+| `.stale` | The server's event log let the copy's cursor go. The rows are those the copy held then, and some have since changed or gone on the server. |
+
+```swift
+let notes = try await copy.items.list(ListFilters(type: "core.note"))
+if notes.completeness == .partial { /* show the notes, and a note that more are coming */ }
+for note in notes { print(note.title ?? "") }
+```
+
+`Rows` is a random-access collection, so a loop or a `ForEach` takes it as it is. **Some calls can't be answered until the copy is whole and throw `hydrationIncomplete`:** `items.get(_:)` for an item that hasn't arrived, edge reads, and every edge write (`edges.create`, `edges.update` and `edges.delete`). A write to an item the copy holds, or to a new item, is queued during a fill, and so is a file item or an `attach`; a write to an item that hasn't arrived throws `hydrationIncomplete`, because there is no version to base it on.
+
+A `hydrate` call announces `.refreshed(.hydrated)` when it completes, and `.refreshed(.unfinished)` when it throws or is canceled, because the rows it wrote stay. A `LiveQuery` over a partial copy therefore reads again after a fill that failed part way, and a follow failure it shows stays shown.
+
+### Refill a stale copy
+
+When a catch-up learns that the event log no longer reaches the copy's cursor, `status().hydration` becomes `.stale` and the catch-up throws `copyExpired` with the reason `catchup_too_old`. A drain of a stale copy is refused the same way. Reads answer what the copy held, with `completeness` `.stale`, and item writes are queued as during a first fill.
+
+Call `hydrate(types:tier:edgeTypes:)` with the slice the copy was built under. The copy is filled again **over the rows it holds**, without clearing them first, and the `HydrateReport` says `inPlace`. The copy stays `.stale` through the fill, and through one that was interrupted, until a fill completes. If the instance, the credential's reach or the slice differs, the copy is cleared first and `inPlace` is `false`.
 
 ## Read and write
 
@@ -103,7 +135,7 @@ guard let id = created.itemId, let note = try await copy.items.get(id) else { re
 _ = try await copy.items.update(id, Edit(properties: ["title": "Books to read"], baseVersion: note.version))
 ```
 
-Besides `items`, the copy has `edges` (`from`, `to`, `ofType`, `create`, `update`, `delete`), `tags` (`add` and `remove`, one at a time), `metadata` (`replaceTags`, which writes the whole set), `extensions` (`write` and `delete` by namespace), `blobs` (`put(file:mimeType:)` queues an upload, and `get(_:)` returns where a blob's bytes are held, fetching them first if they aren't, within the copy's [file budget](/devices#the-working-copy)), the [`queue`](#send-writes-and-read-verdicts) and the `catalog` of types the copy holds.
+Besides `items`, the copy has `edges` (`from`, `to`, `ofType`, `create`, `update`, `delete`), `tags` (`add` and `remove`, one at a time), `metadata` (`replaceTags`, which writes the whole set), `extensions` (`write` and `delete` by namespace), `blobs` (`put(file:mimeType:)` queues an upload, and `withFile(_:_:)` calls your closure with where a blob's bytes are held, fetching them first if they aren't, within the copy's [file budget](#keep-files-on-the-device)), the [`queue`](#send-writes-and-read-verdicts) and the `catalog` of types the copy holds.
 
 | `items` call | What it does |
 | --- | --- |
@@ -121,6 +153,21 @@ Besides `items`, the copy has `edges` (`from`, `to`, `ofType`, `create`, `update
 - **A whole number Marfa would serve back with other digits** is refused `validation` with the code `invalid_properties` before anything is queued, as [Field kinds](/catalog/field-kinds) explains.
 - **An `EdgeEdit`** changes an edge's properties, moves one end with `.source(id)` or `.target(id)`, or both. A move is refused `validation` when the end that stays can hold more than one edge of the type, or when the edge would point to itself. An edit that changes nothing returns `nil`. Once the server accepts a source move the copy expires, so hydrate again.
 
+## Keep files on the device
+
+The copy keeps the files it fetched within a **file budget**, and keeps the files attached to items it holds outside it. [Devices](/devices#the-working-copy) explains both. These `blobs` calls work them:
+
+| Call | What it does |
+| --- | --- |
+| `cache()` | The `FileCache`: the setting, the budget in force, the bytes held and, in `attachments`, how many attachments the copy holds and how many it doesn't. |
+| `setBudget(_:)` | Chooses `.default`, `.bytes(bytes:)` or `.keepAll`, trims to it at once and returns the bytes removed. The copy keeps the choice. |
+| `trim()` | Trims to the budget now and returns the bytes removed. |
+| `fetchKept()` | Fetches the attachments the copy doesn't hold, smallest first, and returns a `KeptFetch` with the count and bytes fetched and the `attachments` after. |
+
+`fetchKept()` fits after a hydration and after each catch-up. Cancel its task to stop it: the files already fetched stay. Each attachment the copy doesn't hold has a `NotHeldReason` in `status().fileCache.attachments`, such as `.notFetched`, `.noRoom` or `.rateLimited`. A reader refuses `setBudget(_:)`, `trim()` and `fetchKept()` with `readingHandle`, and can still call `cache()`. A copy without a server refuses `fetchKept()` with `noServer`.
+
+`withFile(_:_:)` refuses with `noRoom` a file the volume has no room for. The file stays where it is only while the closure runs, so open it inside the closure.
+
 ## Declare an app type
 
 An app brings its own types under its own namespace, such as `app.readinglist.entry`:
@@ -189,15 +236,15 @@ A `LiveQuery` reads from the working copy and reads again whenever the copy chan
 ```swift
 @MainActor @Observable
 final class Library {
-    let notes: LiveQuery<[Item]>
+    let notes: LiveQuery<Rows<Item>>
 
     init(copy: WorkingCopy) {
         notes = LiveQuery(copy, reading: .list(ListFilters(type: "core.note")))
     }
 }
 ```
 
-A `Reading` is `.list(_:sort:)` or `.search(_:filters:limit:)` for `[Item]`, `.item(_:)` for `Item?`, `.queue` for `[QueuedWrite]`, or `Reading(placeholder:_:)` for any other value, from a closure that only reads.
+A `Reading` is `.list(_:sort:)` or `.search(_:filters:limit:)` for `Rows<Item>` (its placeholder is empty and `.partial`), `.item(_:)` for `Item?`, `.queue` for `[QueuedWrite]`, or `Reading(placeholder:_:)` for any other value, from a closure that only reads.
 
 | Property | Meaning |
 | --- | --- |
@@ -228,7 +275,7 @@ A layout's `kind` says how it shows items, such as `.board` or `.chart(x:y:)`, a
 @MainActor @Observable
 final class Board {
     let groups: LiveQuery<[ViewGroup]>
-    let tasks: LiveQuery<[Item]>
+    let tasks: LiveQuery<Rows<Item>>
 
     init(copy: WorkingCopy, viewID: String) {
         groups = LiveQuery(copy, reading: .groups(inView: viewID, layout: "board"))