fix(core): finish a push after a key change, keep waits at hand-over

conformance/spec/folders.md

@@ -3626,7 +3626,7 @@ When the person changes or moves a folder file whose row the copy no longer hold
 
 While a folder file whose row the copy no longer holds stays unchanged at its path, a device MUST report the file as lost.
 
-**Tests:** `device/folders-identity.test.ts › queues a file whose row the copy lost again once it changes, and says so`, `device/folders-identity.test.ts › keeps a file the folder never wrote whose create was refused, inside one push`.
+**Tests:** `device/folders-identity.test.ts › queues a file whose row the copy lost again once it changes, and says so`, `device/folders-identity.test.ts › keeps a file the folder never wrote whose create was refused, inside one push`, `device/folders.test.ts › pulls again after a rebuild at the drain after the pull, writing what the new view holds and keeping a file it no longer reads`.
 
 ### `folders/lost-create-not-repeated`
 
@@ -3690,6 +3690,44 @@ When a folder’s cursor has aged out of the server log, a device MUST hydrate a
 
 **Tests:** `device/folders.test.ts › hydrates again at a push whose cursor the log has aged past`, `device/folders.test.ts › hydrates again while watching when the log ages past its cursor`.
 
+### `folders/view-changed-push-rebuilds`
+
+When any drain of a folder push or a send-only run, the drain after the push's pull included, ends `copy_expired` with the reason `read_view_changed`, a device MUST hydrate the copy again and drain once more in the same run, sending each write still waiting once and reporting the verdict of every write that either drain settled.
+
+**Reason:** A key that is replaced or narrowed changes the read view the copy was built under (`device/view-changed`), and a run that stopped there would leave every write behind it waiting for no cause the person can act on. The rebuild reads under the new view, so nothing the new key cannot read stays in the copy. A verdict the first drain settled is final in the queue, and a report without it would tell the person that a refused write was taken.
+
+**Tests:** `device/folders.test.ts › builds the copy again and delivers every waiting write in the same $run`, `device/folders.test.ts › reports a refusal from before the view changed with the writes sent after the rebuild`, `device/folders.test.ts › builds the copy again where the drain after the pull meets the changed view, and finishes the push`, `device/folders.test.ts › reports a placement refused where its own read meets the changed view, and does not send it again`.
+
+### `folders/view-changed-push-pulls-again`
+
+When the drain after a folder push's pull hydrates the copy again under `folders/view-changed-push-rebuilds`, a device MUST pull once more in the same run, and drain once more where that pull queued the placement of a file it wrote.
+
+**Reason:** The first pull wrote the directory out from the copy built under the old read view. Pulling from the rebuilt copy writes what the new view holds in the same push, rather than leaving it for the next one. A file bound to a row the new key cannot read is not removed by that pull; the next scan reports it as lost (`folders/lost-row-unchanged-held`).
+
+**Tests:** `device/folders.test.ts › pulls again after a rebuild at the drain after the pull, writing what the new view holds and keeping a file it no longer reads`.
+
+### `folders/view-changed-push-once`
+
+If a drain of a folder push or a send-only run that has already rebuilt its copy for a changed read view also ends `copy_expired` with the reason `read_view_changed`, then a device MUST end the run `copy_expired` without hydrating again, keeping the writes still waiting.
+
+**Reason:** A view that changes again at once is changing under the run, and rebuilding without end would spend the key's rate limit and never finish.
+
+**Tests:** `device/folders.test.ts › builds the copy again only once in a push, and ends it copy_expired where the view changes again`.
+
+### `folders/view-changed-rebuild-fails`
+
+If the hydration a folder push takes for a changed read view fails, then the CLI MUST end the push with that hydration's own error and exit code, keeping the writes still waiting for the next push.
+
+**Reason:** The person acts on why the rebuild failed, such as a server that cannot be reached, not on the changed view the rebuild was answering.
+
+**Tests:** `device/folders.test.ts › ends the push with the rebuild's own failure, keeping the writes for the next push`.
+
+### `folders/view-changed-watch-rebuilds`
+
+When a folder watch's drain ends `copy_expired` with the reason `read_view_changed`, a device MUST hydrate the copy again and send the writes still waiting without ending the watch.
+
+**Tests:** `device/folders.test.ts › builds the copy again while watching and delivers every waiting write, still watching`.
+
 ### `folders/failed-hydration-retried`
 
 When a folder push previously failed to hydrate a usable copy, a device MUST retry hydration at the next push.
@@ -4996,6 +5034,14 @@ When a folder watch's pass leaves a write undelivered and its drain reports `ret
 
 **Tests:** `device/folders.test.ts › sends again when the Retry-After a rate limit named is over, without waiting for a change`.
 
+### `folders/watch-rate-limit-handover`
+
+When a folder watch takes back a folder it let go to a command, a device MUST NOT start a pass of that folder before the time named by the `retry_after_seconds` of the most recent drain of that folder that left a write undelivered, whether the watch or the command drained, unless a later drain of the folder has had writes answered with no `retry_after_seconds`.
+
+**Reason:** A hand-over does not reset the server's rate limit, so a watch that forgot the time would send at once and be refused again (`folders/watch-rate-limit-pause`).
+
+**Tests:** `device/folders-handover.test.ts › keeps the server's wait across handing a folder over from $name`.
+
 ### `folders/push-exit-undelivered`
 
 When a folder push ends with its drain reporting `unavailable` or a write `undelivered`, and no refused credential stopped it, the CLI MUST exit 3.

conformance/src/device/cli-adapter.ts

@@ -1260,6 +1260,10 @@ export interface PushReport {
     hydrated?: HydrateReport | null;
     failed?: string;
   };
+  /** The hydration the drain took where the server said the copy's read
+   *  view had changed, before it sent the writes still waiting
+   *  (`folders/view-changed-push-rebuilds`). */
+  rebuilt: HydrateReport | null;
   /** `null` where a failed catch-up left no copy to pull from. */
   pull: PullReport | null;
   /** The exit the command ended with: 0; 3 where it left a write
@@ -1277,6 +1281,7 @@ export interface SendOnlyReport {
   settings: SettingsFileReport;
   scan: ScanReport;
   drain: PushReport["drain"];
+  rebuilt: HydrateReport | null;
   /** The exit the command ended with, added by the adapter. */
   exit: number;
 }

conformance/src/suites/device/folders-handover.test.ts

@@ -488,6 +488,88 @@ describe("handing a folder to a command", () => {
     }, WAIT);
   });
 
+  it.each([
+    {
+      name: "its own watch, to a command that sends nothing",
+      every: false,
+      command: "scan",
+      waits: [8],
+    },
+    {
+      name: "the watch of every folder, to a push the server names a later wait",
+      every: true,
+      command: "push",
+      waits: [4, 8],
+    },
+  ] as const)(
+    "keeps the server's wait across handing a folder over from $name",
+    async ({ every, command, waits }) => {
+      // A monotonic clock, so an adjustment of the wall clock cannot move the
+      // gaps this measures.
+      const attempts: number[] = [];
+      const harness = await oneFolder(
+        `handover-rate-limit-${command}`,
+        (answer) => {
+          attempts.push(performance.now());
+          const wait = waits[attempts.length - 1];
+          return wait === undefined
+            ? answer
+            : {
+                kind: "json",
+                status: 429,
+                headers: { "retry-after": String(wait) },
+                body: {
+                  error: { code: "rate_limited", message: "Too many requests" },
+                },
+              };
+        },
+      );
+      const watch = hold(
+        every ? harness.folder.watchEvery() : harness.folder.watch(),
+      );
+      await vi.waitFor(
+        () =>
+          expect(
+            lines(watch, every ? harness.dir : undefined).some(
+              (line) => line.watching === true,
+            ) || watch.stderr.includes("watching"),
+            watch.stdout + watch.stderr,
+          ).toBe(true),
+        WAIT,
+      );
+      put(harness, "One.md", note("One"));
+      await vi.waitFor(
+        () => expect(attempts, watch.stderr).toHaveLength(1),
+        WAIT,
+      );
+      const worked =
+        command === "scan"
+          ? await harness.folder.scan()
+          : await harness.folder.push();
+      expect(worked.ok, JSON.stringify(worked)).toBe(true);
+      // The push sent the write once more, and was told to wait longer.
+      expect(attempts).toHaveLength(waits.length);
+      await vi.waitFor(() => {
+        expect(handovers(watch, every ? harness.dir : undefined)).toEqual([
+          "let go",
+          "taken back",
+        ]);
+      }, WAIT);
+      await vi.waitFor(
+        () => expect(attempts, watch.stderr).toHaveLength(waits.length + 1),
+        { timeout: 60_000, interval: 100 },
+      );
+      const last = waits.length - 1;
+      expect(
+        attempts[last + 1]! - attempts[last]!,
+        "the watch taken back sent before the wait the server last named",
+      ).toBeGreaterThanOrEqual(waits[last]! * 1000 - 100);
+      expect(sentTitles(harness)).toEqual(
+        Array.from({ length: waits.length + 1 }, () => "One"),
+      );
+    },
+  );
+
   it("lets a folder go only once the write its pass is sending is answered", async () => {
     let release: () => void = () => undefined;
     const gate = new Promise<void>((resolve) => {

conformance/src/suites/device/folders.shared.ts

@@ -155,6 +155,9 @@ export function scriptFolderWrites(
     /** The answer a create is given in place of the door's own, which it is
      *  handed: a gated one, say. */
     creating?: (answer: Answer, request: RecordedRequest) => Answer;
+    /** The answer the read of a row is given in place of the door's own,
+     *  which it is handed: a refusal under a read view that changed, say. */
+    reading?: (answer: Answer, request: RecordedRequest) => Answer;
   } = {},
 ): Map<string, DoorRow> {
   // What the hydration served, so a write to one of those rows is answered
@@ -203,7 +206,12 @@ export function scriptFolderWrites(
       },
     ],
     // Reconciliation reads the current server row after each settled write.
-    read: [(request) => door.read(request.pathname.split("/").at(-1) ?? "")],
+    read: [
+      (request) => {
+        const answer = door.read(request.pathname.split("/").at(-1) ?? "");
+        return options.reading?.(answer, request) ?? answer;
+      },
+    ],
     // The tags a row holds move as the tag doors are told.
     tags: [
       (request) => {
@@ -267,6 +275,9 @@ export class EdgeDoor {
   /** An answer standing in for the server's to a listing of an item's
    *  edges, a refusal say; the door answers where it gives `undefined`. */
   listing?: (source: string) => Answer | undefined;
+  /** The same, for the read of one edge by id: a refusal under a read view
+   *  that changed, say. */
+  reading?: (id: string) => Answer | undefined;
   /** Edges an item's listing serves as they were, once each: a read answered
    *  before a change the copy has since taken (`device.md`, What the real
    *  server cannot be made to produce). */
@@ -373,7 +384,10 @@ export class EdgeDoor {
         this.failListings -= 1;
         return refusal(503, "unavailable", "Try again");
       }
-      const edge = this.edges.get(request.pathname.split("/").at(-1) ?? "");
+      const id = request.pathname.split("/").at(-1) ?? "";
+      const standIn = this.reading?.(id);
+      if (standIn !== undefined) return standIn;
+      const edge = this.edges.get(id);
       return edge === undefined
         ? refusal(404, "edge_not_found", "No such edge")
         : writeAnswers.edge(edge, 200);