diff --git a/src/content/docs/containers/guides/snapshots.mdx b/src/content/docs/containers/guides/snapshots.mdx index 913c5e0b972..9fec631f036 100644 --- a/src/content/docs/containers/guides/snapshots.mdx +++ b/src/content/docs/containers/guides/snapshots.mdx @@ -16,10 +16,10 @@ Snapshots are only supported by Container applications that use the [`durable_ob ## Create a container snapshot -Use `snapshotContainer()` to capture the full container filesystem. Snapshots are immutable. If you restore a snapshot and then change files, create a new snapshot to persist those changes. +Use `snapshotContainer()` to capture the writable root filesystem of the container. Snapshots are immutable. If you restore a snapshot and then change files, create a new snapshot to persist those changes. :::caution[Snapshot image compatibility] -Snapshots capture the Container's complete filesystem state, but not its memory or running processes. A snapshot is tied to the Container image version it was created from and is not portable to a different image. After updating the image, create a new snapshot from a Container running that image. +Snapshots capture the writable root filesystem of the Container. They do not capture separately mounted filesystems, files in `/run`, memory, or running processes. A snapshot is tied to the Container image version it was created from and is not portable to a different image. After updating the image, create a new snapshot from a Container running that image. ::: The returned snapshot handle is a plain data object. You can store it and restore it later, including from another Durable Object: diff --git a/src/content/docs/sandbox/files/back-up-a-directory-to-r2.mdx b/src/content/docs/sandbox/files/back-up-a-directory-to-r2.mdx index d0a2e812719..0d6960aeaf4 100644 --- a/src/content/docs/sandbox/files/back-up-a-directory-to-r2.mdx +++ b/src/content/docs/sandbox/files/back-up-a-directory-to-r2.mdx @@ -329,6 +329,45 @@ An [object lifecycle rule](/r2/buckets/object-lifecycles/) that expires objects A backup that fails does not leave an object. If the Durable Object restarts during a backup, the incomplete multipart upload stays in the bucket until an [object lifecycle rule](/r2/buckets/object-lifecycles/) of the bucket aborts it. +## Send other requests to a catch-all intercept + +If your Durable Object sends the other HTTP requests from the container to an entrypoint with [`interceptAllOutboundHttp()`](/containers/api/durable-object-container/#interceptalloutboundhttp), register the backup route first. A catch-all that a container registers first receives its backup traffic until a new container starts, and `backup()` and `restore()` fail with `BACKUP_TRANSFER`. + +Replace `startSandbox()` with a version that registers both routes on each new container. `Outbound` is a `WorkerEntrypoint` that your Worker exports: + +```ts title="src/index.ts" +export class MyContainer extends DurableObject { + // ... + + // The setup that requests in progress share + private starting?: Promise; + + private startSandbox(): Promise { + if (!this.container.running) { + this.starting ??= this.setUpSandbox().finally(() => { + this.starting = undefined; + }); + } + return this.starting ?? Promise.resolve(); + } + + private async setUpSandbox(): Promise { + this.container.start({ + image: this.container.images.workspace, + enableInternet: false, + }); + await this.container.setInactivityTimeout(INACTIVITY_TIMEOUT_MS); + // Register the backup route before the catch-all + await this.backups.intercept(); + await this.container.interceptAllOutboundHttp( + this.ctx.exports.Outbound({ props: {} }), + ); + } +} +``` + +Requests that arrive during setup wait for it, so no backup starts before both routes exist, and no `intercept()` call replaces the access of a running backup. The routes belong to the container and stay in place when the Durable Object restarts, so a restarted Durable Object that finds the container running registers nothing. + ## Related resources - [DirectoryBackup API](/sandbox/reference/directory-backups/): what a backup keeps, one operation at a time per container, cancellation, and every error. diff --git a/src/content/docs/sandbox/files/manage-files.mdx b/src/content/docs/sandbox/files/manage-files.mdx index 7338ade758f..ff945857c23 100644 --- a/src/content/docs/sandbox/files/manage-files.mdx +++ b/src/content/docs/sandbox/files/manage-files.mdx @@ -216,7 +216,70 @@ The `Files` class from `@cloudflare/sandbox` reads and writes files in a running Compressed files stay in `/workspace` until the container stops. To delete a file sooner, call `remove()` after the caller has read the response. -For other file operations, such as listing a directory or renaming a file, refer to [Files API](/sandbox/reference/files/). +For other file operations, refer to [Files API](/sandbox/reference/files/). + +## Replace a file in one step + +A process that reads a file while `writeFile()` writes it can read part of the file. To replace a file in one step, write the new content to a temporary file in the same directory, then rename the temporary file onto the original. `rename()` replaces the destination in one step, so a reader gets either the old file or the new one. The temporary file must be on the same filesystem, because `rename()` rejects with `EXDEV` across filesystems. + +This example adds a `replace()` method: + +```ts title="src/index.ts" +export class MyContainer extends DurableObject { + // ... + + async replace(path: string, content: string): Promise { + await this.startSandbox(); + const temporary = `${path}.${crypto.randomUUID()}.partial`; + + try { + await this.files.writeFile(temporary, content, { cwd: workspace }); + await this.files.rename(temporary, path, { cwd: workspace }); + } catch (error) { + await this.files.remove(temporary, { cwd: workspace, force: true }); + throw error; + } + } +} +``` + +If a caller chooses `path`, check it first, as [Accept paths from callers](#accept-paths-from-callers) describes. + +## List a directory tree + +`readDirectory()` returns the entries of one directory. To list a tree, read each subdirectory in turn. This example adds a `listTree()` method that returns every path under a directory. It does not follow symbolic links to directories, because their entries have the type `symlink`: + +```ts title="src/index.ts" +export class MyContainer extends DurableObject { + // ... + + async listTree(directory: string): Promise { + await this.startSandbox(); + const paths: string[] = []; + const pending = [directory]; + let current: string | undefined; + + while ((current = pending.pop()) !== undefined) { + const entries = await this.files.readDirectory(current, { + cwd: workspace, + }); + + for (const entry of entries) { + const path = `${current}/${entry.name}`; + paths.push(path); + + if (entry.type === "directory") { + pending.push(path); + } + } + } + + return paths; + } +} +``` + +Each call to `readDirectory()` starts one process in the container. For a large tree, one `find` command through [`exec()`](/containers/api/durable-object-container/#exec) is faster, and its `signal` option can cancel it. The output of `find` counts toward the [memory limit](/workers/platform/limits/#memory) when you read it with `output()`. ## Accept paths from callers @@ -283,6 +346,6 @@ For more information, refer to [Sandbox security](/sandbox/concepts/security/#ev ## Related resources -- [Files API](/sandbox/reference/files/): every method, including `readDirectory()`, `rename()`, and `remove()`. +- [Files API](/sandbox/reference/files/): every method, with its errors and options. - [Save and restore a sandbox with snapshots](/sandbox/files/save-and-restore-a-workspace/): keep the files after the container stops. - [Mount an R2 bucket](/sandbox/files/mount-an-r2-bucket/): write files that other systems read. diff --git a/src/content/docs/sandbox/network/clone-a-private-repository.mdx b/src/content/docs/sandbox/network/clone-a-private-repository.mdx index 5577d2f08d1..269476c4d0c 100644 --- a/src/content/docs/sandbox/network/clone-a-private-repository.mdx +++ b/src/content/docs/sandbox/network/clone-a-private-repository.mdx @@ -114,13 +114,39 @@ Clone a private repository into a sandbox while the access token stays in your W `git-upload-pack` serves clones and fetches. The entrypoint allows it for one repository and rejects every other request. Code in the sandbox cannot push through `git-receive-pack` or read other repositories, even when the token allows it. -5. Add a method to your Durable Object that intercepts `github.com` and clones the repository: +5. Add a method to your Durable Object that intercepts `github.com` and clones the repository, and a constructor that keeps the inactivity timeout after a restart: ```ts title="src/index.ts" + type CloneResult = { exitCode: number; stderr: string }; + + const INACTIVITY_TIMEOUT_MS = 10 * 60 * 1000; + export class MyContainer extends DurableObject { // ... - async clone(): Promise<{ exitCode: number; stderr: string }> { + // The clone that requests in progress share + private cloning?: Promise; + + constructor(ctx: DurableObjectState, env: Env) { + super(ctx, env); + const container = ctx.container; + + // A restarted Durable Object sets the timeout again + if (container?.running) { + void ctx.blockConcurrencyWhile(() => + container.setInactivityTimeout(INACTIVITY_TIMEOUT_MS), + ); + } + } + + clone(): Promise { + this.cloning ??= this.cloneOnce().finally(() => { + this.cloning = undefined; + }); + return this.cloning; + } + + private async cloneOnce(): Promise { const container = this.ctx.container; if (!container) { @@ -139,7 +165,14 @@ Clone a private repository into a sandbox while the access token stays in your W enableInternet: false, }); // Keep the clone for later requests - await container.setInactivityTimeout(10 * 60 * 1000); + await container.setInactivityTimeout(INACTIVITY_TIMEOUT_MS); + } + + // A running container keeps the clone from an earlier request + const cloned = await container.exec(["test", "-d", "/workspace/.git"]); + + if ((await cloned.output()).exitCode === 0) { + return { exitCode: 0, stderr: "" }; } const process = await container.exec( @@ -182,7 +215,7 @@ Clone a private repository into a sandbox while the access token stays in your W The route uses a sandbox name of its own. A container that is already running keeps the image and Internet access it started with, so it would have no Git. - When Git fails, the route responds with `500` and the error from Git. While the sandbox keeps running, a second request fails this way, because `/workspace` already holds the clone. + When Git fails, the route responds with `500` and the error from Git. Requests that arrive during a clone wait for it, and a later request returns without cloning again while the container keeps running. Authenticate callers first, so other people cannot read the repository through the sandbox. For more information, refer to [Sandbox security](/sandbox/concepts/security/#every-opening-is-also-a-way-out). @@ -200,7 +233,7 @@ Clone a private repository into a sandbox while the access token stays in your W Cloning into '/workspace'... ``` - The repository is in `/workspace`, and `git remote get-url origin` prints the URL without a token. A `git push` from the sandbox fails with `403`. The clone lasts until the container stops. For more information, refer to [Sandbox lifetime](/sandbox/concepts/lifetime/#what-stops-an-instance). + The repository is in `/workspace`, and `git remote get-url origin` prints the URL without a token. A `git push` from the sandbox fails with `403`. The clone lasts until the container stops. The constructor sets the inactivity timeout again when a restarted Durable Object finds the container running, because the timeout does not survive a restart. For more information, refer to [Sandbox lifetime](/sandbox/concepts/lifetime/#what-stops-an-instance). diff --git a/src/content/docs/sandbox/reference/directory-backups.mdx b/src/content/docs/sandbox/reference/directory-backups.mdx index 81e4e51ba20..c0d1e7ce64a 100644 --- a/src/content/docs/sandbox/reference/directory-backups.mdx +++ b/src/content/docs/sandbox/reference/directory-backups.mdx @@ -144,43 +144,13 @@ Routes backup traffic from the container to `DirectoryBackupGateway`, which refu {/* Remove the second sentence and the last list item when a hostname intercept registered after interceptAllOutboundHttp() receives its requests. */} -`backup()` and `restore()` add this route themselves. If the Durable Object also calls [`interceptAllOutboundHttp()`](/containers/api/durable-object-container/#interceptalloutboundhttp), `intercept()` must run before it. Otherwise the catch-all receives the backup traffic from the container, and `backup()` and `restore()` fail with `BACKUP_TRANSFER`. +`backup()` and `restore()` add this route themselves. If the Durable Object also calls [`interceptAllOutboundHttp()`](/containers/api/durable-object-container/#interceptalloutboundhttp), `intercept()` must run before it on each new container. Otherwise the catch-all receives the backup traffic from the container, and `backup()` and `restore()` fail with `BACKUP_TRANSFER`. -- Call `intercept()` in the setup that runs after you start the container, and again when a restarted Durable Object finds the container running. Backups and restores in that container then keep their route. +- The routes belong to the container. They stay in place when the Durable Object restarts, so a restarted Durable Object that finds the container running does not need to call `intercept()` again. - Do not call `intercept()` while a backup or restore can be running. It replaces the access of the running operation, and the operation fails. -- A container that registered the catch-all first keeps sending backup traffic to it until the container restarts. +- The order of the routes is fixed when each is first registered on a container. A container that registered the catch-all first keeps sending backup traffic to it until a new container starts. -```ts -#setup: Promise | null = null; - -// Call before each use of the container. -#ensureRunning(container: Container): Promise { - // Set up each new container, and a running container - // after this Durable Object restarts. - if (this.#setup === null || !container.running) { - this.#setup = this.#setUp(container).catch((error) => { - this.#setup = null; - throw error; - }); - } - return this.#setup; -} - -async #setUp(container: Container): Promise { - // `sandbox` is an image in the Wrangler configuration, and `Outbound` - // is a WorkerEntrypoint that the main module of the Worker exports. - if (!container.running) { - container.start({ - image: container.images.sandbox, - enableInternet: false, - }); - } - await this.#backups.intercept(); - await container.interceptAllOutboundHttp( - this.ctx.exports.Outbound({ props: {} }), - ); -} -``` +For setup code that registers both routes, refer to [Send other requests to a catch-all intercept](/sandbox/files/back-up-a-directory-to-r2/#send-other-requests-to-a-catch-all-intercept). ## Types diff --git a/src/content/docs/sandbox/reference/files.mdx b/src/content/docs/sandbox/reference/files.mdx index 1ff8d892777..e8322e31c6a 100644 --- a/src/content/docs/sandbox/reference/files.mdx +++ b/src/content/docs/sandbox/reference/files.mdx @@ -129,12 +129,7 @@ Creates a file, or truncates an existing file, then streams `content` into it. - The file opens before a `ReadableStream` is consumed. A later failure can leave the file created, truncated, or partially written. - If the `content` stream errors, the call rejects with the error from the stream instead of a `SandboxFileError`. -`rename()` replaces an existing destination in one step. Writing a file to a temporary path and then renaming it keeps other processes from reading a partially written file: - -```ts -await files.writeFile(`${path}.partial`, content, { cwd }); -await files.rename(`${path}.partial`, path, { cwd }); -``` +A process that reads the file during the write can read part of it. To replace a file in one step, write a temporary file and rename it, as [Replace a file in one step](/sandbox/files/manage-files/#replace-a-file-in-one-step) shows. ### `stat` @@ -169,24 +164,7 @@ Returns the immediate entries of a directory as [`SandboxDirectoryEntry`](#sandb - A file at `path` rejects with `ENOTDIR`. - An entry name that is not valid UTF-8 rejects the whole call with `EILSEQ`. -A recursive walk with `readDirectory()` lists a directory tree. This walk does not follow symbolic links to directories: - -```ts -import type { Files } from "@cloudflare/sandbox"; - -async function* walk(files: Files, directory: string): AsyncGenerator { - for (const entry of await files.readDirectory(directory)) { - const path = `${directory}/${entry.name}`; - yield path; - - if (entry.type === "directory") { - yield* walk(files, path); - } - } -} -``` - -Each call to `readDirectory()` starts one process in the container. For a large tree, one `container.exec(["find", directory])` call is faster. The `signal` option of `exec()` cancels it. +Each call starts one process in the container. To list a directory tree, refer to [List a directory tree](/sandbox/files/manage-files/#list-a-directory-tree). ### `mkdir` diff --git a/src/content/docs/sandbox/sdk/migrate/tunnels.mdx b/src/content/docs/sandbox/sdk/migrate/tunnels.mdx index ae38682d342..2aeb45892ca 100644 --- a/src/content/docs/sandbox/sdk/migrate/tunnels.mdx +++ b/src/content/docs/sandbox/sdk/migrate/tunnels.mdx @@ -80,10 +80,30 @@ This page replaces `tunnels.get()`, `tunnels.list()`, and `tunnels.destroy()`. R export class MySandbox extends DurableObject { // ... - private async startTunnel( + // The tunnel starts that requests in progress share, by port + private readonly tunnelStarts = new Map>(); + + private startTunnel( port: number, args: string[], env: Record = {}, + ): Promise { + let start = this.tunnelStarts.get(port); + + if (!start) { + start = this.runTunnel(port, args, env).finally(() => { + this.tunnelStarts.delete(port); + }); + this.tunnelStarts.set(port, start); + } + + return start; + } + + private async runTunnel( + port: number, + args: string[], + env: Record, ): Promise { const dir = `${TUNNEL_ROOT}/${port}`; const argv = ["cloudflared", "tunnel", "--no-autoupdate", ...args]; @@ -128,7 +148,7 @@ This page replaces `tunnels.get()`, `tunnels.list()`, and `tunnels.destroy()`. R `openTunnel()` replaces `tunnels.get(port)`, and returns the `trycloudflare.com` URL. The URL can take several seconds to resolve after the method returns. A quick tunnel ends with its container, as in 0.12. - Each call stops any `cloudflared` that already serves the port, and returns a new URL. 0.12 returned the tunnel that already served the port. To keep a URL, look it up with `listTunnels()` from the next step before you call `openTunnel()`. + Requests that arrive while a tunnel for the port is starting wait for the same start and get the same URL. Without the shared start, two requests can each start `cloudflared` in the same directory, and the recorded process ID can belong to the one that exits. A later call stops any `cloudflared` that already serves the port, and returns a new URL. 0.12 returned the tunnel that already served the port. To keep a URL, look it up with `listTunnels()` from the next step before you call `openTunnel()`. 4. Add methods that list and close quick tunnels: