Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ check-sandbox-provider-contract:
.PHONY: check-docs check-ci
check-docs:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/core-distribution-manifest.test.py BundledDocsTests
PYTHONDONTWRITEBYTECODE=1 python3 scripts/benchmark.test.py

check-ci:
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s scripts -p 'ci_*test.py'
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,29 @@ The OpenAI Agents API, open source and self-hosted. Run Codex, Claude Code or Mi
- **Compute that sleeps, workspaces that stay.** On microsandbox, which keeps every workspace on independent storage, idle sandboxes release compute and resume on a compatible node with their files intact. E2B pauses idle sandboxes in place. Docker and your own Linux, macOS or Windows machine also run Sessions. See [Sandbox Provider protocol](https://openagentcore.dev/docs/sandbox-provider).
- **Replaceable parts.** Sandbox Providers, workspace filesystems, Harnesses and model providers plug in through [defined protocols](AGENTS.md#protocols-at-every-boundary).

## Performance

The sandbox lifecycle on one microsandbox node, with E2B's create, pause and resume for comparison. Medians of 7 to 30 samples on 2026-10-25; each row links to its method and raw records, and [`scripts/benchmark.py`](scripts/benchmark.py) reproduces them.

| Operation | OpenAgentCore on microsandbox | E2B | Records |
| --- | --- | --- | --- |
| [Create a sandbox](https://openagentcore.dev/docs/performance#sandbox-start) | 1.22 s | 0.63 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) |
| [Sandbox created → initialization starts](https://openagentcore.dev/docs/performance#sandbox-start) | 0.98 s | — | [core](docs/performance/core.jsonl) |
| [Environment ready, preparation cache hit / miss](https://openagentcore.dev/docs/performance#environment-ready) | 2.50 s / 7.35 s | — | [sessions](docs/performance/sessions.jsonl) |
| [Environment ready on a full node](https://openagentcore.dev/docs/performance#environment-ready) | 5.90 s | — | [sessions](docs/performance/sessions.jsonl), [core](docs/performance/core.jsonl) |
| [Suspend an idle sandbox](https://openagentcore.dev/docs/performance#suspend-and-resume) | 3.33 s | 1.04 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) |
| [Resume it; after dropping the node's page cache](https://openagentcore.dev/docs/performance#suspend-and-resume) | 1.75 s / 1.99 s | 0.64 s | [core](docs/performance/core.jsonl), [cold](docs/performance/cold/core.jsonl), [e2b](docs/performance/e2b.jsonl) |
| [First text of a Turn, suspended / running Session](https://openagentcore.dev/docs/performance#suspend-and-resume) | 3.51 s / 1.26 s | — | [sessions](docs/performance/sessions.jsonl) |

E2B times are SDK calls on the client, each with at least one 0.20 s API round trip; a resume makes two calls. OpenAgentCore's sandbox times are Core's own, over a 0.2 ms local network; Environment ready and first text are timed by an application on the Core host from its event stream, which Core sends in 100 ms steps. [Setup](https://openagentcore.dev/docs/performance#setup) lists the hosts and versions, and [E2B](https://openagentcore.dev/docs/performance#e2b) how the two sides differ.

## Limits

- **Docker is a compatibility backend**, for trusted workloads and hosts without KVM. Containers share the node's kernel, idle sandboxes never suspend, and files stay on the node's disk. See [Choose a sandbox backend](https://openagentcore.dev/docs/getting-started/sandbox-backends).
- **One writer per workspace.** A workspace is attached to one sandbox at a time, and storage keeps only its current files: no file history or forks. See [Workspace filesystem protocol](https://openagentcore.dev/docs/workspace-provider).
- **Workspace storage needs Linux.** The NFS adapter uses a kernel NFS mount on the Core host and on every node. See [Set up workspace storage](https://openagentcore.dev/docs/getting-started/workspace-storage).
- **Suspend is slower than E2B's pause**: 3.33 s against 1.04 s. A Session that starts on a node already running its maximum number of sandboxes waits for an idle one to suspend, as *Environment ready on a full node* above shows. See [Suspension](https://openagentcore.dev/docs/sandbox-provider#suspension).

![Deployment overview](docs/assets/console-overview-en.webp)

## Quickstart
Expand Down
23 changes: 23 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,29 @@
- **计算会休眠,工作区会保留。** 在 microsandbox 上(每个工作区都保存在独立存储中),空闲沙箱释放计算资源,并在兼容节点上带着文件恢复。E2B 会就地暂停空闲沙箱。Docker 以及你自己的 Linux、macOS 或 Windows 机器同样可以运行 Session。详见[沙箱提供商协议](https://openagentcore.dev/zh/docs/sandbox-provider)。
- **部件可替换。** 沙箱提供商、工作区文件系统、Harness 和模型提供商都通过[既定协议](AGENTS.md#protocols-at-every-boundary)接入。

## 性能

单个 microsandbox 节点上的沙箱生命周期,并以 E2B 的创建、暂停和恢复作为对照。2026-10-25 测得,每项为 7 到 30 个样本的中位数;每一行都链接到测量方法和原始记录,[`scripts/benchmark.py`](scripts/benchmark.py) 可以复现这些数据。

| 操作 | OpenAgentCore(microsandbox) | E2B | 原始记录 |
| --- | --- | --- | --- |
| [创建沙箱](https://openagentcore.dev/zh/docs/performance#sandbox-start) | 1.22 s | 0.63 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) |
| [沙箱创建完成 → 开始初始化](https://openagentcore.dev/zh/docs/performance#sandbox-start) | 0.98 s | — | [core](docs/performance/core.jsonl) |
| [Environment 就绪,准备缓存命中 / 未命中](https://openagentcore.dev/zh/docs/performance#environment-ready) | 2.50 s / 7.35 s | — | [sessions](docs/performance/sessions.jsonl) |
| [满载节点上 Environment 就绪](https://openagentcore.dev/zh/docs/performance#environment-ready) | 5.90 s | — | [sessions](docs/performance/sessions.jsonl), [core](docs/performance/core.jsonl) |
| [挂起空闲沙箱](https://openagentcore.dev/zh/docs/performance#suspend-and-resume) | 3.33 s | 1.04 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) |
| [恢复沙箱;清空节点页缓存后恢复](https://openagentcore.dev/zh/docs/performance#suspend-and-resume) | 1.75 s / 1.99 s | 0.64 s | [core](docs/performance/core.jsonl), [cold](docs/performance/cold/core.jsonl), [e2b](docs/performance/e2b.jsonl) |
| [Turn 的首个文本,已挂起 / 运行中的 Session](https://openagentcore.dev/zh/docs/performance#suspend-and-resume) | 3.51 s / 1.26 s | — | [sessions](docs/performance/sessions.jsonl) |

E2B 时间为客户端上 SDK 调用的耗时,每次调用至少含一次 0.20 s 的 API 往返;恢复包含两次调用。OpenAgentCore 的沙箱时间由 Core 自己测量,经过 0.2 ms 的本地网络;Environment 就绪和首个文本由 Core 主机上的应用根据事件流计时,Core 以 100 ms 为间隔推送事件流。主机和版本见[测试环境](https://openagentcore.dev/zh/docs/performance#setup),两边的差异见 [E2B](https://openagentcore.dev/zh/docs/performance#e2b)。

## 适用边界

- **Docker 是兼容选项**,用于可信负载和没有 KVM 的主机。容器共享节点内核,空闲沙箱不会挂起,文件保存在节点磁盘上。详见[选择沙箱提供商](https://openagentcore.dev/zh/docs/getting-started/sandbox-backends)。
- **每个工作区只有一个写入者。** 工作区同一时间只挂载到一个沙箱,存储只保留当前文件,没有文件历史或分叉。详见[工作区文件系统协议](https://openagentcore.dev/zh/docs/workspace-provider)。
- **工作区存储需要 Linux。** NFS 适配器使用 Core 主机和每个节点上的内核 NFS 挂载。详见[设置工作区存储](https://openagentcore.dev/zh/docs/getting-started/workspace-storage)。
- **挂起比 E2B 的暂停慢**:3.33 s 对 1.04 s。节点运行的沙箱已达上限时,新 Session 要等一个空闲沙箱挂起,见上表的“满载节点上 Environment 就绪”。详见[暂停](https://openagentcore.dev/zh/docs/sandbox-provider#suspension)。

![部署概览](docs/assets/console-overview-zh.webp)

## 快速开始
Expand Down
3 changes: 2 additions & 1 deletion docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
"group": "Concepts",
"pages": [
"docs/concepts",
"docs/architecture"
"docs/architecture",
"docs/performance"
]
}
]
Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started/sandbox-backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,19 @@ The installer selects no backend. One installation uses one backend. To change i

## Compare the backends

| | microsandbox (recommended) | Docker | E2B |
| | microsandbox (recommended) | Docker (compatibility) | E2B |
| --- | --- | --- | --- |
| Where sandboxes run | A microVM on your Linux nodes | A container on your Linux nodes | E2B's cloud |
| Nodes to add | Yes | Yes | None |
| You provide | Linux amd64 hosts with systemd and `/dev/kvm` | Linux amd64 hosts with systemd and rootful Docker Engine | An E2B API key and a template; a public HTTPS URL for Core |
| Isolation | Each sandbox has its own kernel | Containers share the node's kernel | E2B's |
| Idle sandboxes suspend and resume | Yes, with checkpoints | No | No |
| Idle sandboxes suspend and resume | Yes, with checkpoints | No | Yes, paused in place by E2B |
| Workspace storage | Required, on NFS | Own disk | Own disk |
| Disk limits | Root disk and workspace quota | None | Set by the template |

The field-level differences are in the [deployment contract](../../contracts/agents-api/sandbox-deployment.md#what-each-field-means-per-sandbox-provider). Host requirements for nodes are in [Add and manage nodes](./nodes.md#host-requirements).

Docker nodes are [root-equivalent](./nodes.md#what-the-installer-sets-up) on their host: add them only on hosts dedicated to sandboxes.
Docker is a compatibility backend, for trusted workloads and for hosts without `/dev/kvm`. Its nodes are [root-equivalent](./nodes.md#what-the-installer-sets-up) on their host: add them only on hosts dedicated to sandboxes.

Sandbox nodes run on Linux amd64. When Core runs on macOS or Windows, connect a Linux node or use E2B. Applications can also run [self-hosted Sessions](./self-hosted.md) on their own machines, with no backend.

Expand Down
109 changes: 109 additions & 0 deletions docs/performance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
title: "Performance"
---

These measurements cover the sandbox lifecycle on microsandbox: starting a sandbox, preparing its environment, suspending it when idle and resuming it. E2B's create, pause and resume are measured with its SDK for comparison, and [E2B](#e2b) lists how the two sides differ. [`scripts/benchmark.py`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/scripts/benchmark.py) produced every number on this page from the [records](https://github.com/MiniMax-AI/OpenAgentCore/tree/main/docs/performance) published beside it.

## Results

| Measurement | Median | Range | n |
| --- | --- | --- | --- |
| Sandbox create | 1.22 s | 1.18–1.37 s | 30 |
| Sandbox created → initialization start | 0.98 s | 0.75–1.48 s | 20 |
| Session created → environment ready, preparation cache miss | 7.35 s | 5.90–16.20 s | 10 |
| npm install, preparation cache miss | 4.67 s | 3.09–13.54 s | 10 |
| Session created → environment ready, preparation cache hit | 2.50 s | 2.30–2.70 s | 10 |
| Session created → environment ready, free slot | 2.50 s | 2.10–2.60 s | 3 |
| Session created → environment ready, full node | 5.90 s | 5.50–15.40 s | 7 |
| Suspend | 3.33 s | 2.88–4.53 s | 10 |
| Resume → sandbox running | 1.75 s | 1.16–4.70 s | 10 |
| Turn first text, resumed Session | 3.51 s | 2.11–6.91 s | 10 |
| Turn first text, running Session | 1.26 s | 1.11–2.31 s | 10 |
| E2B create | 0.63 s | 0.51–1.79 s | 10 |
| E2B pause | 1.04 s | 0.66–2.48 s | 10 |
| E2B resume → first command | 0.64 s | 0.57–4.57 s | 10 |
| E2B API round trip | 0.20 s | 0.20–0.29 s | 10 |

A second run dropped the node's page cache after every Session had suspended, so each resume read its checkpoint from disk. Its resume and first-text rows:

| Measurement | Median | Range | n |
| --- | --- | --- | --- |
| Resume → sandbox running | 1.99 s | 1.69–5.26 s | 10 |
| Turn first text, resumed Session | 3.11 s | 2.71–8.41 s | 10 |
| Turn first text, running Session | 1.26 s | 0.71–2.81 s | 10 |

## What each measurement covers

### Sandbox start

- **Sandbox create**: Core's `create` step, from the Provider call to a booted sandbox.
- **Sandbox created → initialization start**: from the end of `create` until Core starts preparing the environment. It covers the Runtime starting, connecting to Core and Core claiming the initialization. Core logs that start only for an Environment with files or setup work, so this row covers the cold-start Sessions below.

### Environment ready

**Session created → environment ready** runs from `agent.session.created` to `agent.session.environment.ready` as an application on the Core host receives them on its stream, so it includes queueing, sandbox start and preparation. Core sends stream events in 100 ms steps.

Each cold-start sample uses a new Template with one npm package (`prettier@3.3.3`), one file and one setup command. Its first Session misses the [preparation cache](../contracts/agents-api/environments.md#preparation-reuse) and installs everything; **npm install** is that installation's `npm` step as the Runtime timed it, and it depends on the package registry. Its second Session restores the prepared result.

**Free slot** and **full node** cover Sessions without a Template, started one after another. The first ones found a free slot. Once the node ran its maximum number of sandboxes, each waited for Core to [suspend](./sandbox-provider.md#suspension) an idle sandbox.

### Suspend and resume

- **Suspend**: Core's `suspend` step. The node captures the sandbox's memory and disk to a checkpoint and stops it. A Session suspends after the idle timeout, or earlier under capacity pressure as [Suspension](./sandbox-provider.md#suspension) describes.
- **Resume → sandbox running**: from the start of Core's `resume` step until the sandbox is running again. It covers restoring the checkpoint, waking the Runtime, confirming that the Environment resumed and deleting the checkpoint. The first run leaves the node's page cache as it is.
- **Turn first text**: from sending input to the first output text delta, measured by the application like Environment ready, once for the Turn that resumes a suspended Session and once for the next Turn on the same running Session. The difference is the wait a resume adds; the rest is the model.

### E2B

The E2B rows are durations of SDK calls on the client: `Sandbox.create`, `pause`, and `connect` plus one command for a resume. Each call includes at least one API round trip from the client, listed as **E2B API round trip** on a kept-alive connection. Sandboxes on both sides have 2 CPUs and 4 GiB of memory, which `e2b.jsonl` records for E2B, and resume after the same idle wait. They differ in:

- **Clock**: the E2B rows are timed on the client. The sandbox rows for OpenAgentCore come from Core's log, and Core reaches its node over a 0.2 ms network.
- **Start of a create**: E2B's starts at the client's API call. Core's `create` step starts at the Provider call and leaves out Core's scheduling before it.
- **Workload**: before it pauses, each E2B sandbox holds 650 MiB in a process and has written 100 MiB to disk. Each OpenAgentCore sandbox has run one Codex Turn.
- **End of a resume**: E2B's ends when the first command returns, after one call to the API and one to the sandbox. OpenAgentCore's ends when Core records the sandbox running: the Runtime has reconnected, the Environment has resumed and the checkpoint is deleted.

## Setup

- **Revision**: Core at `c5d4205d`; the sandbox node and Runtime at `170a4983`, whose later changes do not touch these steps. Measured on 2026-10-25.
- **Hosts**: Core and one microsandbox node on two Intel Xeon 6982P-C cloud VMs with Ubuntu 22.04 and Linux 5.15, with a 0.2 ms ping between them. Core has 8 vCPUs and 62 GiB; the node has 4 vCPUs and 15 GiB and runs at most 3 sandboxes and retains at most 14.
- **Sandboxes**: 2 CPUs, 4 GiB of memory and an 8 GiB root disk each, with the workspace on NFSv4.2. Idle sandboxes suspend after 300 s.
- **Sessions**: Codex with MiniMax-M2.7 at `https://api.minimaxi.com/v1`, driven from the Core host with `openai` 3.13.0, 10 samples of each phase.
- **E2B**: a template with 2 vCPUs and 4 GiB, made from one that `build-template.py` built from the Runtime image of 2026-10-12, driven with `e2b` 2.51.0 from a client 0.20 s from the API, 10 sandboxes.

## Reproduce

Run on an installation with no other activity: the report reads every lifecycle step Core logs during the run.

1. Install OpenAgentCore, add a microsandbox node with workspace storage, set a default model and issue a Project API key, as in [Install Core and Web](./getting-started/install.md). The Sessions need the default 300 s idle timeout and a node that runs fewer than 10 sandboxes at once, so that the later ones find it full; `report` stops with an error when a measurement has no samples.
2. On the Core host, with Python 3.10 or later and `pip install openai==3.13.0`, set `OPENAI_BASE_URL` and `OPENAI_API_KEY` and run the Sessions. It takes about 15 minutes, 7 of them waiting for the idle timeout:

```sh
python3 scripts/benchmark.py sessions results --model MiniMax-M2.7 --harness codex
```

3. Save the Core log and build the table. The report keeps the log lines it uses in `results/core.jsonl`:

```sh
docker compose -f ~/.oac/core/compose.yaml logs --no-log-prefix core > core.log
python3 scripts/benchmark.py report results core.log
```

4. To resume from disk, repeat step 2 into `results/cold`. About 6 minutes after it prints `waiting 420 s for the idle timeout`, when every Session has suspended, drop the page cache on the node with `sync; echo 3 | sudo tee /proc/sys/vm/drop_caches`. Then repeat step 3 for `results/cold`.
5. For E2B, `pip install e2b==2.51.0`, set `E2B_API_KEY` and build a template from the Runtime image with [Build a template](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/deploy/e2b/README.md#build-a-template) and `--memory-mb 4096`, then run `report` again:

```sh
python3 scripts/benchmark.py e2b results --template <template>
python3 scripts/benchmark.py report results
```

## Records

The [records](https://github.com/MiniMax-AI/OpenAgentCore/tree/main/docs/performance) are the inputs of the tables above; `python3 scripts/benchmark.py report docs/performance` prints the first again, and `report docs/performance/cold` prints the whole second run, whose resume and first-text rows form the second table.

| File | Contents |
| --- | --- |
| `sessions.jsonl` | One line per Turn: `phase` (`cold`, `start`, `resume` or `warm`), `sample`, `cache` for cold starts, the Session ID, `created` (Core's creation time in Unix seconds) for starts, and the seconds after the request at which each event type first arrived |
| `core.jsonl` | The Core log lines of the run that the report reads: lifecycle steps with their durations, phase changes, preparation cache results and initialization steps, with allocation IDs shortened to 8 characters |
| `window.json` | The start and end of the run on Core's clock, in Unix seconds |
| `e2b.jsonl` | One line per E2B sandbox with its `cpu_count`, `memory_mb` and create, pause and resume durations, and one per API round trip |
| `cold/` | `sessions.jsonl`, `core.jsonl` and `window.json` of the run that read checkpoints from disk |
Loading
Loading