Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,14 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
bun-version: 1.4.2
- run: bun install --frozen-lockfile
- run: bun run check
- run: bun run test
node-version: 22
cache: pnpm
- uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-unknown-unknown
- run: pnpm install --frozen-lockfile
- run: pnpm check
- run: pnpm test
15 changes: 10 additions & 5 deletions .github/workflows/worker-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,18 @@ jobs:
steps:
- uses: actions/checkout@v4

- uses: oven-sh/setup-bun@v2
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
bun-version: 1.4.2
node-version: 22
cache: pnpm
- uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-unknown-unknown

- run: bun install --frozen-lockfile
- run: bun run check
- run: bun run test
- run: pnpm install --frozen-lockfile
- run: pnpm check
- run: pnpm test

- name: Deploy worker
uses: cloudflare/wrangler-action@v4
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ node_modules
dist
*.cpuprofile
test/.bundle
test/.bench-bundle
bench/.wrangler
tokenizer-wasm/target
tokenizer-wasm/pkg

__pycache__/
.DS_Store
1 change: 1 addition & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
registry=https://registry.npmjs.org
47 changes: 37 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ curl http://localhost:8787/v1/token/count \

`messages` is a non-empty array. `content` is text.

`tools` on a message is an OpenAI tool list (`{ "type": "function", "function": { ... } }`). The schemas are written into the prompt.
`tools` is an OpenAI tool list (`{ "type": "function", "function": { ... } }`). A list on the request is written into the leading system message, or into a new empty system message when the conversation does not start with one. A list already present on that system message is kept. The schemas are written into the prompt.

`tool_calls` on an assistant message is an OpenAI tool-call list. A `role: "tool"` message (`tool_call_id` plus string `content`) is merged into a user turn as a `<tool_result>` block. Tool `content` must be a string, or a list of content blocks. Objects, numbers, booleans, and null are rejected: the official encoder prints those with Python's `repr`, and a JSON body cannot reproduce that exactly (`22.0` and `22` are the same number after parsing).

Expand All @@ -40,17 +40,36 @@ Other roles: `system`, `developer`, `user`, `assistant`, and `latest_reminder`.

### Options

`thinking_mode` is `chat` (default) or `thinking`. Chat closes the turn with `</think>`. Thinking closes it with `<think>`.
`thinking` is `{ "type": "enabled" }` or `{ "type": "disabled" }`. Enabled opens the turn with `<think>`. Disabled closes it with `</think>`. `thinking_mode` (`thinking` or `chat`) is the same switch. When both are set they must agree. The default is chat.

`drop_thinking` defaults to `true`. In thinking mode, `reasoning_content` on assistant turns before the last user message is left out of the prompt. If any message has `tools`, thinking is kept.

`add_default_bos_token` defaults to `true`. It prepends the BOS token at the start of the conversation.

`reasoning_effort` is `max`, `high`, or `null` (default). In thinking mode, `max` prepends the "Absolute maximum..." instruction. `high` is accepted and adds nothing. That matches current DeepSeek V4 Pro and Flash. It does not match the older Flash-0731 template, where `high` and `max` are different prefixes.
`reasoning_effort` is `low`, `medium`, `high`, `max`, `xhigh`, or `null` (default). In thinking mode, `max` and `xhigh` prepend the "Absolute maximum..." instruction. `low`, `medium`, and `high` are accepted and add nothing. That matches current DeepSeek V4 Pro and Flash, which map `low` and `medium` onto `high` and `xhigh` onto `max`. It does not match the older Flash-0731 template, where `high` and `max` are different prefixes.

### Limits

The raw JSON body must be at most 16 KiB. The rendered prompt must be at most 4 KiB. Over either cap, the response is 413.
The free-plan defaults keep a worst-case request under the 10 ms CPU limit. The raw JSON body must be at most 48 KiB (`49152` bytes). The rendered prompt must be at most 8 KiB (`8192` bytes). Over either cap, the response is 413 `request_too_large`, and the tokenizer does not run. A body over the cap is rejected before `JSON.parse`. A prompt over the cap is rejected after the template is rendered and before tokenization.

A long run of digits is the slowest input measured in workerd. At 8 KiB that request averages about 5 ms and a single warm request stays around 6–7 ms. The next sizes measured, 10 KiB and up, do not leave enough room under 10 ms. The body cap is the size of that prompt written with `\uXXXX` escapes, plus the JSON envelope and the chat-template overhead (66 bytes around an empty user message).

Both caps are [vars](https://developers.cloudflare.com/workers/wrangler/configuration/#vars). A positive integer replaces that default. An unset or invalid value leaves its default in place. The two settings are independent.

Wasm memory stays about 58 MiB through an 8 KiB prompt and about 62 MiB at 128 KiB. A 1 MiB prompt with no whitespace grew it to about 133 MiB, past the 128 MiB isolate limit, so keep a paid prompt at or below 128 KiB.

On the Workers Paid plan the default CPU limit is 30 seconds and `limits.cpu_ms` can raise it up to 5 minutes. Do not set `limits.cpu_ms` on the free plan; that configuration is rejected. A 128 KiB digit run measured about 90 ms (max 95 ms) in workerd. A paid deploy can raise the caps and set the CPU limit above that measurement:

```toml
[vars]
MAX_PROMPT_BYTES = "131072"
MAX_BODY_BYTES = "786432"

[limits]
cpu_ms = 200
```

`786432` is 768 KiB, enough for a `\uXXXX` encoding of a 128 KiB prompt. The isolate encodes a max-size digit run four times at startup so the first request does not pay the one-time cost. That stays inside the 1 second startup limit.

### Errors

Expand All @@ -70,6 +89,14 @@ The raw JSON body must be at most 16 KiB. The rendered prompt must be at most 4

A 429 includes `Retry-After: 60`.

An unpaired UTF-16 surrogate in the rendered prompt is 400. The counter is the Rust `tokenizers` crate, and a Rust string cannot hold that value. Well-formed text, including emoji, is counted.

### Tokenizer

Counts use the Hugging Face [`tokenizers`](https://github.com/huggingface/tokenizers) crate (0.23) compiled to `wasm32-unknown-unknown`. The build turns default features off and enables `unstable_wasm`, so regular expressions use `fancy-regex` rather than the C `onig` library. `config/deepseek-v4/tokenizer.json` is embedded in the wasm module. Special tokens are not added during encode; the chat template writes them into the prompt.

The wasm module is built, not committed. `pnpm test`, `pnpm check`, `pnpm dev`, and `pnpm deploy` run that build first.

### Access

The endpoint is public unless the `API_KEYS` secret is set.
Expand All @@ -87,12 +114,12 @@ Then send `Authorization: Bearer <token>`. A missing or unknown token is 401. Wi
## Develop

```bash
bun install
bun run dev
bun run test
bun run check
pnpm install
pnpm dev
pnpm test
pnpm check
```

Installs use `registry.npmjs.org` (`bunfig.toml`).
Installs use `registry.npmjs.org` (`.npmrc`). The package manager is pnpm 12 (`packageManager` in `package.json`).

`bun run test` runs the node tests and the workerd tests (`@cloudflare/vitest-plugin`).
`pnpm test` runs the node tests and the workerd tests (`@cloudflare/vitest-plugin`). Those commands need Rust, the `wasm32-unknown-unknown` target, and `wasm-bindgen` 0.2.129. `scripts/build-wasm.sh` installs that `wasm-bindgen` release if it is missing. CI installs the Rust toolchain before the tests.
80 changes: 10 additions & 70 deletions bench/encode.ts
Original file line number Diff line number Diff line change
@@ -1,70 +1,10 @@
import { Tokenizer } from "@huggingface/tokenizers";
import { encodeMessages } from "../src/encoding/deepseek-v4";
import { MAX_PROMPT_BYTES } from "../src/limits";

const root = new URL("..", import.meta.url);
const jsonUrl = new URL("config/deepseek-v4/tokenizer.json", root);
const configUrl = new URL("config/deepseek-v4/tokenizer_config.json", root);

const readStarted = performance.now();
const tokenizerJson = await Bun.file(jsonUrl).json();
const tokenizerConfig = await Bun.file(configUrl).json();
const readMs = performance.now() - readStarted;

const constructStarted = performance.now();
const tokenizer = new Tokenizer(tokenizerJson, tokenizerConfig);
const constructMs = performance.now() - constructStarted;

function count(text: string): { tokens: number; ms: number } {
const started = performance.now();
const tokens = tokenizer.encode(text, { add_special_tokens: false }).ids.length;
return { tokens, ms: performance.now() - started };
}

const phrase = "你好,请用一句话介绍深度求索。";
const sizes = [256, 1024, 2048, 4096, 8192, 16384, 32768];

console.log(`read json: ${readMs.toFixed(1)} ms`);
console.log(`construct Tokenizer: ${constructMs.toFixed(1)} ms`);
console.log(`heap: ${(process.memoryUsage().heapUsed / 1024 / 1024).toFixed(1)} MB`);

for (const bytes of sizes) {
const text = phrase.repeat(Math.ceil(bytes / Buffer.byteLength(phrase))).slice(0, bytes);
const sample = count(text);
console.log(
`encode ~${Buffer.byteLength(text)} B -> ${sample.tokens} tokens in ${sample.ms.toFixed(2)} ms`,
);
}

const chat = encodeMessages(
[
{ role: "system", content: "你是助手" },
{ role: "user", content: "你好" },
],
{ thinkingMode: "chat" },
);
const tool = encodeMessages(
[
{
role: "system",
content: "You are helpful.",
tools: [
{
type: "function",
function: { name: "get_weather", description: "天气", parameters: { type: "object" } },
},
],
},
{ role: "user", content: "北京天气" },
],
{ thinkingMode: "chat" },
);
const underCap = "a".repeat(MAX_PROMPT_BYTES - 64);
for (const [name, prompt] of [
["chat", chat],
["tools", tool],
["under-cap", underCap],
] as const) {
const sample = count(prompt);
console.log(`${name}: ${Buffer.byteLength(prompt)} B -> ${sample.tokens} tokens in ${sample.ms.toFixed(2)} ms`);
}
/**
* Token timing is measured inside workerd, which is the runtime that serves the API.
* `pnpm bench` builds the wasm module and runs bench/workerd.mjs under Node.
*/
import { spawnSync } from "node:child_process";

const result = spawnSync("node", [new URL("./workerd.mjs", import.meta.url).pathname], {
stdio: "inherit",
});
process.exit(result.status ?? 1);
29 changes: 29 additions & 0 deletions bench/worker.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import { countTokens } from "../src/models/deepseek-v4-tokenizer";

const loops = 30;

export default {
async fetch(request: Request): Promise<Response> {
const preset = new URL(request.url).searchParams.get("preset") ?? "small";
const text = presetText(preset);
let tokens = 0;
const started = performance.now();
for (let i = 0; i < loops; i++) tokens = countTokens(text);
const encodeMs = (performance.now() - started) / loops;
return Response.json({
preset,
bytes: new TextEncoder().encode(text).byteLength,
tokens,
loops,
encodeMs,
});
},
};

function presetText(preset: string): string {
if (preset === "small") return "Hello";
const phrase = "你好,请用一句话介绍深度求索。";
const phraseBytes = new TextEncoder().encode(phrase).byteLength;
const target = preset === "mid" ? 1024 : 4096 - 64;
return phrase.repeat(Math.ceil(target / phraseBytes)).slice(0, target);
}
103 changes: 103 additions & 0 deletions bench/workerd.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
import { spawnSync } from "node:child_process";
import { readdirSync, readFileSync, rmSync, statSync } from "node:fs";
import { gzipSync } from "node:zlib";
import { unstable_dev } from "wrangler";

const TIMEOUT_MS = 60_000;

let worker;
const timer = setTimeout(() => {
console.error(`bench timed out after ${TIMEOUT_MS}ms, before the report was printed`);
const stop = worker?.stop() ?? Promise.resolve();
void Promise.race([stop, new Promise((resolve) => setTimeout(resolve, 2_000))]).finally(() => {
process.exit(1);
});
}, TIMEOUT_MS);

try {
const bundleDir = new URL("../test/.bench-bundle/", import.meta.url);
const wranglerBin = new URL("../node_modules/wrangler/bin/wrangler.js", import.meta.url);
const configPath = new URL("./wrangler.toml", import.meta.url).pathname;
const scriptPath = new URL("./worker.ts", import.meta.url).pathname;

rmSync(bundleDir, { recursive: true, force: true });
const dry = spawnSync(
process.execPath,
[wranglerBin.pathname, "deploy", "--dry-run", "--outdir", bundleDir.pathname, "--config", configPath],
{ cwd: new URL("./", import.meta.url).pathname, stdio: "inherit", timeout: TIMEOUT_MS },
);
if (dry.error) {
console.error(dry.error.message);
process.exit(1);
}
if (dry.status !== 0) process.exit(dry.status ?? 1);

const bundleFiles = readdirSync(bundleDir).filter((name) => name.endsWith(".js") || name.endsWith(".wasm"));
const bundleBytes = bundleFiles.reduce((sum, name) => sum + statSync(new URL(name, bundleDir)).size, 0);
const bundleGzipBytes = bundleFiles.reduce((sum, name) => sum + gzipSync(readFileSync(new URL(name, bundleDir))).byteLength, 0);

function workerdRss() {
const ps = spawnSync("ps", ["-eo", "pid,rss,comm,args"], { encoding: "utf8" });
const rows = ps.stdout
.split("\n")
.filter((line) => line.includes("workerd"))
.map((line) => {
const parts = line.trim().split(/\s+/);
return { pid: Number(parts[0]), rssKb: Number(parts[1]) };
});
const rssKb = rows.reduce((sum, row) => sum + (Number.isFinite(row.rssKb) ? row.rssKb : 0), 0);
return { processes: rows.length, rssMb: rssKb / 1024 };
}

worker = await unstable_dev(scriptPath, {
config: configPath,
local: true,
logLevel: "error",
experimental: { disableExperimentalWarning: true, forceLocal: true, disableDevRegistry: true, watch: false },
});

async function sample(preset) {
const started = performance.now();
const response = await worker.fetch(`http://example.com/?preset=${preset}`);
const wallMs = performance.now() - started;
const body = await response.json();
return {
status: response.status,
wallMs,
preset: body.preset,
bytes: body.bytes,
tokens: body.tokens,
loops: body.loops,
encodeMs: body.encodeMs,
};
}

const cold = await sample("small");
const warm = [];
for (const preset of ["small", "mid", "cap"]) {
const runs = [];
for (let i = 0; i < 5; i++) runs.push(await sample(preset));
const encode = runs.map((run) => run.encodeMs).sort((a, b) => a - b);
const wall = runs.map((run) => run.wallMs).sort((a, b) => a - b);
warm.push({
preset,
bytes: runs[0].bytes,
tokens: runs[0].tokens,
encodeMsMedian: encode[2],
encodeMsMin: encode[0],
wallMsMedian: wall[2],
});
}

const report = {
bundleBytes,
bundleGzipBytes,
cold,
warm,
memory: workerdRss(),
};
console.log(JSON.stringify(report, null, 2));
} finally {
clearTimeout(timer);
if (worker) await worker.stop();
}
3 changes: 3 additions & 0 deletions bench/wrangler.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
name = "token-count-bench"
main = "worker.ts"
compatibility_date = "2026-10-09"
Loading
Loading