Skip to content

Count tokens with the Rust tokenizers crate on wasm32 - #2

Merged
guuzaa merged 8 commits into
mainfrom
cursor/wasm-tokenizers-cef6
Oct 11, 2026
Merged

guuzaa merged 8 commits into
mainfrom
cursor/wasm-tokenizers-cef6

Conversation

@guuzaa

@guuzaa guuzaa commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Replaces @huggingface/tokenizers with the Hugging Face tokenizers 0.23 crate compiled to wasm32-unknown-unknown. The built .wasm is not committed. scripts/build-wasm.sh produces it, and pnpm test, pnpm check, pnpm dev, and pnpm deploy run that script first. The package manager is pnpm 12. CI installs Node, pnpm, the Rust toolchain, and the wasm32-unknown-unknown target, then runs pnpm install --frozen-lockfile, pnpm check, and pnpm test. This does not deploy.

Free-plan caps are now 8 KiB of rendered prompt and 48 KiB of raw body, sized so a worst-case digit run stays around 5 ms under the 10 ms CPU limit. MAX_PROMPT_BYTES and MAX_BODY_BYTES override those defaults. A paid-plan example (128 KiB prompt, 768 KiB body, cpu_ms = 200) is in the README. limits.cpu_ms is not set in wrangler.toml, because the free plan rejects it.

Choices

  • Crate tokenizers 0.23.2, default-features = false, feature unstable_wasm. That selects fancy-regex and leaves out the C onig and esaxx libraries.
  • config/deepseek-v4/tokenizer.json is embedded in the wasm module with include_bytes. Encode uses encode_fast (same ids, no offset tracking) and does not add special tokens. The chat template still writes those tokens into the prompt.
  • Rayon is forced off. Workerd is single-threaded.
  • ahash asks getrandom for map seeds while the vocab is built. Workerd rejects crypto.getRandomValues on a wasm-memory view (the stock wasm_js backend), so the wasm build uses a custom getrandom backend that copies through a JavaScript Uint8Array.
  • The wasm stack is 16 MiB. Parsing the vocab needs more than the default.
  • Unpaired UTF-16 surrogates are 400 invalid_request_error. A Rust string cannot hold them. The previous JavaScript tokenizer counted U+D800 as one id. Well-formed text, including emoji, is unchanged.
  • Native parity against the previous tokenizer: 0 mismatches on the chat fixtures and 47 other prompts, and encode_fast ids matched encode ids.

Workerd before and after

Same harness (bench/workerd.mjs), wrangler 4.149, compatibility date 2026-10-09. Encode time is the median of 5 warm requests, each averaging 30 encodes inside the isolate. performance.now during module startup does not move in this workerd, so cold start is the first request's wall clock (isolate startup plus one tiny encode plus the local proxy). The JavaScript column is one run. The wasm column is three runs.

JavaScript @huggingface/tokenizers Rust tokenizers 0.23.2 wasm
Bundle, uncompressed 8471 KiB 8707 KiB
Bundle, gzip 2040 KiB 2670 KiB
Cold first request 895 ms 440–475 ms
Encode "Hello" < 0.03 ms 0–0.03 ms
Encode 1035 B, 230 tokens 0.27 ms 0.20 ms
Encode 4050 B, 900 tokens 0.93 ms 0.63–1.1 ms
Wasm memory after init — 57.7 MiB

The API worker dry-run is 8732 KiB uncompressed / 2676 KiB gzip. Both sit under the 64 MiB uncompressed script limit. Wasm memory after init is 57.7 MiB, under the 128 MiB isolate limit. The cold request is under the 1 second startup CPU budget; most of that wall time is outside the encode itself.

Counts on the existing suite match, including the README prompt at 11 tokens.

Open in Web Open in Cursor 

cursoragent and others added 8 commits October 10, 2026 14:11
Replace @huggingface/tokenizers with Hugging Face tokenizers 0.23 compiled to wasm32-unknown-unknown. The vocab is embedded, fancy-regex is used instead of onig, and counts stay the same. Unpaired surrogates are rejected with 400 because a Rust string cannot hold them.

Co-authored-by: Jowell Young <guza@duck.com>
The parity helper was only a local id comparison tool. Nothing in the wasm crate reads JSON, so the direct serde_json dependency goes with it.

Co-authored-by: Jowell Young <guza@duck.com>
Scripts, the bench, and CI run through pnpm and Node. The bench builds the wasm module first and exits if it does not finish within 60 seconds.

Co-authored-by: Jowell Young <guza@duck.com>
An 8 KiB digit run, the slowest input measured in workerd, stays around 5ms. The body cap covers a unicode-escaped prompt of that size. MAX_BODY_BYTES and MAX_PROMPT_BYTES override the defaults.

Co-authored-by: Jowell Young <guza@duck.com>
A positive integer var replaces that default. The caps no longer check each other.

Co-authored-by: Jowell Young <guza@duck.com>
Encode timing stays in the bench worker. Init time and wasm memory are no longer exported from the tokenizer.

Co-authored-by: Jowell Young <guza@duck.com>
@guuzaa
guuzaa marked this pull request as ready for review October 11, 2026 01:37
@guuzaa
guuzaa merged commit d6e3d2b into main Oct 11, 2026
1 check passed
@guuzaa
guuzaa deleted the cursor/wasm-tokenizers-cef6 branch October 11, 2026 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants