Selective, cache-aware context compression and token optimization for OpenCode V2 coding agents.
Note
Implementation status (2026-10-25): the classifier, selection policy, V2 transcript adapter, watermark gate, optimizer/LRU, compaction lane, optional tool-output lane, reversible context_expand tool, literal-preservation check, HTTP worker, and OpenCode Go quality harness are implemented and tested. The tool-output lane is off by default because V2 tool-result persistence is unverified. See docs/architecture.md. The plugin is inert until a compression endpoint is configured.
Important
LLMLingua token-dropping models destroy code syntax, flatten multi-line logs, and drop negative assertions. This project implements a strict, content-classified pipeline that reserves SLM compression exclusively for stale natural-language prose while keeping code, JSON schemas, diffs, errors, and recent dialogue byte-identical.
Docs: Architecture · API reference · Roadmap · ADR 0001 · Quality harness · Research
OpenCode builds prompt context dynamically on every turn. As agentic coding sessions scale beyond 50+ tool invocations, prompt overhead degrades latency, inflates token spend, and risks displacing critical context.
While naive approaches attempt to run LLMLingua across the entire conversation history, empirical benchmarking on Apple Silicon (M3 Pro) reveals:
- Code corruption: Syntax breaks immediately (
tscthrowsTS1005syntax errors due to stripped braces and control tokens). - Log degradation: Structured multi-line compiler/test outputs collapse into unreadable single-line strings.
- Cache thrashing: Continuous mutations to early prompt segments destroy upstream provider KV-prefix caching (OpenAI/Anthropic prompt cache misses).
opencode-context-optimisations solves this with a multi-tiered, selective compression architecture.
Understanding what can and can't reduce billed API tokens on hosted frontier models:
┌────────────────────────────────────────────────────────────────────────┐
│ Layer 1: Pre-Request Client/Gateway Compression (Usable on APIs) │
│ • Deterministic CLI output pruning (RTK) │
│ • Content-addressed cross-turn deduplication (Session-Dedup) │
│ • Tabular JSON array compaction (Headroom / SmartCrusher) │
│ • Reversible caching with retrieval markers (CCR) │
│ • Selective SLM token pruning on aged prose (LLMLingua Request Lane) │
│ → REDUCES BILLED INPUT TOKENS ON OPENAI / ANTHROPIC / GEMINI APIS │
└────────────────────────────────────┬───────────────────────────────────┘
│ (HTTP / JSON)
┌────────────────────────────────────▼───────────────────────────────────┐
│ Layer 2: Soft Parameter / Latent Vectors (White-Box Models Only) │
│ • ICAE (In-Context Autoencoder), AutoCompressor, Gist Tokens │
│ → Requires access to model weights and custom attention masks. │
│ → CANNOT BE USED WITH HOSTED BLACK-BOX APIS. │
└────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────┐
│ Layer 3: Server-Side Inference Engine KV-Eviction (Decode Phase) │
│ • StreamingLLM, H2O, SnapKV, PyramidKV │
│ → Runs inside vLLM / llama.cpp / SGLang during token generation. │
│ → ZERO REDUCTION IN BILLED INPUT TOKENS FOR API USERS. │
└────────────────────────────────────────────────────────────────────────┘
- Conservative Content Classifier (
src/classify.ts): Fast, heuristic inspection of message content. Identifies code, unified diffs, JSON, structured logs, and technical path density. Refuses token compression on any non-prose content. - Aged Selection Policy (
src/policy.ts): Pure deterministic policy engine. Guarantees:- System-role messages are rejected by the policy.
- The latest N user turns (default 2; 1 is an explicit advanced override) remain byte-identical.
- Only old assistant responses and tool prose are eligible for compression.
- Generates deterministic SHA-256 cache keys; the plugin memoizes successful outputs in an LRU.
- Enforces strict token budgets and chunk minimums, with safe fallbacks for non-finite options.
- V2 Transcript Adapter (
src/v2.ts): Understands OpenCode V2Message.contentparts, including thetool-resulttagged union. It extracts textual tool output without touching IDs, tool-call pairing, reasoning parts, JSON, or errors. - Watermark Gate (
src/watermark.ts): Estimates the outgoing request (system, tool schemas, messages, tool-call inputs) and only pays compression latency when estimated usage crosses the model's usable-context threshold (default 70%). - Four-Lane Plugin (
src/plugin.ts):context: watermark-gated, transient request compression.compaction: pre-compresses the transcript before the summarizer; never setsevent.result.execute.after: optional, default-off incremental tool-output compression.context_expand: reversible retrieval tool for compressed spans.
- Reversible Compression (
src/expand.ts): Stores originals in a bounded in-memory store and appends a compact[ctx_expand:ctx_...]marker. The model can retrieve the original or search its lines. - Literal Preservation (
src/literals.ts): Rejects compressed output that drops file paths, URLs, numbers, line references, or error codes. - Optional Warm Worker (
python/llmlingua_worker.py): A small HTTP sidecar that keeps LLMLingua-2 resident and serializes inference. The plugin remains inert unlessendpointis configured. - Quality Harness (
research/eval_quality.mjs): A/B fact-retention test using the OpenCode Go API (defaultdeepseek-v4.1-flash), reported with token and cost metrics. - Gateway Interoperability (OmniRoute 3.8+): Documented configuration for external gateway engines (RTK, Session-Dedup, Headroom); those engines are not bundled in this repository.
The prioritized plan — live V2 validation, a broader quality gate, deterministic pruning, durable state, packaging, and advanced lanes — is in ROADMAP.md. It includes issue-sized tasks and explicit decision gates for default-on behavior.
opencode-context-optimisations/
├── LICENSE # GNU General Public License v3.0
├── FINDINGS.md # Research spike, gateway audit & taxonomy matrix
├── ROADMAP.md # Prioritized future work and decision gates
├── package.json # Exports, test scripts, dependencies
├── tsconfig.json # TypeScript strict configuration
├── docs/
│ ├── architecture.md # System architecture, invariants, lifecycle
│ ├── api-reference.md # Public API reference
│ └── adr/ # Architecture decision records
├── .env.example # OpenCode Go / worker env template (never the key)
├── src/
│ ├── index.ts # Pure package entry point
│ ├── constants.ts # Shared plugin identifier
│ ├── classify.ts # Content-kind heuristic classifier
│ ├── policy.ts # Selection policy & candidate selection engine
│ ├── v2.ts # V2 Message/ContentPart/ToolResult adapter
│ ├── optimizer.ts # LRU, timeout, literals, replacement orchestration
│ ├── compressor.ts # HTTP sidecar client
│ ├── watermark.ts # Context-pressure estimation & gate
│ ├── expand.ts # Reversible original-text store & markers
│ ├── literals.ts # Path/number/URL/error preservation check
│ ├── tool-result.ts # Tool payload extraction & protection
│ └── plugin.ts # Context, compaction, tool, expand wiring
├── tests/
│ ├── classify.test.ts # Classifier suite
│ ├── policy.test.ts # Policy suite
│ ├── v2.test.ts # Adapter & replacement suite
│ ├── optimizer.test.ts # Cache, timeout, literals, fail-open suite
│ ├── compressor.test.ts # Sidecar protocol suite
│ ├── expand.test.ts # Original store & marker suite
│ ├── literals.test.ts # Literal extraction/preservation suite
│ ├── tool-result.test.ts # Tool payload extraction suite
│ ├── watermark.test.ts # Context-pressure suite
│ └── plugin.test.ts # Hook registration & lane suite
├── python/
│ ├── llmlingua_worker.py # Optional warm HTTP sidecar
│ ├── requirements.txt # Sidecar-only Python dependencies
│ └── README.md # Install/run/limits for the worker
└── research/
├── README.md # Research directory overview & benchmark scripts
├── bench.md # M3 Pro CPU/MPS execution benchmarks
├── bench.json # Raw benchmark data & output samples
├── bench_llmlingua.py # Python benchmark harness
├── bench_llmlingua_js.mjs# JS/ONNX TinyBERT benchmark harness
├── eval_quality.mjs # A/B fact-retention harness (OpenCode Go)
├── opencode-v2-hooks.md # OpenCode V2 hook specifications
└── reference-repo.md # Reference audit of context-optimizer
- Node.js >= 22 with built-in TypeScript type stripping (recent 22.x or newer)
- Npm >= 10
# Clone the repository
git clone https://github.com/StrangeSid/opencode-context-optimisations.git
cd opencode-context-optimisations
# Install dependencies
npm install
# Run test suite
npm test
# Verify type safety
npm run typecheckEvaluated on Apple M3 Pro (18 GB RAM):
| Pipeline / Model | Hardware | Throughput / Latency | Compression Ratio | Quality Impact |
|---|---|---|---|---|
| LLMLingua-2 (BERT-base) | CPU | ~0.20 s / 1k tokens ~1.62 s / 8k tokens |
40–50% on prose | Destructive on code/logs; invalidates TypeScript syntax (TS1005). |
| LLMLingua-2 (BERT-base) | Apple MPS | ~0.10 s / 1k tokens ~0.70 s / 8k tokens |
40–50% on prose | Same semantic degradation on technical data. |
| TinyBERT ONNX (JS) | CPU | ~12–14 ms / 1.3k chars ~105 ms / 10k chars |
45–50% on prose | Fast local execution; requires strict fencing around identifiers. |
| RTK Bash Output Pruning | Native Rust | < 10 ms overhead | 60–80% on CLI tools | Lossless for code; strips non-essential bash context while preserving error lines. |
| Session-Dedup | Gateway | < 1 ms / request | ~9.3% across session | 100% Lossless; deduplicates repeated file and log reads across turns. |
Note
For complete taxonomy comparisons (including ICPC, ICAE, AutoCompressor, StreamingLLM, H2O, SnapKV, PyramidKV, and Headroom audits), see FINDINGS.md.
When routing OpenCode through an OmniRoute proxy (http://localhost:20128/v1), safe lossless compression (session-dedup, lite, headroom) is active by default.
To enable lossy tool-output filtering (RTK) and SLM compression:
- Provide the opt-in header in
~/.config/opencode/opencode.json: - Disable
lite.compressToolResultsin OmniRoute settings to prevent silent 2,000-character truncation of historical file reads.
The package now ships a real V2 context hook in src/plugin.ts and an optional warm worker in python/llmlingua_worker.py.
-
Start the sidecar (see
python/README.md):uv venv .venv-worker --python 3.12 uv pip install --python .venv-worker/bin/python -r python/requirements.txt .venv-worker/bin/python python/llmlingua_worker.py --port 8765
-
Point OpenCode at the worker. The plugin is inert without
endpoint:{ "plugins": [ { "package": "./src/plugin.ts", "options": { "endpoint": "http://127.0.0.1:8765/compress", "protectedTurns": 2, "minTokens": 400, "budgetTokens": 2000, "rate": 0.6, "timeoutMs": 1200, "watermarkEnabled": true, "watermarkRatio": 0.7, "compactionEnabled": true, "compactionBudgetTokens": 8000, "compactionTimeoutMs": 12000, "toolOutputEnabled": false, "expandToolEnabled": true, "validateLiterals": true, "debug": false } } ] }When installed as a package, the entry point is
"opencode-context-optimisations/plugin". Relative plugin paths resolve from the config file, so use an absolute path if this config lives outside the repository. -
The lanes:
- context: runs before each model call, but only when the estimated
request crosses
watermarkRatioof the model's usable context. It adaptsevent.messages, selects stale prose, consults the LRU, and replaces only approved text/tool-result spans. - compaction: pre-compresses the transcript before OpenCode's summarizer
runs. It does not set
event.result; it preserves chronology and leaves JSON, errors, code, and the recent window intact. - execute.after: optional, default off. It compresses only completed textual tool output whose class is prose. The research has not verified whether replacement is persisted by every V2 build, so enable it only after testing your version.
- context_expand: registered by default. Compressed spans get a compact
[ctx_expand:ctx_...]marker; the model can retrieve the original text or search its lines. Disable withexpandToolEnabled: false.
The plugin never persists request/compaction edits. Because V2
contextedits are transient, it recomputes against each outgoing request and relies on the content-hash cache to avoid repeat sidecar work. - context: runs before each model call, but only when the estimated
request crosses
Warning
This is still a lossy compressor. The request lane fails open on every
backend error, but quality is only as good as the policy and literal gates.
Run the quality harness before enabling compression for code-adjacent prose,
and leave toolOutputEnabled off unless you have verified persistence in
your OpenCode build.
research/eval_quality.mjs compresses a synthetic coding transcript with the
real worker, asks the same fact questions over the original and compressed
contexts, and reports per-fact retention, token estimates, and API cost.
cp .env.example .env # add OC_GO_KEY
.venv-worker/bin/python python/llmlingua_worker.py --port 8765 &
npm run eval:quality -- --runs 2The default judge is deepseek-v4.1-flash through
https://opencode.ai/inference/go/v1. Override with OC_GO_MODEL /
OC_GO_BASE_URL or --model. The harness disables literal validation so it
measures raw LLMLingua retention; the plugin enables that check by default.
The harness refuses to run without OC_GO_KEY; .env is gitignored.
The included fixture measured 6/6 fact checks retained on both the original and compressed transcripts, with an estimated 32.5% reduction and about $0.0013 of API usage for two runs (2026-10-25). That is a smoke test, not a broad task-success benchmark.
Contributions, bug reports, and benchmark reproductions are welcome. Please ensure all modifications pass unit tests and strict TypeScript checks.
Distributed under the GNU General Public License v3.0 (GPL-3.0-or-later). See LICENSE for the full license text.
{ "provider": { "omniroute": { "npm": "@ai-sdk/openai-compatible", "options": { "baseURL": "http://localhost:20128/v1", "headers": { "x-omniroute-compression": "allow-lossy" } } } } }