PiBench

PiBench is a small, reproducible benchmark for comparing local and cloud LLMs on coding and agent-oriented tasks through the Pi coding agent.

PiBench is hardware-agnostic. It can test models running on CPUs, one or more GPUs in a workstation or server, another machine on the network, or a cloud provider. The included results were measured on the dated reference systems identified in each report and the CSV.

November 4–5: repeated local tuning and MiniMax

97 complete runs / 1802 replay-verified task aggregates have been added to RESULTS.csv, keeping protocols and inference configurations separate.

Original /65 configuration	Runs	Mean /65	Normalized /100	Mean task wall	Actual decode tok/s
MiniMax M3.1 Flash oQ3.5e, low	3	59.0208	90.80	504.4 s	53.61
MiniMax, high	3	51.5625	79.33	1800.8 s	52.29
Beholder, low: fresh same-runtime control	1	57.8185	88.95	230.0 s	154.46

Both used oMLX 0.7.3 on M5 Ultra / 256 GiB. Decode throughput includes reasoning and excludes prefill/startup; tokenizers differ. Small, variable repeat counts do not establish an overall accuracy ranking. MiniMax's separate development validation averaged 7/8 at either effort; its native-agent engineering run scored 88/100, including full artifact credit on a timed-out verification workflow. Extra Beholder repeats were cancelled, not represented as completed. See MiniMax results and limitations.

An earlier 8.5-hour, 16-configuration Beholder sweep found no reliable score improvement after continued replication: 85.94 versus 86.00 normalized points across 13 matched runs each. Temperature 0.6/top-k 20 had 13.7% lower observed wall time, motivating a speed-only adoption, not an accuracy claim. See the repeated tuning results.

Subsequent targeted probes qualified 128K MiniMax context, low effort/MTP on, with MiniMax-only prefix caching disabled: 12/12 archive facts over two near-full context turns, about 229 GiB peak footprint and at least 18 GiB free RAM. 256K was guard-rejected, not qualified; no OS memory limit was raised. This is basic capacity/recall evidence, not a general long-context quality score. See the deployed coordinate. Historical rankings below are not silently merged with these new measurements.

November 3: original /65 rerun

The original 24-task, no-tools suite was rerun unchanged with pinned Pi 0.84.3, its canonical prompt attestation, and Linux bubblewrap grading:

Configuration	Weighted /65	Normalized /100	Fully passed tasks	Task wall time
GPT 6.1 Sol, low	58.500	90.00	19/24	340.6 s
Beholder, low	54.417	83.72	17/24	248.8 s

Normalization is simply weighted score / 65 x 100, not a new rubric. All 48 saved responses regraded identically; there were no CLI errors or sandbox infrastructure failures. Runs 280/281 are in RESULTS.csv. See deductions and execution details. These scores are separate from the tool-enabled engineering experiment below.

November 3: fresh matched /100 comparison

Two new complete evaluations on matched-engineering100-v2 (Pi 1.5.0-max.1, ten workflows, low thinking requested):

Configuration	Score	Normal completions	Agent wall time
GPT 6.1 Sol, low	100/100	10/10	335.2 s
Beholder, low	90/100	9/10	406.8 s

Beholder's original verification attempt ended in native tool-call decoding failure; its successful diagnostic retry is not substituted into the score. Both models earned full credit on the other nine workflows. All 20 primary artifacts replayed consistently. Runs 278/279 are in RESULTS.csv.

This is a new ten-workflow development protocol, not the historical challenge-v1, canonical /65 suite, planned 30-workflow suite or a holdout. It uses constrained native-Pi repository tools and Linux bubblewrap for Python execution. One run per configuration does not establish a general capability ranking. See results, adjudication and limitations.

October 18: newer protocol results

RESULTS.csv now uses schema 2, retaining all historical rows and adding 83 task-level aggregates from 13 complete Pi 1.3.0 runs. Filter by protocol_id, evaluation_tier, and score_policy before comparing results. The corrected six-workflow pilot and the harder seven-workflow candidate are not comparable to the historical /65 suite or each other.

The challenge campaign is paused and incomplete. Completed validation runs are GPT 6.1 Sol — low thinking (98.500 and 97.591/100) and Gandalf (97.667/100, one run). No four-model ranking is established. Interrupted work is retained privately and is not scored as model failure. See protocol results and caveats.

Exact artifact filenames, hashes, byte sizes, derivative repositories, available revisions, draft models and runtime parameters are exported where recorded. Unknown information remains blank; cloud weights cannot be fingerprinted.

Historical October qualification status

October 6 deployment: Gandalf is the owner-selected persistent default, Peregrine is automatic fallback, and Doctor Strange plus the other named llama.cpp profiles remain available for emergency/manual use. Native agent selection uses Pi 1.0.4; canonical benchmarks still pin Pi 0.84.3. This deployment decision is not a new benchmark qualification. Gandalf's restricted reliability remains intermittent, and full native 1.0.4 quality qualification is incomplete. Startup, an actual 1.0.4 response and a manually invoked Peregrine fallback dispatch were checked; that does not establish every fault/reboot scenario. See the reproduction recipes for pinned builds, weights, Pi configuration and recovery policy.

The following score reports describe their original measurement dates.

The leaderboard includes 38 selected profiles, with a cloud-only Top 20 (10 selected profiles) and a local-only Top 20. The October Gandalf llama.cpp candidate scored 63.318/65 in two complete Q8-draft/k4 runs at 21.74 effective t/s. It is an opt-in experiment, not production-qualified: a separate Pi 0.84.3 fresh-prompt screen produced a reasoning-only answer at the output cap. The subsequent 12/12 reliability check passed, but did not catch that failure. Pi 1.0.0 returned visible answers on 8/8 fresh invocations and passed 12/12 reliability checks; its separate exploratory 24-task screen scored 55.247/65 without a validated prompt-version bridge. Full-context, cache-hot and retained-session gates remain open. All nine canonical new-runtime screens, including lower-scoring draft settings, have sanitized task records in RESULTS.csv. See the October findings and Pi 1.0 follow-up.

GPT-6 Astra completed two full runs per reasoning level: 56.241/65 medium, 55.909/65 high (means). Nine valid retired-runtime recovery screens did not meet 60/65 plus 18 effective t/s. TypeSafe is parked; no API experiment or private-data upload was performed. See the September update.

The older GPU-5 b10819 DFlash2 k7 coordinate scored 58.318/65 across three runs, with 22.42–22.44 effective output t/s and 12/12 reliability checks. Those checks do not qualify the newer Gandalf runtime. Peregrine remains the historical production reference; Doctor Strange was its rollback.

A Pi 0.86.1 exact-prompt pilot matched 0.84.3 on 36 no-tool spot-check invocations. This is limited validation, not a general Pi upgrade: the public runner still pins 0.84.3. See results and limitations and the revised leaderboard.

An isolated vLLM 0.29 investigation now reproduces 131K capacity with Peregrine's existing artifacts after targeted compatibility and memory fixes. Two runs scored 57.792/65 at 51.04–51.06 effective output t/s; reliability passed 12/12. This does not beat the historical production reference or establish an upstream release regression. Those experiments did not supersede the historical vLLM 0.28 reference.

What it tests

The main suite contains 24 tasks with a weighted maximum of 65 points. It covers:

• exact instruction following and JSON output
• executable Python and edge cases
• parsers, diffs, rate limiting, retries, and caching
• nginx and systemd configuration
• incident analysis and issue triage
• documentation, architecture decisions, and design review

Hard deterministic tasks carry more weight than smoke tests. See METHODOLOGY.md for the scoring and clean-run controls, LEADERBOARDS.md for the explicit overall and local rankings, INFERENCE_PROFILES.md for tested local settings and portability limits, and RESULTS.md for qualification details.

Peregrine now uses DFlash2 k7 at temperature 0.60 and top-p 0.95. Clean-start runs 232–234 were byte-identical at 57.970/65 and averaged 58.1 effective t/s; the reliability gate passed 12/12. This improves the prior top-p-0.90 coordinate by 1.949 points and 1.0 effective t/s. That historical coordinate used Doctor Strange as rollback.

Run it

Requirements

• Linux or WSL
• Python 3.11+
• Pi 0.84.3, with at least one configured model
• Bubblewrap for safely checking generated Python

No particular GPU is required. PiBench itself has no Python package dependencies.

On Debian or Ubuntu:

sudo apt install bubblewrap
npm install -g @earendil-works/pi-coding-agent@0.84.3
git clone https://github.com/maxmo74/pibench.git
cd pibench

PiBench versions the score protocol, not the Pi dependency. Pi 0.84.3 added a trailing newline to the effective custom prompt relative to the qualified 0.84.1 environment. Full-suite bridge runs reproduced all 24 outcomes and private outputs byte-for-byte on Doctor Strange and Road Runner; cloud controls stayed within their prior variation. The score protocol therefore did not change. Pi version and effective-prompt hash remain recorded on every run. A future Pi upgrade gets the same treatment: bridge at least two deterministic local profiles with complete repeated runs, and change the score protocol only if benchmark semantics or observed outcomes change.

List the models available to Pi:

pi --list-models

Run all 24 tasks against one model:

./pi_agent_bench.py 'provider/model:off'

For a fully local provider:

./pi_agent_bench.py --offline 'local-llama/your-model:off'

Pass several model IDs to compare them in one run, and use --task TASK_NAME for a shorter test. pi_agent_bench.py is the sole current runner. It pins bridge-qualified Pi 0.84.3, attests the complete effective prompt, and refuses unprofiled extension-modified input. Historical runner source remains available through Git history.

PiBench records CPU, memory, detected accelerators, Pi version, and model configuration automatically. For a CPU-only run or a backend that cannot be discovered through Pi, add explicit metadata:

./pi_agent_bench.py 'ollama/model:off' \
  --compute-mode cpu \
  --backend Ollama --backend-version 0.12.3 \
  --model-format GGUF --quantization Q5_K_M \
  --context-size 32768 --kv-cache f16

Repeat --accelerator for the devices actually used in a multi-GPU run. --metadata-file accepts a reusable JSON profile, and --inference-option KEY=VALUE records backend-specific settings. See --help and METHODOLOGY.md.

Results are written locally to results/, including results/pibench.sqlite. Generate a Markdown summary with:

./pibench_report.py

The tracked RESULTS.csv contains the complete sanitized task-level result history. Regenerate it from the local database with:

./pibench_report.py --csv-out RESULTS.csv

The exporter uses a strict field allowlist: scores, timings, model/runtime identifiers, and reproducibility metadata are included; prompts, output, errors, notes, command lines, private paths, and arbitrary JSON are excluded. Incomplete infrastructure runs are explicitly marked.

Before any push, python3 scripts/repository_audit.py --history must pass. It scans the current tree and all reachable Git history; deleting a sensitive file in a later commit is not sufficient. See SECURITY.md.

An OpenAI-compatible endpoint can also be tested directly without Pi:

./openai_endpoint_bench.py \
  --base-url http://127.0.0.1:8080/v1 \
  --model your-model

This diagnostic reuses current PiBench checks and records both effective output speed and native server timing fields when available. Pass --task TASK_NAME repeatedly to select specific tasks.

For a separate tool-enabled, multi-turn operations diagnostic, run configured loopback models through the versioned pi-ops-v1 profile:

PATH=/opt/pibench-pi-0.84.1/node_modules/.bin:$PATH \
  ./pi_ops_bench.py \
  'local-llama/Doctor Strange:low' \
  'local-llama/Road Runner:off'

That profile gives Pi only read, bash, edit, and write inside a disposable Bubblewrap filesystem, uses three sequential turns in one session, hashes its effective tool-enabled system prompt, and scores the resulting code, service unit, documentation, test preservation, and change scope. Its 100-point score is a separate daily-operations diagnostic and is never mixed into the canonical 65-point no-tools ranking. Raw sessions and output remain under ignored local paths.

For current loop/termination qualification, use the Pi 0.84.3 reliability gate:

./pi_agent_reliability_bench.py --repeats 3 \
  'local-peregrine/qwen3.8-27b:low'

This is the sole current reliability runner. Historical harness revisions and evidence remain available through Git history and retained result metadata.

It runs four read-only synthetic investigations twice in fresh isolated sessions: evidence-backed diagnosis, graceful termination when decisive evidence is absent, recovery after a large irrelevant context preamble, and resistance to a deterministic polling trap without searching outside the fixture. A model passes the screen only if every run finishes normally, answers the fixture correctly, respects scenario tool budgets and the 21-message ceiling, makes no exact duplicate tool call, stays inside the repository, and avoids repeated text blocks. This is deliberately a pass/fail screen rather than another score. It remains separate while its scenarios and thresholds are validated; passing is necessary evidence for autonomous use, not a guarantee of universal reliability. Private output records metrics, checks and hashes but not model text. Configured endpoints must remain loopback-only, while built-in cloud providers may use an existing provider-scoped auth.json entry. For cloud runs, only the selected credential is staged in the isolated agent directory, an attested extension blocks tool access outside the fixture, and the parent removes the staged credential after every Pi process—even on timeout.

Earlier Pi 0.84.1 reliability results remain historical qualification evidence. The current vLLM 0.28 Peregrine coordinate uses the merged Pi 0.84.3 reliability gate and passed 12/12 after packaging. Passing is still necessary evidence rather than proof of universal autonomy; the local-model loop guard remains required. Current deployment uses Peregrine as fallback and Doctor Strange for emergency recovery.

Reference results

The tables below preserve the August 29 snapshot. For current ranks and consistently recomputed aggregate throughput, use LEADERBOARDS.md.

All snapshot rows use the same 24-task/65-point score protocol. Pi 0.84.1 and 0.84.3 are recorded execution environments, not separate score protocols: the compatibility bridge found exact 24/24 task and output reproduction for Doctor Strange and Road Runner, while GPT-5.5 medium and high remained within ordinary managed-service variation.

Top 20 overall

Twenty-two complete profiles were selected in this historical snapshot. Repeated equivalent runs use their complete-run mean; no best-run or best-task splicing is used.

Rank	Scope	Model/profile	Runs/evidence	Pi	Score used	Effective output t/s
1	Cloud variant	Claude Opus 4.6, antigravity-v1	198/199/204	0.84.1 + Antigravity	61.506/65	40.5
2	Local retired	Peregrine, vLLM 0.27, low/8K/MTP3	213/214/215	0.84.1	61.006/65	37.81
3	Cloud	GPT-5.5, medium	185/189/216	0.84.1/0.84.3	60.542/65	19.8
4	Cloud	GPT-5.5, high	186/190/216	0.84.1/0.84.3	60.292/65	15.9
5	Cloud variant	Gemini 3.7 Flash, medium, antigravity-v1	194/195	0.84.1 + Antigravity	58.408/65	58.0
6	Local production	Peregrine, vLLM 0.28, low/8K/DFlash2 k7, top-p 0.95	232/233/234	0.84.3	57.970/65	58.1
7	Cloud variant	Gemini 3.1 Pro, high, antigravity-v1	196/197	0.84.1 + Antigravity	57.836/65	13.8
8	Cloud	GPT-5.6 Sol, medium	187/192	0.84.1	57.516/65	18.8
9	Local fallback	Doctor Strange, low/8K/MTP2	180/181/201/217	0.84.1/0.84.3	57.396/65	20.8
10	Cloud	GPT-5.6 Sol, high	188/193	0.84.1	56.305/65	17.5
11	Local predecessor	Peregrine, vLLM 0.28, low/8K/DFlash2 k7, top-p 0.90	229/230/231	0.84.3	56.021/65	57.1
12	Local candidate	Qwen3.8 + Sharp v22.3.1, low/8K/MTP2	208	0.84.1	55.417/65	20.2
13	Local candidate	Cold Fusion, low/8K/MTP2	200/203	0.84.1	55.006/65	19.8
14	Local predecessor	Peregrine, vLLM 0.28, low/8K/MTP3	218/219/220	0.84.3	54.771/65	43.0
15	Cloud	GPT-5.4, medium	184/191	0.84.1	54.277/65	23.1
16	Local bounded	Road Runner, off/4K/MTP3	202/217	0.84.1/0.84.3	54.042/65	159.2
17	Local retained	Spiderman, off/4K/MTP3	206	0.84.1	52.729/65	48.1
18	Local rejected	Qwen3.8-27B Opus Distill v2, low/8K/MTP2	235	0.84.3	52.318/65	32.9
19	Local retained	Thor, thinking/4K/no-spec	207	0.84.1	51.042/65	10.0
20	Local rejected	Road Runner practical, low/8K/MTP3	183	0.84.1	49.542/65	24.2

Top 10 local

Rank	Local profile	Pi	Score used	Effective output t/s	Status
1	Peregrine, vLLM 0.27, low/8K/MTP3	0.84.1	61.006/65	37.81	Retired looping coordinate
2	Peregrine, vLLM 0.28, low/8K/DFlash2 k7, top-p 0.95	0.84.3	57.970/65	58.1	Production qualified
3	Doctor Strange, low/8K/MTP2	0.84.1/0.84.3	57.396/65	20.8	Automatic fallback
4	Peregrine, vLLM 0.28, low/8K/DFlash2 k7, top-p 0.90	0.84.3	56.021/65	57.1	Superseded production coordinate
5	Qwen3.8 + Sharp v22.3.1, low/8K/MTP2	0.84.1	55.417/65	20.2	Rejected
6	Cold Fusion, low/8K/MTP2	0.84.1	55.006/65	19.8	Rejected
7	Peregrine, vLLM 0.28, low/8K/MTP3	0.84.3	54.771/65	43.0	Superseded production coordinate
8	Road Runner, off/4K/MTP3	0.84.1/0.84.3	54.042/65	159.2	Bounded no-tools only
9	Spiderman, off/4K/MTP3	0.84.1	52.729/65	48.1	Retained
10	Qwen3.8-27B Opus Distill v2, low/8K/MTP2	0.84.3	52.318/65	32.9	Rejected

See LEADERBOARDS.md for ranges, throughput, and profile boundaries.

Peregrine is the production daily driver on patched vLLM 0.28.0: Qwen3.8-27B W4A16, a W4A16 DFlash2 k7 drafter, int8-per-token-head attention KV, FP16 recurrent state, aligned prefix caching, synchronous scheduling, two admitted sequences, 131,072 context, and an 8K output ceiling. Clean-start top-p-0.95 runs 232–234 each scored 57.970/65 with byte-identical private outputs at a 58.1 effective t/s mean. Temperature-0.60/top-p-0.95 reliability passed 12/12 using PiBench-owned fixtures. The #48375 cache-tail backport, complete startup patch verification, loop guard, draft-artifact hashes, and hash-bound production certificate are mandatory parts of the coordinate. INFERENCE_PROFILES.md records the exact settings.

A dated public-model candidate survey screened Hugging Face, Hacker News, Reddit, and inference forums for a stronger one-RTX-3090 replacement. Opus Distill v2 was the sole pre-screen winner. Its pinned Q4_K_M coordinate completed run 235 at 52.318/65 and 32.9 effective t/s, missing both the 58.47 quality gate and 35 t/s floor; reliability was also not qualified at 9/12. No challenger qualified, so Peregrine remains production unchanged.

Stable llama.cpp v0.2.0/b10566 and all GGUF profiles remain installed as the controlled fallback runtime; Doctor Strange is the automatic rollback backend. The original vLLM 0.27 MTP3 coordinate later reproduced a cache-hot three-call cycle. Peregrine remains supervised for consequential edits.

Tool-enabled daily-operations result

The separate pi-ops-v1 three-turn profile produced:

Alias	Score	Tool calls	Total wall time
Peregrine	100/100	19	59.3 s
Doctor Strange	100/100	27	225.7 s
Road Runner	95/100	18	20.5 s
Thor	95/100	15	74.0 s
Spiderman	85/100	18	53.6 s

All profiles fixed the retry implementation, preserved tests, produced the fully hardened service, stayed in scope, and completed all turns. Peregrine and Doctor Strange both earned 100/100; Peregrine used 19 tools and completed in 59.3 seconds. Road Runner and Thor omitted the exact requested unittest command; Spiderman missed all three exact README commands. Road Runner remained the fastest older bounded profile. Peregrine used Pi 0.84.1 at PiBench tip 0b0dbf6; the older four used commit b124523. Both used attestor SHA-256 0480e4d9c6e8b2b7905a10c78206b37ac45cd9203068140123e7f08e8c51d013, and effective tool-enabled system-prompt SHA-256 c9f6885987f161b6c530b108b61e2d6b173e1b79dd1caeac2ddc0fb7f18b6cb9.

A subsequent adversarial replay exposed a limitation that pi-ops-v1 does not measure: termination under ambiguous, open-ended investigation. A retained Road Runner session duplicated planning and searches, then repeated one paragraph 43 times until aborted. Replaying the exact branch on b10566 timed out with repeated tools under both MTP3 and target-only execution; repetition penalties reduced verbatim duplication but did not restore termination, and an explicit tool budget was ignored. MTP amplified the failure but was not its root cause. Road Runner should therefore not be used unattended for autonomous tool work; prefer Doctor Strange. Its 148.2 t/s result remains valid for the bounded no-tools profile.

Cloud versus local results

Each run fixes the benchmark input: pinned Pi version, attested effective-system-prompt hash, fixed cwd, task prompts and graders, and no tools, context files, skills, themes, templates, or extensions. The one documented exception is the antigravity-v1 profile (Google Cloud Code Assist via the pinned pi-antigravity 0.3.1 extension), whose fixed three-part system-instruction injection is version-pinned and attested; those runs sit on canonical prompt + fixed injection rather than the pure canonical prompt. It does not make the execution environments equivalent:

• Local runs can attest GGUF/safetensors hashes, quantization, runtime build, server parameters, KV cache, sampler, speculation, and hardware. Seeded local generation can be reproducible, but quantized speculative decoding and runtime changes can still alter output.
• Cloud runs attest the provider/model identifier and request profile, but not weights, backend build, routing, sampler implementation, or service updates. The API does not provide a seed-attested deterministic path, so repeated cloud runs are summarized by mean and range rather than their best score.
• Peregrine's clean-start production runs 213–215 were byte-identical on all 24 tasks. This claim includes server seed 0, omitted request seed, and one greedy startup readiness request; periodic checks are non-inference engine-RPC probes. An extra inference request changes the subsequent unseeded trajectory and is therefore a different request-history coordinate. Doctor Strange remains output-deterministic across its tested b10434/b10566 replay. GPT-5.4 repeated the same score but only 3/24 outputs were byte-identical.
• Effective output t/s is end-to-end visible-output speed. Local figures primarily reflect the reference machine; cloud figures also include network, queueing, and opaque provider execution.
• Output ceilings and reasoning controls are profile properties: historical local coordinates often used 4K, Peregrine and Doctor Strange used 8K, and the newer Mac comparison used 16K/32K with larger exploratory caps. A cloud catalog ceiling is not proof that its adapter serialized or enforced it; consult output_cap_enforcement. Local and cloud reasoning levels are not assumed equivalent.

Date-injected legacy results remain in RESULTS.csv as historical evidence but are not mixed into the pi-agent-24/65 ranking. See RESULTS.md and METHODOLOGY.md for detailed interpretation rules.

Reference systems

November Mac-local runs use Apple M5 Ultra, 30 logical CPUs, 256 GiB unified memory, macOS/Metal. Exact runtime and hardware coordinates remain attached to each measurement. The following table describes the historical RTX 3090 reference, not the Mac:

Component	Specification
OS	Debian GNU/Linux 13 (trixie), kernel 6.12.101+deb13-amd64
CPU	AMD Ryzen 9 7900, 12 cores / 24 threads
RAM	128 GB
GPU	NVIDIA GeForce RTX 3090, 24 GB (24,576 MiB)
NVIDIA driver	595.91.07 (current Peregrine production); earlier local runs used 550.163.01
CUDA toolkit	12.4.131

Most historical local comparisons used a 131,072-token context, quantized KV cache, full GPU offload, and flash attention. That Peregrine coordinate admits two sequences to reduce recurrent-state and scheduler exposure; its llama.cpp fallback uses one parallel slot. Exact settings are part of each result profile.

Contributing results

Runs from other hardware and providers are welcome. To make a result useful, include:

• model and provider, including quantization
• compute mode (CPU, GPU, hybrid, remote, or cloud), hardware, OS, and backend version or commit
• context size, KV-cache type, reasoning mode, temperature, and speculation settings
• exact PiBench command
• weighted score and timing summary from a complete 24-task run

Use --contributor and --source-url to preserve provenance in your local database. Open a pull request that adds a concise row or section to RESULTS.md. Contributions are accepted under AGPL-3.0-or-later. Do not commit results/, metadata profiles, raw transcripts, model weights, credentials, or machine-specific paths. Some historical experiment commits have no file diff because generated result files are intentionally excluded from Git.

Safety and limitations

Generated Python is untrusted. PiBench runs executable checks through Bubblewrap with no network or host-home access and with resource limits. It fails closed when the sandbox is unavailable. Do not set PIBENCH_ALLOW_UNSANDBOXED_EXEC=1 outside a disposable environment.

PiBench is a focused engineering suite, not a complete measure of model quality. Static configuration checks are not deployment tests, models can be stochastic, and local and cloud reasoning controls are not equivalent.

The project was developed with LLM coding assistance; benchmark choices, runs, and summaries were reviewed by the maintainer.

License

GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later)