The question: when an engineer needs to understand real plant logic, is the deterministic engine better than pointing a capable AI model at the raw exported file? We ran the same model both ways — alone vs. with the engine — across 24 runs, 5 vendors, on a real 110 MB production controller, and report the ties, gaps, and wins.
This is the evidence under Nexus, the deterministic engine (and the live/join layers it enables): live break/fix and the cross-system join are only trustworthy if the underlying retrieval is exact, cited, and reproducible. That's what this measures.
Last full refresh: 2026-06-12. Numbers are from each CLI's own footer/session summary and the engine's reproducible answer key — not hand-tallied, and failures are recorded.
A real export is ~29M tokens of XML — too big to paste. So the fair test isn't "paste vs.
engine," it's: when a model points at the raw .L5X and greps, does it match the same model handed
the engine? Every model ran two ways, same three questions, same controller, questions picked from
signal names before anyone saw the answers:
- On its own — a shell,
grep, and the raw file. The real "just use Claude/Gemini on the file." - With the engine — the same model + Nexus tools (a parsed, deterministic model).
| # | Question | Tests | Engine's answer key (<0.3 s, 0 cost, identical every run) |
|---|---|---|---|
| 1 | What turns coil SRT_U000001_OK on? (the file also holds a pending online edit — two versions of the rung; the prompt never mentions it, and noticing which is live is graded as an unprompted bonus) |
right logic, exact rung, edit-awareness | PROG_B/RTN_SORT_02, rung 3 live (rung 2 pending) |
| 2 | Trace motor U000002_ECC_Z1_MTR back to all raw root signals, into the vendor blocks. |
completeness | 37 root signals + 26 flagged-unresolved AOI boundaries (453-node graph) |
| 3 | What drives the outputs of a vendor-encrypted block? | honesty | encrypted — cannot be determined |
On its own, a capable model is slow, reaches a fraction of the root signals before losing the thread, miscites often enough to matter, and can't tell which edit is live. Give that same model the engine and the answer is complete, exactly cited, edit-aware, and near-instant.
| What we measured | 🔴 Model alone | 🟢 Same model + engine |
|---|---|---|
| Found all 37 root-input signals (Q2) | 4–13 by hand, then loses the thread | complete set, deterministically |
| Cited exact program · routine · rung | clean on ~4 of 13 | 11 / 11 |
| Noticed the pending online edit, unprompted (Q1)¹ | 0 of 13 | 11 / 11 |
| Refused to invent logic in the locked block (Q3) | honest, but 1 missed the lock | honest and flagged it, every run |
| Time to answer | 3m17s–18m49s | seconds–~2 min |
| Same answer every run? | no — drifts | yes — the engine's answer is identical |
Coverage: 13 model-alone runs (Anthropic ×7, DeepSeek, Zhipu, Alibaba, Google) + 11 with the engine.
Full per-run receipts in runs/ (earlier pilot runs are kept separately in
runs/pilots/); complete table in kit/scoresheet.md.
¹ Graded, not asked. The prompt (
kit/TASKS.md) only asks what drives the coil; it never mentions the edit. Graders checked which rung version each answer analyzed (per the RUNBOOK grading rules). The engine arm goes 11/11 because the engine resolves and reports the in-control rung by construction — the point being that a pending edit is exactly the kind of thing a tool must surface without being asked.
Completeness, by hand (Q2) — best run per model, where a model was run twice: claude-sonnet-4.6 ~13 · opus-4.5 ~12 · opus-4.7 ~10 · opus-4.8 ~9 (5 on its other run) · sonnet-4.5 ~8 · opus-4.6 / deepseek-3.2 ~7 · gemini-3-flash ~6 · glm-5 / qwen3-coder-next ~5 · sonnet-4 ~4 — vs. the engine's complete 37 every time. With the engine the model's job shifts from finding signals (poor at scale) to explaining them (does well) — and 5 of the 11 engine-arm runs transcribed all 37 into prose while the rest under-read the complete answer they were handed (per-row in the scoresheet; a model-writing limit, not a retrieval one).
Cleanest cost A/B (gemini-3-flash — the one harness reporting token counts on both arms, so the
one pair where before/after cost is in the same hard unit): 2.17M tokens / 18m 49s → 127K tokens /
1m 59s (~17× fewer, ~9× faster) — and the complete set vs. ~6 by hand. A small "flash"-tier model
with the engine beat every model working alone — deliberately the conservative case. Not a
one-off: 8 of the 10 credit-metered pairs also got cheaper with the engine (glm-5 16.91 → 2.15,
opus-4.5 5.67 → 1.75); the two that rose (opus-4.8, sonnet-4.6) spent the extra enumerating the
complete 453-node answer — see the cost note in kit/scoresheet.md.
The engine fixed each weaker model's worst mistake: glm-5 invented a routine that doesn't drive the signal (twice) → cited correctly; deepseek-3.2 never named the containing program → all three exact; qwen3-coder-next wrong routine + missed the lock → correct + flagged.
- One controller. Graded by our own controls engineers, against the engine's output and hand-checked rung by rung on the actual logic. Every receipt is published so anyone can check it.
- Root-signal reference = 37, reproduced deterministically from the export by
make_answer_key.py(0 credits, no model, identical every run). Running the script requires the Nexus engine installed — without it you can read exactly how the key is derived, but not execute it. The fully-reproducible- without-us part of this release is the baseline arm and the grading of the published receipts. - The baseline arm had a 30-tool-call budget (product arm: none stated; engine runs used 4–9 calls). 3 of 13 baseline runs finished at or over the cap (gemini-3-flash 31, sonnet-4.5 31, qwen 30), so read the baseline completeness numbers as "within a realistic effort budget," not an absolute ceiling — though most runs stopped or lost the thread well under it.
- The baseline bars models from writing their own parser. The rules allow shell search and file reads only — so the strongest DIY baseline ("let the model write its own L5X tracer script") is untested here, deliberately, to measure the workflow people actually use. If you run that arm on your own export, we'll link your receipts.
- Edit-awareness (Q1) was graded, not asked — see the footnote under the results table.
- Timings come from each CLI's own session footer, which is the authoritative record for that run. Cost is reported in each harness's native unit; the gemini-3-flash row is the clean token-for-token A/B, since that CLI meters tokens on both arms.
- File size: the controller export is 110 MB (binary MiB); some CLI receipts record it as ~115 MB (decimal MB). Same file.
- What we are NOT claiming: that models "can't read PLC code." On a small program they trace fine. The measured claim is narrower: at real plant scale, a model alone is slow, incomplete, citation- unreliable, and different every run, while the engine hands any model a complete, exactly-cited, reproducible answer in milliseconds — and gets better as models do.
The export is a customer's confidential program, so it is not published (see
ABOUT-THIS-RELEASE.md). What you can do is read exactly how every number
was produced, check the grading against the receipts, and run the same protocol on your own export.
| File | What it is |
|---|---|
kit/RUNBOOK.md |
Start here — step-by-step for both arms, copy-paste prompts. |
kit/TASKS.md · kit/scoresheet.md |
The standard prompt · the full results table. |
kit/make_answer_key.py |
Regenerates the engine answer key (0 credits, deterministic; requires the Nexus engine installed). |
runs/ · agentic-benchmark-spec.md |
Per-run receipts · the benchmark design. |
context-window-comparison.md |
Why a real export can't just be pasted into a model. |
ABOUT-THIS-RELEASE.md |
What is withheld from this release and why. |
# regenerate the engine answer key (no model, no credits; needs the Nexus engine installed)
python kit/make_answer_key.py "path/to/YOUR_EXPORT.L5X"
# → root signals + unresolved AOI boundaries, locked blocks flagged, 0 creditsWhy the engine is the moat and not the model: build deterministic, verifiable engine facts and let the model do prose and judgement. This benchmark keeps that line honest.