Chart your model fleet. Compile the course. Let the recorded tides decide.
An offline compiler that turns recorded LLM traces into auditable routing policies.
No live API. No network. No guesswork — only the evidence already in your logs.
- Why a compiler, not a proxy
- The voyage in one glance
- Chart room: core concepts
- Setting sail: quick start
- Reading the log: real output
- The Pareto compass
- Writing routing policies
- The navigation dashboard
- Deckhand's manual: CLI reference
- How the hull is built
- Determinism: the ship's chronometer
- Testing the rigging
- Project layout
- Design decisions worth defending
- FAQ from the crow's nest
- License
Most "model routers" are live things: they sit in the request path, call providers, and hope the decision they make right now is a good one. That is a runtime gamble dressed up as infrastructure.
ModelMariner takes the opposite tack. It is an offline compiler. You feed it the traces you already recorded — every model, every task, every token, dollar, millisecond, quality score, error, and privacy tier — and it compiles a routing policy the way a build tool compiles source: deterministically, explainably, and with the whole picture in view.
The distinction matters for three reasons a captain cares about:
- Evidence over optimism. Every decision is backed by observations you can
point at. When ModelMariner routes
draft-legal-clausetolighthouse-local, it can show you the 45 recorded calls that justify it. - Hard constraints are actually hard. A budget cap, a latency ceiling, a privacy tier — these disqualify candidates. They are never "soft penalties" that a high enough quality score can bribe past. Regulated workloads demand this, and so does anyone who has ever been paged at 3 a.m.
- Reproducibility. The same traces and the same policy always compile to the same routing table, byte for byte. You can commit the output, diff it in code review, and gate CI on it.
ModelMariner never opens a socket to a provider. It cannot. That is the point.
traces.jsonl ──▶ ingest & validate ──▶ reliability aggregation
│
▼
Pareto frontier per task
│
policy.json ──▶ hard-constraint evaluation ─┤
▼
routing simulation (replay traces)
│
▼
explanations ◀── deterministic report
│
report.json + report.txt + policies.json
│
▼
TypeScript navigation dashboard
Every arrow is a package with tests. Every box is a decision you can inspect.
| Term | What it means aboard ModelMariner |
|---|---|
| Trace | One recorded observation: a model handling a task, with its cost/latency/quality/error/privacy. The atom of everything. |
| Reliability | Per model/task success rate plus a Wilson score lower bound so a lucky 9/10 never outranks a proven 90/100. |
| Pareto frontier | The set of non-dominated models for a task — the only candidates a rational router should even consider. |
| Policy | A bundle of hard constraints (budget, latency, quality, reliability, privacy) plus a weighted preference for ranking survivors. |
| Compilation | Applying a policy to the frontier to pick one winner per task, then replaying recorded traces to measure what that choice actually delivers. |
| Explanation | The human-readable "why": margins, score components, replayed evidence, and the exact constraint each rejected model tripped over. |
The bundled synthetic corpus sails five vessels, each a caricature of a real trade-off you will recognize:
harbor-nano— the dinghy. Cheapest and fastest, but its quality founders on hard generative work.harbor-mini— the sloop. A superb all-rounder that wins most cost-sensitive routes.clipper-pro— the clipper. Fast and accurate, at a real price.galleon-max— the galleon. Top quality, heaviest cost and latency.lighthouse-local— the on-prem lighthouse. Modest metrics, but the only vessel cleared to carry restricted cargo without leaving harbor.
No single vessel wins everywhere. That is by design — a router with an obvious answer is not worth compiling.
You need Go 1.24+ and (for the dashboard) Node 20+. Nothing else.
# 1. Build the compiler
make build # produces ./bin/modelmariner
# 2. Compile a report + routing tables from the sample fleet
make report # writes testdata/output/{report.json,report.txt,policies.json}
# 3. Explore it in the dashboard
make demo # builds the TS dashboard and prints the fleet overviewOr drive the binary directly:
./bin/modelmariner analyze \
--traces testdata/fleet.jsonl \
--policy testdata/policies.json \
--out testdata/output \
--format bothThis is actual output from modelmariner analyze over the 1,400-line sample
corpus — not a mock-up.
========================================================================
RELIABILITY (per model / task)
========================================================================
task model n success reliab.LB p95 ms quality
classify-intent clipper-pro 70 97.1% 0.902 1040 0.926
classify-intent galleon-max 70 98.6% 0.923 1696 0.953
classify-intent harbor-mini 70 97.1% 0.902 512 0.908
classify-intent harbor-nano 70 95.7% 0.881 309 0.778
classify-intent lighthouse-lo. 70 98.6% 0.923 584 0.784
draft-legal-clause clipper-pro 45 100.0% 0.921 925 0.939
draft-legal-clause galleon-max 45 100.0% 0.921 1634 0.964
draft-legal-clause harbor-mini 45 97.8% 0.884 417 0.819
draft-legal-clause harbor-nano 45 84.4% 0.712 302 0.589
draft-legal-clause lighthouse-lo. 45 97.8% 0.884 550 0.779
Notice harbor-nano on draft-legal-clause: only 84.4% success and a
reliability lower bound of 0.712. The dinghy is out of its depth on legal
drafting, and the numbers say so before any policy is applied.
• route "classify-intent" to harbor-mini (score 0.8157)
- harbor-mini beat runner-up lighthouse-local by a margin of 0.0698 in weighted score
- score components: cost=0.5573, quality=0.1860, reliability=0.0724
evidence: replayed 70 recorded call(s): 97.1% success, mean quality 0.908, mean latency 259 ms, total cost $12.9234
evidence: versus cheapest-model baseline harbor-nano: $7.5293 more cost, 0.908 vs 0.778 mean quality
rejected: clipper-pro — max_cost_usd (limit 0.35, observed 0.6692)
rejected: galleon-max — max_cost_usd (limit 0.35, observed 1.612)
Every line is defensible. The winner, the runner-up and the margin between them, the weighted breakdown, the replayed evidence, the baseline comparison, and the precise reason each disqualified vessel was left at the dock.
• route "draft-legal-clause" to lighthouse-local (score 0.6630)
- lighthouse-local was the only model to satisfy every hard constraint
rejected: clipper-pro — max_privacy (task data reaches "confidential"
but policy caps at "internal" and model is not privacy-safe)
rejected: galleon-max — max_privacy (task data reaches "confidential"
but policy caps at "internal" and model is not privacy-safe)
Under the privacy-lockdown policy, confidential data cannot leave on-prem
infrastructure. The cloud vessels are disqualified outright — not down-weighted,
disqualified — and the only privacy-safe model wins by default.
Before any policy is applied, ModelMariner draws the Pareto frontier for each task. A model is dominated when some other model is at least as good on every objective — cheaper-or-equal, faster-or-equal, higher-or-equal quality, more-reliable-or-equal — and strictly better on at least one. Dominated models are vessels no rational captain would ever choose, so they are flagged and explained:
PARETO FRONTIERS (non-dominated models per task)
classify-intent frontier: harbor-nano, lighthouse-local, harbor-mini, clipper-pro, galleon-max
harbor-nano cost $0.0774 p95 309ms q 0.778 rel 0.881 on-frontier
harbor-mini cost $0.1867 p95 512ms q 0.908 rel 0.902 on-frontier
galleon-max cost $1.6117 p95 1696ms q 0.953 rel 0.923 on-frontier
The compass needle in the image above sweeps exactly this space: cyan vessels ride the frontier; a dominated vessel drifts grey and is never chosen. Objectives minimized (cost, latency) point one way; objectives maximized (quality, reliability) point the other. The frontier is the coastline between them.
A policy set is JSON. Constraints are hard; the preference ranks whoever
survives. Here is the bundled budget-guard:
{
"name": "budget-guard",
"description": "Minimize spend while keeping quality acceptable across all tasks.",
"constraints": {
"max_cost_usd": 0.35,
"min_quality": 0.7,
"min_reliability": 0.8
},
"preference": {
"weights": [
{ "objective": "cost", "weight": 0.6 },
{ "objective": "quality", "weight": 0.25 },
{ "objective": "reliability", "weight": 0.15 }
]
}
}constraints— any candidate that breachesmax_cost_usd,max_latency_ms(checked against p95),min_quality,min_reliability, or the privacy cap is disqualified. Adddeny_models,allow_models, andprivacy_safe_modelsto steer the fleet by name.preference.weights— rank the survivors. Weights normalize to 1, so only the ratios matter. Cost and latency are inverted (lower is better) and every objective is scaled across the candidate set before weighting, so a policy never accidentally compares dollars to milliseconds.tasks(optional) — scope a policy to specific tasks; omit it to govern the whole voyage.
The full schema — input traces, policy language, and every output artifact —
lives in docs/TRACE.md.
The dashboard/ directory holds a strictly-typed TypeScript consumer of
report.json. It has zero runtime dependencies (only the TypeScript
compiler and Node's built-in test runner at dev time) and renders entirely in the
terminal — no browser, no remote assets.
──────────────────────────────────────────────────────────────────────
MODELMARINER DASHBOARD — fleet overview
──────────────────────────────────────────────────────────────────────
Traces: 1400 accepted / 1400 lines | 5 models | 5 tasks | 4 policies
ROUTING WINNERS BY TASK
task budget-guard latency-critical privacy-lockdown quality-first
classify-intent harbor-mini harbor-nano — —
draft-legal-clause lighthouse-local — lighthouse-local clipper-pro
extract-pii-redaction lighthouse-local — lighthouse-local —
POLICY ECONOMICS (realized vs cheapest-model baseline)
budget-guard spent $23.4950 for +0.137 quality (5 routed, 0 unrouted)
privacy-lockdown spent $3.8977 for +0.167 quality (2 routed, 0 unrouted)
quality-first spent $100.1142 for +0.297 quality (2 routed, 0 unrouted)
Views:
| Command | Shows |
|---|---|
overview |
Fleet summary + winners-by-task grid + policy economics. |
task <name> |
One task: reliability table, frontier, and each policy's pick. |
model <name> |
One model across every task, and how often it is chosen. |
policy <name> |
A policy's compiled decisions with full explanations. |
routes <name> |
Just the compiled winner-per-task routing table. |
cd dashboard
npm install && npm run build
node dist/dashboard.js ../testdata/output/report.json task draft-legal-clausemodelmariner analyze --traces FILE [--policy FILE] [--out DIR]
[--format json|text|both] [--strict] [--with-timestamp]
modelmariner validate --traces FILE [--strict]
modelmariner version
| Flag | Purpose |
|---|---|
--traces, -t |
Path to the JSONL trace file (required). |
--policy, -p |
Path to a JSON policy set. Omit for reliability + Pareto only. |
--out, -o |
Directory for report.json, report.txt, policies.json. |
--format |
What goes to stdout: json, text, or both (default both). |
--strict |
Treat any invalid trace line as a fatal error. |
--with-timestamp |
Stamp the report with wall-clock time (breaks determinism). |
validate ingests and reports accepted/rejected counts without compiling —
handy in a pre-commit hook to keep your trace logs seaworthy.
ModelMariner is pure Go standard library — not a single third-party import. Each plank of the hull is an internal package with a single responsibility:
| Package | Responsibility |
|---|---|
internal/trace |
JSONL ingestion, validation, normalization, privacy tiers. |
internal/reliability |
Success rates, Wilson bounds, latency percentiles. |
internal/pareto |
Multi-objective dominance and frontier extraction. |
internal/policy |
Policy language, hard-constraint evaluation, scoring. |
internal/routing |
Winner selection + trace replay + baseline economics. |
internal/explain |
Human-readable rationale for every decision. |
internal/report |
Deterministic JSON/text assembly + compiled artifacts. |
cmd/modelmariner |
The CLI that wires the voyage together. |
A router you cannot reproduce is a router you cannot trust. ModelMariner's report JSON is byte-for-byte identical across runs on identical inputs:
- Traces are sorted by
(model, task, timestamp, line)after ingestion. - Every map is emitted through sorted keys.
- The frontier, evaluations, decisions, and policies all carry stable ordering.
- HTML escaping is disabled so
<,>, and&survive diffs unmangled. - The generation timestamp is omitted by default (opt in with
--with-timestamp).
The CI smoke test runs analyze twice and diffs the output; a mismatch fails
the build. Determinism is not an aspiration here — it is enforced.
make test # Go unit + integration tests
make cover # with per-package coverage
make dashboard # TypeScript type-check + build + node:test
make ci # fmt-check + vet + test + dashboard (what CI runs)Go coverage sits comfortably above the waterline across every engine package
(Pareto and reliability above 96%), and the dashboard ships twelve node:test
cases covering the navigator and every render view. Tests assert the behavior
that matters: budgets disqualify, privacy caps hold, dominated models drop off
the frontier, Wilson bounds reward evidence, and reports stay deterministic.
modelmariner/
├── cmd/modelmariner/ # CLI entrypoint + integration tests
├── internal/
│ ├── trace/ # ingestion, validation, privacy tiers
│ ├── reliability/ # aggregation, Wilson bounds, percentiles
│ ├── pareto/ # dominance & frontier analysis
│ ├── policy/ # policy language + hard-constraint evaluator
│ ├── routing/ # winner selection + trace replay
│ ├── explain/ # decision rationale
│ └── report/ # deterministic JSON/text + artifacts
├── dashboard/ # TypeScript navigation dashboard
│ ├── src/ # types, navigator, renderer, tests
│ ├── package.json
│ └── tsconfig.json
├── docs/
│ ├── TRACE.md # authoritative format reference
│ └── assets/ # two animated SVGs (no remote media)
├── testdata/
│ ├── fleet.jsonl # 1,400-line synthetic corpus
│ ├── gen_traces.go # deterministic corpus generator
│ ├── policies.json # four example policies
│ └── output/ # committed sample report artifacts
├── .github/workflows/ci.yml
├── Makefile
├── go.mod
├── LICENSE
├── CHANGELOG.md
└── README.md
Why the Wilson lower bound instead of raw success rate?
Because a model that succeeded 9 out of 10 times has not proven itself the way
one that succeeded 90 out of 100 has. The Wilson score interval's lower bound
folds sample size into the estimate, so thinly-sampled vessels cannot bluff their
way onto the frontier. wilsonLowerBound(9,10) < wilsonLowerBound(90,100), and a
test pins exactly that.
Why replay traces instead of trusting the aggregates? Aggregates answer "how does this model behave on average?" Replay answers "what would this specific routing decision have delivered against the record we hold?" The second is the honest question. Simulation never invents a data point; it only re-reads observations already in hand, which is why it stays deterministic.
Why are constraints hard rather than weighted? Because a budget is a budget and a privacy tier is the law. A soft-penalty router will, given a high enough quality score, quietly route restricted data to a cloud model. ModelMariner structurally cannot. Constraints disqualify; preferences only rank the survivors.
Why only the standard library? Fewer dependencies mean fewer surprises, a smaller supply-chain surface, and a binary that compiles anywhere Go does. For a tool whose whole value proposition is trustworthy, reproducible decisions, that discipline is the point.
Does ModelMariner call any model provider? No. It has no networking code in the decision path and needs none. It reasons purely over recorded traces.
Can it route in real time?
It is a compiler. It produces a routing table (policies.json) that a runtime
can load and apply. ModelMariner itself makes the decision offline, once, so you
can review it before it ever touches production traffic.
What happens when no model satisfies a policy?
The task is reported as unrouted with the reason, both in the report and in the
compiled artifact. Silence would be a bug; a loud, explained non-decision is a
feature.
How do I bring my own traces?
Emit newline-delimited JSON matching the schema in
docs/TRACE.md and point --traces at it. Run validate first
to catch malformed lines.
Released under the MIT License. Fair winds and following seas. ⚓