#333 child: Establish end-to-end time-to-answer baselines and regression gates

[Joncallim]

Parent: #333. Blocked by #332.

Goal

Create the performance authority DockerMap currently lacks outside the Atlas: controlled, reproducible measurements for the actual operator path from process start to a useful Review answer.

This issue owns measurement, fixtures, evidence format and promotion gates only. Do not optimize production code here unless a tiny instrumentation hook is strictly required to make a stage measurable.

Required measurements

Use controlled 25 / 100 / 250-container reference fixtures and measure at least:

1. daemon process start → HTTP listener accepting requests;
2. listener ready → first authoritative Docker publication;
3. Docker observation wall time;
4. Compose correlation/enrichment wall time as a separate stage;
5. daemon publication → Node/SSE observation;
6. browser notification → coherent model accepted;
7. coherent model accepted → Home/Review useful content rendered;
8. buildModel();
9. Findings derivation for representative 25/100/250 topology/evidence sizes;
10. legacy Home topology-layout cost;
11. Cmd-K open + query-to-results for representative query classes;
12. production bundle/startup asset budget.

Methodology

• Pin runner class, CPU class, OS image, browser revision/flags, font environment, production build mode, fixture revision and source revision, following the discipline already used by Atlas performance evidence.
• Use warmed repeated samples and a reviewed percentile/median policy; do not gate on one noisy CI duration.
• Keep raw timings in the evidence artifact; derive summaries during review.
• Ordinary unit tests may validate evidence shape/math but must not pretend to be the controlled benchmark.
• Create an explicit baseline artifact/version so later candidates can be compared under an equivalent environment.
• Define regression limits from measured variance; do not invent aspirational absolute milliseconds before the baseline exists.

Fixture requirements

At minimum:

• 25-container Compose-heavy host;
• 100-container mixed relationship/mount/network density;
• 250-container stress host;
• provider-only revision change with Docker inventory unchanged;
• Docker topology change with providers unchanged;
• slow-but-bounded Compose projection;
• unavailable/slow optional provider.

Fixtures must be deterministic, secret-free and representative of existing contract bounds.

Acceptance criteria

• One documented command/job produces the complete controlled timing matrix.
• Results clearly distinguish backend collection, transport/notification, browser model work, rendering and search.
• Atlas's existing renderer performance evidence is reused where appropriate rather than duplicated.
• The baseline exposes whether Compose, SSE polling, model rebuilds or legacy layout dominate actual time.
• Promotion rules fail closed on incompatible runner/environment metadata.
• No production runtime telemetry/analytics are added.
• No optimization claim is accepted later without comparing against this baseline.
• Full normal checks remain green.

Deliverables

• performance contract/evidence schema;
• benchmark fixture definitions;
• baseline capture procedure;
• checked-in baseline metadata or reviewed external artifact location according to existing repo policy;
• a short architecture/performance document explaining exactly what each number does and does not prove.

Non-goals

• No performance fix itself.
• No browser analytics.
• No user tracking.
• No loosening of security/coherence to make numbers look better.

Risk: LOW · read_only_product_behavior: true · security_sensitive: false · data_migration: false · routing_mode: stable

[Joncallim]

First increment pushed for review as a draft PR #346 — issue stays open, this is not a completion claim.

Landed so far (all green under npm run check):

• the closed time-to-answer evidence contract: 12 stages with bucket + explicit measures/doesNotProve, the fixture set, the exact fixture × stage matrix, the pinned environment allowlist (incl. Docker revision), raw-sample validation, p95/median-of-three math, and the max(baseline × 1.25, baseline + 2) promotion gate — summaries are recomputed at review time so a supplied summary cannot influence a result;
• the deterministic fixture source: a byte-identical 25/100/250 (+4 scenario) Docker inventory generator and a fake Docker Engine API over a unix socket serving only /containers/json, /networks, /volumes — proven end to end against the real daemon build (25 containers / 1 network / 5 volumes, mode: docker, dockerReachable: true);
• docs/testing/TIME_TO_ANSWER_EVIDENCE.md (what each number does and does not prove);
• npm run test:perf wired into check:js.

Still outstanding before #335 can close: the one documented capture command driving all three controlled runs, the browser-side stage probes (reusing the existing Playwright harness), genuine daemon-side separation of Docker observation from Compose enrichment, and the first captured baseline artifact. No performance number is quoted and no optimization may be claimed until that baseline exists.

[Joncallim]

#335 — completion evidence

Head: 71494a0 (PR #346, base b6904d5^). Product code unchanged since the measured revision — git diff b6904d5..71494a0 -- apps crates packages shows only the new perf test file.

1. One documented capture command

npm run perf:metadata        -- --output <metadata.json>
npm run perf:time-to-answer  -- --metadata <metadata.json> --output <artifact.json> [--baseline <reviewed.json>]
npm run perf:summarize       -- --artifact <artifact.json>

perf:time-to-answer owns the whole lifecycle: deterministic fixture Docker daemon, real daemon with bench attribution, real API (tsx), production web build, benchmark-only probe build, real Chromium. Private OS-reserved ports; process-group teardown (no pattern killing); explicit readiness waits; fail-closed on any incomplete cell; raw samples preserved when assembly fails; live deployment never touched.

2. Complete matrix — 12 stages × 7 fixtures × 3 runs × 15 samples = 21 cells

Captured in 16.5 min on the pinned runner. Two harness defects were found by running it for real and fixed before the final capture: samples were nested one level too deep (a run-shape bug), and raw samples were not preserved when artifact assembly failed.

3. Baseline artifact

• dockermap-v1/time-to-answer-baseline-1
• source revision b6904d553bf9062da05e0375e40765820104b70f
• fixture revision dockermap-v1/time-to-answer-fixtures-1
• artifact sha256 67f9b78b4e358c77d3980fbc8752dbcd33141e1898bb9112a9b9312224eaf9d2
• stored outside the repo: /srv/jonas/evidence/dockermap/time-to-answer-baseline-1.json (+ -environment.json, -summary.txt)
• raw samples only; every summary recomputed at review time via perf:summarize

Environment fingerprint: linux-x86_64-dedicated / cpus-16vcpu / ubuntu-26.04 / kernel 7.0.0-31-generic / Node 22.23.2 / rustc 1.88.0 / Docker 29.8.1 / Chromium 1.61.0 (flags pinned) / system-default fonts / production build / effective ssePollIntervalMs = 2000 (the API's own default, never overridden).

4. Interpretation (median of three run p95, ms)

stage	25	100	250
daemon start → listener	41.99	109.84	254.00
listener → first Docker model	12.71	59.90	165.76
Docker observation	7.26	9.10	13.07
Compose projection	1.15	1.18	1.25
publication → Node observation	1307.47	636.12	1623.47
notification → coherent model	12.70	20.60	34.60
coherent → useful render	12.70	20.60	34.60
buildModel()	0.50	0.90	1.50
findings derivation	0.07	0.20	0.92
legacy topology layout	2.10	25.80	174.40
Cmd-K open + query	55.10	45.40	34.90
production bundle load	49.30	59.30	50.20

Scenarios: provider-only revision change observation 1964.50; docker topology change 701.60; slow bounded Compose projection 8.46; unavailable optional provider 1918.03.

Top three contributors

1. Publication → Node observation (the poll floor) — 87 / 64 / 68 % of summed stage medians, spread 0.42–1976 ms. That is a phase-of-poll distribution, explicitly not network latency. #333 child: Add revision-scoped selective refresh and remove avoidable SSE polling latency #337 owns removing it.
2. Cold start — process bring-up plus first publication; ~55 / 170 / 420 ms at 25 / 100 / 250.
3. Legacy Home topology layout at scale — 174 ms at 250 containers, scaling super-linearly (10× containers → 83× layout). #333 child: Remove avoidable frontend work from Home, topology and route loading #338 owns that decision.

Compose contribution, measured: 1.15 / 1.18 / 1.25 ms on the reference fixtures, 8.46 ms on the deliberately large bounded project, while it still executes inside the Docker publication budget. Nothing is decoupled here — #336 owns the change, and this baseline says the win there is real but small, not dominant. #336 should be judged against these numbers rather than an assumed large win.

Browser model work is not a bottleneck: buildModel 0.5–1.5 ms, findings derivation 0.07–0.92 ms, notification→render 12.7–34.6 ms.

High variance: publicationToNodeObservationMs (min 0.42–396 ms vs max ~1976 ms). Also daemonStartToListenerMs and commandQueryMs (2–4× spread). Stable: layout, bundle, Docker observation.

What these numbers do not prove: the fixture daemon is deterministic and local (so Docker observation says nothing about a real engine/host); daemon, API, web and Chromium all ran on one runner over loopback (no network claim); Compose trees are synthetic (40 / 400 services); fixtures are synthetic topologies; one pinned runner class.

5. Promotion/regression gate

timeToAnswerPromotion.test.ts — 24 RED-checks, all failing for evidence reasons: mismatched runner class / CPU / OS image / kernel / Node / Rust / Chromium / Docker revision / fixture revision / SSE poll interval / font environment / browser flags / build mode; missing stage; undeclared stage; malformed sample count; wrong run count; negative or non-finite sample; supplied or fabricated summary; unknown baseline id; candidate above max(baseline × 1.25, baseline + 2 ms); plus a compatible candidate passing and boundary cases just inside and just outside the budget.

6. Production-isolation proof

productionIsolation.test.mjs asserts the shipped bundle contains none of __dockermapProbe, __dockermapBench, __dockermapBenchHelpers, measureModelAcceptance, browserProbe, benchVite, perf:time-to-answer; that no production source or build script reaches the benchmark; that the daemon hook requires an absolute path, is enabled only by DOCKERMAP_BENCH_STAGE_TIMING_PATH and is referenced only by cache_refresh.rs / main.rs; and that the shipped bundle carries no analytics markers. No response, route, severity, cadence or generated schema changed.

7. Gates

• npm run check green.
• npm run test:perf 13/13 (8 fixture + 5 isolation); vitest src/lib/performance 30/30.
• Review routes: independent diff-scoped adversarial review dispatched on the explicit DeepSeek overflow lane (deleg_191f0357, 2 parallel read-only reviewers) — one contract/claims reviewer, one harness-vacuity/reproducibility reviewer.

Unavailable gates, recorded rather than waived: Architect×2 and the Sol/frontier FINAL gate are not runnable — the canonical router is terminal (ModelArk and OpenCode Go disabled in provider state; Sol reserve-gated; delegate_task refuses the Claude/Opus lane, whose launcher is session-limited). Dispositions are recorded in the delegation override receipt. The chatgpt-codex-connector bot is not active on this repository (PRs #327–#331 and #345 have zero bot reviews), so its silence is not a pass.

PR #346 stays draft until the review round is clean; #336 and #337 are untouched by this run.

[Joncallim]

Review round 1 — dispositions (PR #346 @ 71494a0)

Two independent read-only adversarial reviews ran on the explicit DeepSeek
overflow lane (deleg_191f0357): one contract/claim-truth reviewer, one
harness-vacuity/reproducibility reviewer.

Verdicts: REJECT (P1×2, P2×3, P3×7) and APPROVE-WITH-FOLLOW-UPS (P2×2, P3×5).
No P0. The validation machinery, the closed artifact and the recomputed numbers
were verified strong — 46/46 cells exactly 3 runs × 15 samples, all 46 doc rows
recompute exactly, no vacuous cell, no min==max run. Baseline 1 is therefore
NOT merged and must not gate #336/#337/#338 until the items below are fixed and
the baseline is re-captured.

Confirmed defects to remediate

1. P1 — commandQueryMs never measured query-to-results. It ended on
   "any li exists", and the palette renders the full unfiltered list
   synchronously on open, so the check passes before React's filtered re-render —
   it would pass even with filtering entirely broken. Also 8080 matches nothing
   in the fixture topology. Fix: derive the representative query from the fixture
   (a port it actually publishes), require the filtered list to differ from the
   pre-query list and to contain a matching item, and fail closed otherwise.
2. P1 — the "random phase inside the interval" claim is false for its own
   data. Reference-25 samples all sit in a 66 ms band of a 2000 ms interval and
   the topology-change runs drift monotonically: the gap is set by the phase
   offset between the daemon's fixed 2 s loop and the API's fixed 2 s poll, locked
   by the harness's fixed startup sequence. Consequence: the headline
   "poll floor = 87/64/68 %" is partly harness-ordering dependent, and the doc's
   central distribution claim is wrong. Fix: jitter each trigger by a uniform
   sub-interval delay so samples actually describe the poll-wait distribution,
   and state the phase relationship and autocorrelation honestly in the doc.
3. P2 — bench-sink provenance mixed for stages 3/4/9. The 15 transient
   stage-1/2 probe daemons shared DOCKERMAP_BENCH_STAGE_TIMING_PATH, so the
   "warmed" samples are an uncontrolled interleave of cold-start first
   observations and resident steady-state passes. Fix: give probe daemons no
   sink.
4. P2 — baseline 1 is reproducible from no committed revision. The artifact
   pins sourceRevision: b6904d5, but the harness that produced it existed only
   as uncommitted working-tree changes (the 15-sample loops land in 71494a0).
   Fix: commit the harness before capturing, refuse to capture from a dirty tree,
   and record product and harness revision separately.
5. P2 — findingsDerivationMs is daemon-side work bucketed as browser-model,
   so the bucket's "where the time went" claim is wrong. Fix: bucket it as
   backend-collection and correct both docs.
6. P2 — stages 6 and 7 are byte-identical duplicates and unattributed. Stage 6
   ends on any DOM mutation (a spinner or status pill would qualify) and stage 7
   duplicates the same instant, carrying zero independent information. Fix:
   require a rendered-text change for stage 6, end stage 7 on a Home-content
   region repaint, and declare stage 7 not applicable to the two fixtures whose
   revisions need not repaint Home.
7. P2 — "21 cells" is wrong; the closed matrix and the artifact both have
   46 cells.
8. P3 — ssePollIntervalMs hardcoded rather than derived from or asserted
   against the API; the isolation "inert by default" check is string-based
   (sink()'s real env lookup has no behavioural test); the promotion RED-checks
   do not pin the median-of-three aggregation; --fixtures lacks the debug guard
   and is a guaranteed-waste footgun; the doc's variance ranges contradict its own
   table; dockerRevision pins a host dimension the fixture never exercises;
   scenario-fixture premises (provider-only advance, provider unavailability) are
   constructed but never asserted.

Verification the reviewers confirmed

• 46/46 matrix cells, exactly 3 runs × 15 finite non-negative samples; every doc
  row recomputes exactly from the raw samples; no run where min == max.
• Stage 5 genuinely includes the API's real 2 s poll and resolves the publication
  instant externally; stage 12 is a cold, uninstrumented context; fixture daemon
  is byte-deterministic and its control route changes only inventory; teardown is
  process-group based with per-cell socket/workdir/ports.
• packages/contracts and apps/api: zero changes — no generated schema, no
  HTTP response, no production behaviour change. The daemon hook is inert when
  unset and landed before this PR's base.
• Harness writes only gitignored build outputs and tmp paths; the daemon cannot
  reach the real Docker socket; the sole real-Docker contact is a read-only
  version query for environment pinning.

Next: remediate 1-8, commit the harness, re-capture baseline 1 (same pinned
environment), re-run the promotion/isolation gates, then a fresh verification
round on the new head before merge. Draft PR #346 stays draft; #336 and #337
remain untouched.

[Joncallim]

Review round 2 — dispositions (PR #346 @ c0baa05, artifact 38e0c650…)

Fresh independent review of the reworked head (deleg_c9f2f9ed), 2 read-only
reviewers, prior verdict explicitly disregarded.

Verdicts: APPROVE-WITH-FOLLOW-UPS (P0 none, P1 none, P2×2, P3×4) and REJECT
(P0 none, P1×2, P2×1, P3×6). Baseline 2 is not merged and still must not be
the performance authority until the items below are fixed and recaptured.

Confirmed fixed (both reviewers, at code level)

• Cmd-K now genuinely measures query-to-results: unfiltered snapshot taken
  first, success requires the list to change and still contain the token, and
  the token is fixture-derived and not self-fulfilling. Mutation that would make
  it vacuous: a reorder-only filter (see F5).
• Stage 5 is genuinely de-correlated: every single run now spans nearly the
  whole 2000 ms interval (reference-25 35.0–1991.2, reference-100 0.34–1869.1,
  reference-250 0.27–1920.1, provider-only 0.54–1954.0), medians ≈ 776–859 ms.
  Contract and docs correctly say only the measurement is de-correlated and the
  two fixed 2 s cycles are unchanged. Jitter adds no inclusion bias (the sleep
  precedes both timestamps) and the resulting phase is uniform.
• Shape/recomputation: 44 records (not 46 — see F3), every cell exactly
  3 runs × 15 samples, all 44 doc rows recompute exactly, bucket sums exact.
• Provenance: artifact names sourceRevision = harnessRevision =
  0714c870…, verified to exist and to contain the harness; the dirty-tree and
  revision guards fail closed and the debug hatch cannot emit an artifact.
• Premise assertions are real and non-vacuous (sorted container-id comparison
  before/after for provider-only; emptied PATH plus a non-fresh provider state
  assertion for unavailable-provider).
• Bucketing: findingsDerivationMs is backend-collection in the contract
  and in the baseline doc. Guard hardening: the dockerRevision
  informational change does not weaken the gate (14 other fields stay pinned, no
  measured stage touches the host daemon, rationale is RED-tested). Probe-sink
  separation is proven by the code path. No behavioural regression from the
  jitter.

Defects to fix before the next capture

1. Cold first observation inside the "warmed" window (P1/P2, both reviewers).
   main.rs:141 runs refresh_cache before the listener binds, so the daemon's
   first-ever observation lands in the bench sink; capture.ts slices the first
   15 lines and nearest-rank p95 of 15 is the maximum, so the published
   stage-3 numbers (7.91/9.36/15.34) are cold-start samples vs warmed medians of
   1.6/3.3/5.3 ms. Same contamination on the slow-Compose fixture's stage 4
   (p95 = first sample in all three runs) and a sign-flipped one on stage 9. Fix:
   discard the first observation per stage (slice [k..count+k)) and document the
   discard. This is the last remnant of the defect class the rework targeted, and
   it loosens that stage's promotion limit ~3×.
2. Daemon binary provenance (P1/P3). emit-metadata checks only that the
   release daemon exists; nothing binds it to sourceRevision, and stages
   1/2/3/4/5/9 all come from it. Fix: record the binary's sha256 in the
   environment, verify it at capture start, fail closed on mismatch.
3. Stage 7 is degenerate (P2). Stage-6 and stage-7 arrays are byte-identical
   in all 180 samples because the first text-changing commit always lands in Home
   for the four declared fixtures, so stage 7 carries no independent information.
   Decide and implement: either make it a genuinely distinct boundary (a fixture
   whose published change alters Home's content, measured from the coherent
   commit) or stop declaring it a separate stage.
4. Doc corrections (P3). 44 cells, not 46; the evidence doc's bucket table
   still lists findingsDerivationMs under browser-model; the promotion
   sentence must mention dockerRevision; the reproduction section must state the
   required checkout of the pinned revision; stage 6/7 coincidence must be
   explained where the table presents stage 7.
5. Cmd-K hardening (P3). Also assert the filtered item count is strictly less
   than the unfiltered count, so a reorder-only filter cannot satisfy it.
6. Disclosures (P3). The jitter does nothing for the two fixtures with no
   harness trigger (their spread comes from provider-schedule drift) —
   browserRevision records the Playwright package version rather than the
   Chromium build revision — crash teardown relies on process groups.

Next: fix 1-6, commit the harness, recapture baseline 2 (same pinned
environment), rerun all gates, then a third, fresh independent review before
merge. Draft PR #346 stays draft; no production behaviour changes; #336 and #337
remain untouched and must be judged against the corrected baseline.