Expose exact packet byte costs and adopt SDK fitting

.agents/pm/issues/pm-http-33ck.toon

@@ -2,10 +2,32 @@ id: pm-http-33ck
 title: Distinguish exact serialized byte costs from heuristic token estimates in difference packets
 description: "Tracks gh:unbraind/pm-http#5 (filed 2026-10-29, missed by the blind issue watch). src/core.ts computes tokens(value) as ceil(max(JSON, TOON UTF-8 bytes)/4) and reports it as estimatedTokens with its budget; the epic's bounded-difference-packet contract is met, but agent consumers cannot distinguish an enforceable serialized-byte ceiling from an approximate model-token cost (ceil(bytes/4) is neither a tokenizer count nor a guaranteed upper bound). Request: expose the exact packet byte costs alongside the estimate, and name the estimator explicitly in receipts/docs so a context budget claim cannot be read as a measured token count. Static proposal; no transport defect reproduced."
 type: Issue
-status: open
+status: in_progress
 priority: 2
 tags[3]: api,"gh:unbraind/pm-http#5",measurement
 created_at: "2026-11-02T09:17:27.180Z"
-updated_at: "2026-11-02T09:17:27.180Z"
+updated_at: "2026-11-06T15:46:40.919Z"
 author: pi-glm
+comments[5]{created_at,author,text}:
+  "2026-11-06T14:11:37.107Z",codex-sol,"Confirmed current main retains the local byte/4 estimator and independent packet fitting loop. Issue 5 remains applicable; companion G6 requests the SDK fitter with the existing skip policy. Plan: failing real no-git tracker tests, exact self-inclusive JSON/TOON byte accounting including newline, explicit heuristic budget naming, SDK fitting, boundary/revert proofs, documentation and full review loop. Installed SDK already exports the fitter, so no pin change is required."
+  "2026-11-06T14:16:32.947Z",codex-sol,"Failing-first proof on main: both initial real no-git tracker tests failed because exact byte fields were absent (measured JSON 2301 / TOON 1775 bytes for the Unicode/digest fixture). Fixed packets use the installed SDK fitter, skip admission, canonical estimation, self-inclusive renderer bytes, and typed safe fitting refusals. Targeted packet tests pass, including minimum receipts and decimal/budget boundaries. Typecheck, lint, docstring and unchanged duplication gates pass. Historical 817 and raw-body figures are explicitly scoped as heuristics; no current hosted or public data was accessed."
+  "2026-11-06T14:20:29.145Z",codex-sol,"Validation: packet linked test 4/4 passed; focused HTTP/command tests 29/29 passed. Behavioral revert proofs 5/5 passed in disposable copies: original main differencePacket fails all three packet regressions and native-host byte assertions; restoring skip admission, JSON newline accounting or SDK refusal mapping makes its corresponding regression pass after the intentional revert fails. npm typecheck, lint, docstring (96 declarations) and duplication (0 percent) passed at unchanged gates. The full npm gate is also running with isolated HOME and Git identity; CI will run full npm/Bun gates on the PR head. No tokenizer or end-to-end task-cost claim is made."
+  "2026-11-06T15:42:42.063Z",codex-sol,"Full isolated npm and Bun release gates passed: each ran 143/143 tests, 100 percent statements/branches/functions/lines across 20 configured source files and 2/2 packed npm/Bun investigations against the declared minimum SDK. Node 22/26 CI and CodeQL also passed on 12b71dc. The required explicit review remains queued with no findings received. Follow-up correction: pin the revert harness to historical main e2074b1, so its original-function proofs remain reproducible after the fix is merged; production implementation is unchanged. Re-running all five disposable proofs and targeted static gates before the correction push. PR: https://github.com/unbraind/pm-http/pull/21."
+  "2026-11-06T15:46:37.785Z",codex-sol,"Pinned-baseline correction verified: all five disposable revert proofs passed again; typecheck and lint passed. A real disposable git clone advanced origin/main to the fixed function: the pinned e2074b1 harness passed, while reverting the harness to moving origin/main failed its identity check. This proves the harness remains usable after main advances. Refreshed current main and independently verified its tracker history remains a byte prefix of this branch. Production source is byte-identical to the implementation that passed both full local release gates and prior-head CI. The correction push will receive fresh exact-head CI and the queued review."
+files[8]{path,scope}:
+  index.ts,project
+  src/commands.ts,project
+  src/core.ts,project
+  src/derive.ts,project
+  test/commands.test.ts,project
+  test/packet.test.ts,project
+  test/real-data.acceptance.ts,project
+  test/revert.acceptance.ts,project
+tests[2]{command,scope,provenance{author,created_at,source_kind,source_ref}}:
+  node --test --test-concurrency=2 test/packet.test.ts,project,codex-sol,"2026-11-06T14:16:42.421Z",local_mutation,exact-packet-byte-costs
+  node test/revert.acceptance.ts issue5-packet issue5-packet-host issue5-skip issue5-byte-framing issue5-sdk-refusals,project,codex-sol,"2026-11-06T14:20:25.211Z",local_mutation,exact-packet-byte-costs
+docs[3]{path,scope}:
+  ARCHITECTURE.md,project
+  docs/real-data.md,project
+  README.md,project
 body: ""

ARCHITECTURE.md

@@ -50,7 +50,7 @@ Contracts define status alternatives, required selected headers (redacted values
 
 For each probe, observations sort by timestamp and stable ID. Find the first contract violation and its preceding last satisfying observation. If no violation exists, compare the latest pair; absent predecessors remain explicit null identities. Compare status, outcome, body digest/length, total timing, retry receipts, selected headers and each structural shape field. Include source observation IDs/timestamps, verdict reasons and exact per-field differences. Unknown evidence never becomes last-good evidence. Independent missing rows identify each absent observation, contract or complete capture.
 
-The packet prioritizes missing-evidence rows, then deterministic per-probe difference rows. It estimates the larger actual JSON/TOON wire size, including newline and the self-inclusive estimate/receipt, as bytes divided by four rounded up. Rows that do not fit are omitted; receipts state exact omitted/returned counts and commands to restore the full diff or missing-evidence list. The estimate is conservative across these two serializations, not model-token exact. Source reads are complete even when packet rows are omitted.
+The packet prioritizes missing-evidence rows, then deterministic per-probe difference rows. The SDK `fitOutputEnvelope` owns whole-envelope convergence, work ceilings and row admission with `policy: 'skip'`; the package supplies its inert JSON/TOON renderers including the final newline. `bytes.json` and `bytes.toon` report exact self-inclusive UTF-8 lengths; `output_fit.output_bytes` is their maximum. To populate both lengths without another fixed-point loop, envelope construction subtracts each format's length difference from the SDK maximum. The accounting fields are scalar integers spelled identically in both formats, so that difference is invariant under their digit widths. The SDK converges and verifies the final maximum including all domain and SDK receipts. Its canonical default estimator gives `estimatedTokens = ceil(max(bytes.json,bytes.toon)/4)`, retained as a compatibility field and explicitly labeled heuristic units. This is no model-token measurement or upper-bound guarantee. Rows that do not fit are omitted; receipts state exact omitted/returned counts and commands to restore the full diff or missing-evidence list. Source reads remain complete. Typed infeasible, unstable and work-limit refusals produce no partially certified packet.
 
 ## Limits and backend boundary
 

CHANGELOG.md

@@ -11,6 +11,10 @@
 - Produce bounded difference packets and missing evidence ([pm-http-s2xn](https://github.com/unbraind/pm-http/blob/main/.agents/pm/features/pm-http-s2xn.toon))
 - Recover concurrent journaled sends ([pm-http-95t1](https://github.com/unbraind/pm-http/blob/main/.agents/pm/features/pm-http-95t1.toon))
 
+### Fixed
+
+- Distinguish exact serialized byte costs from heuristic token estimates in difference packets ([pm-http-33ck](https://github.com/unbraind/pm-http/blob/main/.agents/pm/issues/pm-http-33ck.toon))
+
 ### Other
 
 - Build task-scoped HTTP investigation context ([pm-http-76jn](https://github.com/unbraind/pm-http/blob/main/.agents/pm/epics/pm-http-76jn.toon))

README.md

@@ -107,7 +107,7 @@ const packet = await differencePacket(itemId, { ...options, budget: 1500 });
 const missing = await missingEvidence(itemId, options);
 ```
 
-`observations(item, options)` returns complete stored observations and a completeness receipt. `compare(item, {...options, probe?})` returns historical verdicts and reasons. `differencePacket(item, {...options, budget?})` returns a self-accounted bounded packet, including omitted-row counts and commands to restore full context. `missingEvidence(item, options)` names each missing observation, contract or complete response. `listProbes`, `preview`, `reviewProbe` and the input/record interfaces are also exported. `Options` supports an injected native `PmClient`, an AbortSignal, expected revision, stable operation ID, protocol-boundary and SDK transition observers.
+`observations(item, options)` returns complete stored observations and a completeness receipt. `compare(item, {...options, probe?})` returns historical verdicts and reasons. `differencePacket(item, {...options, budget?})` returns a self-accounted bounded packet with exact `bytes.json` and `bytes.toon` UTF-8 counts, including the final newline, all receipts, and the byte counts themselves. The SDK `output_fit` receipt records the maximum byte cost and retained source indices. Omitted-row counts and commands restore full context. `missingEvidence(item, options)` names each missing observation, contract or complete response. `listProbes`, `preview`, `reviewProbe` and the input/record interfaces are also exported. `Options` supports an injected native `PmClient`, an AbortSignal, expected revision, stable operation ID, protocol-boundary and SDK transition observers.
 
 ## Safety and limits
 
@@ -118,7 +118,7 @@ const missing = await missingEvidence(itemId, options);
 - Capture defaults: 64 KiB body, 64 headers, 5 seconds total across DNS, connection, TLS, body and redirects. Hard ceilings: 1 MiB body/request body, 128 response headers, 16 KiB parser headers, 60 seconds total. Caps produce capture omissions or uncertainty. No body decompression. Cancellation through SIGINT/AbortSignal leaves no partial observation; a cancelled operation requires a new explicit identity.
 - Response scalar values, text and property names are never stored. JSON properties use SHA-256 identities; only structural types remain. Headers retain a narrow set of canonical nonopaque semantics. Set-Cookie, authorization-like, arbitrary/token-shaped values and known credential echoes are redacted. Other headers retain no opaque values. Digests cover captured bytes and explicitly identify prefix truncation.
 - Shape admission examines at most 64 properties/items per level, 256 structural nodes, and 12 levels. Omissions never count as a satisfied contract. Response controls and instruction delimiters render as inert escaped data.
-- Each task holds at most 32 probes, 128 operations/observations and a 1 MiB state envelope. Move a continuing investigation to a new task when full. Packet budgets range from 256 to 1,000,000 estimated tokens; default 1,500. Estimation is the larger JSON/TOON UTF-8 size including receipt and newline, divided by four and rounded up; it is not a model tokenizer.
+- Each task holds at most 32 probes, 128 operations/observations and a 1 MiB state envelope. Move a continuing investigation to a new task when full. Packet `--budget` / SDK `budget` ranges from 256 to 1,000,000 **heuristic units**; default 1,500. One unit is four serialized UTF-8 bytes: `estimatedTokens` (the compatibility field name) is `ceil(max(bytes.json, bytes.toon)/4)`. Thus both complete rendered packets, including newline and every accounting field, are bounded by `4 * budget` bytes. This is neither a tokenizer measurement nor a guaranteed upper bound on model tokens. No tokenizer or model encoding is used. `receipt.budgetUnit` names these heuristic units. The SDK fitter retains whole rows in order, skipping oversized rows and considering later rows. It refuses an infeasible mandatory receipt or exhausted fitting work with an actionable typed error.
 
 Boundary declarations are authorization by the operator, not network discovery or a monitoring schedule. Historical observations never assert current health. The tool does not retain raw bodies, excerpts, cookies or credential values, and does not load credential files.
 
@@ -128,7 +128,7 @@ All probe, send, contract, compare, diff, missing and SDK operations work in an
 
 ## Evidence and development
 
-See [measured ten-request investigation](docs/real-data.md) and [architecture](ARCHITECTURE.md). The measured packet contains 817 estimated tokens versus 25,847 for 103,388 captured raw body bytes; ten observations, 90 header redactions, ten HTTP calls, 65,151 ms including SDK setup/recording. No credentials were sent, and the temporary tracker was deleted.
+See [measured ten-request investigation](docs/real-data.md) and [architecture](ARCHITECTURE.md). The historical packet measured 817 bytes/4 heuristic units versus 25,847 body-only heuristic units for 103,388 captured raw body bytes; these are not model-token counts or an end-to-end cost comparison. Exact renderer bytes were not retained for that historical run, and the new receipt changes packet sizes. The run recorded ten observations, 90 header redactions, ten HTTP calls, 65,151 ms including SDK setup/recording. No credentials were sent, and the temporary tracker was deleted.
 
 ```sh
 npm run release:check

docs/real-data.md

@@ -7,10 +7,14 @@ Measured 2026-10-29T00:58:16.455Z. Ten explicit read-only requests in a temporar
 | Observations | 10 |
 | HTTP calls | 10 (2 public, 8 fixture) |
 | Captured raw response body bytes | 103388 |
-| Raw body estimated tokens (bytes/4, rounded up) | 25847 |
-| Difference packet estimated tokens | 817 |
+| Raw body heuristic units (bytes/4, rounded up) | 25847 |
+| Historical packet heuristic units (bytes/4, rounded up) | 817 |
 | Packet omitted rows | 0 |
 | Header redactions | 90 |
 | Total elapsed milliseconds, including SDK setup and recording | 65151 |
 
 Packet costs include the omission receipt and the larger serialized JSON/TOON format. Raw cost includes response bodies only, excluding headers and transcript/request overhead. Body values and keys are never retained. Historical captures are not current health. The incident changes from a successful numeric revision to a failing string revision; the packet identifies the first bad and preceding last good observations. Missing evidence: [{"probe":"pm-evidence","evidence":"complete-response"},{"probe":"pm-hypothesis","evidence":"complete-response"},{"probe":"pending","evidence":"observation"},{"probe":"pending","evidence":"contract"}]. Public schemas wider than 64 properties produce explicit structural omissions.
+
+The historical 817 figure is a byte-derived heuristic, not a tokenizer count or guaranteed model-token bound. Exact JSON and TOON lengths were not saved, so they cannot be reconstructed from that rounded value. The SDK fitter and new accounting fields change current packet costs; this table is an immutable historical measurement, not a current-size claim.
+
+Successful-task scope: this SDK harness performs tracker initialization and task creation, four probe declarations, three contract declarations, ten explicit `send` operations (two public and eight fixture observations), then `observations`, `differencePacket`, and `missingEvidence` reads. Native SDK recording/journal receipts, request/response metadata, setup returns, the observation-list and missing-evidence responses, and this report are excluded from the body-only baseline. The packet figure includes only its own historical receipt and framing. No automatic request retries are implemented, and this run recorded ten calls for ten sends; no recovery/retry transcript or all-command serialized costs were measured. Consequently these numbers do not establish an end-to-end token reduction. Public responses are not re-fetched for this correction.

index.ts

@@ -19,7 +19,7 @@ export const setContract = guarded(api.setContract);
 export const observations = guarded(derive.observations);
 /** Compare stored observations to current explicit contracts. */
 export const compare = guarded(derive.compare);
-/** Produce historical differences under a self-inclusive token budget. */
+/** Produce historical differences under a self-inclusive heuristic-unit budget with exact JSON/TOON UTF-8 byte costs. */
 export const differencePacket = guarded(derive.differencePacket);
 /** Enumerate exact outstanding evidence requests. */
 export const missingEvidence = guarded(derive.missingEvidence);

src/commands.ts

@@ -4,7 +4,7 @@ import { defineCommand } from '@unbrained/pm-cli/sdk/authoring';
 import type { CommandHandlerContext, ExtensionApi, FlagDefinition } from '@unbrained/pm-cli/sdk/authoring';
 import { addProbe, listProbes, preview, reviewProbe, send, setContract } from './api.ts';
 import { compare, differencePacket, missingEvidence } from './derive.ts';
-import { HttpError, requireInput, wire } from './core.ts';
+import { HttpError, failure, requireInput, wire } from './core.ts';
 import { FIELDS } from './store.ts';
 
 /** Reject absent or array operands without echoing untrusted data. */
@@ -15,7 +15,7 @@ function texts(value: unknown): string[] { return value === undefined ? [] : (Ar
 function number(value: unknown): number | undefined { if (value === undefined) return undefined; requireInput(typeof value === 'string' && /^\d+$/.test(value), 'Numeric flag requires a nonnegative integer'); const parsed = Number(value); requireInput(Number.isSafeInteger(parsed), 'Numeric flag exceeds the safe integer range'); return parsed; }
 /** Convert one refusal into the native host's safe expected-error contract. */
 function expected(error: unknown) {
-  const refusal = error instanceof HttpError ? error : new HttpError('sdk', 'HTTP context command failed', 'Check tracker health, schema file and access before retrying');
+  const refusal = failure(error);
   return createPmCliExpectedError(`http ${refusal.code}: ${refusal.message}. Remedy: ${refusal.remedy}`, { exitCode: refusal.code === 'cancelled' ? 130 : refusal.code === 'conflict' ? 3 : 2, context: { code: refusal.code, ...refusal.details, required: refusal.remedy, recovery: { attempted_command: 'pm http [arguments omitted]', normalized_args: [], provided_fields: [] } } });
 }
 /** Execute through the native host renderer and structured expected-error boundary. */