Hex-escape the byte a decode error names, and drop the asyncio figures no table here carries

A DecodeError message spliced a raw input byte into the rejected ack, so a malformed
frame produced an ack the Python poller could not decode as UTF-8. Both splice sites now
render the byte as hex, with a test that feeds the three frames that reached it.

The 5.0x and 1.2us figures appear in no table and no CSV in this repository, so the
headline is now the uvloop numbers that are committed. Also: the v2 golden frame asserts
the two later fields it was missing, the 84/5 percent split is gone because both inputs
round to 100ns, and the second benchmark run is named as unrecorded instead of checkable.

Author: anaborne

README.md

@@ -8,16 +8,16 @@ against the same wire protocol, so the Python poller drives either one and the t
 each other in the same run on the same machine.
 
 The Python poller drove this executor across 4400 frames with `poller_client.py` unmodified, every
-frame accepted and no telemetry row dropped. Executor-side `wake_recv` dropped by a factor of 2.4 to
-5.0 depending on the event loop and the percentile, 0.0045ms to 0.0016ms at p50 under uvloop. The
-caveat is that the two configurations differ by more than language: the Python baseline runs the
-poller and the executor on one event loop and the C++ configuration spawns a second process, so
-every ratio here is an upper bound on what the rewrite bought. [`RESULTS.md`](RESULTS.md) is where
-those numbers live and it carries the confound in full.
+frame accepted and no telemetry row dropped. Executor-side `wake_recv` dropped by a factor of 2.4 at
+p99 and 2.8 at p50 under uvloop, 0.0045ms to 0.0016ms. The caveat is that the two configurations
+differ by more than language: the Python baseline runs the poller and the executor on one event loop
+and the C++ configuration spawns a second process, so every ratio here is an upper bound on what the
+rewrite bought. [`RESULTS.md`](RESULTS.md) is where those numbers live and it carries the confound
+in full.
 
 The pre-registered expectation in [`BENCHMARK.md`](BENCHMARK.md) section 6 was written before any
 code existed, and it was wrong in both halves. It predicted `wake_recv` would fall by roughly an
-order of magnitude, and 2.4x to 5.0x is not that. It predicted end-to-end `wake_send` could not move
+order of magnitude, and 2.4x to 2.8x is not that. It predicted end-to-end `wake_send` could not move
 because the span brackets the executor with Python work on both sides, and `wake_send` halved,
 because the span ends at `drain()` and never contained the executor at all.
 
@@ -31,12 +31,14 @@ is a production A/B, and no number here will be presented as one.
 
 The frames the C++ is tested against were produced by running the Python, not by reading it. Ten of
 them: five `WakeMessage` shapes, the v1 through v3 legacy frames, and both `WakeAck` statuses. The
-C++ suite asserts field-for-field equality against each, and the legacy three assert that every
-field added since arrives at its Python default, two of which are refusals rather than permissions.
-The rejection cases are separate, in `test_protocol.cpp` and `test_executor.cpp`: an oversize length
+C++ suite asserts field-for-field equality against the five message shapes and the two acks. The v1
+frame asserts that every field added since arrives at its Python default, two of which are refusals
+rather than permissions; the v2 and v3 frames assert the fields added after each of them. The
+rejection cases are separate, in `test_protocol.cpp` and `test_executor.cpp`: an oversize length
 prefix closes the connection, and a malformed body produces a `rejected` ack whose reason names the
-field. The reason text is this port's own, where the Python sends `str(exc)`; the poller only logs
-it, and [`PORT-FIDELITY.md`](PORT-FIDELITY.md) records the difference.
+field when the failure is a missing or unexpected one, and names the parse error otherwise. The
+reason text is this port's own, where the Python sends `str(exc)`; the poller only logs it, and
+[`PORT-FIDELITY.md`](PORT-FIDELITY.md) records the difference.
 
 The encoder is byte identical to `orjson.dumps` over `WakeMessage` and `WakeAck`. That is a stricter
 claim than valid JSON, because `orjson` serializes a dataclass in field-declaration order, so the
@@ -77,13 +79,14 @@ n=2000 after 200 discarded warm-up iterations:
 | cpp-to-cpp | 0.0019 | 0.0020 | 0.0026 |
 
 Those bottom two rows are one executor measured from two different clients, and they should agree,
-because the executor does not know what is on the other end of the socket. Adding the asyncio-driven
-run, the same binary reports 1.2us, 1.6us and 1.9us across three clients. The 0.7us spread is the
-resolution of this measurement at this scale, and any claim about `wake_recv` finer than "between 1
-and 2 microseconds" is reading noise.
-
-Inside the C++ executor the span decomposes cleanly. `decode` is 84% of `wake_recv`, `encode_ack` is
-5%, and the remainder is the ack's `send`. The decoder is the only place with anything in it.
+because the executor does not know what is on the other end of the socket. The same binary reports
+1.6us and 1.9us across the two. The 0.3us spread is the resolution of this measurement at this
+scale, and any claim about `wake_recv` finer than "between 1 and 2 microseconds" is reading noise.
+
+Inside the C++ executor, `decode` measured standalone accounts for most of `wake_recv` and
+`encode_ack` for a small remainder, with the ack's `send` inside the difference. Both figures are
+rounded to 100ns and the two are separate measurements, so no percentage split should be read off
+them. The decoder is the only place with anything in it.
 
 The signer, benchmarked in the same session, turned out not to compare what it was built to
 compare. That row is 2.8x in this port's favour and it is not a language result: `cryptography`

RESULTS.md

@@ -22,8 +22,9 @@ average was 1.2 at the start of the run rather than the sub-1.0 that was wanted.
 
 What stands in for the reboot is a repeat. The C++ benchmark was run twice back to back and the two
 runs agree: `sign` within 1.3%, `roundtrip` within 3.8%, `wake_recv` within 5.3%, `decode` within
-12.5% on an absolute difference of 200ns. Only the first run was written to the CSV. A reader can
-check that agreement; nobody can check an assertion that the machine was quiet.
+12.5% on an absolute difference of 200ns. Only the first run was written to the CSV, so the
+percentages above are the only record of the second; nobody can check them, and nobody can check an
+assertion that the machine was quiet either.
 
 ## The three configurations
 
@@ -64,17 +65,16 @@ such spans:
 Section 6 said: "Executor-side `wake_recv` drops by roughly an order of magnitude, and end-to-end
 `wake_send` barely moves."
 
-**`wake_recv` did not drop by an order of magnitude.** It dropped by a factor of 2.4 to 5.0
-depending on the event loop and the percentile: 2.4x at p99 under uvloop, 5.0x at p99 under stock
-asyncio. The prediction was off by roughly half in log terms, and it was off in the direction that
-flattered the port.
+**`wake_recv` did not drop by an order of magnitude.** It dropped by a factor of 2.4 at p99 and 2.8
+at p50 under uvloop. The prediction was off by roughly half in log terms, and it was off in the
+direction that flattered the port.
 
 There is a floor under this, and it is worth naming because it bounds how much better the number
-could get. The same C++ executor, measured three ways, reports `wake_recv` at 1.2us, 1.6us and
-1.9us: driven by the asyncio poller, by the uvloop poller, and by the C++ client. Those three
-should be identical, because the executor does not know what is on the other end of the socket.
-They differ by 0.7us, which is the resolution of this measurement at this scale. Any claim about
-`wake_recv` finer than "between 1 and 2 microseconds" is reading noise.
+could get. The same C++ executor, measured two ways, reports `wake_recv` at 1.6us and 1.9us: driven
+by the uvloop poller and by the C++ client. Those two should be identical, because the executor does
+not know what is on the other end of the socket. They differ by 0.3us, which is the resolution of
+this measurement at this scale. Any claim about `wake_recv` finer than "between 1 and 2
+microseconds" is reading noise.
 
 **`wake_send` moved, and it should not have.** It halved at p50, 0.0257 to 0.0123 under uvloop.
 Section 6 predicted it would barely move, and the reasoning given was that the span brackets the
@@ -136,9 +136,11 @@ comparable to any number in the `wake_send` column. PORT-FIDELITY.md carries the
 
 ## What the run does support
 
-Inside the C++ executor the span decomposes cleanly. `decode` at 1.6us is 84% of `wake_recv` at
-1.9us; `encode_ack` at 0.1us is 5%; the remainder is the ack's `send`. If the executor's half of
-the wire is ever worth optimizing further, the decoder is the only place with anything in it.
+Inside the C++ executor, `decode` measured standalone at 1.6us accounts for most of `wake_recv` at
+1.9us and `encode_ack` at 0.1us for a small remainder, with the ack's `send` inside the difference.
+Both figures are rounded to 100ns and the two are separate measurements, so no percentage split
+should be read off them. If the executor's half of the wire is ever worth optimizing further, the
+decoder is the only place with anything in it.
 
 `sign` moved 0.3% across the executor swap, 0.9227 to 0.9256. That is the control working. Signing
 lives in `rest_client.py`, past the `dispatch()` boundary, so it is in neither executor process and
@@ -162,7 +164,7 @@ three rows in `history.csv` from before 2026-08-30 are not comparable to anythin
    be separated from the process separation.
 2. Both signers against one OpenSSL build, or the `sign` row dropped as a cross-language comparison
    and kept only as a per-library note.
-3. A measurement floor better than 0.7us if `wake_recv` is ever to be reported more precisely than
+3. A measurement floor better than 0.3us if `wake_recv` is ever to be reported more precisely than
    an order of magnitude band.
 
 None of these change the protocol result, and none of them are done here.

include/hotpath/clock.hpp

@@ -3,7 +3,7 @@
 // Ported from `runtime/clock.py`. The poller stamps `recv_ns` and this process measures against
 // it, so the two must agree on which monotonic clock they mean. `clock_domain()` is the token that
 // says so, and it has to be built the same way on both sides or every cross-process span is
-// silently dropped.
+// silently dropped. Nothing in this process consumes the token yet.
 
 #include <cstdint>
 #include <string>
@@ -14,7 +14,8 @@ namespace hotpath {
 
 [[nodiscard]] std::int64_t wall_clock_ms() noexcept;
 
-// Hostname and quantized boot epoch, as `gethostname():1787290000`. Computed once on first call.
+// Hostname and boot epoch floored to a 60-second bucket, as `gethostname():1787289960`. Computed
+// once on first call.
 [[nodiscard]] const std::string& clock_domain();
 
 }  // namespace hotpath

src/executor.cpp

@@ -200,7 +200,7 @@ void ExecutorServer::serve_connection(int fd) {
 
         // Stamped after the read returns, not before it. The read blocks until a frame arrives, so
         // a stamp taken ahead of it measures the gap between wakes rather than the cost of one. In
-        // the Python that mistake put `wake_recv` at 1181 ms against a sub-millisecond budget,
+        // the Python that mistake put `wake_recv` at 1186 ms against a sub-millisecond budget,
         // tracking wake inter-arrival time, and the burst-shaped benchmark never showed it.
         const std::int64_t started_at_ms = wall_clock_ms();
         const std::int64_t started_ns = monotonic_ns();

src/json.cpp

@@ -26,6 +26,15 @@ constexpr std::byte as_byte(char c) noexcept {
     return static_cast<std::byte>(static_cast<unsigned char>(c));
 }
 
+// A `DecodeError` message goes back out on the wire inside a rejected ack, and the frame that
+// produced it is not trusted to be UTF-8. A byte lifted out of one goes out as hex, because a raw
+// 0xFF spliced into the reason makes the whole ack undecodable to the poller.
+std::string hex_byte(char c) {
+    static constexpr std::string_view kByteHexDigits = "0123456789abcdef";
+    const auto value = static_cast<std::size_t>(static_cast<unsigned char>(c));
+    return std::string{kByteHexDigits[value >> 4U], kByteHexDigits[value & 0x0FU]};
+}
+
 // `std::string_view::compare(pos, ...)` throws when `pos` is past the end, which the parser never
 // does but the compiler cannot know, and one potentially-throwing call is enough to cost `at_null`
 // its noexcept.
@@ -251,7 +260,7 @@ bool Reader::expect(char c) {
     }
     if (seen != c) {
         return fail(DecodeErrorCode::MalformedJson,
-                    std::string("expected '") + c + "', got '" + seen + "'");
+                    std::string("expected '") + c + "', got byte 0x" + hex_byte(seen));
     }
     ++pos_;
     return true;
@@ -531,7 +540,8 @@ bool Reader::append_escape(std::string& out) {
         case 'u':
             return append_code_point(out);
         default:
-            return fail(DecodeErrorCode::MalformedJson, std::string("invalid escape: \\") + c);
+            return fail(DecodeErrorCode::MalformedJson,
+                        std::string("invalid escape: byte 0x") + hex_byte(c));
     }
 }
 

src/json.hpp

@@ -26,7 +26,7 @@
 
 namespace hotpath::json {
 
-// Longest output `format_double` can produce is 25 bytes, from a 17-digit mantissa laid out in
+// Longest output `format_double` can produce is 24 bytes, from a 17-digit mantissa laid out in
 // fixed notation at the smallest exponent that still uses it.
 inline constexpr std::size_t kDoubleBufferSize = 32;
 

tests/test_executor.cpp

@@ -230,6 +230,54 @@ hotpath::WakeAck ack_from(const std::vector<std::byte>& body) {
     return decoded.value();
 }
 
+// The poller calls `orjson.loads` on the whole ack frame before it reaches the `reason` it logs,
+// so a rejected ack has to be valid UTF-8 even when the frame that caused it was not. Strict, the
+// way CPython's decoder is: an overlong form is refused, and so is any code point in the surrogate
+// range or past U+10FFFF.
+bool is_valid_utf8(std::span<const std::byte> body) {
+    std::size_t i = 0;
+    while (i < body.size()) {
+        const auto lead = static_cast<unsigned char>(body[i]);
+        if (lead < 0x80U) {
+            ++i;
+            continue;
+        }
+        std::size_t extra = 0;
+        std::uint32_t code_point = 0;
+        if ((lead & 0xE0U) == 0xC0U) {
+            extra = 1;
+            code_point = lead & 0x1FU;
+        } else if ((lead & 0xF0U) == 0xE0U) {
+            extra = 2;
+            code_point = lead & 0x0FU;
+        } else if ((lead & 0xF8U) == 0xF0U) {
+            extra = 3;
+            code_point = lead & 0x07U;
+        } else {
+            return false;
+        }
+        if (i + extra >= body.size()) {
+            return false;
+        }
+        for (std::size_t k = 1; k <= extra; ++k) {
+            const auto continuation = static_cast<unsigned char>(body[i + k]);
+            if ((continuation & 0xC0U) != 0x80U) {
+                return false;
+            }
+            code_point = (code_point << 6U) | (continuation & 0x3FU);
+        }
+        const bool overlong = (extra == 1 && code_point < 0x80U) ||
+                              (extra == 2 && code_point < 0x800U) ||
+                              (extra == 3 && code_point < 0x10000U);
+        if (overlong || (code_point >= 0xD800U && code_point <= 0xDFFFU) ||
+            code_point > 0x10FFFFU) {
+            return false;
+        }
+        i += extra + 1;
+    }
+    return true;
+}
+
 }  // namespace
 
 TEST_CASE("a wake is acked with its own correlation id and then fired", "[executor]") {
@@ -331,6 +379,37 @@ TEST_CASE("a frame that does not decode is rejected and the connection survives"
     CHECK(harness.fires().size() == 1);
 }
 
+// Three frames that make the decoder build a reason string out of a byte from the frame itself.
+// Each byte goes out as hex, so the ack stays decodable to a poller that only wanted to log the
+// reason.
+TEST_CASE("a rejected ack is valid UTF-8 when the frame that caused it is not", "[executor]") {
+    Harness harness("reject-utf8");
+    const int client = connect_to(harness.socket_path());
+
+    const std::array<std::string, 3> bodies{
+        std::string("{\"schema_version\":4\xff}"),
+        std::string("{\"asset\"\xc3:1}"),
+        std::string("{\"asset\":\"\\\xff\"}"),
+    };
+
+    for (const std::string& body : bodies) {
+        INFO(body);
+        send_all(client, frame_of(body));
+        const std::vector<std::byte> rejection = read_frame(client);
+        REQUIRE_FALSE(rejection.empty());
+        CHECK(is_valid_utf8(rejection));
+        const hotpath::WakeAck ack = ack_from(rejection);
+        CHECK(ack.status == hotpath::WakeAckStatus::Rejected);
+        CHECK_FALSE(ack.reason.value_or("").empty());
+    }
+
+    ::close(client);
+    harness.join_after_disconnect();
+
+    CHECK(harness.stats().acks_rejected == 3);
+    CHECK(harness.fires().empty());
+}
+
 TEST_CASE("an oversize length prefix closes the connection", "[executor]") {
     Harness harness("oversize");
     const int client = connect_to(harness.socket_path());

tests/test_protocol_golden.cpp

@@ -204,6 +204,9 @@ TEST_CASE("a v2 frame decodes with an untradeable wire price", "[golden]") {
     CHECK(decoded.value().clock_domain == "host-a:1787290000");
     CHECK(decoded.value().wire_price_yes_dollars == 0.0);
     CHECK(decoded.value().exchange_index == -1);
+    CHECK(decoded.value().available_size_contracts == 0.0);
+    CHECK(decoded.value().price_ranges ==
+          std::vector<hotpath::PriceRange>{hotpath::kDefaultPriceRange});
     CHECK(decoded.value().correlation_group.empty());
 }