Prove pinned HTTPS SNI on a loopback handshake

DECISIONS.md

@@ -162,6 +162,19 @@ Fix: each rendered field is scored on its own. A fixed not-run marker in the sam
 ### 2026-08-19: screen --help assertions must ignore Rich markup
 CI `checks` on 3.11-3.13 went red from the cache UX PR (`test_screen_help_includes_refresh`) while ruff and mypy stayed green. Rich styles a long option as two bold spans (`-` then `-refresh`), so a raw `--refresh` substring is missing from colored help on the runner even though the flag is present. Local pytest without `color=True` did not insert those spans. The test now uses the existing `flat_output` helper, which already exists for width-independent CLI wording, and invokes help with color on so the CI shape is what the assertion sees.
 
+### 2026-08-19: hermetic TLS SNI pin; keep the httpx pool assignment fail-closed
+The site pin still installs a custom httpcore `NetworkBackend` by assigning `pool._network_backend` after construction. The steps are `pool = getattr(self, "_pool", None)`, `isinstance(pool, httpcore.ConnectionPool)`, then that assignment. That is a private coupling.
+
+Why it exists: installed `httpx.HTTPTransport.__init__` (httpx 0.28.1) has no `network_backend` parameter. Its parameters are `verify`, `cert`, `trust_env`, `http1`, `http2`, `limits`, `proxy`, `uds`, `local_address`, `retries`, and `socket_options`. It constructs `httpcore.ConnectionPool` internally and does not pass a backend in. Installed `httpcore.ConnectionPool.__init__` (httpcore 1.0.9) does accept `network_backend`, but that hook is not reachable through httpx's public constructor. Pinning has to live below Host and TLS SNI so those stay on the origin hostname (httpcore `start_tls` uses `server_hostname` from the origin host). pyproject lower bounds remain `httpx>=0.27` and `httpcore>=1.0`. There is no public hook in the installed httpx signature. This entry cites those signatures, not a GitHub issue.
+
+Fail-closed: if `_pool` is missing or is not a `ConnectionPool`, constructing `_PinnedTransport` raises `RuntimeError` and the site stage refuses to run rather than fetching unpinned.
+
+Chosen path: keep the private assignment, the isinstance guard, and the hermetic HTTP Host-header plus TLS SNI tests, until httpx grows a public hook. Then switch and delete the private assign.
+
+A loopback HTTPS handshake now proves server-observed SNI. The test binds `127.0.0.1` only, uses a fictional hostname, a runtime openssl cert with `SAN DNS:hostname`, and client `verify=` against that cert (never `verify=False`). `_PinnedTransport` forwards constructor kwargs to `HTTPTransport` so the test can pass `verify=`. Production `_SiteFetcher` still constructs `_PinnedTransport(pinner)` and keeps default verify-on against system CAs.
+
+Alternatives considered and rejected this sprint: dropping httpx for hand-rolled TLS (more surface), pinning an upper httpx bound with no public-API evidence (churn without a hook), adding a third-party TLS test extra (`cryptography` or `trustme`).
+
 ## Section 16 verification log
 
 Findings are recorded here as verification completes, each with source URL and retrieval date.

FUTURE.md

@@ -16,8 +16,7 @@ Ideas that are out of scope for v0.1. The non-goals in ARCHITECTURE.md section 1
 Deferred from the pre-launch hardening review:
 
 - Residual: unseen future model phrasing can still miss the stage-honesty phrase set, and an honest gap sentence that uses a clean-claim substring without a not-run marker still fails closed. Observed not-run miss classes are in the tuples; a same-field not-run marker suppresses a clean-claim substring. The set stays deliberately narrow so honest gap sentences still pass.
-- Add a hermetic TLS test for SNI preservation through the pinned network backend. Preservation is proven by design (httpcore derives server_hostname from the origin, which the pin never touches) and by a plain-HTTP Host-header test, but no HTTPS handshake test exists yet.
-- Revisit the httpx-internals coupling in the pinned transport if httpx changes its connection-pool shape. It is pinned to the tested versions and fails closed (refuses to run the site stage) otherwise; a public-API path would remove the coupling.
+- Switch the pinned transport off the private httpx `_pool._network_backend` assignment when `HTTPTransport` grows a public constructor hook for a custom httpcore `NetworkBackend`. Until then the assignment stays, the isinstance guard fails closed, and the hermetic HTTP and TLS tests hold the wiring. See the 2026-08-19 DECISIONS entry.
 
 Ideas deferred from the weekend 3 build and review:
 

HANDOFF.md

@@ -8,7 +8,7 @@ The repository is at github.com/samrusani/coldscreen. Branch `main` is current a
 
 A CLI that turns a UK company name into a first-pass screening memo built entirely from public sources, with every finding traceable to evidence. It is not due diligence. It is the screen that decides whether due diligence is worth anyone's time. The same pipeline is also an MCP stdio server, so the screen runs inside agent workflows without a second implementation of it.
 
-Current state: 734 tests, 28 modules, roughly 9,600 lines of source, green on Python 3.11 through 3.13. All three milestone success tests passed, two of them against the live Companies House API. Feature-complete for v0.1; not yet published to PyPI. Rubric 0.3 adds a mechanical R4 floor for origin-year contradictions ("operating since 2015" against incorporation in 2019), so that class of claims-bearing case now anchors unconditionally. Cache UX (`--refresh`, `coldscreen cache path|clear|stats`) landed after charges pagination. The stage-honesty phrase set grew from observed not-run lies; the gate is still a substring check, still sanctions and media only, and still arms only when those stages are recorded not run or failed.
+Current state: 736 tests, 28 modules, roughly 9,600 lines of source, green on Python 3.11 through 3.13. All three milestone success tests passed, two of them against the live Companies House API. Feature-complete for v0.1; not yet published to PyPI. Rubric 0.3 adds a mechanical R4 floor for origin-year contradictions ("operating since 2015" against incorporation in 2019), so that class of claims-bearing case now anchors unconditionally. Cache UX (`--refresh`, `coldscreen cache path|clear|stats`) landed after charges pagination. The stage-honesty phrase set grew from observed not-run lies; the gate is still a substring check, still sanctions and media only, and still arms only when those stages are recorded not run or failed. SNI through the pinned backend is proven by a loopback HTTPS handshake; the httpx pool assignment stays fail-closed by choice.
 
 ## The five non-negotiables
 
@@ -78,10 +78,9 @@ Findings this loop caught that the test suite did not: pagination that silently
 
 FUTURE.md holds remaining items. My recommended ordering:
 
-1. **A hermetic TLS test for SNI preservation** through the pinned network backend, and a plan for the httpx internals coupling in `site.py` (it fails closed if httpx changes shape, but it is coupled).
-2. Registry adapters for other jurisdictions. Design the adapter interface when a second registry forces it, not before.
+1. Registry adapters for other jurisdictions. Design the adapter interface when a second registry forces it, not before.
 
-The stage-honesty phrase set grew from observed not-run lies (PEP-first order, evidence-of phrasing, and the `ran and returned no` leak). It is still a substring check, still sanctions and media only, and still arms only when those stages are recorded not run or failed. A clean-claim substring in the same field as a not-run marker (`not performed`, `did not run`) is not treated as a clean result; a marker in another field does not excuse a lying narrative; citing SAN-000 is not a marker. Unseen future phrasing remains a residual by design. The five non-negotiables above are untouched.
+The hermetic TLS SNI test now proves the pinned backend preserves server-observed SNI on a loopback HTTPS handshake, not only the Host header on plain HTTP. The httpx `_pool._network_backend` assignment stays fail-closed by choice until a public constructor hook exists; see the dated DECISIONS.md entry. The stage-honesty phrase set grew from observed not-run lies (PEP-first order, evidence-of phrasing, and the `ran and returned no` leak). It is still a substring check, still sanctions and media only, and still arms only when those stages are recorded not run or failed. A clean-claim substring in the same field as a not-run marker (`not performed`, `did not run`) is not treated as a clean result; a marker in another field does not excuse a lying narrative; citing SAN-000 is not a marker. Unseen future phrasing remains a residual by design. The five non-negotiables above are untouched.
 
 ## Known open items that are not code
 

src/coldscreen/site.py

@@ -64,6 +64,7 @@
 from dataclasses import dataclass, field
 from datetime import UTC, datetime
 from html.parser import HTMLParser
+from typing import Any
 from urllib.parse import urljoin, urlsplit
 from urllib.robotparser import RobotFileParser
 
@@ -470,13 +471,15 @@ class _PinnedTransport(httpx.HTTPTransport):
     transport's connection pool to install it. The isinstance guard fails
     closed if httpx internals ever change shape: the site stage must not
     run without connection pinning, and the hermetic pinning tests exercise
-    this wiring against a real socket.
+    this wiring against a real socket. Extra keyword arguments are
+    forwarded to HTTPTransport so a test can pass verify=; production
+    construction keeps the default verify-on against system CAs.
     """
 
-    def __init__(self, pinner: _HostPinner) -> None:
-        super().__init__()
+    def __init__(self, pinner: _HostPinner, **kwargs: Any) -> None:
+        super().__init__(**kwargs)
         pool = getattr(self, "_pool", None)
-        if not isinstance(pool, httpcore.ConnectionPool):  # pragma: no cover - httpx drift
+        if not isinstance(pool, httpcore.ConnectionPool):
             raise RuntimeError(
                 "httpx internals changed: the pinned network backend cannot be"
                 " installed, so the site stage refuses to run without its"

tests/test_site.py

@@ -7,18 +7,23 @@
 path, or a cross-host redirect target, must never be fetched at all).
 Resolution is always a fake resolver: no real DNS in tests, enforced by the
 conftest autouse guard. The pinning tests at the bottom run against a real
-loopback HTTP server with sockets re-enabled for 127.0.0.1 only, because
-the property under test lives below the mock layer: the TCP connection
-must go to the address the pinner validated, never to a second resolution.
+loopback HTTP or HTTPS server with sockets re-enabled for 127.0.0.1 only,
+because the property under test lives below the mock layer: the TCP
+connection must go to the address the pinner validated, never to a second
+resolution, and TLS SNI must stay on the origin hostname.
 """
 
 from __future__ import annotations
 
 import json
+import shutil
 import socket
+import ssl
+import subprocess
 import threading
 from datetime import UTC, datetime
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
 from typing import cast
 
 import httpx
@@ -489,8 +494,8 @@ def test_robots_redirecting_off_host_means_nothing_is_fetched(
 
 # -- connection pinning: check and connect share one resolution -----------------------
 #
-# These tests are hermetic but real: a loopback HTTP server, sockets
-# re-enabled for 127.0.0.1 only (any other connect is refused by
+# These tests are hermetic but real: a loopback HTTP or HTTPS server,
+# sockets re-enabled for 127.0.0.1 only (any other connect is refused by
 # pytest-socket before a packet leaves), and injected resolvers. They prove
 # the F1 property below the respx mock layer, where the TOCTOU lived.
 
@@ -516,6 +521,77 @@ def __init__(self, body: bytes) -> None:
         self.hits: list[tuple[str | None, str]] = []
 
 
+class _LoopbackTLSServer(_LoopbackServer):
+    """Same recorder as the HTTP pin server, with a TLS handshake."""
+
+    def __init__(self, body: bytes, ssl_context: ssl.SSLContext) -> None:
+        super().__init__(body)
+        self.socket = ssl_context.wrap_socket(self.socket, server_side=True)
+
+
+def _self_signed_hostname_cert(directory: Path, hostname: str) -> tuple[Path, Path]:
+    """Self-signed cert for hostname, generated into directory via openssl.
+
+    A temp config sets SAN DNS:hostname so the file works on Ubuntu OpenSSL
+    and macOS LibreSSL without relying on -addext.
+    """
+    openssl = shutil.which("openssl")
+    if openssl is None:
+        pytest.skip("openssl is required to generate the hermetic TLS test certificate")
+    key_path = directory / "tls-sni-key.pem"
+    cert_path = directory / "tls-sni-cert.pem"
+    config_path = directory / "tls-sni-openssl.cnf"
+    config_path.write_text(
+        (
+            "[req]\n"
+            "default_bits = 2048\n"
+            "prompt = no\n"
+            "default_md = sha256\n"
+            "distinguished_name = dn\n"
+            "x509_extensions = v3_ext\n"
+            "req_extensions = v3_ext\n"
+            "\n"
+            "[dn]\n"
+            f"CN = {hostname}\n"
+            "\n"
+            "[v3_ext]\n"
+            f"subjectAltName = DNS:{hostname}\n"
+            "basicConstraints = CA:TRUE\n"
+        ),
+        encoding="utf-8",
+    )
+    completed = subprocess.run(
+        [
+            openssl,
+            "req",
+            "-x509",
+            "-newkey",
+            "rsa:2048",
+            "-nodes",
+            "-keyout",
+            str(key_path),
+            "-out",
+            str(cert_path),
+            "-days",
+            "1",
+            "-config",
+            str(config_path),
+        ],
+        check=False,
+        capture_output=True,
+        text=True,
+        timeout=30,
+    )
+    if completed.returncode != 0:
+        message = completed.stderr.strip() or completed.stdout.strip() or str(completed.returncode)
+        raise RuntimeError(
+            f"openssl failed to generate the hermetic TLS test certificate: {message}"
+        )
+    if not cert_path.is_file() or not key_path.is_file():
+        raise RuntimeError("openssl did not write the hermetic TLS test certificate files")
+    return cert_path, key_path
+
+
 def test_connect_time_enforcement_blocks_a_host_the_precheck_never_saw() -> None:
     """The pinner verdict is enforced at connect, not only at the polite
     pre-check: a request straight through the transport is refused before
@@ -529,6 +605,20 @@ def test_connect_time_enforcement_blocks_a_host_the_precheck_never_saw() -> None
         client.close()
 
 
+def test_pinned_transport_fails_closed_when_pool_is_not_a_connection_pool(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """If httpx no longer exposes a ConnectionPool on _pool, refuse to run."""
+
+    def broken_init(self: httpx.HTTPTransport, **_kwargs: object) -> None:
+        self._pool = object()  # type: ignore[assignment]
+
+    monkeypatch.setattr(httpx.HTTPTransport, "__init__", broken_init)
+    pinner = _HostPinner(resolver=lambda host: [PUBLIC_TEST_ADDRESS])
+    with pytest.raises(RuntimeError, match="httpx internals changed"):
+        _PinnedTransport(pinner)
+
+
 @pytest.mark.enable_socket
 @pytest.mark.allow_hosts(["127.0.0.1"])
 def test_pinned_transport_dials_the_validated_address_and_keeps_the_host_header() -> None:
@@ -581,6 +671,64 @@ def spying_getaddrinfo(host: object, *args: object, **kwargs: object) -> object:
     assert "pinned-host.example" not in getaddrinfo_hosts
 
 
+@pytest.mark.enable_socket
+@pytest.mark.allow_hosts(["127.0.0.1"])
+def test_pinned_transport_preserves_tls_sni_on_the_origin_hostname(tmp_path: Path) -> None:
+    """HTTPS pinning proof: TCP goes to the validated address, and the TLS
+    ClientHello SNI the server observes is the fictional hostname, not the
+    pinned IP. Handshake verifies against a cert for that name."""
+    hostname = "sni-pin.example"
+    cert_path, key_path = _self_signed_hostname_cert(tmp_path, hostname)
+    observed_sni: list[str | None] = []
+
+    def capture_sni(_sock: object, server_name: str | None, _ctx: object) -> int | None:
+        observed_sni.append(server_name)
+        return None
+
+    server_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+    server_ctx.load_cert_chain(certfile=str(cert_path), keyfile=str(key_path))
+    server_ctx.set_servername_callback(capture_sni)
+
+    server = _LoopbackTLSServer(b"PINNED-TLS-SNI-OK", server_ctx)
+    thread = threading.Thread(target=server.serve_forever, daemon=True)
+    thread.start()
+    port = server.server_address[1]
+    resolver_calls: list[str] = []
+
+    def resolver(host: str) -> list[str]:
+        resolver_calls.append(host)
+        return ["127.0.0.1"]
+
+    getaddrinfo_hosts: list[object] = []
+    real_getaddrinfo = socket.getaddrinfo
+
+    def spying_getaddrinfo(host: object, *args: object, **kwargs: object) -> object:
+        getaddrinfo_hosts.append(host)
+        return real_getaddrinfo(host, *args, **kwargs)  # type: ignore[arg-type]
+
+    pinner = _HostPinner(resolver=resolver, classifier=lambda address: False)
+    verify_ctx = ssl.create_default_context(cafile=str(cert_path))
+    client = httpx.Client(
+        transport=_PinnedTransport(pinner, verify=verify_ctx),
+        timeout=5.0,
+    )
+    try:
+        socket.getaddrinfo = spying_getaddrinfo  # type: ignore[assignment]
+        response = client.get(f"https://{hostname}:{port}/hello")
+    finally:
+        socket.getaddrinfo = real_getaddrinfo
+        client.close()
+        server.shutdown()
+        server.server_close()
+    assert response.status_code == 200
+    assert response.text == "PINNED-TLS-SNI-OK"
+    assert server.hits == [(f"{hostname}:{port}", "/hello")]
+    assert observed_sni == [hostname]
+    assert "127.0.0.1" not in observed_sni
+    assert resolver_calls == [hostname]
+    assert hostname not in getaddrinfo_hosts
+
+
 @pytest.mark.enable_socket
 @pytest.mark.allow_hosts(["127.0.0.1"])
 def test_dns_rebinding_cannot_deliver_an_internal_body(