Add 52 public API operations, archive retrieval, and four live adapters

CLAUDE.md

@@ -3,7 +3,9 @@
 TinyERCOT preserves its legacy public Python API. The opt-in `tinyercot.catalog`
 module supplies offline metadata only. Read `docs/public-foundation.md` before
 adding current data capabilities. Read `docs/public-retrieval.md` for the opt-in
-four-endpoint registry client, annual file adapter, and ESR website feed.
+registry client, annual file adapter, and ESR website feed. Read
+`docs/public-metadata.md` for reusable discovery, product facades, and evidence
+accounting for the expanded public contracts.
 
 ## Local checks
 

docs/api-archive-installed-task-note.md

@@ -0,0 +1,33 @@
+# Installed archive verification
+
+Commands run on 2026-09-06 UTC from the current shared worktree unless noted:
+
+```bash
+uv build
+uv pip install --python /tmp/tinyercot-retrieval-wheel/bin/python --force-reinstall 'dist/tinyercot-0.2.2-py3-none-any.whl[files]'
+```
+
+Both passed. The probe ran with `/tmp` as its working directory:
+
+```bash
+/tmp/tinyercot-retrieval-wheel/bin/python -I /Users/kevin/.treehouse/tinyercot-1bad13/1/tinyercot/tools/probe_api_archives.py --live --credentials-file /Users/kevin/.treehouse/tinyercot-1bad13/1/tinyercot/.env --evidence /Users/kevin/.treehouse/tinyercot-1bad13/1/tinyercot/.local-evidence/catalog --output /Users/kevin/.treehouse/tinyercot-1bad13/1/tinyercot/.local-evidence/installed-api-archives --retired-history --download-oldest-bundle
+```
+
+Required checks passed with Python 3.14 from installed `site-packages`:
+116 root products; 278 DME documents; four typed exact-text rows per current and
+oldest cached DME ZIP; 103 DAM bundles; two first-page public retired-history
+listings. The optional oldest-bundle transfer returned `SourceUnavailableError`;
+its exact HTTP status was not retained. No bundle download or typed bundle rows
+are claimed. Original DME ZIPs were not downloaded again. Source receipts and
+selected document identities are checked in separately from ignored raw files.
+
+```bash
+uv run --extra files pytest -q tests/test_probe_api_archives.py tests/test_api_archives.py tests/test_api_bundles.py tests/test_resource_dme.py tests/test_rt_archives.py
+uv run ruff check tools/probe_api_archives.py tests/test_probe_api_archives.py
+uv run ruff format --check tools/probe_api_archives.py tests/test_probe_api_archives.py
+```
+
+The test command passed **48 tests**. Lint and format checks passed after the
+test import grouping was corrected. No legacy runtime files, exports, coverage
+records, secrets, or credential contents were edited or copied in this subtask.
+No commit, push, merge, or no-mistakes action was performed.

docs/api-archives.md

@@ -0,0 +1,127 @@
+# Selected public API archives
+
+`tinyercot.public.api_archives.APIArchiveClient` adds typed **document metadata**
+and one selected public ZIP download. It inherits explicit public-account
+credentials and bounded transport from `MetadataClient`. The generic transfer
+keeps `APIArchiveFile.row_schema` as `"unknown"`; the separate, optional
+`iter_resource_dme` decoder supports the verified NP3-988-ER CSV format below.
+
+```python
+from tinyercot.public import Credentials
+from tinyercot.public.api_archives import APIArchiveClient
+
+with APIArchiveClient(Credentials.from_env()) as client:
+    products, root_receipt = client.products()
+    page = client.archives("NP3-988-ER")
+    selected = page.documents[0]
+    file = client.download(selected)
+    assert file.row_schema == "unknown"
+```
+
+Calling `products()` establishes this instance's source-backed access gate and
+invalidates previous download selections. An archive request requires an exact
+product from that fetched root with explicit Public classification. Lifecycle is
+separate: inactive or audit-retired products are marked `retired-or-inactive`,
+and their historical public documents remain selectable when the current listing
+verifies availability. That does not reinstate a retired live service. Secure,
+Certified, missing classifications, unknown products, and unlisted document IDs
+cannot authorize downloads.
+
+The source preserves publication timestamps without offsets. They remain exact
+strings in `post_datetime`; the adapter does not invent UTC or market-time
+semantics. Listing counts refer to documents, never decoded data rows. The source
+advertised 278 NP3-988-ER archive records on one page during discovery. The adapter
+returns source counters and does not follow further pages automatically.
+Archive pagination remains deferred pending a verified query contract.
+
+Downloads use the
+[primary OpenAPI contract](https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api?export=true&api-version=2022-04-01-preview):
+`POST /archive/{emilId}/download` with JSON `{"docIds": [selected_id]}`. Selection
+is limited to one unchanged document returned by this client. The response byte
+budget, ZIP member count, expansion size, paths, and member integrity are checked;
+members are never extracted to local paths. The returned document identity is
+retained beside the receipt because a POST receipt URL alone cannot identify the
+selected document.
+
+The bounded source discovery POST selected document 1270779234 on 2026-09-06
+UTC. ERCOT returned 22,983 ZIP bytes containing one CSV member (212,038 expanded
+bytes). The source receipt and checked directory metadata are recorded in
+`docs/evidence/api-archive-download.json`. One additional bounded historical check
+selected document 1166614741, dated 2025-12-01, returning 21,892 ZIP bytes. Its
+matching seven-column header and source rows are pinned alongside the current
+sample in `tests/fixtures/public/resource-dme-excerpt.json`. This oldest listed
+observation does not prove stable retention or complete history.
+
+## Optional resource decision-making entity CSV rows
+
+```python
+from contextlib import closing
+from tinyercot.public.resource_dme import iter_resource_dme
+
+# file is one previously selected NP3-988-ER APIArchiveFile.
+with closing(iter_resource_dme(file, max_rows=None)) as records:
+    for record in records:
+        process(record.row.dme_duns)  # Original string, including leading zeros.
+```
+
+The exact source columns are OWNER RE, RESOURCE NAME, TYPE, SPLIT GEN RESOURCE,
+DME, DME DUNS, and RMR. Every cell remains a string: blanks, whitespace, quoting
+results, flags, and identifier zeros are preserved. No string is silently parsed
+as a number, boolean, or enumeration. `ResourceDmeRow` gives these observed
+columns explicit field names. Receipt, source product, document/member identity,
+UTF-8 decoding, ZIP expansion, header, and column checks fail explicitly on an
+unknown schema epoch. `max_rows=None` iterates the complete selected CSV;
+the default 1,000-row safeguard raises if more records remain.
+
+## Explicit historical bundles
+
+`client.bundles("NP4-190-CD")` returns a separate `APIBundlePage`. The actual
+source response advertised 103 monthly bundles with signed negative document
+identities, spanning the observed 2018-01 through 2026-08 labels. This is listing
+evidence, not a claim of uninterrupted monthly data. Bundle metadata excerpts
+and their source receipt are pinned in `tests/fixtures/public/api-bundle-excerpt.json`.
+
+`client.download_bundle(page.documents[0])` uses only
+`POST /bundle/{emilId}/download` with that one source ID. Ordinary archive
+documents cannot substitute for bundle identities, and a failed route never
+falls back to another service. The caller's HTTP byte budget remains in force;
+bundle defaults allow 512 ZIP members and 256 MB of advertised expansion.
+Bundle CSV member-row schemas remain unknown. An installed-wheel attempt to
+retrieve the oldest listed monthly bundle (-593019152, 2018-01) returned
+`SourceUnavailableError` within a 16 MB response ceiling and one attempt. The
+initial probe did not retain the exact HTTP status; no successful bundle bytes
+or decoded rows are claimed. This does not establish that every bundle is
+unavailable. There is no route fallback or repeated bundle request.
+
+## Installed-wheel evidence
+
+`docs/evidence/installed-api-archive-receipts.json` records real installed-client
+root metadata (116 products), the 278-document DME listing, the 103-document DAM
+bundle listing, and four typed DME rows from each of the two original ZIP caches.
+Cache decoding checks original document publication/selection identity and
+receipt hashes. The installed client did not download those files again.
+
+Two additional installed metadata requests verified public history listings for
+audit-retired NP4-179-CD and NP3-990-EX: 1,000 of 4,236 and 1,000 of 2,602
+document entries respectively. Their lifecycle remains `retired-or-inactive`.
+No document from either retired collection was downloaded. Further listing pages
+remain outside this adapter's verified pagination contract.
+
+The explicit probe is `tools/probe_api_archives.py`. It requires an isolated
+installed interpreter, `--live`, an existing mode-0600 `--credentials-file`,
+`--evidence` pointing at the original ignored catalog evidence, and `--output`.
+`--retired-history` adds only the two bounded listings. The separately opt-in
+`--download-oldest-bundle` selects one bundle, with no monthly row extraction.
+It refuses to overwrite an existing bundle output. The offline suite tests cache
+binding and never executes authenticated requests.
+
+`tests/fixtures/public/api-archive-excerpt.json` records the exact primary spec
+hash, public root receipt, one source product, archive-list receipt, and the first
+two unmodified archive records. Source pagination counters remain unchanged in
+the fixture; tests adjust them only when constructing a synthetic two-row page.
+Offline checks cover metadata drift, public access gates, known retirement,
+selection invalidation, JSON POST shape, untyped generic member data, malformed
+archives, and byte/ZIP budgets. The optional DME decoder has current/historical
+source-cell tests, identifier preservation, malformed schema tests, and complete
+iteration beyond the sampling budget. Other binary row schemas remain unsupported;
+these file adapters add no count to the typed REST data-path denominator.

docs/evidence/aggregate-dashboard-fixture-provenance.json

@@ -0,0 +1,43 @@
+{
+  "description": "Anonymous rolling outage and DC-tie website source captures; no historical date queries or continuous polling.",
+  "source_pages": [
+    {
+      "url": "https://www.ercot.com/gridmktinfo/dashboards/generationoutages",
+      "observed_route": "/api/1/services/read/dashboards/generation-outages.json"
+    },
+    {
+      "url": "https://www.ercot.com/gridmktinfo/dashboards/dctieflows",
+      "observed_route": "/api/1/services/read/dashboards/dc-tie-flows.json"
+    }
+  ],
+  "fixtures": [
+    {
+      "fixture": "tests/fixtures/public/generation-outages-live.json",
+      "fixture_sha256": "feec97543fe9b88cee73b32cd5419eee67a6bd588d4dd0f114f52a8af1be6933",
+      "source_receipt": {
+        "source_url": "https://www.ercot.com/api/1/services/read/dashboards/generation-outages.json",
+        "retrieved_at": "2026-09-06T00:34:49.810383+00:00",
+        "sha256": "93b58993e27f231852ea7fe9f91d078455c9e9e095c5e60a6f75ea4581494e9e",
+        "byte_count": 495264,
+        "status": 200
+      },
+      "transformation": "Retain first and last two observations in each current/previous section; retain all metadata and source categories; no value transformations.",
+      "authentication": "none; one anonymous snapshot request",
+      "observed_nullability": "No source nulls observed; models reject nulls."
+    },
+    {
+      "fixture": "tests/fixtures/public/dc-tie-flows-live.json",
+      "fixture_sha256": "e2274cd4dcace70592400f63e5ebc75e14a82960ae36adcb7a5f8b8508cc3992",
+      "source_receipt": {
+        "source_url": "https://www.ercot.com/api/1/services/read/dashboards/dc-tie-flows.json",
+        "retrieved_at": "2026-09-06T00:34:51.838286+00:00",
+        "sha256": "a5ff699999b2ef3974a1100c087839579dbd39b71441530793c32deac07ac137",
+        "byte_count": 1347948,
+        "status": 200
+      },
+      "transformation": "Retain first and last two observations; preserve all fields and serialize JSON numbers losslessly.",
+      "authentication": "none; one anonymous snapshot request",
+      "observed_nullability": "No source nulls observed; models reject nulls."
+    }
+  ]
+}

docs/evidence/api-archive-download.json

@@ -0,0 +1,26 @@
+{
+  "emil_id": "NP3-988-ER",
+  "document_id": 1270779234,
+  "selection_body": {
+    "docIds": [
+      1270779234
+    ]
+  },
+  "receipt": {
+    "source_url": "https://api.ercot.com/api/public-reports/archive/np3-988-er/download",
+    "retrieved_at": "2026-09-06 00:41:23.171960+00:00",
+    "sha256": "98d58b35701a712bab1ca82449063ac512a8b63851fae51cf5096388773cd455",
+    "byte_count": 22983,
+    "status": 200
+  },
+  "members": [
+    {
+      "name": "1270779234.rpt.00010036.0000000000000000.20260905.050243.ResDMEList_05092026.csv",
+      "expanded_bytes": 212038,
+      "compressed_bytes": 22709
+    }
+  ],
+  "zip_crc_verified": true,
+  "member_row_schema": "unknown",
+  "transformation": "Receipt and ZIP directory metadata only; no CSV contents retained or typed."
+}

docs/evidence/dashboard-fixture-provenance.json

@@ -0,0 +1,41 @@
+{
+  "description": "Two bounded public dashboard captures; snapshots only, no historical extraction or polling.",
+  "source_pages": [
+    {
+      "url": "https://www.ercot.com/gridmktinfo/dashboards/fuelmix",
+      "observed_route": "/api/1/services/read/dashboards/fuel-mix.json"
+    },
+    {
+      "url": "https://www.ercot.com/gridmktinfo/dashboards/gridconditions",
+      "observed_route": "/api/1/services/read/dashboards/daily-prc.json"
+    }
+  ],
+  "fixtures": [
+    {
+      "fixture": "tests/fixtures/public/fuel-mix-live.json",
+      "fixture_sha256": "e8fcebd6ef548bf9fefe3d81fafe689feb2b2e91504b3973314012e2aa0d1602",
+      "source_receipt": {
+        "source_url": "https://www.ercot.com/api/1/services/read/dashboards/fuel-mix.json",
+        "retrieved_at": "2026-09-06T00:27:11.954853+00:00",
+        "sha256": "9d969e4f9ba0cd976cc210b8ae1dadff6ee62f6da306a945a36fc9a2aae32de5",
+        "byte_count": 162715,
+        "status": 200
+      },
+      "transformation": "Retain first two and last two observations of each source day; retain all categories and monthlyCapacity; serialize numbers losslessly.",
+      "authentication": "none; anonymous ERCOT website request"
+    },
+    {
+      "fixture": "tests/fixtures/public/daily-prc-live.json",
+      "fixture_sha256": "7c9d9684f91bbf1e50216222b4d6f8a55685a2098039c9eb70e448c7dbecb550",
+      "source_receipt": {
+        "source_url": "https://www.ercot.com/api/1/services/read/dashboards/daily-prc.json",
+        "retrieved_at": "2026-09-06T00:27:11.989638+00:00",
+        "sha256": "e7e48bf1454fc0a50e0dd8d3c817c0f332ce59c08567808f4b9457b70b751aef",
+        "byte_count": 777495,
+        "status": 200
+      },
+      "transformation": "Retain first two and last two PRC observations and all current_condition fields; serialize numbers losslessly.",
+      "authentication": "none; anonymous ERCOT website request"
+    }
+  ]
+}