diff --git a/docs/esr-archive-task-note.md b/docs/esr-archive-task-note.md new file mode 100644 index 00000000..d1c4bb4d --- /dev/null +++ b/docs/esr-archive-task-note.md @@ -0,0 +1,200 @@ +# ESR and public archive contracts + +This independent contribution starts at PR #4 commit +`11e20c84b11a38186ef1fe3e05a615de2cb8571d`. Its branch is +`feat/esr-archive-contracts`. The PR base is `feat/public-metadata-expansion`. +Firstmate authorized this review-base substitution because the requested +`feat/public-coverage-batches` was unpublished and resolved locally to the +same PR #4 head, `dea83409ad8666545f518154e90722f1b42ff69c`. +Only new assigned paths change. No shared registry, generator, package import, +dependency file, or owner branch changes. + +## Result and limits + +`tinyercot.public.esr_api.ESRAPIClient.current(size=1)` addresses the separate +`GET /api/public-data/rptesr-m/4_sec_esr_charging_mw` operation. It returns an +explicitly raw `RawESRPage` after envelope checks. It never claims typed ESR +rows. The source schema defines generic `Report.data` and generic fields. +The adapter retains object data or a bounded field-width-checked raw matrix. +Unknown field types remain raw. An invalid envelope fails closed. + +The installed wheel reached this route but received HTTP 401, 143 bytes. +A narrow same-token diagnostic returned the source message: +"Access denied due to invalid subscription key. Make sure to provide a valid +key for an active subscription." The same token and key successfully retrieved +Public Reports. Account credentials are not globally invalid. The evidence +supports a service-specific subscription boundary. The exact account-side +subscription state remains unknown. No successful ESR current rows, oldest +ESR history, or typed ESR schema are claimed. + +`tinyercot.public.archive_api.ArchiveAPIClient` reuses the pinned archive +metadata and ZIP implementation. It adds media-type checks and exact public +attempt receipts, including failed transfers. It stops after a 401 or 403. +It does not reacquire a token and replay a denied operation. + +| Public operation or file | Installed result | Remaining boundary | +| --- | --- | --- | +| Public Reports root | 116 Public products, including 19 BINARY products | Root metadata alone does not verify file or row access | +| NP3-988-ER archive | 278 listed documents; document 1270779234 downloaded, HTTP 200, 22,983 bytes | Four typed DME rows verified with the existing decoder; other documents remain untested | +| NP4-190-CD bundles | 103 listed bundles | Oldest listed ID -593019152, January 2018, returns HTTP 400, 174 bytes | +| NP4-188-CD bundle | 103 bundles; August 2026 ID -1257192846 downloaded, HTTP 200, 34,664 bytes | Four typed rows from one selected nested CSV; no full-month row claim | +| Retired NP4-179-CD | 1,000 first-page documents; ID 1167729099 downloaded, HTTP 200, 1,148 bytes | Four typed offer rows for December 5, 2025; other historical schemas remain unknown | +| Retired NP3-990-EX | 1,000 first-page documents | File transfers and rows remain outstanding | +| BINARY NP4-765-ER | 1,000 first-page documents | File transfers and rows remain outstanding | + +The oldest DAM bundle diagnostic used POST, `application/json`, and +`{"docIds": [-593019152]}`. ERCOT returned code 400, status BAD_REQUEST, and +a generic message that the request may be malformed or the URL may not exist. +The capacity bundle succeeded with the same construction and a negative ID. +Thus negative IDs and this JSON construction can work. The evidence does not +establish the exact cause of the older document failure or prove that its +history is unavailable. There was no alternate verb, route, or request-key +fallback. + +## Source contracts and provenance + +The primary operation documents specify: + +- GET `/archive/{emilId}` and `/bundle/{emilId}` return `application/json`. + They declare no query parameters. This adapter fetches the first listing + page and preserves the source counters. Further listing pages remain an + outstanding contract; no invented pagination query is sent. +- POST `/archive/{emilId}/download` and `/bundle/{emilId}/download` accept + `application/json` with a required `docIds` array of int64 values and return + `application/zip`. This adapter deliberately selects one listed document + per transfer. Multi-document collections are documented but not implemented. +- The source documents 400, 403, and 404 JSON error responses. Runtime receipts + also preserve unexpected statuses. Transport failure before response headers + uses status 0. A truncated response hashes only its retained decoded prefix + and records `complete=false`. + +Source links: +[ESR operation](https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api/operations/getData?api-version=2022-04-01-preview), +[archive operation](https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductArchives?api-version=2022-04-01-preview), +[bundle operation](https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductBundle?api-version=2022-04-01-preview), +[authentication documentation](https://developer.ercot.com/applications/pubapi/user-guide/registration-and-authentication/). +The authentication documentation requires selection of the API product before +subscription. The ESR and Public Reports API configurations both require a +subscription. Backend service URLs from those configurations were not used. + +The original `export=true` URL returned API configuration, not operation +schemas. `format=openapi` returned empty paths. Bounded direct operation and +schema GET requests supplied the usable source contracts. The fixtures retain +safe projections and source receipts. They omit example subscription headers. + +- `docs/evidence/esr-archive-sources.json`: primary source identities, times, + statuses, byte counts, SHA-256 hashes, and the empty-export boundary. +- `docs/evidence/esr-archive-installed.json`: installed network retrievals, + public product metadata, selected document identities, exact failed-transfer + receipts, and the same-token diagnostic. Root `file_access=outstanding` + records the state at root retrieval; later file checks add evidence. +- `docs/evidence/esr-archive-typed-samples.json`: installed typed decoding of + exact cached live bytes. These checks made zero network requests. +- `tests/fixtures/public/archive_api/rows.json`: two source CSV excerpts with + original outer receipts and inner CSV hashes. Tests rebuild synthetic ZIPs. + Synthetic ZIPs are not installed live evidence. + +The new `archive_rows` module samples only two observed formats. It validates +the outer receipt and both ZIP layers. It requires exact CSV headers and row +widths. Dates and finite decimals are typed. Hour-ending and DST text remain +source values. No UTC market interval, correction priority, or full historical +schema claim is inferred. The DME check uses the existing seven-column decoder. + +## Bounds and credentials + +Each live run used at most 12 public-client HTTP attempts, one attempt per +operation, a 4,000,000-byte decoded response ceiling, a 30-second timeout, and +2.1-second pacing. ESR used a separate three-attempt total budget with one +attempt per operation. Successful archive and bundle transfers underwent ZIP +path, member-count, advertised expansion, and CRC checks. No files were +extracted. The new CSV samplers cap input at 4,000,000 bytes, each ZIP layer at +4,000,000 expanded bytes, and samples at 1,000 rows. Live sample proofs use four +rows. The diagnostic capped each error body at 4,096 bytes and omitted auth +headers. It retained only public error fields after checking local secrets. + +The vault CLI is `/Applications/Automic Vault.app/Contents/MacOS/av`, also +available as `/usr/local/bin/av`. `av --help`, `av inject --help`, and +`av list --help` supplied CLI syntax. No unfiltered vault list was run. +The attempted three-name injection stopped because `ERCOT_PASSWORD` is not +a saved vault entry. The actual vault password and subscription key names +remain unconfirmed; the captain was asked for names only. + +Live checks instead used the already authorized local vault export at +`/Users/kevin/.treehouse/tinyercot-1bad13/1/tinyercot/.env`. Metadata checks proved +it was a regular mode-0600 file. `git -C` on that worktree with +`check-ignore -q .env` returned zero. No credential file was copied or created. +The probe reads only the three ERCOT environment names from that export. +No values, tokens, auth responses, or request headers appear in retained +evidence. No unrelated vault entry, restricted data source, EWS request, +private telemetry, paid source, or bulk history download was used. + +## Commands and verification + +Commands ran from this worktree unless specified otherwise: + +```sh +git switch -c feat/esr-archive-contracts 11e20c84b11a38186ef1fe3e05a615de2cb8571d +uv sync --frozen --group dev --extra files +uv run python tools/generate_client.py --check +uv run python tools/generate_public.py --check +uv run pytest -q +uv run ruff check tools tinyercot/catalog.py tinyercot/public tests +uv run ruff format --check tools tinyercot/catalog.py tinyercot/public tests +uv build --quiet +uv venv /tmp/tinyercot-esr-archive-wheel +uv pip install --python /tmp/tinyercot-esr-archive-wheel/bin/python dist/tinyercot-0.2.2-py3-none-any.whl +uv pip install --python /tmp/tinyercot-esr-archive-wheel/bin/python --reinstall --no-deps dist/tinyercot-0.2.2-py3-none-any.whl +uv pip install --python /tmp/tinyercot-esr-archive-wheel/bin/python pandas-stubs +uv run python tools/check_typing.py --python /tmp/tinyercot-esr-archive-wheel/bin/python +``` + +Final offline result: 1,318 passed and one installed integration test skipped. +The 59 dedicated tests cover status/media failures, request and retry budgets, +response-prefix receipts, ZIP traversal, symlinks, CRC and expansion failures, +unknown/restricted access, retired BINARY file access, CSV schema drift, invalid +values, and raw ESR unknown handling. Generation, full scoped lint, and format +checks passed. The installed PEP 561 check initially needed `pandas-stubs` in +the isolated environment. It passed after that test dependency was installed. +No repository dependency changed. + +Installed commands used these exact path bindings: + +```sh +ESR_TASK=/Users/kevin/.treehouse/tinyercot-1bad13/2/tinyercot +ESR_PY=/tmp/tinyercot-esr-archive-wheel/bin/python +ESR_CREDS=/Users/kevin/.treehouse/tinyercot-1bad13/1/tinyercot/.env +"$ESR_PY" -I "$ESR_TASK/tools/probe_esr_archive.py" --live --credentials-file "$ESR_CREDS" --output "$ESR_TASK/.local-evidence/installed-final" --bundle-transfer +"$ESR_PY" -I "$ESR_TASK/tools/probe_esr_archive.py" --live --credentials-file "$ESR_CREDS" --output "$ESR_TASK/.local-evidence/installed-followup" --followup +"$ESR_PY" -I "$ESR_TASK/tools/probe_esr_archive.py" --sample-cache "$ESR_TASK/.local-evidence/installed-final" --output "$ESR_TASK/.local-evidence/typed-dme" +"$ESR_PY" -I "$ESR_TASK/tools/probe_esr_archive.py" --sample-cache "$ESR_TASK/.local-evidence/installed-followup" --output "$ESR_TASK/.local-evidence/typed-followup" +cd /tmp +"$ESR_TASK/.venv/bin/mypy" --strict --no-incremental --follow-imports=silent --python-executable "$ESR_PY" "$ESR_TASK/tests/fixtures/public/esr_api/typing_contract.py" +``` + +Direct installed-module typing passed. Installed module hashes are in the +receipts. Earlier bounded discovery and `installed-v1` checks preceded the +final run. A separate `python -I -` HTTPX diagnostic used the two exact public +requests and error-body bounds recorded in the installed evidence. It reused +the token from its successful public-root request and did not inspect tokens. + +## Integration handoff and task status + +Use direct imports from `tinyercot.public.esr_api`, +`tinyercot.public.archive_api`, and `tinyercot.public.archive_rows`. +No package-data registration is required for these Python modules. An optional +future shared-package export can expose `ESRAPIClient`, `RawESRPage`, +`ArchiveAPIClient`, `sample_capacity_bundle`, and `sample_retired_offers`. +The registry owner must make that change separately if desired. + +Keep ESR `row_schema=unknown` and service subscription access unverified. +Keep the old DAM bundle transfer unresolved, with HTTP 400 and its document +identity. Mark the sampled capacity bundle and retired NP4-179-CD document as +supported file access with observed typed samples. Keep NP3-990-EX and other +unretrieved products outstanding. Never translate `binary-row-schema-unsupported` +or `retired-or-inactive` into unavailable history. Restricted and unknown access +classifications remain fail-closed gates, independent of retirement and format. + +The contribution remains incomplete for successful ESR current retrieval. +It needs an authorized ESR-product subscription key through the local/vault +flow and actual live schema evidence. No merge or no-mistakes action ran. +The task status will record the PR URL and pushed commit after publication. diff --git a/docs/evidence/esr-archive-installed.json b/docs/evidence/esr-archive-installed.json new file mode 100644 index 00000000..3369d02a --- /dev/null +++ b/docs/evidence/esr-archive-installed.json @@ -0,0 +1,1761 @@ +{ + "primary_run": { + "installed_module": "/private/tmp/tinyercot-esr-archive-wheel/lib/python3.14/site-packages/tinyercot/__init__.py", + "checks": [ + { + "operation": "GET ESR current size=1", + "outcome": "AuthenticationError", + "reason": "Public API authentication failed", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-data/rptesr-m/4_sec_esr_charging_mw?page=1&size=1&sort=AGCExecTime&dir=desc", + "retrieved_at": "2026-09-06 01:35:56.178050+00:00", + "sha256": "f5ca18fbac30a6080c8d590062d1e5b3d0c7f507df2b141503e896b02949f920", + "byte_count": 143, + "status": 401 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + } + ] + }, + { + "operation": "GET public product root", + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/", + "retrieved_at": "2026-09-06 01:35:58.389946+00:00", + "sha256": "1f67b1c4c7e6d51d1814c6ce15112756f9a3320b85d46bcd0d5867e9d4f255ec", + "byte_count": 204591, + "status": 200 + }, + "products": [ + { + "id": "COPG-316", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "EIA-930-CD", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "EIA-930-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "GEN-55-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP1-300", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP1-301", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP1-302", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP1-346-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-108", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-161-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-162-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-233-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-257-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-560-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-561-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-562-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-565-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-566-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-763-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-764-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-765-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-906-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-907-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-908-ER", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-909-ER", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-910-ER", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-911-ER", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-914-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-915-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-916-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-965-ER", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-966-ER", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-987-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-988-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-990-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP3-991-EX", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-158-SG", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-159-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-179-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-183-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-188-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-190-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-191-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-192-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-193-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-194-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-196-M", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-197-M", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-19-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-200-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-212-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-213-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-214-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-215-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-231-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-33-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-412-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-442-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-443-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-494-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-523-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-532-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-722-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-732-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-733-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-737-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-738-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-742-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-743-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-745-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-746-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-751-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-752-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-765-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-790-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP4-791-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-108-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-520-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-525-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-526-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-527-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-528-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-754-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP5-755-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-235-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-322-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-323-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-324-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-325-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-326-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-327-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-328-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-329-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-331-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-332-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-344-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-345-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-346-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-569-CD", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-625-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-626-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-655-CD", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-787-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-788-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-792-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-793-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-794-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-795-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-796-ER", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-86-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-905-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-913-CD", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-915-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP6-970-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP7-464-CD", + "access": "public", + "status": "Active", + "content_type": "DATA", + "row_schema": "not_assessed", + "file_access": "outstanding" + }, + { + "id": "NP7-535-SG", + "access": "public", + "status": "Active", + "content_type": "BINARY", + "row_schema": "not_assessed", + "file_access": "outstanding" + } + ], + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/", + "retrieved_at": "2026-09-06 01:35:58.389946+00:00", + "sha256": "1f67b1c4c7e6d51d1814c6ce15112756f9a3320b85d46bcd0d5867e9d4f255ec", + "byte_count": 204591, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + } + ] + }, + { + "operation": "GET archive/NP3-988-ER", + "product_id": "NP3-988-ER", + "lifecycle": "active", + "documents": 278, + "total_records": 278, + "page_size": 1000, + "total_pages": 1, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-988-er", + "retrieved_at": "2026-09-06 01:36:01.906957+00:00", + "sha256": "fb7e721472adabcd6850e71c096f29eff591f97811425835879fc84f3de39cf7", + "byte_count": 59354, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np3-988-er", + "document_id": 1270779234, + "friendly_name": "ResDMEList_05092026", + "post_datetime": "2026-09-05T05:02:43.000" + }, + "last_document": { + "emil_id": "np3-988-er", + "document_id": 1166614741, + "friendly_name": "ResDMEList_01122025", + "post_datetime": "2025-12-01T05:13:53.000" + } + }, + { + "operation": "selected_document", + "emil_id": "np3-988-er", + "document_id": 1270779234, + "friendly_name": "ResDMEList_05092026", + "post_datetime": "2026-09-05T05:02:43.000" + }, + { + "operation": "public archive representative transfer", + "product_id": "NP3-988-ER", + "lifecycle": "active", + "documents": 278, + "total_records": 278, + "page_size": 1000, + "total_pages": 1, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-988-er", + "retrieved_at": "2026-09-06 01:36:01.906957+00:00", + "sha256": "fb7e721472adabcd6850e71c096f29eff591f97811425835879fc84f3de39cf7", + "byte_count": 59354, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_zip", + "first_document": { + "emil_id": "np3-988-er", + "document_id": 1270779234, + "friendly_name": "ResDMEList_05092026", + "post_datetime": "2026-09-05T05:02:43.000" + }, + "last_document": { + "emil_id": "np3-988-er", + "document_id": 1166614741, + "friendly_name": "ResDMEList_01122025", + "post_datetime": "2025-12-01T05:13:53.000" + }, + "document": { + "emil_id": "np3-988-er", + "document_id": 1270779234, + "friendly_name": "ResDMEList_05092026", + "post_datetime": "2026-09-05T05:02:43.000" + }, + "members": [ + "1270779234.rpt.00010036.0000000000000000.20260905.050243.ResDMEList_05092026.csv" + ], + "download_receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-988-er/download", + "retrieved_at": "2026-09-06 01:36:02.961263+00:00", + "sha256": "f9fc2fd3eca0aaf528a78403ec3c3f8926390ee15c8a07880965d5350ff608a5", + "byte_count": 22983, + "status": 200 + }, + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-988-er", + "retrieved_at": "2026-09-06 01:36:01.906957+00:00", + "sha256": "fb7e721472adabcd6850e71c096f29eff591f97811425835879fc84f3de39cf7", + "byte_count": 59354, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + }, + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-988-er/download", + "retrieved_at": "2026-09-06 01:36:02.961263+00:00", + "sha256": "f9fc2fd3eca0aaf528a78403ec3c3f8926390ee15c8a07880965d5350ff608a5", + "byte_count": 22983, + "status": 200 + }, + "method": "POST", + "content_type": "application/zip", + "complete": true, + "document_ids": [ + 1270779234 + ] + } + ] + }, + { + "operation": "GET bundle/NP4-190-CD", + "product_id": "NP4-190-CD", + "lifecycle": "active", + "documents": 103, + "total_records": 103, + "page_size": 1000, + "total_pages": 1, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-190-cd", + "retrieved_at": "2026-09-06 01:36:05.232334+00:00", + "sha256": "c8f081661149de29141e7d4cf1431d11d0e6739951f4452f0536b3261c3aef47", + "byte_count": 22196, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np4-190-cd", + "document_id": -1257192853, + "friendly_name": "DAMSPNP4190_2026-08", + "post_datetime": "2026-08-31T23:59:59.000" + }, + "last_document": { + "emil_id": "np4-190-cd", + "document_id": -593019152, + "friendly_name": "DAMSPNP4190_2018-01", + "post_datetime": "2018-01-31T23:59:59.000" + } + }, + { + "operation": "selected_document", + "emil_id": "np4-190-cd", + "document_id": -593019152, + "friendly_name": "DAMSPNP4190_2018-01", + "post_datetime": "2018-01-31T23:59:59.000" + }, + { + "operation": "public bundle boundary", + "outcome": "SourceUnavailableError", + "reason": "Public transfer failed (HTTP 400)", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-190-cd", + "retrieved_at": "2026-09-06 01:36:05.232334+00:00", + "sha256": "c8f081661149de29141e7d4cf1431d11d0e6739951f4452f0536b3261c3aef47", + "byte_count": 22196, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + }, + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-190-cd/download", + "retrieved_at": "2026-09-06 01:36:07.239179+00:00", + "sha256": "4401a903c9191b747ee7f607835f136c74ff0bd52bc4b4b9a1e3a3bb61f9458f", + "byte_count": 174, + "status": 400 + }, + "method": "POST", + "content_type": "application/json", + "complete": true, + "document_ids": [ + -593019152 + ] + } + ] + }, + { + "operation": "GET archive/NP4-179-CD", + "product_id": "NP4-179-CD", + "lifecycle": "retired-or-inactive", + "documents": 1000, + "total_records": 4236, + "page_size": 1000, + "total_pages": 5, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd", + "retrieved_at": "2026-09-06 01:36:14.997568+00:00", + "sha256": "ee4d528a7e9f30d96dfa4ca24b628a44e92d5cf084fd21fc0a5446c60b21f47a", + "byte_count": 206581, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + "last_document": { + "emil_id": "np4-179-cd", + "document_id": 901405767, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2023-03-11T12:38:21.189" + } + }, + { + "operation": "public file metadata NP4-179-CD", + "product_id": "NP4-179-CD", + "lifecycle": "retired-or-inactive", + "documents": 1000, + "total_records": 4236, + "page_size": 1000, + "total_pages": 5, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd", + "retrieved_at": "2026-09-06 01:36:14.997568+00:00", + "sha256": "ee4d528a7e9f30d96dfa4ca24b628a44e92d5cf084fd21fc0a5446c60b21f47a", + "byte_count": 206581, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + "last_document": { + "emil_id": "np4-179-cd", + "document_id": 901405767, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2023-03-11T12:38:21.189" + }, + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd", + "retrieved_at": "2026-09-06 01:36:14.997568+00:00", + "sha256": "ee4d528a7e9f30d96dfa4ca24b628a44e92d5cf084fd21fc0a5446c60b21f47a", + "byte_count": 206581, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + } + ] + }, + { + "operation": "GET archive/NP3-990-EX", + "product_id": "NP3-990-EX", + "lifecycle": "retired-or-inactive", + "documents": 1000, + "total_records": 2602, + "page_size": 1000, + "total_pages": 3, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-990-ex", + "retrieved_at": "2026-09-06 01:36:16.909420+00:00", + "sha256": "da1fffd15e5c0d1219981205a27b43c82b2668eaec99d3e62f09d1def4428b80", + "byte_count": 214693, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np3-990-ex", + "document_id": 1189109876, + "friendly_name": "60_Day_SASM_Disclosure", + "post_datetime": "2026-02-02T05:17:13.000" + }, + "last_document": { + "emil_id": "np3-990-ex", + "document_id": 912669633, + "friendly_name": "60_Day_SASM_Disclosure", + "post_datetime": "2023-05-02T05:08:51.542" + } + }, + { + "operation": "public file metadata NP3-990-EX", + "product_id": "NP3-990-EX", + "lifecycle": "retired-or-inactive", + "documents": 1000, + "total_records": 2602, + "page_size": 1000, + "total_pages": 3, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-990-ex", + "retrieved_at": "2026-09-06 01:36:16.909420+00:00", + "sha256": "da1fffd15e5c0d1219981205a27b43c82b2668eaec99d3e62f09d1def4428b80", + "byte_count": 214693, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np3-990-ex", + "document_id": 1189109876, + "friendly_name": "60_Day_SASM_Disclosure", + "post_datetime": "2026-02-02T05:17:13.000" + }, + "last_document": { + "emil_id": "np3-990-ex", + "document_id": 912669633, + "friendly_name": "60_Day_SASM_Disclosure", + "post_datetime": "2023-05-02T05:08:51.542" + }, + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-990-ex", + "retrieved_at": "2026-09-06 01:36:16.909420+00:00", + "sha256": "da1fffd15e5c0d1219981205a27b43c82b2668eaec99d3e62f09d1def4428b80", + "byte_count": 214693, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + } + ] + }, + { + "operation": "GET archive/NP4-765-ER", + "product_id": "NP4-765-ER", + "lifecycle": "active", + "documents": 1000, + "total_records": 1004, + "page_size": 1000, + "total_pages": 2, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-765-er", + "retrieved_at": "2026-09-06 01:36:17.326106+00:00", + "sha256": "4af0bca3c9e7e120cac4e03c65f932f9215eed81004d2351db43baf33a492666", + "byte_count": 222121, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np4-765-er", + "document_id": 1270852173, + "friendly_name": "ESRIntegrationReport 09-04-26", + "post_datetime": "2026-09-05T09:02:42.000" + }, + "last_document": { + "emil_id": "np4-765-er", + "document_id": 964407639, + "friendly_name": "ESRIntegrationReport 12-09-23", + "post_datetime": "2023-12-10T08:04:51.000" + } + }, + { + "operation": "public file metadata NP4-765-ER", + "product_id": "NP4-765-ER", + "lifecycle": "active", + "documents": 1000, + "total_records": 1004, + "page_size": 1000, + "total_pages": 2, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-765-er", + "retrieved_at": "2026-09-06 01:36:17.326106+00:00", + "sha256": "4af0bca3c9e7e120cac4e03c65f932f9215eed81004d2351db43baf33a492666", + "byte_count": 222121, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np4-765-er", + "document_id": 1270852173, + "friendly_name": "ESRIntegrationReport 09-04-26", + "post_datetime": "2026-09-05T09:02:42.000" + }, + "last_document": { + "emil_id": "np4-765-er", + "document_id": 964407639, + "friendly_name": "ESRIntegrationReport 12-09-23", + "post_datetime": "2023-12-10T08:04:51.000" + }, + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-765-er", + "retrieved_at": "2026-09-06 01:36:17.326106+00:00", + "sha256": "4af0bca3c9e7e120cac4e03c65f932f9215eed81004d2351db43baf33a492666", + "byte_count": 222121, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + } + ] + } + ], + "bulk_history": false, + "runtime_sha256": { + "esr_api.py": "7ac7687985dc91652104812c5c187d1c1ef845cecb3e50828823a620e2a17e2a", + "archive_api.py": "653130ee8c1cedc0725fabc665a79e1d836a4b611ae359f6c83d4af864a03bba", + "archive_http.py": "42a920907c353a193891b5bb558488283322ca482d0324b7e3588d7a36250e81" + } + }, + "followup_run": { + "installed_module": "/private/tmp/tinyercot-esr-archive-wheel/lib/python3.14/site-packages/tinyercot/__init__.py", + "checks": [ + { + "operation": "GET public product root", + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/", + "retrieved_at": "2026-09-06 01:37:27.577333+00:00", + "sha256": "1f67b1c4c7e6d51d1814c6ce15112756f9a3320b85d46bcd0d5867e9d4f255ec", + "byte_count": 204591, + "status": 200 + }, + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/", + "retrieved_at": "2026-09-06 01:37:27.577333+00:00", + "sha256": "1f67b1c4c7e6d51d1814c6ce15112756f9a3320b85d46bcd0d5867e9d4f255ec", + "byte_count": 204591, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + } + ], + "product_count": 116, + "projection": "Product entries omitted; full earlier root projection is in primary run." + }, + { + "operation": "GET bundle/NP4-188-CD", + "product_id": "NP4-188-CD", + "lifecycle": "active", + "documents": 103, + "total_records": 103, + "page_size": 1000, + "total_pages": 1, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd", + "retrieved_at": "2026-09-06 01:37:30.193506+00:00", + "sha256": "b9d97a5a37c96c36be05a1357777091340655212a9a449668ef6112d7ae7614d", + "byte_count": 22304, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np4-188-cd", + "document_id": -1257192846, + "friendly_name": "DAMCPCNP4188_2026-08", + "post_datetime": "2026-08-31T23:59:59.000" + }, + "last_document": { + "emil_id": "np4-188-cd", + "document_id": -593019147, + "friendly_name": "DAMCPCNP4188_2018-01", + "post_datetime": "2018-01-31T23:59:59.000" + } + }, + { + "operation": "selected_document", + "emil_id": "np4-188-cd", + "document_id": -1257192846, + "friendly_name": "DAMCPCNP4188_2026-08", + "post_datetime": "2026-08-31T23:59:59.000" + }, + { + "operation": "public capacity bundle transfer", + "product_id": "NP4-188-CD", + "lifecycle": "active", + "documents": 103, + "total_records": 103, + "page_size": 1000, + "total_pages": 1, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd", + "retrieved_at": "2026-09-06 01:37:30.193506+00:00", + "sha256": "b9d97a5a37c96c36be05a1357777091340655212a9a449668ef6112d7ae7614d", + "byte_count": 22304, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_zip", + "first_document": { + "emil_id": "np4-188-cd", + "document_id": -1257192846, + "friendly_name": "DAMCPCNP4188_2026-08", + "post_datetime": "2026-08-31T23:59:59.000" + }, + "last_document": { + "emil_id": "np4-188-cd", + "document_id": -593019147, + "friendly_name": "DAMCPCNP4188_2018-01", + "post_datetime": "2018-01-31T23:59:59.000" + }, + "document": { + "emil_id": "np4-188-cd", + "document_id": -1257192846, + "friendly_name": "DAMCPCNP4188_2026-08", + "post_datetime": "2026-08-31T23:59:59.000" + }, + "members": [ + "1261505456.cdr.00012329.0000000000000000.20260812.132338.DAMCPCNP4188_csv.zip", + "1265005095.cdr.00012329.0000000000000000.20260821.124910.DAMCPCNP4188_csv.zip", + "1265774856.cdr.00012329.0000000000000000.20260823.132102.DAMCPCNP4188_csv.zip", + "1260311788.cdr.00012329.0000000000000000.20260809.125216.DAMCPCNP4188_csv.zip", + "1261105814.cdr.00012329.0000000000000000.20260811.125501.DAMCPCNP4188_csv.zip", + "1267361142.cdr.00012329.0000000000000000.20260827.132013.DAMCPCNP4188_csv.zip", + "1257975678.cdr.00012329.0000000000000000.20260803.130438.DAMCPCNP4188_csv.zip", + "1259538996.cdr.00012329.0000000000000000.20260807.123249.DAMCPCNP4188_csv.zip", + "1266957708.cdr.00012329.0000000000000000.20260826.124145.DAMCPCNP4188_csv.zip", + "1268915186.cdr.00012329.0000000000000000.20260831.123351.DAMCPCNP4188_csv.zip", + "1260704536.cdr.00012329.0000000000000000.20260810.124629.DAMCPCNP4188_csv.zip", + "1258377831.cdr.00012329.0000000000000000.20260804.130624.DAMCPCNP4188_csv.zip", + "1259156998.cdr.00012329.0000000000000000.20260806.124123.DAMCPCNP4188_csv.zip", + "1262648386.cdr.00012329.0000000000000000.20260815.123251.DAMCPCNP4188_csv.zip", + "1268138590.cdr.00012329.0000000000000000.20260829.125211.DAMCPCNP4188_csv.zip", + "1266152543.cdr.00012329.0000000000000000.20260824.131126.DAMCPCNP4188_csv.zip", + "1258768357.cdr.00012329.0000000000000000.20260805.125622.DAMCPCNP4188_csv.zip", + "1264621740.cdr.00012329.0000000000000000.20260820.131954.DAMCPCNP4188_csv.zip", + "1262280176.cdr.00012329.0000000000000000.20260814.132336.DAMCPCNP4188_csv.zip", + "1266561943.cdr.00012329.0000000000000000.20260825.124239.DAMCPCNP4188_csv.zip", + "1263834167.cdr.00012329.0000000000000000.20260818.130255.DAMCPCNP4188_csv.zip", + "1267748273.cdr.00012329.0000000000000000.20260828.123357.DAMCPCNP4188_csv.zip", + "1265396711.cdr.00012329.0000000000000000.20260822.132322.DAMCPCNP4188_csv.zip", + "1264223179.cdr.00012329.0000000000000000.20260819.125133.DAMCPCNP4188_csv.zip", + "1263414914.cdr.00012329.0000000000000000.20260817.123916.DAMCPCNP4188_csv.zip", + "1257575263.cdr.00012329.0000000000000000.20260802.123243.DAMCPCNP4188_csv.zip", + "1268525445.cdr.00012329.0000000000000000.20260830.130059.DAMCPCNP4188_csv.zip", + "1257192846.cdr.00012329.0000000000000000.20260801.123249.DAMCPCNP4188_csv.zip", + "1259929151.cdr.00012329.0000000000000000.20260808.123706.DAMCPCNP4188_csv.zip", + "1261881464.cdr.00012329.0000000000000000.20260813.130154.DAMCPCNP4188_csv.zip", + "1263033921.cdr.00012329.0000000000000000.20260816.125151.DAMCPCNP4188_csv.zip" + ], + "download_receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd/download", + "retrieved_at": "2026-09-06 01:37:31.990062+00:00", + "sha256": "78467014a630a0b2f41729a1971067593d8a67116e0541f688a3b03cb46b9f3b", + "byte_count": 34664, + "status": 200 + }, + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd", + "retrieved_at": "2026-09-06 01:37:30.193506+00:00", + "sha256": "b9d97a5a37c96c36be05a1357777091340655212a9a449668ef6112d7ae7614d", + "byte_count": 22304, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + }, + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd/download", + "retrieved_at": "2026-09-06 01:37:31.990062+00:00", + "sha256": "78467014a630a0b2f41729a1971067593d8a67116e0541f688a3b03cb46b9f3b", + "byte_count": 34664, + "status": 200 + }, + "method": "POST", + "content_type": "application/zip", + "complete": true, + "document_ids": [ + -1257192846 + ] + } + ] + }, + { + "operation": "GET archive/NP4-179-CD", + "product_id": "NP4-179-CD", + "lifecycle": "retired-or-inactive", + "documents": 1000, + "total_records": 4236, + "page_size": 1000, + "total_pages": 5, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd", + "retrieved_at": "2026-09-06 01:37:35.226805+00:00", + "sha256": "ee4d528a7e9f30d96dfa4ca24b628a44e92d5cf084fd21fc0a5446c60b21f47a", + "byte_count": 206581, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": { + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + "last_document": { + "emil_id": "np4-179-cd", + "document_id": 901405767, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2023-03-11T12:38:21.189" + } + }, + { + "operation": "selected_document", + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + { + "operation": "retired public file transfer", + "product_id": "NP4-179-CD", + "lifecycle": "retired-or-inactive", + "documents": 1000, + "total_records": 4236, + "page_size": 1000, + "total_pages": 5, + "current_page": 1, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd", + "retrieved_at": "2026-09-06 01:37:35.226805+00:00", + "sha256": "ee4d528a7e9f30d96dfa4ca24b628a44e92d5cf084fd21fc0a5446c60b21f47a", + "byte_count": 206581, + "status": 200 + }, + "row_schema": "unknown", + "file_access": "supported_zip", + "first_document": { + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + "last_document": { + "emil_id": "np4-179-cd", + "document_id": 901405767, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2023-03-11T12:38:21.189" + }, + "document": { + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + "members": [ + "1167729099.cdr.00012327.0000000000000000.20251204.123457.TASONP4179_csv.zip" + ], + "download_receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd/download", + "retrieved_at": "2026-09-06 01:37:35.912158+00:00", + "sha256": "1ef611d15c003a49d5e17448ae64d826eef6fa7e0972e9678c1a95c08ddfdcc6", + "byte_count": 1148, + "status": 200 + }, + "outcome": "supported", + "exchanges": [ + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd", + "retrieved_at": "2026-09-06 01:37:35.226805+00:00", + "sha256": "ee4d528a7e9f30d96dfa4ca24b628a44e92d5cf084fd21fc0a5446c60b21f47a", + "byte_count": 206581, + "status": 200 + }, + "method": "GET", + "content_type": "application/json", + "complete": true, + "document_ids": [] + }, + { + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd/download", + "retrieved_at": "2026-09-06 01:37:35.912158+00:00", + "sha256": "1ef611d15c003a49d5e17448ae64d826eef6fa7e0972e9678c1a95c08ddfdcc6", + "byte_count": 1148, + "status": 200 + }, + "method": "POST", + "content_type": "application/zip", + "complete": true, + "document_ids": [ + 1167729099 + ] + } + ] + } + ], + "bulk_history": false, + "runtime_sha256": { + "esr_api.py": "7ac7687985dc91652104812c5c187d1c1ef845cecb3e50828823a620e2a17e2a", + "archive_api.py": "653130ee8c1cedc0725fabc665a79e1d836a4b611ae359f6c83d4af864a03bba", + "archive_http.py": "42a920907c353a193891b5bb558488283322ca482d0324b7e3588d7a36250e81" + } + }, + "diagnostic": [ + { + "method": "GET", + "source_url": "https://api.ercot.com/api/public-data/rptesr-m/4_sec_esr_charging_mw?page=1&size=1&sort=AGCExecTime&dir=desc", + "request_json": null, + "request_content_type": null, + "status": 401, + "content_type": "application/json", + "byte_count": 143, + "sha256": "f5ca18fbac30a6080c8d590062d1e5b3d0c7f507df2b141503e896b02949f920", + "retrieved_at": "2026-09-06T01:38:40.202000+00:00", + "complete": true, + "same_token_as_successful_public_root": true, + "public_error": { + "statusCode": 401, + "message": "Access denied due to invalid subscription key. Make sure to provide a valid key for an active subscription." + } + }, + { + "method": "POST", + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-190-cd/download", + "request_json": { + "docIds": [ + -593019152 + ] + }, + "request_content_type": "application/json", + "status": 400, + "content_type": "application/json", + "byte_count": 174, + "sha256": "890aac529cb45456caab355a1683e65b717bb3081b240a57b33539701f7350ff", + "retrieved_at": "2026-09-06T01:38:40.245964+00:00", + "complete": true, + "same_token_as_successful_public_root": true, + "public_error": { + "code": 400, + "status": "BAD_REQUEST", + "message": "Your request could not be processed. The request may be malformed or the URL does not exist." + } + } + ], + "authentication_document_receipt": { + "source_url": "https://developer.ercot.com/applications/pubapi/user-guide/registration-and-authentication/", + "retrieved_at": "2026-09-06T01:48:07.516642+00:00", + "status": 200, + "byte_count": 127337, + "sha256": "23bf92c67e619662d62153a6274268664a29705f345a2760d92783f80e0194f0" + } +} diff --git a/docs/evidence/esr-archive-sources.json b/docs/evidence/esr-archive-sources.json new file mode 100644 index 00000000..7350ade1 --- /dev/null +++ b/docs/evidence/esr-archive-sources.json @@ -0,0 +1,161 @@ +{ + "source_scope": "Primary ERCOT public documentation only", + "retrieval_policy": "One GET per listed source URL; no restricted URLs or subscription-key query parameters", + "receipts": [ + { + "local_source_identity": "archive-guide", + "receipt": { + "source_url": "https://developer.ercot.com/applications/pubapi/user-guide/using-api/", + "retrieved_at": "2026-09-06T01:23:55.467268+00:00", + "status": 200, + "byte_count": 129217, + "sha256": "9e412546019e580c8517845c894de0bab4ee86b23bd2023a1ce8239ddd72dcb5" + } + }, + { + "local_source_identity": "archive-schema", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/schemas/6a90e81cb9a2561f7875661e?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:35:07.009742+00:00", + "status": 200, + "byte_count": 709341, + "sha256": "e4fef3f8022efc391e3d3a51fcafff41e74cc68f8a569f4d779f954c8352cb94" + } + }, + { + "local_source_identity": "esr-operations", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api/operations?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:23:55.335639+00:00", + "status": 200, + "byte_count": 2055, + "sha256": "14e7ef9e13ea47f420bac8919273e59578ad4c02147dda69d2b1a27e6b31430a" + } + }, + { + "local_source_identity": "esr-schema", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api/schemas/6838a2f0ea971914cc572cd3?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:28:51.075469+00:00", + "status": 200, + "byte_count": 8403, + "sha256": "80a535738b9e8ece59c36027cdb51251b20a3034a1195af0bc1dbca6182633c8" + } + }, + { + "local_source_identity": "esrapi-export", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api?export=true&format=openapi&api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:23:16.035225+00:00", + "status": 200, + "byte_count": 522, + "sha256": "2b3f0f2f5461f4614f402eb25f6e00617020fc4fa8221baf8940b8f4801dbb0c" + } + }, + { + "local_source_identity": "esrapi-full", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api?export=true&format=openapi+json&api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:23:35.808536+00:00", + "status": 200, + "byte_count": 983, + "sha256": "ae77fa88c9bf8bc997186c2f384f60d845f8c722ff9e4cb6fd465aff7a3aee68" + } + }, + { + "local_source_identity": "esrapi-getData", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api/operations/getData?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.079818+00:00", + "status": 200, + "byte_count": 5463, + "sha256": "94bd075d4914a8329d6c0ffb6dc03605ffa4f5c55b30c5ca52d8b3eb30569de1" + } + }, + { + "local_source_identity": "esrapi", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api?export=true&api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:22:46.199197+00:00", + "status": 200, + "byte_count": 983, + "sha256": "ae77fa88c9bf8bc997186c2f384f60d845f8c722ff9e4cb6fd465aff7a3aee68" + } + }, + { + "local_source_identity": "pubapi-export", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api?export=true&format=openapi&api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:23:16.209077+00:00", + "status": 200, + "byte_count": 547, + "sha256": "8e36db5d54e42555349121bbb3e42db54339300e562442fae05bbbbe7a9b6de4" + } + }, + { + "local_source_identity": "pubapi-full", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api?export=true&format=openapi+json&api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:23:36.003359+00:00", + "status": 200, + "byte_count": 1028, + "sha256": "4f47536b67ffe5f45e807480a8b10077fcaa7651a7ec974c3345297cdfe80f50" + } + }, + { + "local_source_identity": "pubapi-getProductArchives", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductArchives?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.434362+00:00", + "status": 200, + "byte_count": 2505, + "sha256": "a4d39f2fbd8708decc47c6bba1e87e0f6ac16ba4e62902cd317a57b40fd3f000" + } + }, + { + "local_source_identity": "pubapi-getProductBundle", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductBundle?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.540151+00:00", + "status": 200, + "byte_count": 2487, + "sha256": "842d46909c9a2a326c1177b0b4c0752d3068b1477d01e3fef9d5e19f4d4cd4be" + } + }, + { + "local_source_identity": "pubapi-getProductHistory", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductHistory?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.199177+00:00", + "status": 200, + "byte_count": 3106, + "sha256": "8a50c478aae63a61070350417b9605d0c662ea8e211f2c02e1ee053b4e87969e" + } + }, + { + "local_source_identity": "pubapi-getProductHistoryBundles", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductHistoryBundles?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.297200+00:00", + "status": 200, + "byte_count": 3132, + "sha256": "a515bdb92a45889782f1d415d0b030322d105d5d4cce871059dd6e95e976485d" + } + }, + { + "local_source_identity": "pubapi", + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api?export=true&api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:22:46.363046+00:00", + "status": 200, + "byte_count": 1028, + "sha256": "4f47536b67ffe5f45e807480a8b10077fcaa7651a7ec974c3345297cdfe80f50" + } + } + ], + "schema_findings": { + "esr": "getData documents GET /rptesr-m/4_sec_esr_charging_mw. Report.data is generic object. Field has no concrete names. Row schema remains unknown.", + "archive": "Four operation documents specify GET listings with no query parameters and POST application/json docIds arrays returning application/zip. ArchiveRequest.docIds is an array of int64; no positive minimum is declared.", + "export": "The export=true URL returned API configuration, not OpenAPI paths. format=openapi returned paths: {}. Direct public operation and schema documents provided the retained contracts." + } +} diff --git a/docs/evidence/esr-archive-typed-samples.json b/docs/evidence/esr-archive-typed-samples.json new file mode 100644 index 00000000..bed8a0b5 --- /dev/null +++ b/docs/evidence/esr-archive-typed-samples.json @@ -0,0 +1,223 @@ +{ + "dme": { + "checks": [ + { + "document": { + "emil_id": "np3-988-er", + "document_id": 1270779234, + "friendly_name": "ResDMEList_05092026", + "post_datetime": "2026-09-05T05:02:43.000" + }, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np3-988-er/download", + "retrieved_at": "2026-09-06 01:36:02.961263+00:00", + "sha256": "f9fc2fd3eca0aaf528a78403ec3c3f8926390ee15c8a07880965d5350ff608a5", + "byte_count": 22983, + "status": 200 + }, + "network_download": false, + "installed_module": "/tmp/tinyercot-esr-archive-wheel/lib/python3.14/site-packages/tinyercot/__init__.py", + "typed_rows": 4, + "decoder": "iter_resource_dme", + "rows": [ + { + "owner_re": "1370 SHANNON LLC (RE)", + "resource_name": "SHANNONW_UNIT_1", + "resource_type": "Generation", + "split_gen_resource": "Not Applicable", + "dme": "1370 SHANNON LLC (DME)", + "dme_duns": "1450336317000", + "rmr": "N" + }, + { + "owner_re": "224WB 8ME LLC (RE)", + "resource_name": "GALLOWAY_SOLAR2", + "resource_type": "Generation", + "split_gen_resource": "Not Applicable", + "dme": "224WB 8ME LLC (DME)", + "dme_duns": "0815527087000", + "rmr": "N" + }, + { + "owner_re": "225DD 8ME LLC (RE)", + "resource_name": "GALLOWAY_SOLAR1", + "resource_type": "Generation", + "split_gen_resource": "Not Applicable", + "dme": "225DD 8ME LLC (DME)", + "dme_duns": "0815490207000", + "rmr": "N" + }, + { + "owner_re": "226HC 8ME LLC (RE)", + "resource_name": "HOLSTEIN_SOLAR1", + "resource_type": "Generation", + "split_gen_resource": "Not Applicable", + "dme": "226HC 8ME LLC (DME)", + "dme_duns": "1169540657000", + "rmr": "N" + } + ] + } + ], + "network_requests": 0, + "runtime_sha256": { + "archive_rows.py": "e3ad6f1a12ee3a5be4026f6b6d30d1a3aadc03f0c15c8488ee7f27d7b5f264ba", + "resource_dme.py": "5fac7c6a0829e47f7c077382ba8ee725986ac479509c7a5ea808379b6599499c" + } + }, + "bundle_and_retired": { + "checks": [ + { + "document": { + "emil_id": "np4-188-cd", + "document_id": -1257192846, + "friendly_name": "DAMCPCNP4188_2026-08", + "post_datetime": "2026-08-31T23:59:59.000" + }, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd/download", + "retrieved_at": "2026-09-06 01:37:31.990062+00:00", + "sha256": "78467014a630a0b2f41729a1971067593d8a67116e0541f688a3b03cb46b9f3b", + "byte_count": 34664, + "status": 200 + }, + "network_download": false, + "installed_module": "/tmp/tinyercot-esr-archive-wheel/lib/python3.14/site-packages/tinyercot/__init__.py", + "typed_rows": 4, + "decoder": "sample_capacity_bundle", + "sample": { + "rows": [ + { + "delivery_date": "2026-08-13", + "hour_ending": "01:00", + "ancillary_type": "NSPIN", + "mcpc": "1.64", + "dst_flag": "N" + }, + { + "delivery_date": "2026-08-13", + "hour_ending": "01:00", + "ancillary_type": "RRS", + "mcpc": "0.19", + "dst_flag": "N" + }, + { + "delivery_date": "2026-08-13", + "hour_ending": "01:00", + "ancillary_type": "REGDN", + "mcpc": "1", + "dst_flag": "N" + }, + { + "delivery_date": "2026-08-13", + "hour_ending": "01:00", + "ancillary_type": "REGUP", + "mcpc": "0.39", + "dst_flag": "N" + } + ], + "outer_member": "1261505456.cdr.00012329.0000000000000000.20260812.132338.DAMCPCNP4188_csv.zip", + "inner_member": "cdr.00012329.0000000000000000.20260812.132338.DAMCPCNP4188.csv", + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd/download", + "retrieved_at": "2026-09-06 01:37:31.990062+00:00", + "sha256": "78467014a630a0b2f41729a1971067593d8a67116e0541f688a3b03cb46b9f3b", + "byte_count": 34664, + "status": 200 + }, + "truncated": true + } + }, + { + "document": { + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd/download", + "retrieved_at": "2026-09-06 01:37:35.912158+00:00", + "sha256": "1ef611d15c003a49d5e17448ae64d826eef6fa7e0972e9678c1a95c08ddfdcc6", + "byte_count": 1148, + "status": 200 + }, + "network_download": false, + "installed_module": "/tmp/tinyercot-esr-archive-wheel/lib/python3.14/site-packages/tinyercot/__init__.py", + "typed_rows": 4, + "decoder": "sample_retired_offers", + "sample": { + "rows": [ + { + "delivery_date": "2025-12-05", + "hour_ending": "01:00", + "regdn": "5677.8", + "regup": "6695.8", + "rrspfr": "8676.7", + "rrsffr": "150.4", + "rrsufr": "892.2", + "ecrssd": "8449.1", + "ecrsmd": "268.5", + "nspin": "8259", + "dst_flag": "N" + }, + { + "delivery_date": "2025-12-05", + "hour_ending": "02:00", + "regdn": "4762.3", + "regup": "6725.8", + "rrspfr": "8736.9", + "rrsffr": "100", + "rrsufr": "888.9", + "ecrssd": "8567.5", + "ecrsmd": "272.3", + "nspin": "8275", + "dst_flag": "N" + }, + { + "delivery_date": "2025-12-05", + "hour_ending": "03:00", + "regdn": "4743.7", + "regup": "6701.5", + "rrspfr": "8638.2", + "rrsffr": "150.4", + "rrsufr": "893.7", + "ecrssd": "8515.8", + "ecrsmd": "272.3", + "nspin": "8335.9", + "dst_flag": "N" + }, + { + "delivery_date": "2025-12-05", + "hour_ending": "04:00", + "regdn": "5178.9", + "regup": "6518.2", + "rrspfr": "8523.9", + "rrsffr": "150.4", + "rrsufr": "893.7", + "ecrssd": "8394.9", + "ecrsmd": "272.3", + "nspin": "8124.8", + "dst_flag": "N" + } + ], + "outer_member": "1167729099.cdr.00012327.0000000000000000.20251204.123457.TASONP4179_csv.zip", + "inner_member": "cdr.00012327.0000000000000000.20251204.123457.TASONP4179.csv", + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd/download", + "retrieved_at": "2026-09-06 01:37:35.912158+00:00", + "sha256": "1ef611d15c003a49d5e17448ae64d826eef6fa7e0972e9678c1a95c08ddfdcc6", + "byte_count": 1148, + "status": 200 + }, + "truncated": true + } + } + ], + "network_requests": 0, + "runtime_sha256": { + "archive_rows.py": "e3ad6f1a12ee3a5be4026f6b6d30d1a3aadc03f0c15c8488ee7f27d7b5f264ba", + "resource_dme.py": "5fac7c6a0829e47f7c077382ba8ee725986ac479509c7a5ea808379b6599499c" + } + } +} diff --git a/tests/fixtures/public/archive_api/rows.json b/tests/fixtures/public/archive_api/rows.json new file mode 100644 index 00000000..6e51f89f --- /dev/null +++ b/tests/fixtures/public/archive_api/rows.json @@ -0,0 +1,109 @@ +{ + "origin": "Installed-wheel public file retrieval, selected source CSV excerpts; tests rebuild synthetic ZIP containers.", + "samples": [ + { + "document": { + "emil_id": "np4-188-cd", + "document_id": -1257192846, + "friendly_name": "DAMCPCNP4188_2026-08", + "post_datetime": "2026-08-31T23:59:59.000" + }, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/bundle/np4-188-cd/download", + "retrieved_at": "2026-09-06 01:37:31.990062+00:00", + "sha256": "78467014a630a0b2f41729a1971067593d8a67116e0541f688a3b03cb46b9f3b", + "byte_count": 34664, + "status": 200 + }, + "outer_member": "1261505456.cdr.00012329.0000000000000000.20260812.132338.DAMCPCNP4188_csv.zip", + "inner_member": "cdr.00012329.0000000000000000.20260812.132338.DAMCPCNP4188.csv", + "inner_sha256": "f44d111e75e3ced3b145497ade9b087b55e7044ba43afd8e9aa5f5e22f02848b", + "inner_byte_count": 3527, + "header": [ + "DeliveryDate", + "HourEnding", + "AncillaryType", + "MCPC", + "DSTFlag" + ], + "rows_excerpt": [ + [ + "08/13/2026", + "01:00", + "NSPIN", + "1.64", + "N" + ], + [ + "08/13/2026", + "01:00", + "RRS", + "0.19", + "N" + ] + ], + "total_csv_rows": 120 + }, + { + "document": { + "emil_id": "np4-179-cd", + "document_id": 1167729099, + "friendly_name": "TASONP4179_csv", + "post_datetime": "2025-12-04T12:34:57.000" + }, + "receipt": { + "source_url": "https://api.ercot.com/api/public-reports/archive/np4-179-cd/download", + "retrieved_at": "2026-09-06 01:37:35.912158+00:00", + "sha256": "1ef611d15c003a49d5e17448ae64d826eef6fa7e0972e9678c1a95c08ddfdcc6", + "byte_count": 1148, + "status": 200 + }, + "outer_member": "1167729099.cdr.00012327.0000000000000000.20251204.123457.TASONP4179_csv.zip", + "inner_member": "cdr.00012327.0000000000000000.20251204.123457.TASONP4179.csv", + "inner_sha256": "53b1989fb94b9bf0257d49153355428e294c8046e863cf3498042a4357cf893b", + "inner_byte_count": 1758, + "header": [ + "DeliveryDate", + "HourEnding", + "REGDN", + "REGUP", + "RRSPFR", + "RRSFFR", + "RRSUFR", + "ECRSSD", + "ECRSMD", + "NSPIN", + "DSTFlag" + ], + "rows_excerpt": [ + [ + "12/05/2025", + "01:00", + "5677.8", + "6695.8", + "8676.7", + "150.4", + "892.2", + "8449.1", + "268.5", + "8259", + "N" + ], + [ + "12/05/2025", + "02:00", + "4762.3", + "6725.8", + "8736.9", + "100", + "888.9", + "8567.5", + "272.3", + "8275", + "N" + ] + ], + "total_csv_rows": 24 + } + ] +} diff --git a/tests/fixtures/public/archive_api/source-contracts.json b/tests/fixtures/public/archive_api/source-contracts.json new file mode 100644 index 00000000..93302e50 --- /dev/null +++ b/tests/fixtures/public/archive_api/source-contracts.json @@ -0,0 +1,195 @@ +{ + "operations": [ + { + "id": "getProductHistory", + "method": "GET", + "urlTemplate": "/archive/{emilId}", + "description": "Available report archives for a specified EMIL product.", + "query": [], + "request_media": [], + "responses": [ + { + "status": 200, + "media": [ + "application/json" + ] + }, + { + "status": 400, + "media": [ + "application/json" + ] + }, + { + "status": 403, + "media": [ + "application/json" + ] + }, + { + "status": 404, + "media": [ + "application/json" + ] + } + ], + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductHistory?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.199177+00:00", + "status": 200, + "byte_count": 3106, + "sha256": "8a50c478aae63a61070350417b9605d0c662ea8e211f2c02e1ee053b4e87969e" + } + }, + { + "id": "getProductHistoryBundles", + "method": "GET", + "urlTemplate": "/bundle/{emilId}", + "description": "Available historical report archives for a specified EMIL product.", + "query": [], + "request_media": [], + "responses": [ + { + "status": 200, + "media": [ + "application/json" + ] + }, + { + "status": 400, + "media": [ + "application/json" + ] + }, + { + "status": 403, + "media": [ + "application/json" + ] + }, + { + "status": 404, + "media": [ + "application/json" + ] + } + ], + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductHistoryBundles?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.297200+00:00", + "status": 200, + "byte_count": 3132, + "sha256": "a515bdb92a45889782f1d415d0b030322d105d5d4cce871059dd6e95e976485d" + } + }, + { + "id": "getProductArchives", + "method": "POST", + "urlTemplate": "/archive/{emilId}/download", + "description": "Download a collection of EMIL Product archives by generating a JSON request with a single 'docIds' element containing the document identifiers you wish to download.", + "query": [], + "request_media": [ + "application/json" + ], + "responses": [ + { + "status": 200, + "media": [ + "application/zip" + ] + }, + { + "status": 400, + "media": [ + "application/json" + ] + }, + { + "status": 403, + "media": [ + "application/json" + ] + }, + { + "status": 404, + "media": [ + "application/json" + ] + } + ], + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductArchives?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.434362+00:00", + "status": 200, + "byte_count": 2505, + "sha256": "a4d39f2fbd8708decc47c6bba1e87e0f6ac16ba4e62902cd317a57b40fd3f000" + } + }, + { + "id": "getProductBundle", + "method": "POST", + "urlTemplate": "/bundle/{emilId}/download", + "description": "Download an EMIL Product bundle archive by generating a JSON request with a single 'docIds' element containing the document identifier you wish to download.", + "query": [], + "request_media": [ + "application/json" + ], + "responses": [ + { + "status": 200, + "media": [ + "application/zip" + ] + }, + { + "status": 400, + "media": [ + "application/json" + ] + }, + { + "status": 403, + "media": [ + "application/json" + ] + }, + { + "status": 404, + "media": [ + "application/json" + ] + } + ], + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/operations/getProductBundle?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.540151+00:00", + "status": 200, + "byte_count": 2487, + "sha256": "842d46909c9a2a326c1177b0b4c0752d3068b1477d01e3fef9d5e19f4d4cd4be" + } + } + ], + "selection_schema": { + "required": [ + "docIds" + ], + "type": "object", + "properties": { + "docIds": { + "type": "array", + "items": { + "type": "integer", + "format": "int64" + } + } + }, + "description": "Represents a multi-document download request in JSON format." + }, + "selection_schema_receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/pubapi-apim-api/schemas/6a90e81cb9a2561f7875661e?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:35:07.009742+00:00", + "status": 200, + "byte_count": 709341, + "sha256": "e4fef3f8022efc391e3d3a51fcafff41e74cc68f8a569f4d779f954c8352cb94" + } +} diff --git a/tests/fixtures/public/esr_api/source-contract.json b/tests/fixtures/public/esr_api/source-contract.json new file mode 100644 index 00000000..b404b406 --- /dev/null +++ b/tests/fixtures/public/esr_api/source-contract.json @@ -0,0 +1,226 @@ +{ + "operation": { + "id": "getData", + "method": "GET", + "urlTemplate": "/rptesr-m/4_sec_esr_charging_mw", + "description": "Energy Storage Resource Four Second Data", + "query": [ + { + "name": "AGCExecTimeFrom", + "type": "string", + "description": "Format - yyyy-MM-ddTH24:mm:ss." + }, + { + "name": "AGCExecTimeTo", + "type": "string", + "description": "Format - yyyy-MM-ddTH24:mm:ss." + }, + { + "name": "DSTFlag", + "type": "boolean", + "description": "Format - true | false." + }, + { + "name": "AGCExecTimeUTCFrom", + "type": "string", + "description": "Format - yyyy-MM-ddTH24:mm:ss." + }, + { + "name": "AGCExecTimeUTCTo", + "type": "string", + "description": "Format - yyyy-MM-ddTH24:mm:ss." + }, + { + "name": "systemDemandFrom", + "type": "number", + "description": "Format - ####.###." + }, + { + "name": "systemDemandTo", + "type": "number", + "description": "Format - ####.###." + }, + { + "name": "ESRChargingMWFrom", + "type": "number", + "description": "Format - ####.###." + }, + { + "name": "ESRChargingMWTo", + "type": "number", + "description": "Format - ####.###." + }, + { + "name": "page", + "type": "integer", + "description": "Format - ###. Page number of returned values in the collection." + }, + { + "name": "size", + "type": "integer", + "description": "Format - ###. Number of returned items per page." + }, + { + "name": "sort", + "type": "string", + "description": "Format - abc123. Single or comma separated list of fields by which to sort the returned resource values. The default sort order for each field is ascending, but can be changed to descending by adding a '-' (minus) suffix to the field name." + }, + { + "name": "dir", + "type": "string", + "description": "Format - abc123. Defines sort order of returned values based on a single field specified by the 'sort' parameter, or the primary business key of the resource if no sort field is specified. Valid values are 'asc' and 'desc'" + } + ], + "request_media": [], + "responses": [ + { + "status": 200, + "media": [ + "application/json" + ] + }, + { + "status": 400, + "media": [ + "application/json" + ] + }, + { + "status": 403, + "media": [ + "application/json" + ] + }, + { + "status": 404, + "media": [ + "application/json" + ] + } + ], + "receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api/operations/getData?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:24:20.079818+00:00", + "status": 200, + "byte_count": 5463, + "sha256": "94bd075d4914a8329d6c0ffb6dc03605ffa4f5c55b30c5ca52d8b3eb30569de1" + } + }, + "schemas": { + "Report": { + "properties": { + "_meta": { + "$ref": "#/components/schemas/ResultMetadata" + }, + "report": { + "$ref": "#/components/schemas/ReportMetadata" + }, + "fields": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Field" + } + }, + "data": { + "type": "object" + }, + "links": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Link" + } + } + }, + "description": "Represents report data search results from any artifact report query." + }, + "Field": { + "properties": { + "name": { + "type": "string" + }, + "label": { + "type": "string" + }, + "cardinality": { + "type": "integer", + "format": "int32" + }, + "dataType": { + "enum": [ + "BOOLEAN", + "VARCHAR", + "INTEGER", + "LONG", + "DOUBLE", + "DATE", + "DATETIME", + "TIME" + ], + "type": "string" + }, + "searchable": { + "type": "boolean" + }, + "sortable": { + "type": "boolean" + }, + "hasRange": { + "type": "boolean" + } + }, + "description": "Represents report data field." + }, + "ReportMetadata": { + "properties": { + "reportProductId": { + "type": "string" + }, + "reportName": { + "type": "string" + }, + "reportDisplayName": { + "type": "string" + }, + "reportId": { + "type": "string" + }, + "downloadLimit": { + "type": "integer", + "format": "int32" + } + }, + "description": "Represents general metadata for the specified report artifact, including the parent Data Product identifier." + }, + "ResultMetadata": { + "properties": { + "totalRecords": { + "type": "integer", + "format": "int64" + }, + "pageSize": { + "type": "integer", + "format": "int32" + }, + "totalPages": { + "type": "integer", + "format": "int32" + }, + "currentPage": { + "type": "integer", + "format": "int32" + }, + "query": { + "$ref": "#/components/schemas/QueryMetadata" + } + }, + "description": "Represents record and paging summary for the specified query results." + } + }, + "schema_receipt": { + "source_url": "https://apiexplorer.ercot.com/developer/apis/esrapi-apim-api/schemas/6838a2f0ea971914cc572cd3?api-version=2022-04-01-preview", + "retrieved_at": "2026-09-06T01:28:51.075469+00:00", + "status": 200, + "byte_count": 8403, + "sha256": "80a535738b9e8ece59c36027cdb51251b20a3034a1195af0bc1dbca6182633c8" + } +} diff --git a/tests/fixtures/public/esr_api/typing_contract.py b/tests/fixtures/public/esr_api/typing_contract.py new file mode 100644 index 00000000..8186f3e3 --- /dev/null +++ b/tests/fixtures/public/esr_api/typing_contract.py @@ -0,0 +1,36 @@ +"""Check direct installed-wheel module types without running a data request.""" + +from typing import assert_type + +from tinyercot.public.api import Credentials +from tinyercot.public.api_archives import APIArchiveFile, APIArchivePage, APIBundlePage +from tinyercot.public.archive_api import ArchiveAPIClient +from tinyercot.public.archive_http import Exchange +from tinyercot.public.archive_rows import ( + ArchiveRowSample, + CapacityRow, + sample_capacity_bundle, +) +from tinyercot.public.esr_api import ESRAPIClient, RawESRPage + + +def contracts(credentials: Credentials) -> None: + """Check public return types without calling this function. + + Args: + credentials: Explicit account credentials required by both clients. + """ + with ESRAPIClient(credentials) as esr: + assert_type(esr.current(), RawESRPage) + assert_type(esr.exchanges, tuple[Exchange, ...]) + with ArchiveAPIClient(credentials) as archive: + archive.products() + page = archive.archives("NP3-988-ER") + assert_type(page, APIArchivePage) + assert_type(archive.bundles("NP4-190-CD"), APIBundlePage) + assert_type(archive.download(page.documents[0]), APIArchiveFile) + file = archive.download(page.documents[0]) + assert_type( + sample_capacity_bundle(file, member="selected.zip"), + ArchiveRowSample[CapacityRow], + ) diff --git a/tests/test_public_archive_api.py b/tests/test_public_archive_api.py new file mode 100644 index 00000000..e427c456 --- /dev/null +++ b/tests/test_public_archive_api.py @@ -0,0 +1,381 @@ +"""Verify generic archive receipts and failure boundaries without a network.""" + +import copy +import csv +import datetime +import hashlib +import io +import json +import stat +import zipfile +from dataclasses import replace +from decimal import Decimal +from pathlib import Path + +import httpx +import pytest + +from tinyercot.public._http import ( + AccessDeniedError, + AuthenticationError, + LimitError, + RateLimitError, + Receipt, + SchemaMismatchError, + SourceUnavailableError, +) +from tinyercot.public.api import Credentials +from tinyercot.public.api_archives import APIArchive, APIArchiveFile, APIBundle +from tinyercot.public.archive_api import ArchiveAPIClient, Limits +from tinyercot.public.archive_rows import sample_capacity_bundle, sample_retired_offers + +FIXTURES = Path(__file__).parent / "fixtures/public" +SOURCE = json.loads((FIXTURES / "api-archive-excerpt.json").read_text()) +BUNDLE = json.loads((FIXTURES / "api-bundle-excerpt.json").read_text()) + + +def zip_bytes(name="unknown.csv", body=b"unknown\n1\n", mode=None): + stream = io.BytesIO() + with zipfile.ZipFile(stream, "w") as archive: + info = zipfile.ZipInfo(name) + if mode is not None: + info.external_attr = mode << 16 + archive.writestr(info, body) + return stream.getvalue() + + +def make_client(*, kind="archive", response=None, limits=None, change=None): + evidence = BUNDLE if kind == "bundle" else SOURCE + product = copy.deepcopy(evidence["product"]) + listing = copy.deepcopy(evidence[kind + "_excerpt"]) + listing["_meta"]["totalRecords"] = len(listing[kind + "s"]) + if change: + change(product, listing) + calls = [] + + def handle(request): + calls.append(request) + if request.url.path.endswith("/token"): + return httpx.Response( + 200, json={"id_token": "synthetic-secret", "expires_in": 3600} + ) + if request.url.path.endswith("/public-reports/"): + return httpx.Response(200, json={"_embedded": {"products": [product]}}) + if request.method == "POST": + return ( + response(request) + if response + else httpx.Response( + 200, + content=zip_bytes(), + headers={"Content-Type": "application/zip"}, + ) + ) + return httpx.Response(200, json=listing) + + return ArchiveAPIClient( + Credentials("synthetic-user", "synthetic-password", "synthetic-key"), + limits=limits or Limits(min_interval=0, attempts=1), + transport=httpx.MockTransport(handle), + ), calls + + +@pytest.mark.parametrize("kind", ["archive", "bundle"]) +def test_document_selection_verb_mime_receipt_and_route(kind): + client, calls = make_client(kind=kind) + identity = "NP4-190-CD" if kind == "bundle" else "NP3-988-ER" + with client: + client.products() + page = ( + client.bundles(identity) if kind == "bundle" else client.archives(identity) + ) + document = page.documents[0] + downloaded = ( + client.download_bundle(document) + if kind == "bundle" + else client.download(document) + ) + request = calls[-1] + assert request.method == "POST" + assert request.url.path.endswith(f"/{kind}/{identity.lower()}/download") + assert json.loads(request.content) == {"docIds": [document.document_id]} + assert request.headers["content-type"] == "application/json" + assert downloaded.row_schema == "unknown" + assert downloaded.members == ("unknown.csv",) + assert downloaded.receipt.sha256 == hashlib.sha256(downloaded.body).hexdigest() + assert len(client.exchanges) == 3 + assert client.exchanges[-1].document_ids == (document.document_id,) + assert client.exchanges[-1].complete + assert "synthetic-secret" not in repr(client.exchanges) + assert "synthetic-key" not in repr(client.exchanges) + assert all("token" not in x.receipt.source_url for x in client.exchanges) + + +@pytest.mark.parametrize( + "status,error", + [ + (400, SourceUnavailableError), + (401, AuthenticationError), + (403, AccessDeniedError), + (404, SourceUnavailableError), + (302, SourceUnavailableError), + (429, RateLimitError), + (500, SourceUnavailableError), + (502, SourceUnavailableError), + ], +) +def test_exact_failed_bundle_status_without_fallback(status, error): + body = b'{"message":"synthetic error"}' + client, calls = make_client( + kind="bundle", + response=lambda r: httpx.Response( + status, + content=body, + headers={ + "Content-Type": "application/json", + "Location": "https://other.invalid/", + }, + ), + ) + with client: + client.products() + document = client.bundles("NP4-190-CD").documents[0] + with pytest.raises(error): + client.download_bundle(document) + receipt = client.exchanges[-1].receipt + assert receipt.status == status + assert receipt.sha256 == hashlib.sha256(body).hexdigest() + assert receipt.byte_count == len(body) + assert len(calls) == 4 + + +@pytest.mark.parametrize( + "media", ["text/html", "application/json", "application/octet-stream", ""] +) +def test_zip_requires_documented_content_type(media): + client, _ = make_client( + response=lambda r: httpx.Response( + 200, + content=zip_bytes(), + headers={"content-type": media}, + ) + ) + with client: + client.products() + document = client.archives("NP3-988-ER").documents[0] + with pytest.raises(SchemaMismatchError, match="content type"): + client.download(document) + assert client.exchanges[-1].receipt.status == 200 + + +@pytest.mark.parametrize( + "raw", + [ + b"not a ZIP", + zip_bytes("../outside"), + zip_bytes("/absolute"), + zip_bytes("C:drive"), + zip_bytes("a\\b"), + zip_bytes("link", mode=stat.S_IFLNK | 0o777), + ], +) +def test_malformed_zip_and_paths(raw): + client, _ = make_client( + response=lambda r: httpx.Response( + 200, + content=raw, + headers={"content-type": "application/zip"}, + ) + ) + with client: + client.products() + with pytest.raises(SchemaMismatchError): + client.download(client.archives("NP3-988-ER").documents[0]) + + +def test_zip_crc_and_expansion_limits(): + good = zip_bytes(body=b"unique payload") + bad = good.replace(b"unique payload", b"broken payload", 1) + for raw, error, kwargs in [ + (bad, SchemaMismatchError, {}), + (good, LimitError, {"max_expanded_bytes": 1}), + ]: + client, _ = make_client( + response=lambda r, raw=raw: httpx.Response( + 200, + content=raw, + headers={"content-type": "application/zip"}, + ) + ) + with client: + client.products() + with pytest.raises(error): + client.download(client.archives("NP3-988-ER").documents[0], **kwargs) + + +def test_response_size_prefix_receipt_and_no_retry(): + raw = b"x" * 70000 + client, calls = make_client( + response=lambda r: httpx.Response( + 200, + content=raw, + headers={"content-type": "application/zip"}, + ), + limits=Limits(max_bytes=16000, min_interval=0, attempts=3), + ) + with client: + client.products() + document = client.archives("NP3-988-ER").documents[0] + with pytest.raises(LimitError): + client.download(document) + exchange = client.exchanges[-1] + assert not exchange.complete + assert exchange.receipt.byte_count == 16000 + assert exchange.receipt.sha256 == hashlib.sha256(raw[:16000]).hexdigest() + assert len(calls) == 4 + + +def test_retry_and_total_request_budgets(): + client, calls = make_client( + response=lambda r: httpx.Response( + 503, + content=b"busy", + headers={"Retry-After": "0"}, + ), + limits=Limits(min_interval=0, attempts=3, max_requests=5), + ) + with client: + client.products() + document = client.archives("NP3-988-ER").documents[0] + with pytest.raises(LimitError, match="request budget"): + client.download(document) + assert len(calls) == 5 + assert [x.receipt.status for x in client.exchanges[-2:]] == [503, 503] + + +def test_retry_after_cannot_extend_budget(): + client, calls = make_client( + response=lambda r: httpx.Response( + 429, + content=b"busy", + headers={"Retry-After": "999"}, + ), + limits=Limits(min_interval=0, attempts=3, max_retry_wait=1), + ) + with client: + client.products() + with pytest.raises(RateLimitError, match="wait budget"): + client.download(client.archives("NP3-988-ER").documents[0]) + assert len(calls) == 4 + + +@pytest.mark.parametrize("access", ["Secure", "Certified", "unknown", None]) +def test_access_is_fail_closed(access): + client, calls = make_client( + change=lambda p, l: p.update(securityClassification=access) + ) + with client: + client.products() + with pytest.raises(AccessDeniedError): + client.archives("NP3-988-ER") + assert len(calls) == 2 + + +def test_binary_retired_history_is_supported_file_access(): + client, _ = make_client( + change=lambda p, l: p.update(contentType="BINARY", status="Retired") + ) + with client: + products, _ = client.products() + assert products[0].content_type == "BINARY" + page = client.archives("NP3-988-ER") + assert page.lifecycle == "retired-or-inactive" + assert client.download(page.documents[0]).row_schema == "unknown" + with pytest.raises(AccessDeniedError): + client.download(replace(page.documents[0], document_id=1)) + client.products() + with pytest.raises(AccessDeniedError): + client.download(page.documents[0]) + + +def test_source_contract_has_no_listing_query_or_get_download(): + source = json.loads((FIXTURES / "archive_api/source-contracts.json").read_text()) + assert source["selection_schema"]["required"] == ["docIds"] + for op in source["operations"]: + assert op["query"] == [] + if op["urlTemplate"].endswith("/download"): + assert op["method"] == "POST" + assert op["request_media"] == ["application/json"] + assert op["responses"][0] == {"status": 200, "media": ["application/zip"]} + + +def sample_file(index, change=None, inner_path=None): + source = json.loads((FIXTURES / "archive_api/rows.json").read_text())["samples"][ + index + ] + rows = [source["header"], *source["rows_excerpt"]] + if change: + change(rows) + stream = io.StringIO(newline="") + csv.writer(stream).writerows(rows) + inner = zip_bytes(inner_path or source["inner_member"], stream.getvalue().encode()) + raw = zip_bytes(source["outer_member"], inner) + document = (APIBundle if index == 0 else APIArchive)(**source["document"]) + receipt = Receipt( + source["receipt"]["source_url"], + datetime.datetime.now(datetime.UTC), + hashlib.sha256(raw).hexdigest(), + len(raw), + ) + return APIArchiveFile(document, receipt, (source["outer_member"],), raw) + + +@pytest.mark.parametrize( + "index,decoder", [(0, sample_capacity_bundle), (1, sample_retired_offers)] +) +def test_source_backed_nested_csv_sample(index, decoder): + file = sample_file(index) + sample = decoder(file, member=file.members[0], max_rows=1) + assert sample.truncated and len(sample.rows) == 1 + assert sample.receipt == file.receipt + assert sample.rows[0].hour_ending == "01:00" + if index == 0: + assert sample.rows[0].mcpc == Decimal("1.64") + assert sample.rows[0].delivery_date == datetime.date(2026, 8, 13) + else: + assert sample.rows[0].regdn == Decimal("5677.8") + assert sample.rows[0].delivery_date == datetime.date(2025, 12, 5) + + +@pytest.mark.parametrize( + "change", + [ + lambda rows: rows[0].__setitem__(0, "UnknownHeader"), + lambda rows: rows[1].__setitem__(3, "NaN"), + lambda rows: rows[1].__setitem__(0, "02/30/2026"), + lambda rows: rows[1].__setitem__(1, "25:00"), + lambda rows: rows[1].__setitem__(-1, "unknown"), + lambda rows: rows[1].pop(), + ], +) +def test_nested_csv_schema_and_values_fail_closed(change): + file = sample_file(0, change) + with pytest.raises(SchemaMismatchError): + sample_capacity_bundle(file, member=file.members[0]) + + +def test_nested_zip_path_receipt_and_selection_checks(): + file = sample_file(0, inner_path="../outside.csv") + with pytest.raises(SchemaMismatchError): + sample_capacity_bundle(file, member=file.members[0]) + file = sample_file(0) + with pytest.raises(AccessDeniedError): + sample_retired_offers(file, member=file.members[0]) + with pytest.raises(SchemaMismatchError): + sample_capacity_bundle( + replace(file, body=file.body + b"changed"), member=file.members[0] + ) + with pytest.raises(SchemaMismatchError): + sample_capacity_bundle(file, member="unlisted.zip") + with pytest.raises(ValueError): + sample_capacity_bundle(file, member=file.members[0], max_rows=0) diff --git a/tests/test_public_esr_api.py b/tests/test_public_esr_api.py new file mode 100644 index 00000000..40a02f79 --- /dev/null +++ b/tests/test_public_esr_api.py @@ -0,0 +1,130 @@ +"""Test raw ESR contracts with synthetic envelopes, never live row evidence.""" + +import copy +import json +from pathlib import Path + +import httpx +import pytest + +from tinyercot.public._http import AuthenticationError, Limits, SchemaMismatchError +from tinyercot.public.api import Credentials +from tinyercot.public.esr_api import BASE, PATH, ESRAPIClient, RawESRPage + + +def raw_envelope(): + return { + "_meta": {"currentPage": 1, "pageSize": 1, "totalPages": 1, "totalRecords": 1}, + "report": {"reportProductId": "rptesr-m"}, + "fields": [{"name": "unverified", "dataType": "VARCHAR"}], + "data": {}, + } + + +def client_for(body=None, status=200, media="application/json"): + calls = [] + + def handle(request): + calls.append(request) + if request.url.path.endswith("/token"): + return httpx.Response( + 200, json={"id_token": "synthetic", "expires_in": 3600} + ) + return httpx.Response( + status, + content=json.dumps(body if body is not None else raw_envelope()).encode(), + headers={"content-type": media}, + ) + + return ESRAPIClient( + Credentials("synthetic", "synthetic", "synthetic"), + limits=Limits(min_interval=0, attempts=1), + transport=httpx.MockTransport(handle), + ), calls + + +def test_direct_module_raw_current_route_and_unknown_schema(): + client, calls = client_for() + with client: + page = client.current() + assert isinstance(page, RawESRPage) + assert page.row_schema == "unknown" + assert page.envelope["data"] == {} + assert str(calls[-1].url).startswith(BASE + PATH) + assert dict(calls[-1].url.params) == { + "page": "1", + "size": "1", + "sort": "AGCExecTime", + "dir": "desc", + } + assert "unverified" not in repr(page) + assert len(client.exchanges) == 1 + + +def test_unrecognized_fields_remain_raw_not_typed(): + body = raw_envelope() + body["fields"] = [{"name": "futureField", "dataType": "UNKNOWN_TYPE"}] + body["data"] = [["raw"]] + client, _ = client_for(body) + with client: + assert client.current().row_schema == "unknown" + + +@pytest.mark.parametrize( + "change", + [ + lambda b: b.pop("fields"), + lambda b: b.update(fields=[]), + lambda b: b.update(fields=[{"name": "same"}, {"name": "same"}]), + lambda b: b.update(data=[[1, 2]]), + lambda b: b.update(data=[[1], [2]]), + lambda b: b.update(data="not data"), + lambda b: b.update(report=[]), + lambda b: b["_meta"].update(currentPage=True), + lambda b: b["_meta"].update(currentPage=2), + lambda b: b["_meta"].update(pageSize=1001), + ], +) +def test_envelope_mismatch_fails_closed_with_receipt(change): + body = copy.deepcopy(raw_envelope()) + change(body) + client, _ = client_for(body) + with client: + with pytest.raises(SchemaMismatchError): + client.current() + assert client.exchanges[-1].complete + + +def test_no_service_fallback_after_subscription_denial(): + client, calls = client_for(status=401) + with client: + with pytest.raises(AuthenticationError): + client.current() + assert len(calls) == 2 + assert client.exchanges[-1].receipt.status == 401 + + +def test_esr_bad_content_type(): + client, _ = client_for(media="text/html") + with client, pytest.raises(SchemaMismatchError, match="content type"): + client.current() + + +@pytest.mark.parametrize("size", [0, -1, True, 1001, 1.5]) +def test_size_checks_precede_authentication(size): + client, calls = client_for() + with client, pytest.raises(ValueError): + client.current(size=size) + assert not calls + + +def test_primary_esr_schema_has_no_concrete_rows(): + source = json.loads( + ( + Path(__file__).parent / "fixtures/public/esr_api/source-contract.json" + ).read_text() + ) + assert source["operation"]["method"] == "GET" + assert source["operation"]["urlTemplate"] == PATH + assert source["schemas"]["Report"]["properties"]["data"] == {"type": "object"} + assert "AGCExecTimeFrom" in {q["name"] for q in source["operation"]["query"]} diff --git a/tinyercot/public/archive_api.py b/tinyercot/public/archive_api.py new file mode 100644 index 00000000..06eae6f3 --- /dev/null +++ b/tinyercot/public/archive_api.py @@ -0,0 +1,83 @@ +"""Public archive and bundle contracts with exact failed-transfer receipts. + +Import this module directly. Root metadata controls public file access. +Binary content and retirement do not disable publicly listed file access. +""" + +import httpx + +from ._http import Limits, Payload +from .api import BASE, Credentials +from .api_archives import APIArchiveClient +from .archive_http import EvidenceHTTP, Exchange + + +class ArchiveAPIClient(APIArchiveClient): + """Fetch public metadata and one selected archive or bundle ZIP. + + The inherited products(), archives(), bundles(), download(), and + download_bundle() methods keep their document and ZIP checks. Listings + have no documented query parameters. This client does not invent paging. + Each download sends POST JSON with exactly one source docIds entry. + All completed public attempts, including failures, have an exchange receipt. + """ + + def __init__( + self, + credentials: Credentials, + *, + limits: Limits | None = None, + transport: httpx.BaseTransport | None = None, + ) -> None: + """Create a bounded client without a request. + + Args: + credentials: Explicit authorized public account credentials. + limits: Request, byte, retry, and pacing budgets. + transport: Optional offline transport. + """ + self._credentials = credentials + self._http = EvidenceHTTP(limits or Limits(), transport) + self._token = None + self._expires_at = 0.0 + + @property + def exchanges(self) -> tuple[Exchange, ...]: + """Return public attempt receipts without authentication payloads.""" + return tuple(self._http.exchanges) + + def _authenticated_payload( + self, + path: str, + params: dict, + *, + method: str = "GET", + json_body: dict | None = None, + ) -> Payload: + """Fetch a checked public route without replay after a denial. + + Args: + path: Route already checked by the public metadata adapter. + params: Validated public query parameters. + method: Documented GET or POST verb. + json_body: Selected public document IDs. + + Returns: + Bounded public bytes and their receipt. + + Raises: + PublicDataError: Authentication, transfer, or limits fail. + """ + with self._http.lock: + if self._token is None or self._http.clock() >= self._expires_at: + self.refresh_token() + return self._http.request( + method, + BASE + path, + params=params, + json_body=json_body, + headers={ + "Authorization": f"Bearer {self._token}", + "Ocp-Apim-Subscription-Key": self._credentials.subscription_key, + }, + ) diff --git a/tinyercot/public/archive_http.py b/tinyercot/public/archive_http.py new file mode 100644 index 00000000..7b5bd49f --- /dev/null +++ b/tinyercot/public/archive_http.py @@ -0,0 +1,161 @@ +"""Retain bounded public HTTP receipts, including failed transfers.""" + +import datetime +import hashlib +from dataclasses import dataclass + +import httpx + +from ._http import ( + _HTTP, + AccessDeniedError, + AuthenticationError, + LimitError, + Payload, + RateLimitError, + Receipt, + SchemaMismatchError, + SourceUnavailableError, +) + + +@dataclass(frozen=True) +class Exchange: + """Identify one public HTTP attempt without headers or response text. + + Attributes: + receipt: Hash and byte count of the retained decoded prefix. + method: HTTP verb. + content_type: Response media type without parameters. + complete: Whether the hash covers the complete response. + document_ids: Explicit public POST selection, if present. + """ + + receipt: Receipt + method: str + content_type: str + complete: bool + document_ids: tuple[int, ...] = () + + +class EvidenceHTTP(_HTTP): + """Check public media types and retain receipts for each bounded attempt.""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.exchanges: list[Exchange] = [] + + def request( + self, + method, + url, + *, + params=None, + headers=None, + data=None, + json_body=None, + authentication=False, + ) -> Payload: + """Fetch public bytes with receipts or use the existing secret auth flow. + + Args: + method: Public GET or archive POST verb. + url: Fixed public adapter URL. + params: Public query filters. + headers: Authentication headers; never retained. + data: Authentication form; never retained. + json_body: Public document selection. + authentication: Use the existing auth path without receipts. + + Returns: + Complete bounded bytes after status and media type checks. + + Raises: + PublicDataError: Transport, status, media type, or limits fail. + """ + if authentication: + return super().request( + method, + url, + params=params, + headers=headers, + data=data, + json_body=json_body, + authentication=True, + ) + expected = "application/zip" if method == "POST" else "application/json" + with self.lock: + for attempt in range(self.limits.attempts): + self._pace() + raw = bytearray() + status = 0 + media = "" + complete = False + oversized = False + retry_after = None + source = str(httpx.URL(url, params=params)) + try: + with self.client.stream( + method, + url, + params=params, + headers=headers, + json=json_body, + follow_redirects=False, + ) as response: + status = response.status_code + media = ( + response.headers.get("content-type", "") + .split(";", 1)[0] + .strip() + .lower() + ) + retry_after = response.headers.get("retry-after") + for chunk in response.iter_bytes(chunk_size=65536): + remaining = self.limits.max_bytes - len(raw) + raw.extend(chunk[:remaining]) + if len(chunk) > remaining: + oversized = True + break + complete = not oversized + except httpx.HTTPError: + pass + receipt = Receipt( + source, + datetime.datetime.now(datetime.UTC), + hashlib.sha256(raw).hexdigest(), + len(raw), + status, + ) + self.exchanges.append( + Exchange( + receipt, + method, + media, + complete, + tuple((json_body or {}).get("docIds", ())), + ) + ) + if oversized: + raise LimitError("Public response byte budget exceeded") + if status == 401: + raise AuthenticationError("Public API authentication failed") + if status == 403: + raise AccessDeniedError("Source denied public access") + if status == 200 and complete: + if media != expected: + raise SchemaMismatchError( + "Public response content type differs" + ) + return Payload(bytes(raw), receipt) + retryable = not complete or status in (429, 502, 503, 504) + if not retryable or attempt + 1 == self.limits.attempts: + if status == 429: + raise RateLimitError( + "Public rate-limit attempt budget exhausted" + ) + raise SourceUnavailableError( + f"Public transfer failed (HTTP {status})" + ) + self.sleep(self._retry_delay(retry_after, attempt)) + raise AssertionError("Unreachable request state") diff --git a/tinyercot/public/archive_rows.py b/tinyercot/public/archive_rows.py new file mode 100644 index 00000000..4f2bd2d9 --- /dev/null +++ b/tinyercot/public/archive_rows.py @@ -0,0 +1,268 @@ +"""Sample two observed public nested CSV formats with exact source identities.""" + +import csv +import datetime +import hashlib +import io +import re +import zipfile +import zlib +from dataclasses import dataclass +from decimal import Decimal, InvalidOperation +from typing import Generic, TypeVar + +from ._http import AccessDeniedError, LimitError, Receipt, SchemaMismatchError +from .api import BASE +from .api_archives import APIArchiveFile, APIBundle +from .archives import checked_zip + +CAPACITY_HEADER = ("DeliveryDate", "HourEnding", "AncillaryType", "MCPC", "DSTFlag") +OFFERS_HEADER = ( + "DeliveryDate", + "HourEnding", + "REGDN", + "REGUP", + "RRSPFR", + "RRSFFR", + "RRSUFR", + "ECRSSD", + "ECRSMD", + "NSPIN", + "DSTFlag", +) + + +@dataclass(frozen=True) +class CapacityRow: + """Preserve one observed DAM capacity-price CSV row. + + Attributes: + delivery_date: Source delivery date, without a timezone assumption. + hour_ending: Source hour-ending text, including 24:00. + ancillary_type: Original source ancillary-service identifier. + mcpc: Finite decimal source price. + dst_flag: Original source DST flag, without an inferred UTC interval. + """ + + delivery_date: datetime.date + hour_ending: str + ancillary_type: str + mcpc: Decimal + dst_flag: str + + +@dataclass(frozen=True) +class OfferedCapacityRow: + """Preserve one retired NP4-179-CD total ancillary-offer CSV row. + + Attributes: + delivery_date: Source delivery date. + hour_ending: Original hour-ending text. + regdn: REGDN source quantity. + regup: REGUP source quantity. + rrspfr: RRSPFR source quantity. + rrsffr: RRSFFR source quantity. + rrsufr: RRSUFR source quantity. + ecrssd: ECRSSD source quantity. + ecrsmd: ECRSMD source quantity. + nspin: NSPIN source quantity. + dst_flag: Original DST flag; no UTC conversion is asserted. + """ + + delivery_date: datetime.date + hour_ending: str + regdn: Decimal + regup: Decimal + rrspfr: Decimal + rrsffr: Decimal + rrsufr: Decimal + ecrssd: Decimal + ecrsmd: Decimal + nspin: Decimal + dst_flag: str + + +Row = TypeVar("Row", CapacityRow, OfferedCapacityRow) + + +@dataclass(frozen=True) +class ArchiveRowSample(Generic[Row]): + """Bind a partial typed sample to both ZIP members and the outer receipt. + + Attributes: + rows: Typed source rows up to the caller's limit. + outer_member: Explicitly selected nested ZIP name. + inner_member: Validated CSV name inside that ZIP. + receipt: Exact outer public file identity. + truncated: Whether the selected CSV has another row. + """ + + rows: tuple[Row, ...] + outer_member: str + inner_member: str + receipt: Receipt + truncated: bool + + +def _sample(file, member, max_rows, *, product, kind, header, model): + """Check source identity and both ZIP layers before sampling one CSV. + + Args: + file: Complete bounded public archive with a source receipt. + member: Explicit selected outer ZIP member name. + max_rows: Positive sample ceiling, at most 1000. + product: Exact supported public product. + kind: Exact archive or bundle download route. + header: Pinned source CSV header. + model: Supported row constructor. + + Returns: + A typed partial sample with nested member and receipt identities. + + Raises: + PublicDataError: Identity, size, ZIP, or CSV checks fail. + ValueError: The sample budget is invalid. + """ + if type(max_rows) is not int or not 1 <= max_rows <= 1000: + raise ValueError("max_rows must be between 1 and 1000") + if ( + file.document.emil_id != product + or isinstance(file.document, APIBundle) != (kind == "bundle") + or file.receipt.source_url != f"{BASE}/{kind}/{product}/download" + or file.receipt.status != 200 + ): + raise AccessDeniedError("Unsupported public file identity") + if len(file.body) > 4_000_000: + raise LimitError("CSV sample input byte budget exceeded") + if ( + file.receipt.byte_count != len(file.body) + or file.receipt.sha256 != hashlib.sha256(file.body).hexdigest() + ): + raise SchemaMismatchError("Public file differs from its receipt") + try: + with checked_zip( + file.body, max_members=64, max_expanded_bytes=4_000_000 + ) as outer: + if member not in file.members or not member.endswith(".zip"): + raise SchemaMismatchError("Select an observed nested ZIP member") + nested = outer.read(member) + with checked_zip(nested, max_members=1, max_expanded_bytes=4_000_000) as inner: + entries = inner.infolist() + if ( + len(entries) != 1 + or entries[0].is_dir() + or not entries[0].filename.endswith(".csv") + ): + raise SchemaMismatchError("Expected one CSV inside the selected ZIP") + inner_name = entries[0].filename + text = inner.read(entries[0]).decode("utf-8-sig") + reader = csv.reader(io.StringIO(text, newline=""), strict=True) + if tuple(next(reader, ())) != header: + raise SchemaMismatchError("CSV header differs from the observed schema") + rows = [] + truncated = False + for raw in reader: + if len(rows) == max_rows: + truncated = True + break + if len(raw) != len(header): + raise ValueError + if not re.fullmatch(r"\d{2}/\d{2}/\d{4}", raw[0]): + raise ValueError + month, day, year = (int(value) for value in raw[0].split("/")) + date = datetime.date(year, month, day) + if not re.fullmatch(r"(?:0[1-9]|1\d|2[0-4]):00", raw[1]) or raw[-1] not in { + "Y", + "N", + }: + raise ValueError + numbers = [ + Decimal(v) for v in (raw[3:4] if model is CapacityRow else raw[2:-1]) + ] + if any(not n.is_finite() for n in numbers): + raise ValueError + if model is CapacityRow: + if not raw[2] or len(raw[2]) > 64: + raise ValueError + rows.append(CapacityRow(date, raw[1], raw[2], numbers[0], raw[-1])) + else: + rows.append(OfferedCapacityRow(date, raw[1], *numbers, raw[-1])) + return ArchiveRowSample( + tuple(rows), member, inner_name, file.receipt, truncated + ) + except ( + KeyError, + ValueError, + UnicodeError, + csv.Error, + InvalidOperation, + zipfile.BadZipFile, + zlib.error, + NotImplementedError, + RuntimeError, + ): + raise SchemaMismatchError( + "Public nested CSV differs from its observed contract" + ) from None + + +def sample_capacity_bundle( + file: APIArchiveFile, + *, + member: str, + max_rows: int = 4, +) -> ArchiveRowSample[CapacityRow]: + """Sample one observed NP4-188-CD nested CSV from a selected bundle. + + Args: + file: Public capacity bundle with a verified receipt. + member: Exact selected outer member name. + max_rows: Sample ceiling from 1 through 1000. + + Returns: + Typed partial rows with source and member identities. + + Raises: + PublicDataError: Source identity, ZIP, or CSV checks fail. + ValueError: The row budget is invalid. + """ + return _sample( + file, + member, + max_rows, + product="np4-188-cd", + kind="bundle", + header=CAPACITY_HEADER, + model=CapacityRow, + ) + + +def sample_retired_offers( + file: APIArchiveFile, + *, + member: str, + max_rows: int = 4, +) -> ArchiveRowSample[OfferedCapacityRow]: + """Sample one observed retired NP4-179-CD archive CSV. + + Args: + file: Public retired offer archive with a verified receipt. + member: Exact selected outer member name. + max_rows: Sample ceiling from 1 through 1000. + + Returns: + Typed partial rows without a claim about other retired formats. + + Raises: + PublicDataError: Source identity, ZIP, or CSV checks fail. + ValueError: The row budget is invalid. + """ + return _sample( + file, + member, + max_rows, + product="np4-179-cd", + kind="archive", + header=OFFERS_HEADER, + model=OfferedCapacityRow, + ) diff --git a/tinyercot/public/esr_api.py b/tinyercot/public/esr_api.py new file mode 100644 index 00000000..77e14d5d --- /dev/null +++ b/tinyercot/public/esr_api.py @@ -0,0 +1,138 @@ +"""Bounded raw access to the separate public ESR REST operation. + +The primary operation example has generic fields and data, not a row schema. +This module preserves raw data and validates only the response envelope. +It does not infer rows from the separate website ESR dashboard. +""" + +from dataclasses import dataclass, field +from typing import Any, Literal + +import httpx + +from ._http import Limits, Receipt, SchemaMismatchError +from .api import Credentials, PublicClient +from .archive_http import EvidenceHTTP, Exchange + +BASE = "https://api.ercot.com/api/public-data" +PATH = "/rptesr-m/4_sec_esr_charging_mw" + + +@dataclass(frozen=True) +class RawESRPage: + """Preserve an untyped public ESR page and its exact source identity. + + Attributes: + envelope: Original decoded public JSON; no typed row claim. + receipt: Exact received byte identity. + raw: Original bounded response bytes. + row_schema: Unknown until actual source fields and types are verified. + """ + + envelope: dict[str, Any] = field(repr=False) + receipt: Receipt + raw: bytes = field(repr=False) + row_schema: Literal["unknown"] = "unknown" + + +class ESRAPIClient(PublicClient): + """Fetch only the documented ESR data route using public credentials. + + Subscription access is service-specific. A denial stops the request. + No alternate key, service, or entitlement route is tried. + """ + + def __init__( + self, + credentials: Credentials, + *, + limits: Limits | None = None, + transport: httpx.BaseTransport | None = None, + ) -> None: + """Create the separate ESR client without a request. + + Args: + credentials: Authorized public ESR account and subscription. + limits: Request, byte, retry, and pacing budgets. + transport: Optional offline transport. + """ + self._credentials = credentials + self._http = EvidenceHTTP(limits or Limits(), transport) + self._token = None + self._expires_at = 0.0 + + @property + def exchanges(self) -> tuple[Exchange, ...]: + """Return public attempt receipts without authentication payloads.""" + return tuple(self._http.exchanges) + + def current(self, *, size: int = 1) -> RawESRPage: + """Fetch one page sorted by the documented AGC execution time. + + Args: + size: Maximum requested raw records, from 1 through 1000. + + Returns: + Raw source envelope with unknown row schema and exact receipt. + + Raises: + ValueError: The page size is invalid. + PublicDataError: Status, content type, envelope, or limits fail. + """ + if type(size) is not int or not 1 <= size <= 1000: + raise ValueError("size must be between 1 and 1000") + with self._http.lock: + if self._token is None or self._http.clock() >= self._expires_at: + self.refresh_token() + payload = self._http.request( + "GET", + BASE + PATH, + params={"page": 1, "size": size, "sort": "AGCExecTime", "dir": "desc"}, + headers={ + "Authorization": f"Bearer {self._token}", + "Ocp-Apim-Subscription-Key": self._credentials.subscription_key, + }, + ) + body = payload.json() + try: + if not isinstance(body, dict): + raise TypeError + meta = body["_meta"] + if any( + type(meta[k]) is not int or meta[k] < 0 + for k in ( + "currentPage", + "totalPages", + "pageSize", + "totalRecords", + ) + ): + raise ValueError + if meta["currentPage"] != 1 or not 1 <= meta["pageSize"] <= size: + raise ValueError + if not isinstance(body["report"], dict) or not isinstance( + body["fields"], list + ): + raise TypeError + if body["report"].get("reportProductId", "").lower() != "rptesr-m": + raise ValueError + if not isinstance(body["data"], (dict, list)): + raise TypeError + if isinstance(body["data"], list) and len(body["data"]) > size: + raise ValueError + fields = body["fields"] + names = [f["name"] for f in fields] + if ( + not names + or any(type(n) is not str or not n for n in names) + or len(set(names)) != len(names) + ): + raise ValueError + for row in body["data"] if isinstance(body["data"], list) else (): + if not isinstance(row, list) or len(row) != len(names): + raise ValueError + except (KeyError, TypeError, ValueError, AttributeError): + raise SchemaMismatchError( + "ESR raw envelope differs; row schema remains unknown" + ) from None + return RawESRPage(body, payload.receipt, payload.body) diff --git a/tools/probe_esr_archive.py b/tools/probe_esr_archive.py new file mode 100644 index 00000000..f2238337 --- /dev/null +++ b/tools/probe_esr_archive.py @@ -0,0 +1,305 @@ +"""Probe bounded ESR and archive contracts with an isolated installed wheel.""" + +import argparse +import datetime +import hashlib +import json +import logging +import stat +from contextlib import closing +from dataclasses import asdict +from itertools import islice +from pathlib import Path + +import tinyercot +from tinyercot.public._http import Limits, PublicDataError, Receipt +from tinyercot.public.api import Credentials +from tinyercot.public.api_archives import APIArchive, APIArchiveFile, APIBundle +from tinyercot.public.archive_api import ArchiveAPIClient +from tinyercot.public.archive_rows import sample_capacity_bundle, sample_retired_offers +from tinyercot.public.esr_api import ESRAPIClient +from tinyercot.public.resource_dme import iter_resource_dme + + +def sample_cache(cache: Path, output: Path) -> None: + """Verify installed decoders against exact prior public file receipts. + + Args: + cache: Prior live probe output directory. + output: New directory for typed sample evidence. + + Raises: + SystemExit: The cache or receipt file is unsafe or too large. + PublicDataError: Source bytes or CSV schemas differ. + """ + path = cache / "receipts.json" + if not stat.S_ISREG(path.lstat().st_mode) or path.stat().st_size > 1_000_000: + raise SystemExit("Invalid cached receipt file") + checks = json.loads(path.read_text())["checks"] + records = [] + for check in checks: + if "download_receipt" not in check: + continue + raw_receipt = dict(check["download_receipt"]) + raw_receipt["retrieved_at"] = datetime.datetime.fromisoformat( + raw_receipt["retrieved_at"] + ) + receipt = Receipt(**raw_receipt) + kind = "bundle" if "/bundle/" in receipt.source_url else "archive" + document = (APIBundle if kind == "bundle" else APIArchive)(**check["document"]) + path = cache / f"{kind}-{document.emil_id.upper()}.zip" + if not stat.S_ISREG(path.lstat().st_mode) or path.stat().st_size > 4_000_000: + raise SystemExit("Invalid cached public file") + file = APIArchiveFile( + document, receipt, tuple(check["members"]), path.read_bytes() + ) + record = { + "document": asdict(document), + "receipt": asdict(receipt), + "network_download": False, + "installed_module": tinyercot.__file__, + } + if document.emil_id == "np3-988-er": + with closing(iter_resource_dme(file)) as iterator: + rows = list(islice(iterator, 4)) + record.update( + typed_rows=len(rows), + decoder="iter_resource_dme", + rows=[r.row.model_dump() for r in rows], + ) + else: + decoder = ( + sample_capacity_bundle if kind == "bundle" else sample_retired_offers + ) + sample = decoder(file, member=file.members[0], max_rows=4) + record.update( + typed_rows=len(sample.rows), + decoder=decoder.__name__, + sample=asdict(sample), + ) + records.append(record) + output.mkdir(parents=True) + result = { + "checks": records, + "network_requests": 0, + "runtime_sha256": { + name: hashlib.sha256( + (Path(tinyercot.__file__).parent / "public" / name).read_bytes() + ).hexdigest() + for name in ("archive_rows.py", "resource_dme.py") + }, + } + (output / "samples.json").write_text( + json.dumps(result, indent=2, default=str) + "\n" + ) + print("Verified installed typed samples against cached public file receipts.") + + +def main() -> None: + """Write safe public receipts for explicitly enabled bounded live requests. + + Raises: + SystemExit: The wheel, credential file, or output path is invalid. + """ + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--live", action="store_true") + parser.add_argument("--credentials-file", type=Path) + parser.add_argument("--sample-cache", type=Path) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--bundle-transfer", action="store_true") + parser.add_argument( + "--followup", + action="store_true", + help="Probe one capacity bundle and one retired public file", + ) + args = parser.parse_args() + installed = Path(tinyercot.__file__).resolve() + if "site-packages" not in installed.parts: + raise SystemExit("Use an isolated installed wheel with python -I") + if args.output.exists(): + raise SystemExit("Select a new output directory") + if args.sample_cache: + sample_cache(args.sample_cache, args.output) + return + if not args.live or args.credentials_file is None: + raise SystemExit("Live requests require --live and --credentials-file") + path = args.credentials_file + if ( + not stat.S_ISREG(path.lstat().st_mode) + or stat.S_IMODE(path.stat().st_mode) != 0o600 + or path.stat().st_size > 16000 + ): + raise SystemExit("Use a regular mode-0600 authorized local credential file") + logging.disable(logging.CRITICAL) + values = { + k: json.loads(v) + for line in path.read_text().splitlines() + for k, v in [line.split("=", 1)] + } + credentials = Credentials( + *( + values[k] + for k in ( + "ERCOT_USERNAME", + "ERCOT_PASSWORD", + "ERCOT_SUBSCRIPTION_KEY", + ) + ) + ) + del values + args.output.mkdir(parents=True) + result = { + "installed_module": str(installed), + "checks": [], + "bulk_history": False, + "runtime_sha256": { + name: hashlib.sha256( + (installed.parent / "public" / name).read_bytes() + ).hexdigest() + for name in ("esr_api.py", "archive_api.py", "archive_http.py") + }, + } + + def save(): + (args.output / "receipts.json").write_text( + json.dumps(result, indent=2, default=str) + "\n" + ) + + def check(client, name, action): + start = len(client.exchanges) + record = {"operation": name} + try: + record.update(action()) + record["outcome"] = "supported" + except PublicDataError as error: + record.update(outcome=type(error).__name__, reason=str(error)) + record["exchanges"] = [asdict(x) for x in client.exchanges[start:]] + result["checks"].append(record) + save() + + with ESRAPIClient(credentials, limits=Limits(max_requests=3, attempts=1)) as esr: + + def current(): + page = esr.current(size=1) + (args.output / "esr-current.json").write_bytes(page.raw) + return { + "row_schema": page.row_schema, + "meta": page.envelope["_meta"], + "fields": page.envelope["fields"], + "raw_list_entries": len(page.envelope["data"]) + if isinstance(page.envelope["data"], list) + else None, + } + + if not args.followup: + check(esr, "GET ESR current size=1", current) + + with ArchiveAPIClient( + credentials, + limits=Limits(max_requests=12, max_bytes=4_000_000, attempts=1), + ) as client: + + def root(): + products, receipt = client.products() + return { + "receipt": asdict(receipt), + "products": [ + { + "id": p.emil_id, + "access": p.access.value, + "status": p.status, + "content_type": p.content_type, + "row_schema": "not_assessed", + "file_access": "outstanding" + if p.access.value == "public" + else "blocked", + } + for p in products + ], + } + + check(client, "GET public product root", root) + + def history(identity, kind, transfer=False): + page = ( + client.bundles(identity) + if kind == "bundle" + else client.archives(identity) + ) + data = { + "product_id": identity, + "lifecycle": page.lifecycle, + "documents": len(page.documents), + "total_records": page.total_records, + "page_size": page.page_size, + "total_pages": page.total_pages, + "current_page": page.current_page, + "receipt": asdict(page.receipt), + "row_schema": "unknown", + "file_access": "supported_listing", + "first_document": asdict(page.documents[0]) if page.documents else None, + "last_document": asdict(page.documents[-1]) if page.documents else None, + } + result["checks"].append({"operation": f"GET {kind}/{identity}", **data}) + save() + if transfer and page.documents: + document = ( + (max if args.followup else min)( + page.documents, key=lambda d: d.post_datetime + ) + if kind == "bundle" + else page.documents[0] + ) + result["checks"].append( + {"operation": "selected_document", **asdict(document)} + ) + save() + downloaded = ( + client.download_bundle(document) + if kind == "bundle" + else client.download(document) + ) + data.update( + document=asdict(document), + members=downloaded.members, + download_receipt=asdict(downloaded.receipt), + file_access="supported_zip", + ) + (args.output / f"{kind}-{identity}.zip").write_bytes(downloaded.body) + return data + + if args.followup: + check( + client, + "public capacity bundle transfer", + lambda: history("NP4-188-CD", "bundle", True), + ) + check( + client, + "retired public file transfer", + lambda: history("NP4-179-CD", "archive", True), + ) + print("Wrote bounded follow-up installed-wheel receipts.") + return + + check( + client, + "public archive representative transfer", + lambda: history("NP3-988-ER", "archive", True), + ) + check( + client, + "public bundle boundary", + lambda: history("NP4-190-CD", "bundle", args.bundle_transfer), + ) + for identity in ("NP4-179-CD", "NP3-990-EX", "NP4-765-ER"): + check( + client, + f"public file metadata {identity}", + lambda identity=identity: history(identity, "archive"), + ) + print("Wrote bounded installed-wheel receipts; inspect each operation outcome.") + + +if __name__ == "__main__": + main()