Add typed historical market and operational data

.gitattributes

@@ -5,6 +5,11 @@ tools/inputs/operations.json linguist-generated=true
 tools/inputs/esr-operations.json linguist-generated=true
 tools/inputs/products.json linguist-generated=true
 tools/inputs/*-sample.json linguist-generated=true
+# Keep captured HTML byte-for-byte, including ERCOT's trailing whitespace.
+tools/inputs/*-index.html linguist-generated=true -whitespace
 tools/inputs/samples/** linguist-generated=true
 tools/inputs/dashboards/** linguist-generated=true
 # Hand-maintained mappings, overrides and evidence remain visible.
+tools/inputs/generation-profiles/csv-header-samples.json linguist-generated=true
+tinyercot/_generated_esr.py linguist-generated=true
+tools/inputs/esr/operations.json linguist-generated=true

CLAUDE.md

@@ -3,7 +3,12 @@
 Keep the SDK small and fully typed. Product methods and row models are generated;
 handwritten code handles shared transport, typed metadata, and public dashboards.
 
+Focus the release on actionable market and operational time series for retailers,
+battery operators, and energy traders. Planning studies, profile-key workbooks,
+and document-specific summary extraction are outside the release scope.
+
 - `tinyercot/_client.py`: authentication, sync/async retrieval, pagination, metadata, ZIP downloads.
+- `tinyercot/_history.py`: shared typed CSV and nested ZIP readers. Verified mappings live in `tools/inputs/history-formats.json`.
 - `tinyercot/_generated.py`: generated typed product namespaces; never edit manually.
 - `tinyercot/_dashboards.py`: typed public dashboard payloads and retrieval.
 - `tools/generate_client.py`: offline generation from `tools/inputs/operations.json`

MANIFEST.in

@@ -1,3 +1,5 @@
 include api_response_fields.json
-recursive-include tools *.py *.json
+recursive-include tools *.py *.md *.json *.csv *.xlsx *.pdf *.zip *.html
 recursive-include tests *.py
+recursive-include examples *.py *.md
+recursive-include docs *.md

README.md

@@ -1,79 +1,109 @@
 # tinyercot
 
-A small, fully typed client for ERCOT public data.
+A small, fully typed ERCOT data client for retailers, battery operators, and
+energy traders. Query recent data, stream historical archives and monthly
+bundles, or read public load and generation files. Coverage focuses on data for market
+decisions, settlement and resource planning.
 
-**Development status:** all 242 Public Reports endpoints have typed methods and captured-response tests. The separate ESR API is awaiting an enabled subscription key. MIS is not included.
+## Install and connect
+
+This development branch contains the API described below. Install from its checkout:
+
+```sh
+pip install .
+```
+
+Set `ERCOT_USERNAME`, `ERCOT_PASSWORD`, and `ERCOT_SUBSCRIPTION_KEY`, or pass
+`username`, `password`, and `subscription_key` to `Client`. Credentials are read
+lazily; anonymous public downloads do not need them.
 
 ```python
 from datetime import date
 from tinyercot import Client
 
 with Client() as ercot:
-    prices = ercot.np4_190_cd.dam_stlmnt_pnt_prices_iter(
-        deliveryDateFrom=date(2026, 1, 1),
-        deliveryDateTo=date(2026, 1, 2),
+    for price in ercot.np4_190_cd.dam_stlmnt_pnt_prices_iter(
+        deliveryDateFrom=date(2026, 9, 1),
+        deliveryDateTo=date(2026, 9, 1),
         settlementPoint="HB_HOUSTON",
-    )
-    for price in prices:
+    ):
         print(price.deliveryDate, price.settlementPointPrice)
 ```
 
-Pass `username`, `password`, and `subscription_key` to `Client`, or set
-`ERCOT_USERNAME`, `ERCOT_PASSWORD`, and `ERCOT_SUBSCRIPTION_KEY`.
-Authentication is lazy; the client refreshes tokens and retries transient failures.
+Methods have named, typed filters and Pydantic row models. A method returns one
+`Page[Row]`; `_iter` streams every page. `_async` and `_iter_async` provide async
+access. Dates, decimals, missing values, and DST flags retain their source meaning.
 
-Each product exposes generated methods with named, typed filters and Pydantic row
-models. A method returns one `Page[Row]`; its `_iter` variant streams all pages.
-`_async` and `_iter_async` provide asynchronous access. Date and decimal values
-retain their Python types. Nullable source cells are represented by `None`.
+## Get the available history
 
-```python
-from pathlib import Path
-from tinyercot import Client
+Use the same method name with `_history`. `backfill()` combines monthly bundles
+with individual archives, including older and bundle-only reports:
 
+```python
 with Client() as ercot:
-    products = ercot.products()
-    history = ercot.archives("np4-190-cd")
-    document = history.archives[0]
-    Path("prices.zip").write_bytes(
-        ercot.download("np4-190-cd", [document.docId])
-    )
-
-    bundles = ercot.bundles("np4-190-cd")
-    Path("monthly.zip").write_bytes(
-        ercot.download("np4-190-cd", [bundles.bundles[0].docId], kind="bundle")
-    )
+    history = ercot.np4_190_cd.dam_stlmnt_pnt_prices_history
+    for price in history.backfill(
+        where=lambda row: row.settlementPoint == "HB_HOUSTON",
+        batch_size=25,
+    ):
+        print(price.deliveryDate, price.hourEnding, price.settlementPointPrice)
 ```
 
-Archive and bundle listings expose typed metadata. Downloads return the original
-ZIP bytes. Listing metadata includes the server's total and page counts; `iter_documents()` streams all listing pages.
+This requests all retained publications and can take considerable time. The
+predicate filters typed rows after download. Original document IDs prevent
+fetching the same publication twice; distinct corrections and repeated source
+rows remain intact. Output is streamed without sorting.
+
+`posted_from` and `posted_to` select **publication times**, which can differ from
+operating dates. Omit them for the widest history, including bundle-only reports.
+Use `publications()` to retain each archive's original posting metadata for
+forecast backtests. A bundle's timestamp does not establish when its forecasts
+were available. `read(zip_bytes)` decodes saved downloads with the same types.
+
+Runnable exports from this repository:
+
+```sh
+uv run python -m examples.market_day 2026-09-01 --point HB_HOUSTON --output market-data
+uv run python -m examples.price_history dam --point HB_HOUSTON --output dam-history.jsonl
+uv run python -m examples.price_history rt --point HB_HOUSTON --output rt-history.jsonl
+```
+
+See the [market-data guide](docs/market-data.md) to choose price, load, forecast,
+ancillary-service, outage, and disclosure readers, and the
+[examples](examples/README.md) for export details and forecast vintages.
+
+## Coverage
+
+The September 7, 2026 inventory check matched **242 Public Reports queries** and
+**249 HTTP operations** including shared metadata/download operations. Generated
+history readers cover **289 tables across 113 products**. This establishes API
+and known-format coverage, not uninterrupted history for every dataset.
+
+Full-source comparisons cover retained DAM settlement prices, DAM ancillary
+prices, and weather-zone actual load. Other products have sampled historical
+layout checks; see the [coverage evidence](docs/data-coverage.md). Available
+history and source gaps differ by product. The separate ESR service adds one
+generated query and its history reader through `ESRClient`, using
+`ERCOT_ESR_SUBSCRIPTION_KEY` with the same username/password. See the
+[ESR workflow](docs/usage.md#energy-storage-four-second-data) for verified retention
+and source freshness. MIS remains outside the current scope.
+
+Direct public services include `hourly_load`, `fuel_mix`, `load_profiles`,
+`loss_factors`, `load_forecast_performance`, and `dashboards`. Install `tinyercot[files]` for XLS/XLSX/XLSB readers
+or `tinyercot[pdf]` for supported PDF tables. Detailed source-specific examples
+remain in the [usage reference](docs/usage.md).
 
 ## Development
 
-Runtime dependencies are `httpx`, `httpx-retries`, and `pydantic`. The handwritten
-core handles authentication, transport, pagination, and metadata. Product methods
-and row models are generated from saved ERCOT operation and field definitions.
+Product methods and row models are generated from saved ERCOT definitions. The
+runtime depends on `httpx`, `httpx-retries`, and `pydantic`; optional file parsers
+are loaded when used. Source fixtures support tests and generation and are
+excluded from the installed wheel.
 
 ```sh
 uv sync --group dev
 uv run python tools/generate_client.py
-uv run mypy tinyercot tests/typing_client.py --strict --follow-untyped-imports
+uv run mypy tinyercot tests/typing_client.py examples --strict --follow-untyped-imports
 uv run pytest
 uv build
 ```
-
-Generation fails if an endpoint has no field schema. During discovery only,
-`--allow-incomplete` emits available endpoints and reports the missing count.
-Public dashboards are available through `ercot.dashboards`: fuel mix, grid
-conditions, energy storage, generation outages, DC-tie flows, system prices,
-supply/demand, combined wind/solar, system demand, ancillary services, weather,
-and the current locational price map.
-
-Source-specific corrections live in `tools/inputs/type-overrides.json`,
-`row-fields.json`, and `query-defaults.json`. ERCOT declares hour-ending labels
-and offer identifiers as numeric on two reports. The ESR-west report includes
-two metadata-only columns absent from its rows and has an invalid default sort;
-its generated method uses `SCEDTimestamp` by default. Captured responses support
-these small corrections. Large forecasts may need a date filter or a longer
-client timeout. Specify `sort` when supplying `dir`. Tests use saved public responses and mocked
-HTTP; credentials are not part of the test suite.