Ingestion: live tests, cookbook recipe, re-verified docs (#17, PR B)

[craigmcchesney]

PR B of #17 (plan/tickets/17/plan.md, "PR B — live tests and docs that need real data"). PR A was #67.

Live tests

• test_ingestion_client_integration.py (rewritten): every ingest confirmed through await_request_statuses().
  • Unary ack + SUCCESS + exact read-back.
  • Unknown providerId and a duplicate PV/first-timestamp re-ingest: both acked, then ERROR.
  • ingest_data_stream() with one server-rejected request (a frame over the one-day span cap): rejected_request_ids, the others SUCCESS, the reject's status REJECTED.
  • Bidi: in-order per-request results including the reject.
  • A split_data_frame()-chunked frame (3+ requests) reading back whole.
  • Array / image / struct / serialized columns reaching SUCCESS.
  • Oversized request → RESOURCE_EXHAUSTED with the split_data_frame() hint, unary and streamed.
• test_query_client_integration.py: test_closed_loop_round_trip is no longer skipped; TestQueryClosedLoop asserts exact values/timestamps, half-open trimming at both bounds, dense alignment across two clocks (gaps are unset values, not zeros), and in-order paging at limit=7.
• test_sample_status_client_integration.py: new TestSampleStatusQueryFiltering labels ingested samples and asserts exclude() / include() return exactly the right rows.
• test_datasets_annotations_integration.py, test_query_helper_relaxations_integration.py: the hand-written stub ingests and probe loops are gone; both use the new tests/integration/ingest_support.py and wait for SUCCESS.

Docs

• New doc/cookbook/ingestion.md, in the worked example's PVs. It was run end to end as one continuous script against the live stack, and the snippet checker's preamble was extended for it.
• query.md and sample-status.md re-verified against real data. Three claims were wrong; I confirmed each in dp-service or dp-grpc:
  1. A config-only query is rejected by the server. QueryParams now rejects it as well (see Code changes). QuerySpec.pvSelector is required, and QueryV2Resolver rejects a query without one.
  2. Sample query results carry no column metadata. dp-service's sample path populates none ("the tabular path carries no column metadata"), so the df.attrs["column_metadata"] example was fiction.
  3. include() without status_codes keeps every densely labeled sample. A model that scores every sample gives each one a status, including its "normal" code 0.
• Other doc updates: the Add ingestion API client (full surface: ingestData + streaming) #17 pointers in the cookbook index and datasets-and-annotations.md are replaced; CLAUDE.md, README.md and NEXT.md are updated.

Code changes

• QueryParams now requires pv_selector, which is a behavior change. It used to accept config_criteria alone, a query the server always rejects. The parameter no longer has a default, so leaving it out raises TypeError at the call, and type checkers flag it too. Passing None or an empty PvSelector() raises ValueError, whose message points at PvQuery.pattern(".*"). NEXT.md lists it as an upgrade item. _build_query_spec() now copies the selector unconditionally.
• iter_ingest_data_bidi_stream() is annotated Generator instead of Iterator. The snippet checker caught that the documented contextlib.closing(...) pattern failed mypy for callers. Runtime behavior is unchanged.

Verification

• ruff, ruff format --check, mypy src/, the cookbook snippet checker and the release-notes checker are all clean.
• Unit: 885 passed.
• Integration against the live ecosystem: 62 passed, three consecutive runs.

Closes #17

🤖 Generated with Claude Code

https://claude.ai/code/session_01QkZu3MPfUopTisahmvhakD

[Copilot]

Copilot review overview

🟢 Approval recommended

The changes consistently implement PR B’s testing and documentation scope without unresolved correctness issues.

Review effort: Balanced
Findings: None

What changed in this PR

Completes issue #17’s live verification and documentation for ingestion workflows.

Changes:

• Adds shared confirmed-ingestion helpers and comprehensive live integration coverage.
• Adds the ingestion cookbook and corrects query/sample-status documentation.
• Exposes bidi streams as Generator for typed contextlib.closing() support.

File	Description
src/​dp_python_lib/​client/​ingestion_client.py	Corrects bidi stream return typing.
tests/​integration/​ingest_support.py	Adds shared confirmed-ingestion helpers.
tests/​integration/​test_ingestion_client_integration.py	Adds end-to-end ingestion scenarios.
tests/​integration/​test_query_client_integration.py	Adds closed-loop query verification.
tests/​integration/​test_sample_status_client_integration.py	Tests live sample-status filtering.
tests/​integration/​test_query_helper_relaxations_integration.py	Migrates setup to confirmed ingestion.
tests/​integration/​test_datasets_annotations_integration.py	Replaces raw-stub ingestion.
doc/​cookbook/​ingestion.md	Adds the ingestion recipe.
doc/​cookbook/​query.md	Corrects and verifies query behavior.
doc/​cookbook/​sample-status.md	Corrects filtering guidance.
doc/​cookbook/​datasets-and-annotations.md	Updates column-builder guidance.
doc/​cookbook/​README.md	Adds ingestion to the cookbook index.
.dev/​tools/​check-cookbook-snippets.py	Supports ingestion snippets.
README.md	Links ingestion documentation.
doc/​release-notes/​NEXT.md	Records user-visible changes.
plan/​tickets/​17/​plan.md	Records implementation findings.
CLAUDE.md	Documents verified behavior and tests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.