#16 PR A: bucket query client and bucket_conversions

[craigmcchesney]

Refs #16. This is PR A of the two-PR plan in plan/tickets/16/plan.md (merged in #71): the client, the conversions, and unit tests. PR B will add the live tests and the cookbook, and will close #16.

What's in it

query_client.py

• New methods query_buckets(), iter_query_buckets(), and iter_query_buckets_stream(). They take the same QueryParams as the three sample methods and keep their exact contracts (D1).
• A sample_status_filter is refused with a ValueError before any RPC (D2).
• useSerializedColumns is forced False. excludeColumnMetadata is passed through (D3).
• QueryBucketsApiResult exposes .data_buckets and .next_page_token, plus .to_dataframes(). It deliberately has no .to_dataframe().
• Both stream senders now go through a private _iter_stream() (D9). The sample stream's yielded error texts are unchanged, and the existing tests pass untouched.
• Docstrings updated: limit counts buckets on this path, and the "future release" and seam notes are rewritten.

bucket_conversions.py (new, D4–D8, D10)

• bucket_column() and bucket_to_data_frame() view a bucket as a one-column common.DataFrame.
• bucket_timestamps() and bucket_values() read its axis and values.
• trim_bucket():
  • exact and half-open, using bisect_left on both bounds, so repeated timestamps are handled;
  • reuses _slice_frame(), so a SamplingClock keeps its clock form;
  • raises on a decreasing axis, a serialized bucket, or begin >= end.
• buckets_by_pv() groups buckets by PV.
• [analysis] extra: buckets_to_dataframes() returns dict[pv, DataFrame], and query_buckets_to_dataframes() adds max_buckets.
• Read-side checks only: nothing calls validate_data_frame() or timestamp_count(). Before slicing, trim_bucket() runs the read path's alignment check, so an array column with absent dims raises instead of being cut in the wrong-sized blocks.

Docs:

• A NEXT.md section.
• CLAUDE.md Key Files entries.
• Stale "interface to v2 bucket-oriented query API (queryBuckets / queryBucketsStream) #16 is future work" docstrings in data_frame_conversions and its tests.

One addition beyond the plan, for review

The plan's D7 lists enum_ids as the structural attr carried even under exclude_column_metadata. The per-PV frames also carry dimensions, image_descriptors, and schema_ids in attrs, keyed by PV. Without them, an array, image, or struct column's object cells can't be interpreted, and the consistency check already guarantees they are uniform across a PV's buckets. Dropping them would repeat the #6 "carry it, then lose it" defect. This is easy to remove if you'd rather keep strictly to the plan.

trim_bucket() finds both bounds with bisect_left for both axis forms, a SamplingClock included, rather than computing them arithmetically for a clock. The plan allowed either (it said "or"), and the results are identical.

Verification

• pytest tests/unit/: 960 passed. This includes 53 new tests in test_bucket_conversions.py and the new bucket classes in test_query_client.py.
• ruff check ., ruff format --check ., mypy src/, the cookbook snippet checker, and the release-notes checker are all clean.
• Typed-stub check (T9): generated .pyi stubs from dp-grpc e775244 with grpcio-tools==1.84.0 and mypy-protobuf==5.1.0, using generate-python-stubs.yml's flags and import fixup, and ran mypy src/ against them. It caught one issue: WhichOneof() is typed str | None. That is fixed, and the check now passes.
• Not run live. That is PR B.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CajTMkjkkzeoWXLSgMpn5k

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation matches the planned PR A scope and has thorough unit coverage for request handling, conversion behavior, and failure cases.

Review effort: Balanced
Findings: None

What changed in this PR

Adds bucket-oriented v2 query support and conversions for whole stored DataBucket records.

Changes:

• Adds unary, paged, and streaming bucket-query APIs.
• Adds pure-Python and pandas bucket conversions with exact trimming.
• Adds comprehensive unit tests and documentation.

File	Description
src/​dp_python_lib/​client/​query_client.py	Implements bucket-query APIs and shared stream handling.
src/​dp_python_lib/​client/​bucket_conversions.py	Adds bucket reading, trimming, grouping, and pandas assembly.
src/​dp_python_lib/​client/​data_frame_conversions.py	Updates bucket-query documentation.
src/​dp_python_lib/​client/​__init__.py	Exports QueryBucketsApiResult.
tests/​unit/​test_query_client.py	Tests bucket requests, paging, streaming, and errors.
tests/​unit/​test_bucket_conversions.py	Tests all bucket conversion paths and edge cases.
tests/​unit/​test_data_frame_conversions.py	Updates conversion test documentation.
doc/​release-notes/​NEXT.md	Documents the new public functionality.
CLAUDE.md	Records bucket-query architecture and testing details.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.