Lazy asset message parsing

.docs/packages.d2

@@ -43,7 +43,3 @@ public.tilebox-workflows -> public.tilebox-core: {
 public.tilebox-storage -> public.tilebox-core: {
   label: depends on
 }
-
-public.tilebox-grpc -> public.tilebox-core: {
-  label: compatibility dependency only
-}

.github/workflows/publish_tilebox_core.yml

@@ -17,9 +17,9 @@ jobs:
           fetch-depth: 0
           lfs: true
       - name: Set up uv
-        uses: astral-sh/setup-uv@v10.1.0
+        uses: astral-sh/setup-uv@v10.3.3
         with:
-          python-version: "3.13"
+          python-version: "3.14"
           enable-cache: true
           cache-python: true
           cache-dependency-glob: "uv.lock"

AGENTS.md

@@ -6,10 +6,9 @@ This repository is the Python monorepo for Tilebox SDK packages. It is organized
 
 ## Monorepo Architecture
 
-The workspace contains four implementation packages and a compatibility package:
+The workspace contains four implementation packages:
 
 - `tilebox-core`: Public `tilebox.query` value types and `tilebox.errors` exceptions; private shared infrastructure, generated messages, conversions, transports, pagination, replay, and concurrency helpers under `_tilebox`. Only this distribution packages `_tilebox`.
-- `tilebox-grpc`: Empty compatibility package depending on `tilebox-core>=0.68.0`, retained for one release.
 - `tilebox-datasets`: Datasets SDK (`tilebox.datasets`) with sync + async clients, query/data model helpers, protobuf conversion, and generated datasets-related protobuf stubs.
 - `tilebox-workflows`: Workflows SDK (`tilebox.workflows`) with job/cluster/automation clients, task model + runner, observability/tracing, and generated workflows protobuf stubs.
 - `tilebox-storage`: Storage SDK (`tilebox.storage`) for provider-specific payload downloads (ASF, Copernicus, Umbra, Landsat, local FS), built mostly on async internals with sync wrappers.

CHANGELOG.md

@@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.69.0] - 2026-11-09
+
+### Changed
+
+- `tilebox-datasets`: Return compact `LazyMessage` wrappers for `Assets`, `Links`, `Storage`, and
+  `Authentication` fields in query and `find()` results, including repeated elements. Call `.get()`
+  to decode and cache the generated message; `AssetCollection.from_datapoint()` unwraps these fields
+  automatically. Ingestion accepts both wrappers and generated messages, forwarding untouched payloads
+  without decoding and preserving edits to decoded messages. Nested decoding errors are deferred until
+  decoding. Display uses a formatter to decode sampled cells and show counts such as `Assets(23)`,
+  falling back to `Assets(…)` on `ValueError`. Equality comparisons can also trigger decoding. Primitive fields
+  keep their existing xarray representation.
+- `tilebox-datasets`: Reduce protobuf-to-xarray conversion overhead with batched geometry decoding,
+  direct UUID formatting, and a class-check fast path that retains support for dynamic message types.
+
+### Removed
+
+- `tilebox-grpc`: Remove the compatibility package and its release workflow after the 0.68 release.
+  Use `tilebox-core` directly instead.
+
 ## [0.68.0] - 2026-10-29
 
 ### Added
@@ -734,7 +754,8 @@ the first client that does not cache data (since it's already on the local file
 - Released under the [MIT](https://opensource.org/license/mit) license.
 - Released packages: `tilebox-datasets`, `tilebox-workflows`, `tilebox-storage`, `tilebox-grpc`
 
-[Unreleased]: https://github.com/tilebox/tilebox-python/compare/v0.68.0...HEAD
+[Unreleased]: https://github.com/tilebox/tilebox-python/compare/v0.69.0...HEAD
+[0.69.0]: https://github.com/tilebox/tilebox-python/compare/v0.68.0...v0.69.0 
 [0.68.0]: https://github.com/tilebox/tilebox-python/compare/v0.67.0...v0.68.0
 [0.67.0]: https://github.com/tilebox/tilebox-python/compare/v0.66.0...v0.67.0
 [0.65.0]: https://github.com/tilebox/tilebox-python/compare/v0.64.0...v0.65.0

CONTRIBUTING.md

@@ -11,7 +11,6 @@ organize our code into multiple packages that then share the common `tilebox` py
 `_tilebox` infrastructure namespace. Datasets, workflows, and storage depend on core;
 scientific dependencies stay in datasets. Core and workflows have no runtime dependency
 on datasets. Their cross-SDK tests run in the all-packages workspace environment.
-`tilebox-grpc` is a dependency-only compatibility package retained for the 0.68 release.
 
 ## Required tooling
 
@@ -88,7 +87,7 @@ Raise these minimums only when a package needs a newer sibling. Make core availa
 and datasets before storage. The release workflows run independently; do not announce installation availability
 until all packages have published successfully.
 Before the first core release, configure PyPI trusted publishing for `tilebox-core` using this repository's
-`publish_tilebox_core.yml` workflow. The compatibility `tilebox-grpc` wheel must contain no Python modules.
+`publish_tilebox_core.yml` workflow.
 All that is needed is to create a github release for a given tag. This can be done in the github web UI or on the
 command line.
 

README.md

@@ -52,7 +52,6 @@ The SDKs share types and native Connect RPC infrastructure in `tilebox-core`, in
 automatically with datasets, workflows, and storage. Use `tilebox.query` for shared
 intervals and pagination, and `tilebox.errors` for SDK exceptions. Infrastructure under
 `_tilebox` is private. Core and workflows do not require the datasets scientific stack.
-`tilebox-grpc` is a dependency-only compatibility package for the 0.68 release.
 
 ## Getting started
 

prek.toml

@@ -19,7 +19,7 @@ hooks = [
 
 [[repos]]
 repo = "https://github.com/charliermarsh/ruff-pre-commit"
-rev = "v0.17.3"
+rev = "v0.17.4"
 hooks = [
   {
     id = "ruff-check",

pyproject.toml

@@ -37,15 +37,13 @@ bypass-selection = true
 [tool.uv.workspace]
 members = [
     "tilebox-core",
-    "tilebox-grpc",
     "tilebox-datasets",
     "tilebox-storage",
     "tilebox-workflows",
 ]
 
 [tool.uv.sources] # don't install from pypi, use the workspace packages
 tilebox-core = { workspace = true }
-tilebox-grpc = { workspace = true }
 tilebox-datasets = { workspace = true }
 tilebox-storage = { workspace = true }
 tilebox-workflows = { workspace = true }

tilebox-datasets/README.md

@@ -87,6 +87,44 @@ results = s2a_l1c.query(
 print(f"Found {results.sizes['time']} datapoints")  # Found 979 datapoints
 ```
 
+## Lazy metadata fields
+
+Queries and `find()` return `LazyMessage` wrappers for fields of type `Assets`, `Links`,
+`Storage`, and `Authentication`, including elements of repeated fields. Other fields
+keep their existing xarray representation. Wrappers display counts such as `Assets(23)`,
+`Links(8)`, or `Storage(1 scheme)`. Displaying a wrapper decodes and caches its message;
+xarray's default display formats only a sample of rows, not the entire result.
+
+For a result with an `assets` field:
+
+```python
+from tilebox.datasets.assets import AssetCollection
+
+row = results.isel(time=0)
+message = row.assets.item().get()  # Decode to the generated Assets message.
+assets = AssetCollection.from_datapoint(row)  # Unwraps assets and their storage/authentication context.
+```
+
+`get()` caches the mutable message in that wrapper and releases its encoded bytes.
+Subsequent calls return the same message. Different datapoints do not share decoded
+messages, but an xarray shallow copy shares its wrappers with the original.
+Missing fields remain missing; a present empty message still has a wrapper.
+Malformed nested messages raise when decoded, rather than during the query.
+Display catches `ValueError` from decoding or formatting and falls back to `Assets(…)`
+(or the corresponding type name); other errors propagate. Explicit `get()` still raises
+decoding errors. Custom wrappers
+can supply `repr_formatter=` with a function that takes the decoded message and returns
+a string. Without a formatter, display uses the placeholder without decoding.
+Equality comparisons, including xarray merges, decode compatible messages and use
+protobuf equality, including its treatment of NaN values as unequal between
+independently decoded messages.
+
+Ingestion accepts wrappers alongside ordinary generated messages in records, columns,
+DataFrames, and xarray Datasets. Untouched wrappers forward their protobuf bytes without
+decoding; decoded wrappers serialize the current message, including any edits. Lazy
+payload validation is deferred to decoding or the ingestion server. Use `get()` first
+if you need to check that a payload can be decoded before sending it.
+
 ## Enum fields
 
 Define a scalar enum with an ordered list of public strings:

tilebox-datasets/tests/protobuf_conversion/test_protobuf_xarray.py

@@ -19,14 +19,13 @@
     FileDescriptorSet,
     OneofDescriptorProto,
 )
-from shapely import MultiPolygon, Polygon, from_wkb
+from shapely import MultiPolygon, Polygon, from_wkb, get_srid, set_srid
 from xarray.testing import assert_equal
 
 from tests.data.datapoint import example_datapoints
 from tests.example_dataset.schema import DESCRIPTOR, ExampleDatapoint
 from tilebox.datasets.data.datasets import AnnotatedType
 from tilebox.datasets.message_pool import get_message_type
-from tilebox.datasets.protobuf_conversion.field_types import _AssetsDisplay
 from tilebox.datasets.protobuf_conversion.protobuf_xarray import MessageToXarrayConverter
 from tilebox.datasets.protobuf_conversion.to_protobuf import to_messages
 from tilebox.datasets.query.time_interval import timestamp_to_datetime, us_to_datetime
@@ -122,6 +121,33 @@ def test_supplied_imports_preserve_nanoseconds_and_message_values(nanos: int) ->
     assert to_messages(dataset, message_type) == messages
 
 
+@pytest.mark.parametrize("supplied_imports", [False, True])
+def test_geometry_batches_preserve_missing_empty_and_srid(supplied_imports: bool) -> None:
+    files = [well_known_types_pb.desc().proto, DESCRIPTOR] if supplied_imports else [DESCRIPTOR]
+    message_type = get_message_type(AnnotatedType(FileDescriptorSet(file=files), "tests.scientific.Datapoint", []))
+    polygon = set_srid(Polygon([(11, 47), (12, 47), (12, 49), (11, 47)]), 4326)
+    messages = to_messages([{}, {"geometry": Polygon()}, {}, {"geometry": polygon}, {}, {}], message_type)
+    messages = [message_type.from_binary(message.to_binary()) for message in messages]
+    converter = MessageToXarrayConverter(initial_capacity=1)
+    converter.convert(messages[0])
+    converter.convert_all(messages[1:4])
+    converter.convert_all(messages[4:])
+    dataset = converter.finalize("row", skip_empty_fields=True)
+    assert set(dataset.data_vars) == {"geometry"}
+    values = dataset.geometry.to_numpy()
+    assert [value is None for value in values] == [True, False, True, False, True, True]
+    assert values[1].is_empty
+    assert values[3].equals_exact(polygon, tolerance=0)
+    assert get_srid(values[3]) == 4326
+
+
+@pytest.mark.parametrize("length", [0, 15, 17])
+def test_conversion_rejects_invalid_uuid_lengths(length: int) -> None:
+    message = ExampleDatapoint(some_identifier=well_known_types_pb.UUID(uuid=b"x" * length))
+    with pytest.raises(ValueError, match="bytes is not a 16-char string"):
+        MessageToXarrayConverter().convert_all([message])
+
+
 @given(example_datapoints(generated_fields=True, missing_fields=False))
 def test_convert_datapoint(datapoint: ExampleDatapoint) -> None:  # noqa: PLR0915
     converter = MessageToXarrayConverter()
@@ -206,9 +232,10 @@ def test_convert_datapoint(datapoint: ExampleDatapoint) -> None:  # noqa: PLR091
         assert isinstance(dataset.some_repeated_geometry[i].item(), Polygon | MultiPolygon)
 
 
-def test_convert_stac_messages_to_protobuf_py() -> None:
+@pytest.mark.parametrize("dynamic_imports", [False, True])
+def test_convert_stac_messages_to_protobuf_py(dynamic_imports: bool) -> None:
     message_types = {
-        "assets": ("datasets.stac.v1.Assets", Assets(assets=[Asset(key="preview")]), _AssetsDisplay),
+        "assets": ("datasets.stac.v1.Assets", Assets(assets=[Asset(key="preview")]), Assets),
         "authentication": ("datasets.stac.v1.Authentication", Authentication(), Authentication),
         "links": ("datasets.stac.v1.Links", Links(links=[Link(href="https://example.com")]), Links),
         "provider": ("datasets.stac.v1.Provider", Provider(name="Tilebox"), Provider),
@@ -219,6 +246,11 @@ def test_convert_stac_messages_to_protobuf_py() -> None:
         ),
         "storage": ("datasets.stac.v1.Storage", Storage(), Storage),
     }
+    # Unknown field 100 (varint 7) must survive both native reuse and dynamic conversion.
+    message_types = {
+        name: (type_name, target_type.from_binary(value.to_binary() + b"\xa0\x06\x07"), target_type)
+        for name, (type_name, value, target_type) in message_types.items()
+    }
     file_descriptor = FileDescriptorProto(
         name="tests/protobuf_conversion/stac_datapoint.proto",
         package="tests.protobuf_conversion",
@@ -251,15 +283,21 @@ def test_convert_stac_messages_to_protobuf_py() -> None:
                 type_name=f".{message_name}",
             )
         )
-    message_type = get_message_type(
-        AnnotatedType(FileDescriptorSet(file=[file_descriptor]), "tests.protobuf_conversion.StacDatapoint", [])
-    )
+    schema = AnnotatedType(FileDescriptorSet(file=[file_descriptor]), "tests.protobuf_conversion.StacDatapoint", [])
+    message_type = get_message_type(schema)
 
     values = {}
     for field_name, (_, source_value, _) in message_types.items():
         values[field_name] = source_value
         values[f"related_{field_name}"] = [source_value]
-    messages = [message_type(), message_type(**values)]
+    encoded = message_type(**values).to_binary()
+    if dynamic_imports:
+        imports = {
+            target_type.desc().file.name: target_type.desc().file.proto for _, _, target_type in message_types.values()
+        }
+        schema.descriptor_set.file.extend(imports.values())
+        message_type = get_message_type(schema)
+    messages = [message_type(), message_type.from_binary(encoded), message_type.from_binary(encoded)]
 
     converter = MessageToXarrayConverter()
     converter.convert_all(messages)
@@ -269,23 +307,23 @@ def test_convert_stac_messages_to_protobuf_py() -> None:
         assert dataset[field_name].dtype == object
         assert dataset[field_name][0].item() is None
         converted_value = dataset[field_name][1].item()
-        assert isinstance(converted_value, target_type)
+        assert type(converted_value) is target_type
         assert converted_value.to_binary() == source_value.to_binary()
-        if field_name == "assets":
-            assert isinstance(converted_value, Assets)
-            assert str(converted_value) == "1 assets"
-            assert repr(converted_value) == "[preview]"
+        if dynamic_imports:
+            assert not isinstance(getattr(messages[1], field_name), target_type)
+        else:
+            assert converted_value is getattr(messages[1], field_name)
+        # Equal contents from independent datapoints must not become shared mutable messages.
+        assert converted_value is not dataset[field_name][2].item()
 
         repeated_field_name = f"related_{field_name}"
         assert dataset[repeated_field_name].dtype == object
         converted_repeated_value = dataset[repeated_field_name][1, 0].item()
-        assert isinstance(converted_repeated_value, target_type)
+        assert type(converted_repeated_value) is target_type
         assert converted_repeated_value.to_binary() == source_value.to_binary()
-
-    converted_assets = dataset["assets"][1].item()
-    assert isinstance(converted_assets, Assets)
-    assert repr(converted_assets) == "[preview]"
-    assert str(converted_assets) == "1 assets"
+        if not dynamic_imports:
+            assert converted_repeated_value is getattr(messages[1], repeated_field_name)[0]
+        assert converted_repeated_value is not dataset[repeated_field_name][2, 0].item()
 
     roundtripped = to_messages(dataset, message_type)
     assert roundtripped == messages
@@ -294,18 +332,13 @@ def test_convert_stac_messages_to_protobuf_py() -> None:
         input_data = {}
         record = {}
         for field_name, (_, source_value, target_type) in message_types.items():
-            value_type = Assets if field_name == "assets" else target_type
-            target_value = value_type.from_binary(source_value.to_binary())
+            target_value = target_type.from_binary(source_value.to_binary())
             input_data[field_name] = [target_value]
             input_data[f"related_{field_name}"] = [repeated_container([target_value])]
             record[field_name] = target_value
             record[f"related_{field_name}"] = repeated_container([target_value])
-        assert to_messages(input_data, message_type) == [message_type(**values)]
-        assert to_messages([record], message_type) == [message_type(**values)]
-
-    html = dataset._repr_html_()
-    assert "preview" in html
-    assert "access_profiles" not in html
+        assert to_messages(input_data, message_type) == [message_type.from_binary(encoded)]
+        assert to_messages([record], message_type) == [message_type.from_binary(encoded)]
 
 
 def test_convert_scalar_and_repeated_enums_to_names_and_round_trip() -> None:

tilebox-datasets/tests/protobuf_conversion/test_to_protobuf.py

@@ -12,13 +12,15 @@
     FileDescriptorProto,
     FileDescriptorSet,
 )
+from shapely import Polygon, from_wkb, get_srid, set_srid, to_wkb
 
 from tests.data.datapoint import example_datapoints, example_pandas_datapoints
 from tests.example_dataset.schema import ExampleDatapoint
 from tilebox.datasets.data.datasets import AnnotatedType
 from tilebox.datasets.message_pool import get_message_type
 from tilebox.datasets.protobuf_conversion.protobuf_xarray import MessageToXarrayConverter
 from tilebox.datasets.protobuf_conversion.to_protobuf import to_messages
+from tilebox.datasets.v1.well_known_types_pb import Geometry
 
 
 def test_ingestion_preserves_fields_with_escaped_python_names() -> None:
@@ -69,6 +71,24 @@ def test_xarray_dataset_to_protobuf_messages(messages: list[ExampleDatapoint]) -
             )
 
 
+@pytest.mark.parametrize("srid", [0, 4326, 3857])
+def test_geometry_srid_survives_xarray_ingestion_roundtrip(srid: int) -> None:
+    polygon = set_srid(Polygon([(11, 47), (12, 47), (12, 49), (11, 47)]), srid)
+    geometry = Geometry(wkb=to_wkb(polygon, include_srid=True))
+    message = ExampleDatapoint(geometry=geometry, some_repeated_geometry=[geometry])
+
+    dataset = _to_dataset([message])
+    assert get_srid(dataset.geometry.item()) == srid
+    converted = to_messages(dataset, ExampleDatapoint)[0]
+
+    for value in [converted.geometry, *converted.some_repeated_geometry]:
+        restored = from_wkb(value.wkb)
+        assert get_srid(restored) == srid
+        assert restored.equals_exact(polygon, tolerance=0)
+        if srid == 0:
+            assert value.wkb == polygon.wkb
+
+
 def _to_dataset(datapoints: list[ExampleDatapoint]) -> xr.Dataset:
     converter = MessageToXarrayConverter()
     converter.convert_all(datapoints)