Add multi-dataset model run prediction uploads

CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to the [Nucleus Python Client](https://github.com/scaleapi/n
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.20.2](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.20.2) - 2026-08-18
+
+### Added
+- **Multi-dataset model runs.** `Dataset.upload_predictions_for_model_run(model_run_id, predictions, ...)` uploads predictions for an existing run against *this* dataset, adding the dataset to the run's set if it isn't there already. This is what lets a single model run be scored against a benchmark whose items span several datasets. Supports the same `update` / `asynchronous` / `batch_size` / file-batching / `trained_slice_id` arguments as `upload_predictions`.
+  - A run's dataset set only ever grows — a later upload never removes a dataset, so it cannot widen who can read the run.
+  - Access: write on this dataset **and** on every dataset the run already covers. Runs are visible only to users who can read all of their datasets, so adding one can remove the run from a collaborator's view.
+  - `Dataset.upload_predictions` is unchanged and still cannot widen a run: it identifies the run by `(dataset, model)`, so it finds the run already on this dataset or creates a new one.
+
+### Changed
+- **Benchmark evaluations no longer require the run to cover the benchmark's datasets.** `create_benchmark_evaluation_v2` previously failed when the benchmark contained items outside the model run's dataset. Those members are now scored as false negatives like any other uncovered item, so a partial run ranks comparably instead of being rejected. Docstrings on `create_benchmark_evaluation_v2` and `Benchmark.create_evaluation_v2` updated accordingly.
+- `PredictionUploader` accepts `dataset_id` together with `model_run_id` to select the new endpoint. Previously that combination was rejected by an assertion. The other two forms — `(dataset_id, model_id)` and `model_run_id` alone — route exactly as before.
+
+### Deprecated
+- `ModelRun.predict()` (already deprecated with the rest of `ModelRun`) infers its target dataset from the run, so it fails for a run spanning several datasets. Use `Dataset.upload_predictions_for_model_run` instead.
+
+> **Server dependency:** requires the `POST /nucleus/dataset/:datasetId/modelRun/:modelRunId/uploadPredictions` route and the multi-dataset model-run work in scaleapi. Unit tests pass regardless; live calls 404 until that deploys.
+
 ## [0.20.1](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.20.1) - 2026-08-13
 
 ### Added

nucleus/__init__.py

@@ -1324,10 +1324,16 @@ def create_benchmark_evaluation_v2(
         background — call :meth:`EvaluationV2.wait_for_completion`, then
         :meth:`EvaluationV2.charts` or :meth:`EvaluationV2.examples`.
 
+        The benchmark may span datasets the model run has no predictions in at
+        all. Those members are scored as false negatives like any other
+        uncovered item, so a partial run still ranks comparably rather than
+        being rejected. To give a run predictions across several datasets, use
+        :meth:`Dataset.upload_predictions_for_model_run`.
+
         Parameters:
             benchmark_id: Benchmark id (``bm_*``).
-            model_run_id: Model run id (``run_*``). Its predictions must
-                cover items from the benchmark's datasets.
+            model_run_id: Model run id (``run_*``). It need not cover the
+                benchmark's datasets — coverage may be partial, or empty.
             name: Optional display name.
             rollup_groups: Optional rollup classes (the primary label
                 configuration); each :class:`RollupGroup` maps raw labels

nucleus/annotation_uploader.py

@@ -236,27 +236,55 @@ def check_for_duplicate_ids(self, annotations: Iterable[Annotation]):
 
 
 class PredictionUploader(AnnotationUploader):
+    """Routes a prediction upload to one of two ``uploadPredictions`` endpoints.
+
+    Which one depends on the identifiers supplied (``dataset_id`` is required):
+
+    ``dataset_id`` + ``model_run_id``
+        ``dataset/{dataset_id}/modelRun/{model_run_id}/uploadPredictions``. The
+        only route that lets a run span more than one dataset — uploading here
+        adds ``dataset_id`` to the run's dataset set. Requires write access to
+        every dataset the run already covers, not just this one, because a run
+        is only visible to users who can read all of its datasets.
+
+    ``dataset_id`` + ``model_id``
+        ``dataset/{dataset_id}/model/{model_id}/uploadPredictions``. Resolves
+        (or creates) the run for that model on that dataset. Cannot widen an
+        existing run: passing a ``model_run_id`` belonging to a different
+        dataset is rejected server-side.
+
+    ``route`` overrides the id-based selection with an explicit endpoint. It
+    exists only for the deprecated :meth:`ModelRun.predict`, which posts to
+    ``modelRun/{model_run_id}/predict``; new code should not use it.
+    """
+
     def __init__(
         self,
         client: "NucleusClient",
         dataset_id: Optional[str] = None,
         model_id: Optional[str] = None,
         model_run_id: Optional[str] = None,
         trained_slice_id: Optional[str] = None,
+        route: Optional[str] = None,
     ):
         super().__init__(dataset_id, client)
         self._client = client
         self.trained_slice_id = trained_slice_id
+        if route is not None:
+            self._route = route
+            return
+        if dataset_id is None:
+            raise ValueError("dataset_id is required to upload predictions.")
+        if model_run_id is not None and model_id is not None:
+            raise ValueError("Pass either model_id or model_run_id, not both.")
         if model_run_id is not None:
-            assert model_id is None and dataset_id is None
-            self._route = f"modelRun/{model_run_id}/predict"
-        else:
-            assert (
-                model_id is not None and dataset_id is not None
-            ), "Model ID and dataset ID are required if not using model run id."
+            self._route = f"dataset/{dataset_id}/modelRun/{model_run_id}/uploadPredictions"
+        elif model_id is not None:
             self._route = (
                 f"dataset/{dataset_id}/model/{model_id}/uploadPredictions"
             )
+        else:
+            raise ValueError("Either model_id or model_run_id is required.")
 
     def check_for_duplicate_ids(self, annotations: Iterable[Annotation]):
         """Do not allow predictions to have the same (annotation_id, reference_id) tuple"""

nucleus/benchmark.py

@@ -167,6 +167,11 @@ def create_evaluation_v2(
     ) -> EvaluationV2:
         """Evaluate a model run against this benchmark.
 
+        The run need not cover this benchmark's datasets — uncovered members are
+        scored as false negatives, so a partial run still ranks comparably. To
+        give a run predictions across several datasets, use
+        :meth:`Dataset.upload_predictions_for_model_run`.
+
         See :meth:`NucleusClient.create_benchmark_evaluation_v2` for parameter
         details.
 

nucleus/dataset.py

@@ -2043,6 +2043,113 @@ def upload_predictions(
             model_id=model.id,
             client=self._client,
         )
+        return self._upload_predictions(
+            uploader,
+            predictions,
+            async_route=f"dataset/{self.id}/model/{model.id}/uploadPredictions?async=1",
+            update=update,
+            asynchronous=asynchronous,
+            batch_size=batch_size,
+            remote_files_per_upload_request=remote_files_per_upload_request,
+            local_files_per_upload_request=local_files_per_upload_request,
+            trained_slice_id=trained_slice_id,
+        )
+
+    def upload_predictions_for_model_run(
+        self,
+        model_run_id: str,
+        predictions: List[Prediction],
+        update: bool = False,
+        asynchronous: bool = False,
+        batch_size: int = 5000,
+        remote_files_per_upload_request: int = 20,
+        local_files_per_upload_request: int = 10,
+        trained_slice_id: Optional[str] = None,
+    ) -> Union[Dict[str, Any], AsyncJob]:
+        """Uploads predictions for an existing model run against **this** dataset.
+
+        Use this instead of :meth:`upload_predictions` when one model run should
+        hold predictions across several datasets — for example to evaluate the
+        run against a benchmark whose items span more than one dataset. The run
+        does not need to cover this dataset already; uploading here adds it to
+        the run's dataset set.
+
+        :meth:`upload_predictions` cannot do this. It identifies the run by
+        ``(dataset, model)``, so it either finds the run already on this dataset
+        or creates a new one — an existing run belonging to a different dataset
+        is rejected.
+
+        A run's dataset set only ever grows: a later upload never removes a
+        dataset, so it cannot widen who can read the run.
+
+        Access: you need write access to this dataset **and** to every dataset
+        the run already covers. Model runs are visible only to users who can
+        read all of their datasets, so adding a dataset to a run can remove it
+        from a collaborator's view — hence the stricter check.
+
+        Parameters:
+            model_run_id: Nucleus-generated model run ID (starts with ``run_``).
+            predictions: List of prediction objects to upload. Same types as
+              :meth:`upload_predictions`.
+            update: Whether or not to overwrite metadata or ignore on reference
+              ID collision. Default is False.
+            asynchronous: Whether or not to process the upload asynchronously
+              (and return an :class:`AsyncJob` object). Default is False.
+            batch_size: Number of predictions processed in each concurrent
+              batch. Default is 5000. Only relevant for asynchronous=False.
+            remote_files_per_upload_request: Number of remote files to upload in
+              each request. Only relevant for asynchronous=False.
+            local_files_per_upload_request: Number of local files to upload in
+              each request. Maximum is 10. Only relevant for asynchronous=False.
+            trained_slice_id: Nucleus-generated slice ID (starts with ``slc_``)
+              which was used to train the model. Must belong to this dataset.
+
+        Returns:
+            Payload describing the synchronous upload::
+
+                {
+                    "dataset_id": str,
+                    "model_run_id": str,
+                    "predictions_processed": int,
+                    "predictions_ignored": int,
+                }
+        """
+        uploader = PredictionUploader(
+            model_run_id=model_run_id,
+            dataset_id=self.id,
+            client=self._client,
+        )
+        return self._upload_predictions(
+            uploader,
+            predictions,
+            async_route=f"dataset/{self.id}/modelRun/{model_run_id}/uploadPredictions?async=1",
+            update=update,
+            asynchronous=asynchronous,
+            batch_size=batch_size,
+            remote_files_per_upload_request=remote_files_per_upload_request,
+            local_files_per_upload_request=local_files_per_upload_request,
+            trained_slice_id=trained_slice_id,
+        )
+
+    def _upload_predictions(
+        self,
+        uploader: PredictionUploader,
+        predictions: List[Prediction],
+        *,
+        async_route: str,
+        update: bool,
+        asynchronous: bool,
+        batch_size: int,
+        remote_files_per_upload_request: int,
+        local_files_per_upload_request: int,
+        trained_slice_id: Optional[str],
+    ) -> Union[Dict[str, Any], AsyncJob]:
+        """Shared driver for the prediction-upload entry points.
+
+        The public methods differ only in how they build ``uploader`` and in
+        ``async_route``; the duplicate-id check and the sync/async dispatch are
+        identical.
+        """
         uploader.check_for_duplicate_ids(predictions)
 
         if asynchronous:
@@ -2057,7 +2164,7 @@ def upload_predictions(
                     UPDATE_KEY: update,
                     TRAINED_SLICE_ID_KEY: trained_slice_id,
                 },
-                route=f"dataset/{self.id}/model/{model.id}/uploadPredictions?async=1",
+                route=async_route,
             )
             return AsyncJob.from_json(response, self._client)
 

nucleus/model_run.py

@@ -137,6 +137,13 @@ def predict(
     ) -> Union[dict, AsyncJob]:
         """Uploads model outputs as predictions for a model_run.
 
+        Deprecated along with the rest of this class. The target dataset is
+        inferred from the run rather than named, so this fails for a run that
+        spans more than one dataset — there is no single dataset to infer. Use
+        ``dataset.upload_predictions_for_model_run(model_run_id, predictions)``
+        instead: the ``dataset_id`` comes from the :class:`Dataset` you call it
+        on and the ``model_run_id`` is passed explicitly, so both are named.
+
         Args:
             annotations: Predictions to upload for this model run.
             update: If True, existing predictions for the same (reference_id, annotation_id)
@@ -166,8 +173,11 @@ def predict(
                 "predictions_ignored": int,
             }
         """
+
         uploader = PredictionUploader(
-            model_run_id=self.model_run_id, client=self._client
+            client=self._client,
+            dataset_id=self.dataset_id,
+            route=f"modelRun/{self.model_run_id}/predict",
         )
         uploader.check_for_duplicate_ids(annotations)
 

pyproject.toml

@@ -25,7 +25,7 @@ ignore = ["E501", "E741", "E731", "F401"]  # Easy ignore for getting it running
 
 [tool.poetry]
 name = "scale-nucleus"
-version = "0.20.1"
+version = "0.20.2"
 description = "The official Python client library for Nucleus, the Data Platform for AI"
 license =  "MIT"
 authors = ["Scale AI Nucleus Team <nucleusapi@scaleapi.com>"]