Add multi-dataset model run prediction uploads

[luke-e-schaefer]

Summary — v0.20.0: multi-dataset model runs

Stacked on #467 (update-nuc-sdk-for-new-eval-stuff-pt1) — review that first; this PR's diff is only the 7 files below. The SDK-visible half of the scaleapi change that gives a model run a resolved set of datasets instead of one declared dataset.

The one new method

dataset_b.upload_predictions_for_model_run(run_id, predictions)

Posts to dataset/{dataset_id}/modelRun/{model_run_id}/uploadPredictions, adding dataset_b to the run's dataset set. That is what lets a single run be scored against a benchmark whose items span several datasets — previously create_benchmark_evaluation_v2 returned a 400 for exactly that case.

Same update / asynchronous / batch_size / file-batching / trained_slice_id arguments as upload_predictions, and it runs the same duplicate-id check.

What is deliberately not changed

Dataset.upload_predictions still cannot widen a run, and that's the point. It identifies the run by (dataset, model), so it finds the run already on this dataset or creates a new one. Widening got its own endpoint rather than loosening this one — mirroring the server, where the existing route kept its never-widen contract exactly as it was.

ModelRun.predict also stays on the old route. It's deprecated, and switching it would silently turn a stale dataset_id passed to get_model_run() into a widening upload. Its docstring now says it fails for multi-dataset runs and points at the new method.

Routing

PredictionUploader now accepts dataset_id together with model_run_id — previously an assertion rejected the pair — and picks the endpoint from which identifiers are present:

Arguments	Route	Can widen?
dataset_id + model_run_id	dataset/{ds}/modelRun/{run}/uploadPredictions	Yes
dataset_id + model_id	dataset/{ds}/model/{model}/uploadPredictions	No
model_run_id alone	modelRun/{run}/predict	No (deprecated)

The two pre-existing forms route exactly as before.

Access, and the sharp edge

Widening requires write on this dataset and on every dataset the run already covers. That's stricter than it looks necessary, for a reason worth knowing: a run is visible only to users who can read all of its datasets, so adding a dataset to a run can remove that run from a collaborator's view. The set also only ever grows — a later upload never drops a dataset, so it can't widen who can read the run.

Docstring corrections

Both are wrong as of the server change, not merely incomplete:

• create_benchmark_evaluation_v2 / Benchmark.create_evaluation_v2 said the run's predictions "must cover items from the benchmark's datasets". The server enforced that with a 400; it no longer does. Coverage may be partial or empty, and uncovered members score as false negatives.
• ModelRun.predict — see above.

Tests

tests/test_multi_dataset_model_runs.py — 9 mock-based tests pinning all three routes (the whole difference between them is the route), the async route, trained_slice_id forwarding, and that the new entry point runs the duplicate-id check rather than bypassing it.

Ran locally: 68 passed across the new file plus the benchmark / eval-v2 / preset / leaderboard suites. black and isort clean on the changed files. Diff is purely additive — no reformatting churn from a newer local black.

Server dependency: the new route ships with the multi-dataset model-run work in scaleapi (#154100). Unit tests pass regardless; live calls 404 until that deploys.

🤖 Generated with Claude Code

Greptile Summary

Adds multi-dataset model-run prediction uploads while preserving existing upload behavior.

• Adds Dataset.upload_predictions_for_model_run with synchronous and asynchronous widening routes.
• Refactors shared prediction-upload validation and dispatch into _upload_predictions.
• Keeps deprecated ModelRun.predict on its existing endpoint and documents its multi-dataset limitation.
• Updates benchmark-evaluation documentation, package version, changelog, and route-focused tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
nucleus/dataset.py	Adds the public multi-dataset upload entry point and extracts common synchronous/asynchronous upload handling.
nucleus/annotation_uploader.py	Extends prediction uploader routing to support explicit dataset-and-run uploads while validating identifier combinations.
nucleus/model_run.py	Preserves the deprecated prediction route through an explicit override and documents its multi-dataset limitation.
tests/test_multi_dataset_model_runs.py	Covers identifier validation, all upload routes, asynchronous routing, trained-slice forwarding, and duplicate-ID rejection.
nucleus/init.py	Corrects benchmark-evaluation documentation to describe partial or empty model-run coverage.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Prediction upload] --> B{Identifiers supplied}
    B -->|dataset_id + model_run_id| C[dataset/:dataset/modelRun/:run/uploadPredictions]
    B -->|dataset_id + model_id| D[dataset/:dataset/model/:model/uploadPredictions]
    B -->|Deprecated ModelRun.predict| E[modelRun/:run/predict]
    C --> F[Existing run may gain dataset]
    D --> G[Resolve or create run for dataset and model]
    E --> H[Retain legacy single-dataset behavior]

Loading

_{Reviews (7): Last reviewed commit: "updatte" | Re-trigger Greptile}

[jaypsiri]

had a few comments/questions!

[jaypsiri]

a few nits on comment stuff but other than this lgtm!