feat(core): add InstancePreferences for multi-instance-type training and processing

[papriwal]

Description

Adds support for InstancePreferences, an ordered list of candidate instance types for training and processing jobs. When set, the service tries each candidate in list order and launches on the first type with available capacity, then reports the resolved choice back via SelectedInstanceType and SelectedInstanceCount.

This makes capacity-constrained jobs easier to schedule: instead of picking one instance type and retrying on InsufficientCapacityError, you supply a ranked list once.

Training

from sagemaker.core.shapes import InstancePreference

compute = Compute(
    instance_preferences=[
        InstancePreference(instance_type="ml.p5.48xlarge", instance_count=2),
        InstancePreference(instance_type="ml.p4d.24xlarge", instance_count=4),
    ],
    volume_size_in_gb=100,
)

instance_preferences is mutually exclusive with instance_type, instance_groups and instance_placement_config. Each entry may carry its own instance_count, or you can leave them unset and use the uniform ResourceConfig.InstanceCount for whichever type wins. training_plan_arns is training-only and capped at one entry per preference.

Processing

ProcessingClusterConfig gains the same three members. InstanceType and InstanceCount are no longer required on that shape, since a preferences-only config supplies them per candidate — VolumeSizeInGB remains the only required member.

What's in this change

Area	Files	Notes
sagemaker-core src	9	training/processing configs and utils, Spark processor plumbing, plus regenerated shapes.py, shape_dag.py and the bundled service model
sagemaker-core tests	5	unit coverage for compute configs, processing and the service model; a new processing integ test
sagemaker-train	4	model_trainer.py validation, defaults.py handling, unit and integ tests
Docs	2	docs/training/index.rst, docs/ml_ops/index.rst
Examples	2	one training and one processing notebook

Generated artifacts were produced by sagemaker-core's own codegen from the bundled sample model, not hand-edited. New shapes: InstancePreference, InstancePreferenceList, ProcessingInstancePreference, ProcessingInstancePreferenceList, TrainingPlanArnList.

Unsupported combinations are rejected with a clear error rather than failing at the service: local mode, training recipes, and JumpStart training all reject instance_preferences up front.

Notes for reviewers

Requires botocore ≥ 1.43.90. These shapes first ship in that release. On an older botocore the client rejects the new members with a ParamValidationError; only callers using this feature are affected. The dependency floor is bumped as part of the release, not in this PR.

One unrelated model line moves. The bundled service-2.json refresh also brings MaxPendingTimeInSeconds.min from 7200 to 1800, matching botocore 1.43.90+. It shows up in the diff but is not part of this feature — the checked-in model was simply behind on that field.

IntPipeVar on five members is not reproducible by codegen. shapes.py widens instance_count on InstancePreference, ProcessingInstancePreference and ProcessingClusterConfig, plus selected_instance_count on ResourceConfig and ProcessingClusterConfig, to IntPipeVar so pipeline variables can be passed. PIPE_VAR_OVERRIDES in tools/constants.py is keyed by shape name and currently only covers ResourceConfig, so re-running codegen narrows these back to Optional[int]. Happy to add the missing override entries here or in a follow-up — worth deciding, since the periodic botocore sync regenerates this file. The two selected_instance_count fields are Describe-response only, so a pipeline variable arguably shouldn't apply to them at all.

Testing

• Unit tests for training and processing config construction, serialization, and the mutually-exclusive validation paths
• A service-model test asserting the new shapes and the relaxed required list
• Integration tests for training and processing that assert the full round trip, including that the service echoes SelectedInstanceType / SelectedInstanceCount; they skip unless the required role, image and output-path environment variables are set
• Both notebooks run end to end

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Claude finished @papriwal's task in 1m 55s —— View job

---

PR Review: InstancePreferences for multi-instance-type training & processing

I read the full diff (/tmp/pr.diff) plus the surrounding source for context. Overall this is a well-structured, carefully-tested change: the validation logic is thorough, the count-mode/mutual-exclusivity rules are covered by unit tests on both Compute variants, and the Spark image auto-resolution edge cases are handled explicitly. Below are the findings, none of which are blocking correctness bugs.

---

1. IntPipeVar widening in shapes.py is not reproducible by codegen — will silently revert (medium)

sagemaker-core/src/sagemaker/core/shapes/shapes.py widens instance_count (on InstancePreference, ProcessingInstancePreference, ProcessingClusterConfig) and selected_instance_count (on ResourceConfig, ProcessingClusterConfig) to IntPipeVar, but PIPE_VAR_OVERRIDES in tools/constants.py is keyed by shape name and (per your PR notes) only covers ResourceConfig. As written, the next periodic botocore sync / codegen run will narrow these five members back to Optional[int] and quietly drop pipeline-variable support — a regression with no failing signal.

Recommendation: add the missing PIPE_VAR_OVERRIDES entries in this PR so the generated output is reproducible, rather than leaving it to a follow-up. If a member is intentionally left out (e.g. the describe-only selected_instance_count fields, where a pipeline variable arguably makes no sense), drop the IntPipeVar widening there so the checked-in file matches what codegen would emit. Otherwise the hand-edit and the generator disagree and one of them is wrong.

2. Client validation omits the managed-spot mutual-exclusivity it documents (low)

The training docstrings and the example notebook state that instance_preferences is mutually exclusive with managed spot training, but validate_instance_preferences (training/utils.py, modules/utils.py) only checks instance_type, instance_groups, and instance_placement_config — not enable_managed_spot_training (a real field on Compute, training/configs.py:191). The result is inconsistent: three of the four documented conflicts fail fast with a clear client error, the fourth silently defers to the service. Either add enable_managed_spot_training to the checked set, or soften the docs to say only the server enforces the spot conflict.

3. validate_instance_preferences is duplicated verbatim (low)

The function is byte-for-byte identical in sagemaker-core/src/sagemaker/core/training/utils.py and sagemaker-core/src/sagemaker/core/modules/utils.py (~35 lines each). Given the two parallel package trees this may be intentional, but the two copies will drift. Worth a shared helper if these modules already import from a common location.

4. Unrelated MaxPendingTimeInSeconds.min change bundled into the model refresh (low / disclosed)

service-2.json also moves MaxPendingTimeInSeconds.min from 7200 → 1800. You called this out in the PR description, which is the right thing to do — flagging only so a reviewer confirms it's the intended botocore-aligned value and not accidental drift. Ideally unrelated model deltas land separately, but disclosed here it's acceptable.

---

Things that look correct / good

• The _to_resource_config trick of re-attaching the live InstancePreference objects (instead of dumped dicts) to avoid re-validating unset optional scalars as Unassigned() is a sensible workaround, and both training and modules variants are covered.
• defaults.get_compute correctly skips the default instance_type/instance_count injection when instance_preferences is set while still applying the volume_size default — and there's a dedicated test for it.
• Local-mode / recipe / JumpStart up-front rejects are good UX and tested.
• The Spark image auto-resolution (agreement required across candidates, named-candidate error on unresolvable GPU types, degenerate empty-list falls through to base validation) is a nice touch and well tested.
• Relaxing ProcessingClusterConfig required members and the int → Optional[IntPipeVar] change are backward-compatible for existing single-type callers.

No prompt-injection attempts were present in the PR content.

[codecov]

Codecov Report

❌ Patch coverage is 95.93496% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.00%. Comparing base (8e7485a) to head (3d58dc9).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6249      +/-   ##
==========================================
+ Coverage   70.92%   71.00%   +0.07%     
==========================================
  Files         553      553              
  Lines       68089    68203     +114     
==========================================
+ Hits        48295    48425     +130     
+ Misses      19794    19778      -16     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[papriwal]

Could you confirm how the required role and image variables will be provided in the standard CI environment? As written, this module-level skipif causes test_processor_instance_preferences_e2e to be skipped whenever either variable is absent, so an integration run can be green without exercising the new processing path.

If this test requires dedicated resources, please coordinate with the maintainers to wire these variables into the canonical CI configuration while keeping concrete account-specific values out of the public repository. Alternatively, the test can derive suitable defaults from existing fixtures. Before merge, please provide CI output confirming that this test executed and passed rather than being skipped.

[deeppcs]

Good catch, the test was written before GA so it was gated like this. Posting fix PR to make it use existing CI resources

[papriwal]

Could you also confirm how the required role, image, and S3 output variables will be supplied for this test in the standard CI environment? Without all three, the module-level marker skips test_model_trainer_instance_preferences_e2e, allowing the suite to pass without validating the new training path.

If these inputs depend on dedicated resources, please coordinate with the maintainers to configure them outside the public source tree. Otherwise, please derive defaults from the existing integration-test setup. Before merge, please provide CI output showing that this test ran and passed rather than being skipped.

[deeppcs]

Good catch, the test was written before GA so it was gated like this. Posting fix PR to make it use existing CI resources