feat(core): add InstancePreferences for multi-instance-type training and processing

docs/ml_ops/index.rst

@@ -317,6 +317,26 @@ Run data preprocessing with ``ScriptProcessor`` (sklearn) or ``FrameworkProcesso
 
 :doc:`SKLearn example <../v3-examples/ml-ops-examples/v3-processing-job-sklearn>` · :doc:`PyTorch example <../v3-examples/ml-ops-examples/v3-processing-job-pytorch/v3-pytorch-processing-example>`
 
+**Instance Preferences:** pass an ordered list of candidate instance types and the platform runs the job on the first type with available capacity.
+
+.. code-block:: python
+
+   from sagemaker.core.processing import Processor
+
+   processor = Processor(
+       role=role, image_uri=processing_image, volume_size_in_gb=100,
+       instance_preferences=[
+           {"InstanceType": "ml.m5.4xlarge", "InstanceCount": 2},
+           {"InstanceType": "ml.m5.2xlarge", "InstanceCount": 4},
+       ],
+   )
+
+   processor.run(job_name="instance-prefs-processing")
+
+Up to 5 candidates are allowed, each instance type at most once, and exactly one is selected; the list is mutually exclusive with ``instance_type``. Counts use exactly one of two modes — a top-level ``instance_count`` shared by whichever candidate wins, or an ``InstanceCount`` on every candidate — and mixed, partial, or omitted counts are rejected. Selection is based on capacity, not on workload fit, so list only types the job can genuinely run on. The winner is reported as ``SelectedInstanceType`` / ``SelectedInstanceCount`` on the job's ``ClusterConfig``, and billing is for that type and count. Supported on ``Processor``, ``ScriptProcessor``, ``PySparkProcessor``, and ``SparkJarProcessor``; training plans are training-only and do not apply to processing.
+
+:doc:`Instance Preferences example <../v3-examples/ml-ops-examples/v3-processing-instance-preferences>`
+
 
 
 Batch Transform Jobs
@@ -728,5 +748,6 @@ Explore comprehensive MLOps examples:
    ../v3-examples/ml-ops-examples/v3-model-registry-example/v3-model-registry-example
    ../v3-examples/ml-ops-examples/v3-processing-job-pytorch/v3-pytorch-processing-example
    ../v3-examples/ml-ops-examples/v3-processing-job-sklearn
+   ../v3-examples/ml-ops-examples/v3-processing-instance-preferences
    ../v3-examples/ml-ops-examples/v3-emr-serverless-step-example
    ../v3-examples/ml-ops-examples/v3-mlflow-train-inference-e2e-example

docs/training/index.rst

@@ -388,6 +388,80 @@ Key points:
 
 
 
+Instance Preferences
+--------------------
+
+
+Provide an ordered list of candidate instance types and the platform launches the job on the first type with available capacity, instead of failing when one scarce type is unavailable.
+
+**Ordered Candidates:**
+
+.. code-block:: python
+
+   from sagemaker.train.model_trainer import ModelTrainer
+   from sagemaker.core.training.configs import Compute, SourceCode
+   from sagemaker.core.shapes import InstancePreference
+
+   compute = Compute(
+       instance_preferences=[
+           InstancePreference(instance_type="ml.p5.48xlarge"),
+           InstancePreference(instance_type="ml.p4d.24xlarge"),
+           InstancePreference(instance_type="ml.g5.12xlarge"),
+       ],
+       instance_count=2,
+   )
+
+   model_trainer = ModelTrainer(
+       training_image=training_image,
+       source_code=SourceCode(source_dir="./source", entry_script="train.py"),
+       compute=compute,
+       base_job_name="instance-preferences-training",
+   )
+
+   model_trainer.train()
+
+**Per-Candidate Instance Counts and Training Plans:**
+
+.. code-block:: python
+
+   compute = Compute(
+       instance_preferences=[
+           InstancePreference(
+               instance_type="ml.p5.48xlarge",
+               instance_count=2,
+               training_plan_arns=[p5_plan_arn],
+           ),
+           InstancePreference(instance_type="ml.p4d.24xlarge", instance_count=4),
+       ],
+   )
+
+**Identifying the Instance Type That Ran:**
+
+.. code-block:: python
+
+   training_job = model_trainer._latest_training_job
+   training_job.refresh()
+
+   resource_config = training_job.resource_config
+   print(resource_config.selected_instance_type)   # None until a candidate is selected
+   print(resource_config.selected_instance_count)
+
+Key points:
+
+- Up to 5 candidates per job, each instance type at most once, and exactly one type is selected
+- ``instance_preferences`` is mutually exclusive with ``instance_type``, ``instance_groups``, ``instance_placement_config``, and managed spot training
+- Not supported in local mode, with training recipes or JumpStart models, or for jobs submitted through AWS Batch training queues; jobs that leave ``instance_preferences`` unset are unaffected
+- Counts use exactly one of two modes: a top-level ``instance_count`` shared by whichever candidate wins, or an ``instance_count`` on every candidate to size each type differently. Mixed, partial, and omitted counts are rejected
+- A candidate with ``training_plan_arns`` draws from that plan's reserved capacity (one plan, whose instance type must match the candidate's); one without a plan uses on-demand. Per-candidate plans are mutually exclusive with the job-level ``training_plan_arn``, which instead applies to whichever candidate matches its type
+- Selection is based on capacity, not on workload fit: cross-type differences such as architecture or GPU memory are not validated, so list only types your job can genuinely run on
+- While no candidate has capacity the job stays pending and keeps retrying the list. ``max_pending_time_in_seconds`` bounds the total time spent working through the list rather than each candidate, and takes effect only when the list includes an accelerated instance type (``ml.p``, ``ml.g``, ``ml.trn``)
+- The winner is reported as ``selected_instance_type`` / ``selected_instance_count`` on describe; the top-level instance type is not returned
+- Job-level settings such as volume size, volume KMS key, and keep-alive period apply to the selected type, as does billing
+
+:doc:`Full example notebook <../v3-examples/training-examples/instance-preferences-example>`
+
+
+
 AWS Batch Training Queues
 -------------------------
 
@@ -438,6 +512,7 @@ Key points:
 - Batch manages capacity allocation and job scheduling automatically
 - Resources (Service Environments, Job Queues) can be created via console or programmatically
 - Supports FIFO and priority-based scheduling
+- Queued jobs request a single ``instance_type``; ``instance_preferences`` is not supported
 
 :doc:`Full example notebook <../v3-examples/training-examples/aws_batch/sm-training-queues_getting_started_with_model_trainer>`
 
@@ -555,4 +630,5 @@ Training Examples
    Hyperparameter Training <../v3-examples/training-examples/hyperparameter-training-example>
    Training with JumpStart Models <../v3-examples/training-examples/jumpstart-training-example>
    Custom Distributed Training <../v3-examples/training-examples/custom-distributed-training-example>
+   Instance Preferences <../v3-examples/training-examples/instance-preferences-example>
    AWS Batch for Training <../v3-examples/training-examples/aws_batch/sm-training-queues_getting_started_with_model_trainer>

sagemaker-core/sample/sagemaker/2017-07-24/service-2.json

@@ -28787,6 +28787,33 @@
       },
       "documentation":"<p>Defines an instance group for heterogeneous cluster training. When requesting a training job using the <a href=\"https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html\">CreateTrainingJob</a> API, you can configure multiple instance groups .</p>"
     },
+    "InstancePreference":{
+      "type":"structure",
+      "required":["InstanceType"],
+      "members":{
+        "InstanceType":{
+          "shape":"TrainingInstanceType",
+          "documentation":"<p>The ML compute instance type for this candidate.</p>"
+        },
+        "InstanceCount":{
+          "shape":"TrainingInstanceCount",
+          "documentation":"<p>The number of ML compute instances to use for this candidate. Optional; mutually exclusive with the uniform <code>ResourceConfig.InstanceCount</code> (either every preference sets its own count, or the uniform count is used for whichever type wins).</p>",
+          "box":true
+        },
+        "TrainingPlanArns":{
+          "shape":"TrainingPlanArnList",
+          "documentation":"<p>The training plan(s) to target for this candidate type. Optional; list is capped at 1. Training-only; mutually exclusive with the whole-job <code>ResourceConfig.TrainingPlanArn</code>.</p>"
+        }
+      },
+      "documentation":"<p>Defines a single candidate instance type in an ordered <code>InstancePreferences</code> list. When a training job specifies <code>InstancePreferences</code>, the platform tries each candidate type in list order and launches the job on the first type with available capacity.</p>"
+    },
+    "InstancePreferenceList":{
+      "type":"list",
+      "member":{"shape":"InstancePreference"},
+      "documentation":"<p>An ordered list of candidate instance types for a training job (array position is the priority). Currently limited to a minimum of 1 and a maximum of 5 entries; the limit is enforced by the service.</p>",
+      "max":5,
+      "min":1
+    },
     "InstanceGroupHealthCheckConfiguration":{
       "type":"structure",
       "required":[
@@ -35023,7 +35050,7 @@
       "documentation":"<p>Maximum job scheduler pending time in seconds.</p>",
       "box":true,
       "max":2419200,
-      "min":7200
+      "min":1800
     },
     "MaxPercentageOfInputDatasetLabeled":{
       "type":"integer",
@@ -40147,8 +40174,6 @@
     "ProcessingClusterConfig":{
       "type":"structure",
       "required":[
-        "InstanceCount",
-        "InstanceType",
         "VolumeSizeInGB"
       ],
       "members":{
@@ -40167,6 +40192,18 @@
         "VolumeKmsKeyId":{
           "shape":"KmsKeyId",
           "documentation":"<p>The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that Amazon SageMaker uses to encrypt data on the storage volume attached to the ML compute instance(s) that run the processing job. </p> <note> <p>Certain Nitro-based instances include local storage, dependent on the instance type. Local storage volumes are encrypted using a hardware module on the instance. You can't request a <code>VolumeKmsKeyId</code> when using an instance type with local storage.</p> <p>For a list of instance types that support local instance storage, see <a href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes\">Instance Store Volumes</a>.</p> <p>For more information about local instance storage encryption, see <a href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html\">SSD Instance Store Volumes</a>.</p> </note>"
+        },
+        "InstancePreferences":{
+          "shape":"ProcessingInstancePreferenceList",
+          "documentation":"<p>An ordered list of candidate instance types (maximum 5). When set, the platform tries each candidate in list order and launches the job on the first type with available capacity. Mutually exclusive with <code>InstanceType</code> and <code>InstanceCount</code>.</p>"
+        },
+        "SelectedInstanceType":{
+          "shape":"ProcessingInstanceType",
+          "documentation":"<p>The instance type the job was launched on when <code>InstancePreferences</code> is used. Output-only (returned in <code>DescribeProcessingJob</code>); ignored on <code>CreateProcessingJob</code>.</p>"
+        },
+        "SelectedInstanceCount":{
+          "shape":"ProcessingInstanceCount",
+          "documentation":"<p>The resolved number of instances the job was launched with when <code>InstancePreferences</code> is used. Output-only (returned in <code>DescribeProcessingJob</code>); ignored on <code>CreateProcessingJob</code>.</p>"
         }
       },
       "documentation":"<p>Configuration for the cluster used to run a processing job.</p>"
@@ -40237,6 +40274,28 @@
       "max":100,
       "min":1
     },
+    "ProcessingInstancePreference":{
+      "type":"structure",
+      "required":["InstanceType"],
+      "members":{
+        "InstanceType":{
+          "shape":"ProcessingInstanceType",
+          "documentation":"<p>The ML compute instance type for this candidate.</p>"
+        },
+        "InstanceCount":{
+          "shape":"ProcessingInstanceCount",
+          "documentation":"<p>The number of ML compute instances to use for this candidate. Optional; mutually exclusive with the uniform <code>ProcessingClusterConfig.InstanceCount</code> (either every preference sets its own count, or the uniform count is used for whichever type wins).</p>"
+        }
+      },
+      "documentation":"<p>Defines a single candidate instance type in an ordered <code>InstancePreferences</code> list for a processing job. InstanceType is required; InstanceCount is optional per element. Per-type training plans are training-only and do not apply to processing.</p>"
+    },
+    "ProcessingInstancePreferenceList":{
+      "type":"list",
+      "member":{"shape":"ProcessingInstancePreference"},
+      "documentation":"<p>An ordered list of candidate instance types for a processing job (array position is the priority). Currently limited to a minimum of 1 and a maximum of 5 entries; the limit is enforced by the service.</p>",
+      "max":5,
+      "min":1
+    },
     "ProcessingInstanceType":{
       "type":"string",
       "enum":[
@@ -42998,6 +43057,19 @@
         "InstancePlacementConfig":{
           "shape":"InstancePlacementConfig",
           "documentation":"<p>Configuration for how training job instances are placed and allocated within UltraServers. Only applicable for UltraServer capacity.</p>"
+        },
+        "InstancePreferences":{
+          "shape":"InstancePreferenceList",
+          "documentation":"<p>An ordered list of candidate instance types (maximum 5). When set, the platform tries each candidate in list order and launches the job on the first type with available capacity. Mutually exclusive with <code>InstanceType</code>, <code>InstanceGroups</code>, and <code>InstancePlacementConfig</code>.</p>"
+        },
+        "SelectedInstanceType":{
+          "shape":"TrainingInstanceType",
+          "documentation":"<p>The instance type the job was launched on when <code>InstancePreferences</code> is used. Output-only (returned in <code>DescribeTrainingJob</code>); ignored on <code>CreateTrainingJob</code>.</p>"
+        },
+        "SelectedInstanceCount":{
+          "shape":"TrainingInstanceCount",
+          "documentation":"<p>The resolved number of instances the job was launched with when <code>InstancePreferences</code> is used. Output-only (returned in <code>DescribeTrainingJob</code>); ignored on <code>CreateTrainingJob</code>.</p>",
+          "box":true
         }
       },
       "documentation":"<p>Describes the resources, including machine learning (ML) compute instances and ML storage volumes, to use for model training. </p>"
@@ -46934,6 +47006,13 @@
       "type":"list",
       "member":{"shape":"TrainingPlanArn"}
     },
+    "TrainingPlanArnList":{
+      "type":"list",
+      "member":{"shape":"TrainingPlanArn"},
+      "documentation":"<p>The training plan(s) targeted by a single instance preference. Currently limited to exactly 1 training plan per preference; the limit is enforced by the service.</p>",
+      "max":1,
+      "min":1
+    },
     "TrainingPlanDurationHours":{
       "type":"long",
       "box":true,

sagemaker-core/src/sagemaker/core/modules/configs.py

@@ -74,7 +74,7 @@
     "MetricDefinition",
 ]
 
-from sagemaker.core.modules.utils import convert_unassigned_to_none
+from sagemaker.core.modules.utils import convert_unassigned_to_none, validate_instance_preferences
 
 
 class BaseConfig(BaseModel):
@@ -147,6 +147,11 @@ class Compute(shapes.ResourceConfig):
             subsequent training jobs.
         instance_groups (Optional[List[InstanceGroup]]):
             A list of instance groups for heterogeneous clusters to be used in the training job.
+        instance_preferences (Optional[List[InstancePreference]]):
+            An ordered list of candidate instance types (maximum 5). When set, the platform tries
+            each candidate in list order and launches the job on the first type with available
+            capacity. Mutually exclusive with ``instance_type``, ``instance_groups``, and
+            ``instance_placement_config``.
         enable_managed_spot_training (Optional[bool]):
             To train models using managed spot training, choose True. Managed spot training
             provides a fully managed and scalable infrastructure for training machine learning
@@ -159,8 +164,10 @@ class Compute(shapes.ResourceConfig):
 
     @model_validator(mode="after")
     def _model_validator(self) -> "Compute":
-        """Convert Unassigned values to None."""
-        return convert_unassigned_to_none(self)
+        """Convert Unassigned values to None and validate instance_preferences."""
+        converted = convert_unassigned_to_none(self)
+        validate_instance_preferences(converted)
+        return converted
 
     def _to_resource_config(self) -> shapes.ResourceConfig:
         """Convert to a sagemaker_core.shapes.ResourceConfig object."""
@@ -169,6 +176,11 @@ def _to_resource_config(self) -> shapes.ResourceConfig:
         filtered_dict = {
             k: v for k, v in compute_config_dict.items() if k in resource_config_fields
         }
+        # Preserve the nested InstancePreference model objects instead of the
+        # dumped dicts, so pydantic does not re-validate their optional scalar
+        # fields (e.g. an unset per-preference instance_count) as Unassigned().
+        if self.instance_preferences:
+            filtered_dict["instance_preferences"] = self.instance_preferences
         return shapes.ResourceConfig(**filtered_dict)