docs(instance-preferences): resolve example images from a candidate type; make the PipeVar override test tox-safe - #6257
Merged
papriwal merged 3 commits intoSep 10, 2026
Conversation
…e type The training notebook and the training guide resolved the PyTorch image with instance_type="ml.m5.xlarge", which selects the CPU image, and then paired it with GPU candidates (ml.p5.48xlarge, ml.p4d.24xlarge). A reader copying either example would run a GPU job on the CPU image. Both now resolve the image from one of the listed candidates (all three GPU candidates map to the same pytorch-training:2.0.0-gpu-py310 image), and state the rule the feature implies: the image is fixed at submission time while the instance type is not, so every candidate must be able to run it. The processing notebook and guide are updated the same way for consistency; the scikit-learn image is CPU-only and all candidates are m5 types, so their behaviour was already correct.
…he packaged service model TestInstancePreferencesPipeVarOverrides built a ShapesExtractor from the real service JSON, which lives under sample/ in the source tree and is not part of the installed package. Under tox the extractor is imported from site-packages, the file is absent, and all six cases errored at setup with FileNotFoundError. The test now injects a minimal in-memory model that mirrors the real member -> shape wiring for the six count members, so it exercises exactly the PIPE_VAR_OVERRIDES lookup and nothing else. A control shape with the same integer type and no override asserts the widening is targeted rather than blanket. Removing any one override still fails exactly that member's case.
…; define processing_image in the guide The shallow suite README maps each shallow file to its deep counterpart; the instance-preferences pair was missing. The processing guide's instance-preferences snippet referenced processing_image without defining it on the page; it now resolves the image from one of the candidates, matching the training guide.
papriwal
approved these changes
Sep 10, 2026
papriwal
merged commit Sep 10, 2026
ac1dfa7
into
aws:feature-smtj-instance-preferences-rebased
1 of 2 checks passed
deeppcs
requested a deployment
to
manual-approval
September 10, 2026 22:27 — with 
GitHub Actions
Waiting
GitHub Actions
Waiting
papriwal
added a commit
that referenced
this pull request
Sep 11, 2026
…and processing (#6249) * Instance Preferences: full change set replayed onto master tip (#2154) * feat(instance-preferences): Add Instance Preferences (multi-instance-type) support (#2093) * feat(instance-preferences): add multi-instance-type support to PySDK Add customer-specified Instance Preferences (multi-instance-type) support across sagemaker-core and sagemaker-train, mirroring the additive, backward-compatible SageMaker model change. shapes / model (sagemaker-core): - New InstancePreference shape (instance_type, optional per-preference instance_count, optional training_plan_arns capped at 1 via the new TrainingPlanArnList model shape). - New processing-specific ProcessingInstancePreference / ProcessingInstancePreferenceList shapes (ProcessingInstanceType/Count members, no TrainingPlanArns - per-type training plans are training-only), matching the service model where training and processing preferences are distinct shapes. - ResourceConfig: instance_preferences + output-only selected_instance_type / selected_instance_count. - ProcessingClusterConfig: instance_preferences + selected_*; relax required on instance_type / instance_count (XOR instance_preferences, enforced server-side; matches the GA contract). - Mirror all additions into the bundled service-2.json codegen input (also consumed by the runtime botocore loader). consumers: - Compute (modules + training configs) inherit the field; preserve nested InstancePreference objects in _to_resource_config so an unset per-preference instance_count does not fail re-validation. - Processor: instance_preferences param + emit InstancePreferences into ClusterConfig; mutual-exclusivity with instance_type/instance_count. - AWS Batch: reconstruct instance_preferences onto the rebuilt Compute. - ModelTrainer: guard local mode / training recipes / JumpStart against instance_preferences with clear errors (out of scope for M1). tests: round-trip for both Compute classes, selected_* not sent on create, ProcessingClusterConfig with processing-specific shapes, Processor ClusterConfig emission and mutual exclusivity, and AWS Batch reconstruction. * fix(instance-preferences): defaults must not inject instance_type; add integ + contract tests - TrainDefaults.get_compute no longer injects the default top-level instance_type / instance_count when Compute carries instance_preferences (mutually exclusive server-side; the injected default made every instance-preferences ModelTrainer job fail CreateTrainingJob validation with 'InstanceType cannot be specified with InstancePreferences' - caught by the new integ test against a live endpoint). Unit tests added for both count modes. - New integ test (tests/integ/train/test_instance_preferences.py): launches a real training job via ModelTrainer + Compute with an ordered instance_preferences list, asserts the create/describe contract, and asserts the resolved selected_instance_type / selected_instance_count is one of the submitted preferences once the job leaves PENDING. Env-gated (role/image/S3 via INSTANCE_PREFERENCES_TEST_* vars); reports XFAIL with the job name while the server-side scheduling stage is not yet deployed on the target endpoint, and enforces the full contract automatically once it is. - New unit test pinning the Instance Preferences contract in the bundled service-2.json (distinct training/processing preference shapes, TrainingPlanArnList cap 1, list cap 5, Selected* on both Describe shapes, ProcessingClusterConfig required relaxation). * fix(instance-preferences): correct count semantics + client validation; add processing integ test Address review feedback on the Processor mutual-exclusivity check: instance_count is NOT flatly exclusive with instance_preferences - it is the shared uniform count for whichever preference wins. Corrected to the count-mode contract (V4) on the Processor AND, for parity, added the same client-side validation to both training Compute classes (server remains the source of truth): - uniform mode: top-level instance_count set, no element carries a count (Processor emits the uniform count into ClusterConfig); - per-preference mode: EVERY element carries a count, top-level unset; - reject both-set, partial per-preference, and neither; - instance_type / instance_groups / instance_placement_config remain mutually exclusive with instance_preferences (V1/V5). Shared validate_instance_preferences helper in modules/utils.py and training/utils.py (per-package twin pattern); wired into both Compute _model_validator hooks (assignment-safe: no-op without preferences). Unit tests for all modes on Processor and both Compute classes; two pre-existing test fixtures corrected to V4-valid combinations. Add processing integ test (tests/integ/processing/test_instance_preferences_processing.py): launches a real processing job via Processor with InstancePreferences through the bundled service model, asserts the create/describe contract and the resolved Selected* winner; env-gated (PROCESSING_INSTANCE_PREFERENCES_TEST_* vars). Reports XFAIL while the endpoint's public model still hard-requires InstanceType/InstanceCount (the @required relaxation ships with the GA ungating), and enforces the full contract automatically once deployed. * revert(instance-preferences): drop AWS Batch reconstruction support Remove the instance_preferences reconstruction in the AWS Batch TrainingQueuedJob path (and its test) - Batch support for Instance Preferences will ship separately later; this PR intentionally adds no Batch functionality. Files restored to upstream/master state. Also: black-format the new training integ test (line-length 100). * fix(instance-preferences): complete client validation; clearer error wording Complete the client-side validation sweep against the server rule set (server remains the source of truth): - Training plans: the top-level (whole-job) training_plan_arn is mutually exclusive with per-preference training_plan_arns (was missing - review catch). - Duplicate instance types across preferences rejected (both Compute classes + Processor). - List-size limits (max 5 preferences, max 1 plan per preference) are deliberately NOT enforced client-side: they are server-side configurable, so raising them must never require an SDK release (botocore does not enforce the C2J list min/max either - verified - so the service is the single enforcement point for sizes). The current limits are documented on the service-2.json list shapes (InstancePreferenceList, TrainingPlanArnList, ProcessingInstancePreferenceList) instead. Type validity (region-dependent) and count bounds are likewise server-only. Reword thrown error messages to use 'top-level instance_count / training_plan_arn' instead of the design-doc jargon 'uniform', so the error names the actual field the customer must change. Tests: whole-job-plan XOR per-preference-plans (reject) / whole-job-plan-without-per-preference-plans (valid) / duplicate-type cases on both Compute classes and Processor. * docs(instance-preferences): add example notebook; extract Processor validation helper Address review feedback: - Add v3-examples/training-examples/instance-preferences-example.ipynb demonstrating the customer experience end to end: ModelTrainer + Compute with an ordered preference list (shared top-level count), per-preference instance counts, per-preference training plans with an on-demand fallback, reading the resolved winner via selected_instance_type/selected_instance_count on Describe, and the Processor equivalent for processing jobs. All construction patterns verified executable against the SDK. - Extract the Processor's inline instance_preferences validation into a module-level _validate_processing_instance_preferences helper (behavior unchanged; unit tests pass unmodified). * docs(instance-preferences): add per-preference-count Processor example to notebook * docs(instance-preferences): polish example notebook for public customers Align the notebook with the v3-examples conventions: - Session()/get_execution_role()/image_uris.retrieve() setup instead of paste-in placeholders; 'Step N' section headers. - Customer-facing wording: drop client/service enforcement internals; 'instance types must be unique across preferences'. (cherry picked from commit 81758d391bb48aa4676bd59def97fef821fe05f5) * fix(instance-preferences): shape-DAG regen, Spark processor plumbing, integ test (#2099) * test(instance-preferences): XFAIL training integ test under pre-GA dark-launch gating The endpoint's frontend strips internal-gated fields (instance_preferences, selected_*) from customer Describe responses while the surface is dark-launched - the documented pre-GA contract (verified: the long-internal UseReservedCapacity is stripped identically, and the layer below returns the fields correctly with the top-level suppression rule applied). Detect the gating (echo absent from Describe) and report XFAIL with a re-run-after- GA-ungate message instead of failing on the unassertable public contract. Verified live against a pre-GA endpoint: job schedules and completes with the correct winner persisted; test XFAILs cleanly in ~2s. * feat(instance-preferences): plumb through Script/Spark processors ScriptProcessor, _SparkProcessorBase, PySparkProcessor, and SparkJarProcessor now accept instance_preferences and forward it to the base Processor (which already carried the parameter, validation, and ClusterConfig emission). Spark image-URI resolution falls back to the first preference's instance type when no top-level instance_type is set. Unit tests: forwarding on all three public classes, validation reuse, and first-preference image lookup (5 new; full processing suite 99 passing). * fix(instance-preferences): regenerate shape DAG; fix integ wait loop The resource deserializer (codec.transform) walks the generated SHAPE_DAG, which was not regenerated when the InstancePreferences shapes landed - so DescribeTrainingJob/DescribeProcessingJob responses silently DROPPED instance_preferences and Selected* from the typed resources (requests were unaffected; the gap was masked while the beta frontend stripped the fields pre-ungate). Regenerated via ShapesExtractor from the bundled service-2.json (+44 lines, additive). Also fix the integ test's wait loop: gate on terminal-status-or-winner instead of a secondary-status allowlist (Selected* propagation lags the transitions; jobs completing fast broke out pre-resolution). Verified live against ungated beta PDX: full contract PASS in 62s (winner + echo + suppression on the typed resource). * fix(instance-preferences): Spark image resolution requires candidate agreement A job runs one image but the winner among instance_preferences is unknown at create, so resolving from the first preference was silently wrong for candidate lists that mix processor classes. Auto-resolution now resolves per candidate and requires all candidates to yield the SAME image; on divergence it raises with a clear pass-image_uri- explicitly message. No behavior change for homogeneous lists (Spark images are CPU-only today, so real lists agree). * fix(instance-preferences): guard degenerate preference list in Spark image resolution An instance_preferences list with no InstanceType on any element hit StopIteration inside the candidate-agreement resolution (which runs before the base Processor validation). Fall through to the classic resolution path so the base validation owns the clean reject. * fix(instance-preferences): name unresolvable Spark candidates Per-candidate image resolution failures (e.g. GPU candidates: Spark has no gpu image variant, so image_uris.retrieve raises) now surface a message naming the candidate and the pass-image_uri-explicitly remedy. Verified unmocked: mixed CPU families (m5/r5/c5) resolve to the same image and pass agreement; a g5 candidate raises the named error; explicit image_uri bypasses resolution entirely. (cherry picked from commit 574a87d540c26d651d4844fc99737f0e1bc60047) * docs(instance-preferences): document instance preferences for training and processing (#2142) * docs(instance-preferences): document training instance preferences The example notebook shipped in #2093 was referenced by no .rst, so Sphinx never rendered it. Add an Instance Preferences section to the training guide (ordered candidates, per-candidate counts and training plans, reading the selected type) and wire the notebook into the Training Examples toctree. Move the notebook's processing steps out to a dedicated processing example so the training guide stays training-only. * docs(instance-preferences): document processing instance preferences Add instance preferences to the MLOps Processing Jobs guide (dict-shaped candidates, shared vs per-candidate counts, reading the selected type, supported processor classes) with a standalone example notebook split out of the training notebook, wired into the MLOps toctree. * docs(instance-preferences): use canonical imports, tighten count-mode rules - Import from sagemaker.core.training.configs instead of the deprecated sagemaker.train.configs shim (which warns on use), and from sagemaker.core.shapes rather than the nested module path, matching the short form used elsewhere in the docs. - State the count rule as it is enforced: exactly one of a top-level instance_count or an instance_count on every candidate; mixed, partial, and omitted are rejected. - Read the winner off the trainer's job resource instead of a redundant TrainingJob.get round trip. (cherry picked from commit 093b2f005e93cb7e0b7eb5e3c2c4fcc721eda07d) * docs(instance-preferences): align with launch drafts, tighten wording (#2146) Corrects and completes the docs against the API and dev-guide drafts: - managed spot training belongs in the mutual-exclusion list - a candidate's training plan must match that candidate's instance type, and the job-level training_plan_arn applies to whichever candidate matches its type (rather than being unusable with a list) - while nothing has capacity the job stays pending and retries the list; max_pending_time_in_seconds bounds the total, not each candidate, and only applies when the list includes an accelerated type - selection is capacity-based and does not validate workload fit - exactly one type is selected; job-level storage and keep-alive settings apply to it, as does billing - per-candidate training plans are training-only, so that limit does not belong in the processing notes Adds the SDK-level restrictions the public page omitted: local mode, training recipes, JumpStart models, and AWS Batch training queues. The Batch restriction is also noted in the AWS Batch section, where a reader choosing queued submission would look for it. Also trims the notebooks: the intro's table of contents duplicated the step headings (and still listed the processing step that moved to its own notebook), and its usage rules restated what each step and the notes already cover. Rules now live once, at the point of use. (cherry picked from commit b720fa83f03b806828c8a58bd40d95ee10279c3a) * fix(model): lower MaxPendingTimeInSeconds bound to 1800 The bundled service model still carried min=7200. Integer bounds are validated from this model before a request leaves the SDK, so the service's lower bound of 1800 was rejected client-side. One line; max and the field documentation are unchanged. (cherry picked from commit 421478b30cbcfcd4c9e3e41c359641f535298845) * test(instance-preferences): enforce the public contract in integ tests (#2158) Instance Preferences is publicly released, so the integ tests no longer need the pre-release escape hatches: the XFAIL branches for gated Describe responses, model @required rejections, and InternalServerError before type resolution are removed, along with the rollout notes that explained them. Both tests now assert the full contract unconditionally: create accepted, preferences echoed, top-level instance type absent, winner among the submitted preferences with the expected count. The processing test used a private loader for the bundled service model; the public botocore model now carries the shapes, so it uses a plain boto3 client. Candidate instance types are overridable via INSTANCE_PREFERENCES_TEST_INSTANCE_TYPES / PROCESSING_INSTANCE_PREFERENCES_TEST_INSTANCE_TYPES so the tests can run in accounts where the default types are quota-saturated. Verified against prod us-west-2: processing passes (winner resolved, 3 min); training passes (winner resolved, 8 min). * fix(instance-preferences): run integ tests in CI, reject managed spot client-side, stabilize IntPipeVar codegen (#6250) * fix(instance-preferences): reject managed spot client-side; make IntPipeVar codegen-stable The documentation lists managed spot training among the settings that are mutually exclusive with instance_preferences, but the client-side validator only checked instance_type, instance_groups and instance_placement_config, so the spot conflict surfaced as a service error. Both validator copies now reject an explicit enable_managed_spot_training=True; False and unset remain valid. The IntPipeVar annotation on the five instance-preferences count members was a hand edit that generate_shapes could not reproduce: only ResourceConfig.InstanceCount was registered in PIPE_VAR_OVERRIDES, so the next regeneration would have narrowed the others back to Optional[int]. All five are registered, and a unit test asserts the extractor emits IntPipeVar for each so a dropped override fails loudly. * test(instance-preferences): run integ tests in the standard CI account Both integ tests were gated on account-specific environment variables that no CI configuration sets, so every CI run skipped them and the suite passed without exercising the feature. Training coverage: - tests/integ/train/shallow: two accepted cases (uniform and per-preference counts; submit, Describe echo, stop) and four rejected cases (instance_type, managed spot and duplicate types client-side; the five-entry cap server-side). Runs on every pull request. - tests/integ/train/test_instance_preferences.py: a CPU-only end-to-end test that submits a job with a preference list and follows it until the service selects a winner and the job completes on it, asserting the winner is one of the submitted types and the top-level type stays unset. It stops the job in a finally block so it never runs to max_runtime on shared quota. The processing test uses the suite's SageMakerRole and resolves its image through image_uris, as the other sagemaker-core integration tests do, and now stops the job in a finally block once the winner is observed. Candidate types remain overridable via *_TEST_INSTANCE_TYPES for accounts whose default types lack quota; no account-specific values remain. Verified in us-west-2: all six shallow cases pass, the training e2e runs to Completed on a selected winner, and the processing test passes with a resolved winner. * docs(instance-preferences): resolve example images from a candidate type; make the PipeVar override test tox-safe (#6257) * docs(instance-preferences): resolve the example image from a candidate type The training notebook and the training guide resolved the PyTorch image with instance_type="ml.m5.xlarge", which selects the CPU image, and then paired it with GPU candidates (ml.p5.48xlarge, ml.p4d.24xlarge). A reader copying either example would run a GPU job on the CPU image. Both now resolve the image from one of the listed candidates (all three GPU candidates map to the same pytorch-training:2.0.0-gpu-py310 image), and state the rule the feature implies: the image is fixed at submission time while the instance type is not, so every candidate must be able to run it. The processing notebook and guide are updated the same way for consistency; the scikit-learn image is CPU-only and all candidates are m5 types, so their behaviour was already correct. * test(instance-preferences): decouple the PipeVar override test from the packaged service model TestInstancePreferencesPipeVarOverrides built a ShapesExtractor from the real service JSON, which lives under sample/ in the source tree and is not part of the installed package. Under tox the extractor is imported from site-packages, the file is absent, and all six cases errored at setup with FileNotFoundError. The test now injects a minimal in-memory model that mirrors the real member -> shape wiring for the six count members, so it exercises exactly the PIPE_VAR_OVERRIDES lookup and nothing else. A control shape with the same integer type and no override asserts the widening is targeted rather than blanket. Removing any one override still fails exactly that member's case. * docs(instance-preferences): list the shallow test in the suite README; define processing_image in the guide The shallow suite README maps each shallow file to its deep counterpart; the instance-preferences pair was missing. The processing guide's instance-preferences snippet referenced processing_image without defining it on the page; it now resolves the image from one of the candidates, matching the training guide. --------- Co-authored-by: deeppcs <deeppcs@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #6250 on the Instance Preferences branch; targets
feature-smtj-instance-preferences-rebasedso #6249 updates in place.Examples resolve the image from a candidate type. The training notebook and the training guide resolved the PyTorch image with
instance_type="ml.m5.xlarge"(the CPU image) and then paired it with GPU candidates (ml.p5.48xlarge,ml.p4d.24xlarge). A reader copying either example would run a GPU job on the CPU image. Both now resolve from one of the listed candidates — all three GPU candidates in the guide map to the samepytorch-training:2.0.0-gpu-py310image — and state the rule the feature implies: the image is fixed at submission time while the instance type is not, so every candidate must be able to run it. The processing notebook and guide are aligned the same way (the scikit-learn image is CPU-only and all candidates are m5, so they were already correct); the processing guide snippet also now definesprocessing_image, which it previously referenced without defining.unit-tests (sagemaker-core)passes under tox.TestInstancePreferencesPipeVarOverridesconstructedShapesExtractor()from the real service JSON, which lives undersample/in the source tree and is not part of the installed package; under tox all six cases errored at setup withFileNotFoundError. The test now injects a minimal in-memory model mirroring the real member → shape wiring, so it exercises exactly thePIPE_VAR_OVERRIDESlookup. A control shape with the same integer type and no override asserts the widening is targeted, not blanket.Shallow suite README lists the instance-preferences shallow/deep pair alongside the other trainers.
Testing
sample/(the tox condition — the previous fixture reproduces the CIFileNotFoundErrorthere). Removing any one override fails exactly that member's case.image_uris.retrievecall executes, and every listed candidate resolves to the same image as the resolver type. Both.rstfiles parse with docutils.nb_execution_mode = 'off').