Skip to content

[CI only — do not merge] Instance Preferences: run master integration tests - #6251

Open
deeppcs wants to merge 4 commits into
aws:masterfrom
deeppcs:instance-preferences-ci
Open

[CI only — do not merge] Instance Preferences: run master integration tests#6251
deeppcs wants to merge 4 commits into
aws:masterfrom
deeppcs:instance-preferences-ci

Conversation

@deeppcs

@deeppcs deeppcs commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Description

CI-only PR — not for merge. This targets master solely so pr-checks-master runs the integration tests against the Instance Preferences tree; PRs into feature branches do not trigger them. The branch is #6249's linear rebase plus the two review-fix commits from #6250. The merge path stays #6250 → #6249 → master; this PR is closed once its fast-integ-tests run is on record.

Review fixes (on top of #6249)

Integ tests now run in the standard CI account. Both tests were gated on account-specific environment variables that no CI configuration sets, so every run skipped them and the suite passed without exercising the feature. Now:

  • Training coverage lives in sagemaker-train/tests/integ/train/shallow/test_instance_preferences.py, which fast-integ-tests runs on every pull request: two accepted cases (uniform and per-preference counts — submit, Describe echo, stop) and four rejected cases (instance_type, managed spot, and duplicate types client-side; the five-entry cap server-side). The environment-gated deep test is removed.
  • The processing test uses the suite's SageMakerRole and resolves its image via image_uris, like the other sagemaker-core integ tests.
  • Candidate types remain overridable via *_TEST_INSTANCE_TYPES for accounts whose defaults lack quota. No account-specific values remain in the repository.

Managed spot is rejected client-side. The documentation already lists it as mutually exclusive with instance_preferences, but validate_instance_preferences only checked instance_type, instance_groups, and instance_placement_config. Both copies now reject an explicit enable_managed_spot_training=True; False/unset stay valid.

IntPipeVar is codegen-stable. The widening on the five instance-preferences count members was a hand edit that generate_shapes could not reproduce — only ResourceConfig.InstanceCount was in PIPE_VAR_OVERRIDES, so the next regeneration would have narrowed the rest to Optional[int]. All five are registered, and a unit test asserts the extractor emits IntPipeVar for each, so a dropped override fails loudly instead of silently.

Feature (unchanged from #6249)

See #6249 for the full description of the feature, documentation, and notebook changes.

Testing

In us-west-2, with no environment variables beyond region:

  • New shallow tests: 6/6 pass. Under the exact CI command (pytest tests/integ/train/shallow -n 8 -m "not gpu_intensive and not us_east_1") all six pass alongside the existing suite.
  • Processing integ test: pass, winner resolved.
  • Unit: 196 sagemaker-core (compute configs, processing, codec, service model, shapes extractor), 111 sagemaker-train (model_trainer, defaults). The new extractor test was mutation-checked: removing one override fails exactly that member's case.

Related

@deeppcs @papriwal
Instance Preferences: full change set replayed onto master tip (aws#2154
560f0ed 
)

* feat(instance-preferences): Add Instance Preferences (multi-instance-type) support (aws#2093)

* feat(instance-preferences): add multi-instance-type support to PySDK

Add customer-specified Instance Preferences (multi-instance-type)
support across sagemaker-core and sagemaker-train, mirroring the
additive, backward-compatible SageMaker model change.

shapes / model (sagemaker-core):
- New InstancePreference shape (instance_type, optional per-preference
  instance_count, optional training_plan_arns capped at 1 via the new
  TrainingPlanArnList model shape).
- New processing-specific ProcessingInstancePreference /
  ProcessingInstancePreferenceList shapes (ProcessingInstanceType/Count
  members, no TrainingPlanArns - per-type training plans are
  training-only), matching the service model where training and
  processing preferences are distinct shapes.
- ResourceConfig: instance_preferences + output-only
  selected_instance_type / selected_instance_count.
- ProcessingClusterConfig: instance_preferences + selected_*; relax
  required on instance_type / instance_count (XOR instance_preferences,
  enforced server-side; matches the GA contract).
- Mirror all additions into the bundled service-2.json codegen input
  (also consumed by the runtime botocore loader).

consumers:
- Compute (modules + training configs) inherit the field; preserve
  nested InstancePreference objects in _to_resource_config so an unset
  per-preference instance_count does not fail re-validation.
- Processor: instance_preferences param + emit InstancePreferences into
  ClusterConfig; mutual-exclusivity with instance_type/instance_count.
- AWS Batch: reconstruct instance_preferences onto the rebuilt Compute.
- ModelTrainer: guard local mode / training recipes / JumpStart against
  instance_preferences with clear errors (out of scope for M1).

tests: round-trip for both Compute classes, selected_* not sent on
create, ProcessingClusterConfig with processing-specific shapes,
Processor ClusterConfig emission and mutual exclusivity, and AWS Batch
reconstruction.

* fix(instance-preferences): defaults must not inject instance_type; add integ + contract tests

- TrainDefaults.get_compute no longer injects the default top-level
  instance_type / instance_count when Compute carries
  instance_preferences (mutually exclusive server-side; the injected
  default made every instance-preferences ModelTrainer job fail
  CreateTrainingJob validation with 'InstanceType cannot be specified
  with InstancePreferences' - caught by the new integ test against a
  live endpoint). Unit tests added for both count modes.
- New integ test (tests/integ/train/test_instance_preferences.py):
  launches a real training job via ModelTrainer + Compute with an
  ordered instance_preferences list, asserts the create/describe
  contract, and asserts the resolved selected_instance_type /
  selected_instance_count is one of the submitted preferences once the
  job leaves PENDING. Env-gated (role/image/S3 via
  INSTANCE_PREFERENCES_TEST_* vars); reports XFAIL with the job name
  while the server-side scheduling stage is not yet deployed on the
  target endpoint, and enforces the full contract automatically once
  it is.
- New unit test pinning the Instance Preferences contract in the
  bundled service-2.json (distinct training/processing preference
  shapes, TrainingPlanArnList cap 1, list cap 5, Selected* on both
  Describe shapes, ProcessingClusterConfig required relaxation).

* fix(instance-preferences): correct count semantics + client validation; add processing integ test

Address review feedback on the Processor mutual-exclusivity check:
instance_count is NOT flatly exclusive with instance_preferences - it is
the shared uniform count for whichever preference wins. Corrected to the
count-mode contract (V4) on the Processor AND, for parity, added the same
client-side validation to both training Compute classes (server remains
the source of truth):
- uniform mode: top-level instance_count set, no element carries a count
  (Processor emits the uniform count into ClusterConfig);
- per-preference mode: EVERY element carries a count, top-level unset;
- reject both-set, partial per-preference, and neither;
- instance_type / instance_groups / instance_placement_config remain
  mutually exclusive with instance_preferences (V1/V5).
Shared validate_instance_preferences helper in modules/utils.py and
training/utils.py (per-package twin pattern); wired into both Compute
_model_validator hooks (assignment-safe: no-op without preferences).
Unit tests for all modes on Processor and both Compute classes; two
pre-existing test fixtures corrected to V4-valid combinations.

Add processing integ test
(tests/integ/processing/test_instance_preferences_processing.py):
launches a real processing job via Processor with InstancePreferences
through the bundled service model, asserts the create/describe contract
and the resolved Selected* winner; env-gated
(PROCESSING_INSTANCE_PREFERENCES_TEST_* vars). Reports XFAIL while the
endpoint's public model still hard-requires InstanceType/InstanceCount
(the @required relaxation ships with the GA ungating), and enforces the
full contract automatically once deployed.

* revert(instance-preferences): drop AWS Batch reconstruction support

Remove the instance_preferences reconstruction in the AWS Batch
TrainingQueuedJob path (and its test) - Batch support for Instance
Preferences will ship separately later; this PR intentionally adds no
Batch functionality. Files restored to upstream/master state.

Also: black-format the new training integ test (line-length 100).

* fix(instance-preferences): complete client validation; clearer error wording

Complete the client-side validation sweep against the server rule set
(server remains the source of truth):
- Training plans: the top-level (whole-job) training_plan_arn is
  mutually exclusive with per-preference training_plan_arns (was
  missing - review catch).
- Duplicate instance types across preferences rejected (both Compute
  classes + Processor).
- List-size limits (max 5 preferences, max 1 plan per preference) are
  deliberately NOT enforced client-side: they are server-side
  configurable, so raising them must never require an SDK release
  (botocore does not enforce the C2J list min/max either - verified -
  so the service is the single enforcement point for sizes). The
  current limits are documented on the service-2.json list shapes
  (InstancePreferenceList, TrainingPlanArnList,
  ProcessingInstancePreferenceList) instead. Type validity
  (region-dependent) and count bounds are likewise server-only.

Reword thrown error messages to use 'top-level instance_count /
training_plan_arn' instead of the design-doc jargon 'uniform', so the
error names the actual field the customer must change.

Tests: whole-job-plan XOR per-preference-plans (reject) /
whole-job-plan-without-per-preference-plans (valid) / duplicate-type
cases on both Compute classes and Processor.

* docs(instance-preferences): add example notebook; extract Processor validation helper

Address review feedback:
- Add v3-examples/training-examples/instance-preferences-example.ipynb
  demonstrating the customer experience end to end: ModelTrainer +
  Compute with an ordered preference list (shared top-level count),
  per-preference instance counts, per-preference training plans with an
  on-demand fallback, reading the resolved winner via
  selected_instance_type/selected_instance_count on Describe, and the
  Processor equivalent for processing jobs. All construction patterns
  verified executable against the SDK.
- Extract the Processor's inline instance_preferences validation into a
  module-level _validate_processing_instance_preferences helper
  (behavior unchanged; unit tests pass unmodified).

* docs(instance-preferences): add per-preference-count Processor example to notebook

* docs(instance-preferences): polish example notebook for public customers

Align the notebook with the v3-examples conventions:
- Session()/get_execution_role()/image_uris.retrieve() setup instead of
  paste-in placeholders; 'Step N' section headers.
- Customer-facing wording: drop client/service enforcement internals;
  'instance types must be unique across preferences'.

(cherry picked from commit 81758d391bb48aa4676bd59def97fef821fe05f5)

* fix(instance-preferences): shape-DAG regen, Spark processor plumbing, integ test (aws#2099)

* test(instance-preferences): XFAIL training integ test under pre-GA dark-launch gating

The endpoint's frontend strips internal-gated fields (instance_preferences,
selected_*) from customer Describe responses while the surface is
dark-launched - the documented pre-GA contract (verified: the long-internal
UseReservedCapacity is stripped identically, and the layer below returns
the fields correctly with the top-level suppression rule applied). Detect
the gating (echo absent from Describe) and report XFAIL with a re-run-after-
GA-ungate message instead of failing on the unassertable public contract.

Verified live against a pre-GA endpoint: job schedules and completes with
the correct winner persisted; test XFAILs cleanly in ~2s.

* feat(instance-preferences): plumb through Script/Spark processors

ScriptProcessor, _SparkProcessorBase, PySparkProcessor, and
SparkJarProcessor now accept instance_preferences and forward it to the
base Processor (which already carried the parameter, validation, and
ClusterConfig emission). Spark image-URI resolution falls back to the
first preference's instance type when no top-level instance_type is set.

Unit tests: forwarding on all three public classes, validation reuse,
and first-preference image lookup (5 new; full processing suite 99
passing).

* fix(instance-preferences): regenerate shape DAG; fix integ wait loop

The resource deserializer (codec.transform) walks the generated
SHAPE_DAG, which was not regenerated when the InstancePreferences
shapes landed - so DescribeTrainingJob/DescribeProcessingJob responses
silently DROPPED instance_preferences and Selected* from the typed
resources (requests were unaffected; the gap was masked while the beta
frontend stripped the fields pre-ungate). Regenerated via
ShapesExtractor from the bundled service-2.json (+44 lines, additive).

Also fix the integ test's wait loop: gate on terminal-status-or-winner
instead of a secondary-status allowlist (Selected* propagation lags the
transitions; jobs completing fast broke out pre-resolution).

Verified live against ungated beta PDX: full contract PASS in 62s
(winner + echo + suppression on the typed resource).

* fix(instance-preferences): Spark image resolution requires candidate agreement

A job runs one image but the winner among instance_preferences is
unknown at create, so resolving from the first preference was silently
wrong for candidate lists that mix processor classes. Auto-resolution
now resolves per candidate and requires all candidates to yield the
SAME image; on divergence it raises with a clear pass-image_uri-
explicitly message. No behavior change for homogeneous lists (Spark
images are CPU-only today, so real lists agree).

* fix(instance-preferences): guard degenerate preference list in Spark image resolution

An instance_preferences list with no InstanceType on any element hit
StopIteration inside the candidate-agreement resolution (which runs
before the base Processor validation). Fall through to the classic
resolution path so the base validation owns the clean reject.

* fix(instance-preferences): name unresolvable Spark candidates

Per-candidate image resolution failures (e.g. GPU candidates: Spark has
no gpu image variant, so image_uris.retrieve raises) now surface a
message naming the candidate and the pass-image_uri-explicitly remedy.

Verified unmocked: mixed CPU families (m5/r5/c5) resolve to the same
image and pass agreement; a g5 candidate raises the named error;
explicit image_uri bypasses resolution entirely.

(cherry picked from commit 574a87d540c26d651d4844fc99737f0e1bc60047)

* docs(instance-preferences): document instance preferences for training and processing (aws#2142)

* docs(instance-preferences): document training instance preferences

The example notebook shipped in aws#2093 was referenced by no .rst, so
Sphinx never rendered it. Add an Instance Preferences section to the
training guide (ordered candidates, per-candidate counts and training
plans, reading the selected type) and wire the notebook into the
Training Examples toctree.

Move the notebook's processing steps out to a dedicated processing
example so the training guide stays training-only.

* docs(instance-preferences): document processing instance preferences

Add instance preferences to the MLOps Processing Jobs guide (dict-shaped
candidates, shared vs per-candidate counts, reading the selected type,
supported processor classes) with a standalone example notebook split
out of the training notebook, wired into the MLOps toctree.

* docs(instance-preferences): use canonical imports, tighten count-mode rules

- Import from sagemaker.core.training.configs instead of the deprecated
  sagemaker.train.configs shim (which warns on use), and from
  sagemaker.core.shapes rather than the nested module path, matching the
  short form used elsewhere in the docs.
- State the count rule as it is enforced: exactly one of a top-level
  instance_count or an instance_count on every candidate; mixed, partial,
  and omitted are rejected.
- Read the winner off the trainer's job resource instead of a redundant
  TrainingJob.get round trip.

(cherry picked from commit 093b2f005e93cb7e0b7eb5e3c2c4fcc721eda07d)

* docs(instance-preferences): align with launch drafts, tighten wording (aws#2146)

Corrects and completes the docs against the API and dev-guide drafts:

- managed spot training belongs in the mutual-exclusion list
- a candidate's training plan must match that candidate's instance type,
  and the job-level training_plan_arn applies to whichever candidate
  matches its type (rather than being unusable with a list)
- while nothing has capacity the job stays pending and retries the list;
  max_pending_time_in_seconds bounds the total, not each candidate, and
  only applies when the list includes an accelerated type
- selection is capacity-based and does not validate workload fit
- exactly one type is selected; job-level storage and keep-alive settings
  apply to it, as does billing
- per-candidate training plans are training-only, so that limit does not
  belong in the processing notes

Adds the SDK-level restrictions the public page omitted: local mode,
training recipes, JumpStart models, and AWS Batch training queues. The
Batch restriction is also noted in the AWS Batch section, where a reader
choosing queued submission would look for it.

Also trims the notebooks: the intro's table of contents duplicated the
step headings (and still listed the processing step that moved to its own
notebook), and its usage rules restated what each step and the notes
already cover. Rules now live once, at the point of use.

(cherry picked from commit b720fa83f03b806828c8a58bd40d95ee10279c3a)

* fix(model): lower MaxPendingTimeInSeconds bound to 1800

The bundled service model still carried min=7200. Integer bounds are
validated from this model before a request leaves the SDK, so the
service's lower bound of 1800 was rejected client-side.

One line; max and the field documentation are unchanged.

(cherry picked from commit 421478b30cbcfcd4c9e3e41c359641f535298845)
@deeppcs @papriwal
test(instance-preferences): enforce the public contract in integ tests (
3d58dc9 
aws#2158)

Instance Preferences is publicly released, so the integ tests no longer
need the pre-release escape hatches: the XFAIL branches for gated
Describe responses, model @required rejections, and InternalServerError
before type resolution are removed, along with the rollout notes that
explained them. Both tests now assert the full contract unconditionally:
create accepted, preferences echoed, top-level instance type absent,
winner among the submitted preferences with the expected count.

The processing test used a private loader for the bundled service model;
the public botocore model now carries the shapes, so it uses a plain
boto3 client. Candidate instance types are overridable via
INSTANCE_PREFERENCES_TEST_INSTANCE_TYPES /
PROCESSING_INSTANCE_PREFERENCES_TEST_INSTANCE_TYPES so the tests can run
in accounts where the default types are quota-saturated.

Verified against prod us-west-2: processing passes (winner resolved,
3 min); training passes (winner resolved, 8 min).
@deeppcs
fix(instance-preferences): reject managed spot client-side; make IntP…
7d0eb92 
…ipeVar codegen-stable

The documentation lists managed spot training among the settings that are
mutually exclusive with instance_preferences, but the client-side
validator only checked instance_type, instance_groups and
instance_placement_config, so the spot conflict surfaced as a service
error. Both validator copies now reject an explicit
enable_managed_spot_training=True; False and unset remain valid.

The IntPipeVar annotation on the five instance-preferences count members
was a hand edit that generate_shapes could not reproduce: only
ResourceConfig.InstanceCount was registered in PIPE_VAR_OVERRIDES, so the
next regeneration would have narrowed the others back to Optional[int].
All five are registered, and a unit test asserts the extractor emits
IntPipeVar for each so a dropped override fails loudly.
@deeppcs
test(instance-preferences): run integ tests in the standard CI account
1d58ae3 
Both integ tests were gated on account-specific environment variables
that no CI configuration sets, so every CI run skipped them and the suite
passed without exercising the feature.

Training coverage moves to tests/integ/train/shallow, which runs on every
pull request: two accepted cases (uniform and per-preference counts;
submit, Describe echo, stop) and four rejected cases (instance_type,
managed spot and duplicate types client-side; the five-entry cap
server-side). The environment-gated deep test is removed.

The processing test uses the suite's SageMakerRole and resolves its image
through image_uris, as the other sagemaker-core integration tests do.

Candidate types remain overridable via *_TEST_INSTANCE_TYPES for accounts
whose default types lack quota; no account-specific values remain.

Verified in us-west-2: all six shallow cases pass alongside the existing
shallow suite, and the processing test passes with a resolved winner.
@deeppcs
deeppcs deployed to manual-approval September 10, 2026 05:39 — with GitHub Actions Active
@deeppcs deeppcs mentioned this pull request Sep 10, 2026
Open
@deeppcs
deeppcs deployed to manual-approval September 10, 2026 05:39 — with GitHub Actions Active
@deeppcs deeppcs changed the title feat: Instance Preferences for training and processing jobs (Compute.instance_preferences) [CI only — do not merge] Instance Preferences: run master integration tests Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@deeppcs