feat(slurm): integrate Slurm execution for 1.0.0 release by nabinchha · Pull Request #1040 · NVIDIA-NeMo/DataDesigner

nabinchha · 2026-10-15T20:56:59Z

📋 Summary

Integrate the optional Slurm execution package into main for the planned 1.0.0 release. This consolidates the feature-branch implementation and documentation in one reviewable PR. Keep this PR open; do not merge until release review is complete.

🔗 Related Issue

N/A — integration of the existing Slurm feature branch and its previously reviewed changes.

🔄 Changes

✨ Added

Add the data-designer-slurm package and the version-matched data-designer[slurm] extra.
Add declarative Slurm profiles, serving image management, run planning and execution, status/cancel/retry/collection, and benchmark workflows.
Add Slurm examples, architecture notes, and user documentation.

🔧 Changed

Extend the CLI and generation invocation boundaries for the optional Slurm integration.
Update the publish workflow to build and upload the config, engine, Slurm, and interface packages at the same version.

🐛 Fixed

Incorporate feature-branch fixes for startup diagnostics, multi-model placement, shard recovery, partial outputs, and Parquet collection schema handling.

🔍 Attention Areas

Package metadata and publish order — confirm the exact-version dependency relationship and 1.0.0rc1 wheel/install flow.
Slurm run contracts and runtime — review public configuration and allocation behavior.

🧪 Testing

make test-slurm — 1,915 passed after rebasing onto main.
make check-slurm, uv lock --check, and uv pip check passed.
Unit and integration-style tests added for the Slurm package.
make test and final 1.0.0rc1 wheel/install validation are pending release review.

✅ Checklist

Follows commit message conventions.
All commits are signed off (DCO): nine existing feature-branch commits lack a Signed-off-by trailer; resolve with their authors before merge if required.
Architecture docs updated.
Release review complete; do not merge yet.

github-actions · 2026-10-15T20:59:38Z

Fern preview: https://nvidia-preview-pr-1040.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

* feat: add optional Slurm package Add the shared-namespace leaf package and publish it through the same-version data-designer[slurm] extra. Cover resolver, namespace, base-only isolation, and built-wheel installation behavior.\n\nCloses #852 Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: address Slurm package review findings Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: publish Slurm package Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: scope workspace dependency extras Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: publish Slurm before base package Upload the Slurm distribution before data-designer so the exact-version extra is resolvable as soon as the base package is published. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * docs: generalize Slurm package wording Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: define public generation contract * fix: complete public generation contract * docs: clarify public invocation limits * fix: handle mixed-case config extensions Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: complete generation contract edge cases Signed-off-by: Andre Manoel <amanoel@nvidia.com> * test: make inline config regression self-contained Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: classify malformed config paths Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: normalize generation artifact errors Classify malformed inline configs independently of filename suffixes. Wrap unreadable result artifacts in the public storage error across counting and export paths. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: complete artifact error boundaries Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: tighten generation result contract Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: add lazy CLI extensions Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: isolate CLI extension failures Signed-off-by: Andre Manoel <amanoel@nvidia.com> * refactor: order public CLI members first Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com> Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> Co-authored-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: add Slurm state record contracts Define strict, versioned execution, readiness, output, reconciliation, and scheduler records. Enforce cross-record identity, digest, shard-set, and terminal-state invariants with deterministic golden fixtures and built-wheel coverage. Closes #865 * fix: harden Slurm state contracts Reject partial winners and conflicting shard ownership. Require explicit versions, enforce evidence chronology and readiness precedence, and cover every state-contract path with validated fixtures. Refs #865 * fix(slurm): harden state contract invariants Hash persisted record bytes, align state identities with the resolved plan vocabulary, and preserve public model aliases. Add monotonic attempt and scheduler transition checks, including fixed accounting-lag deadlines and UNKNOWN precedence. Refs #865 * refactor(slurm): share contract primitives Make planning and runtime state consume one public immutable model family. Align shard, attempt, range, artifact, and resume workspace shapes so the configuration-plan branch can rebase without preserving duplicate Pydantic types.\n\nRefs #865

* feat: define Slurm config and plan contracts Add strict authored configuration, profile, image, client, benchmark, dependency-lock, and resolved-plan records. Validate cross-record identities and digests with sanitized single-node and multi-node golden fixtures. Closes #873 Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: tighten Slurm contract validation Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: align Slurm F3 contract seam * fix: bind client dataset to effective resume mode * fix: harden Slurm shared contracts Prevent secret material from entering persisted configuration and make nested contract collections immutable. Tighten client and benchmark semantic invariants with focused negative coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: address Slurm contract review findings Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: close Slurm argument validation bypasses Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: close remaining Slurm contract gaps Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com> Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> Co-authored-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: integrate Slurm plan and state Validate plan-aware shards, readiness, attempts, and finalization through a reusable immutable context. Add deterministic goldens and negative coverage for mismatch and failure paths. Closes #880 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: reject unsubmitted planned attempts Require scheduler-bound plan validation to start only after an attempt leaves the created state. Cover created attempts that already carry scheduler metadata and verify the integration validator imports from an isolated Slurm wheel. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: harden Slurm integration join coverage Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: simplify plan state validator Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: remove validator wrapper functions Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* test: add deterministic Slurm fakes Add reusable scheduler, dependency, plugin, serving, clock, and state-transition fixtures for local and CI Slurm integration tests. Keep the harness deterministic and isolated from production lane implementations. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * test: bind Slurm fakes to contracts Promote the merged planning and finalization goldens to package-wide fixtures, exercise dependency doubles with canonical records, and add site-neutral rendered script goldens for downstream lanes. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * test: complete post-F3 Slurm fake coverage Bind rendered scripts and shared fixtures to the merged contracts, tighten scheduler command behavior, and cover retry terminal states and invalid serving restarts. Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: add Slurm image registry foundation Add digest-bound client and serving inspection, atomic workspace-derived registry persistence, and verified alias/path resolution.\n\nPart of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover image inspection boundaries Exercise the production environment reader and verify restrictive atomic registry persistence discovered during review.\n\nPart of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align image registry persistence Persist the package-owned image registry as deterministic safe YAML at the migration plan's canonical path and declare the runtime YAML dependency explicitly.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden image registry file boundaries Reject symlinked registry and lock files, use explicit UTF-8 persistence, and normalize malformed byte content into the package image error hierarchy.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: verify Slurm image wheel integration Exercise the public image service from an isolated installed wheel and assert that the wheel metadata carries the runtime PyYAML dependency.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: verify registry file types before use Treat only a genuinely missing registry as empty and reject non-regular registry or lock nodes before reading, locking, or changing permissions.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: rollback stale image registrations Reverify image bytes while registry mutation locks remain held and restore the previous snapshot if final verification fails. Part of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix image registry review findings Require Data Designer in client image inspections, verify artifacts before publishing registry updates, harden storage directory boundaries, and normalize recursive YAML failures. * harden image registry contracts - Keep the image-registry foundation internal until the public facade lands\n- Require a complete client package inventory and unambiguous registry schema\n- Close SQSH identity races and exercise publication behavior through owned APIs * strengthen image registry test coverage Exercise persisted semantic invariants and exact-byte hashes. Split record, store, and service coverage by ownership. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * address image registry review feedback Bind registry transactions to verified directory descriptors, centralize immutable image fact comparisons, and verify internal image modules from the installed wheel. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * clarify image test helper names Use explicit get prefixes for value-producing test helpers while preserving action verbs for creation, inspection, and filesystem operations. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * split registry filesystem helpers Move descriptor-bound filesystem operations into a focused internal module while keeping registry transactions and YAML semantics together. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: add Slurm command client and renderer Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: tighten Slurm launcher boundaries Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: use native sacct output fields Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: require expanded Slurm array queries Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: sanitize Slurm command diagnostics Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: reject invalid visible GPU memory Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: hash rendered inputs without manifests Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: make Slurm output decoding stable Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: validate finite command timeouts Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm boundary inputs Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: reject malformed Slurm GRES lists Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: isolate submitted Slurm environments Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: verify launcher wheel import Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: pin batch host tool path Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover Slurm launcher boundaries Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: correlate Slurm query responses Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm command boundaries Preserve the caller's PATH as the only ambient lookup input for default Slurm commands while continuing to isolate all other environment variables. Normalize oversized numeric scheduler fields into the launcher parse-error boundary. Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: fall back from empty command path Use the platform default search path when the ambient PATH is absent or empty so bare Slurm executables remain resolvable. Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align Slurm launcher semantics * fix Slurm job observation contracts Support both ordinary jobs and array-task observations while preserving accounting-lag semantics. Bound scheduler numeric fields and diagnostics, and honor unthrottled plan arrays when concurrency is omitted. * refine Slurm launcher boundaries - Internalize implementation APIs and clarify transient types - Harden parsing, rendering, and process error normalization - Extend fakes and executable checksum coverage Part of #868 * fix Slurm selector filtering ownership * simplify Slurm launcher structure --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: resolve Slurm serving deployments Add pure, package-owned vLLM compatibility and deterministic process, readiness, backend, and logical endpoint resolution. Validate topology, launch inputs, and network identities with single-node and multi-node golden coverage. Closes #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm serving contracts Revalidate persisted launch policy inputs, require canonical process ordering, and normalize resolver errors. Reject vLLM CLI aliases and transports that could override compiler-owned topology or endpoints. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: close Slurm serving validation gaps Preserve independent single-node expert-parallel replicas while rejecting cross-node EP. Block attached compiler-owned vLLM topology aliases, require canonical runtime versions, and cover the boundaries with regression tests. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden vLLM launch ownership Reject unmodeled topology, plugin, LoRA, speculative-model, and environment overrides at authored and persisted boundaries. Validate JSON arguments, secret references, readiness paths, and compatibility version lengths. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: bind serving context to plan Derive serving placement and client endpoint records from the same resolved plan so callers cannot mix contracts across plans. Reject vLLM API authentication until the logical endpoint models credential forwarding. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align serving with migration contracts - enforce plan-owned environment and endpoint behavior - limit compatibility to tested vLLM release series - split shared and vLLM configuration from run records - cover multi-group ordering and launch delays Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: clarify option flag extraction Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: share argument and duration helpers Use action-named shared helpers for option extraction and duration conversion while keeping vLLM flag canonicalization backend-specific. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: simplify vLLM compatibility contract Track the tested runtime range and contract revision without repeating baseline vLLM or package-owned behaviors as capability booleans. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: make vLLM resolution version agnostic Preserve inspected runtime versions as digest-bound provenance without hard-coded support gates or redundant compatibility records. Continue to validate persisted text and compiler-owned vLLM arguments. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: treat runtime version as provenance Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: name vLLM serving module explicitly Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: remove serving context wrapper Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: centralize Slurm scalar types Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: use str for model aliases Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: clarify serving replica indices * fix: reject abbreviated vLLM flags * refactor: simplify Slurm serving contracts - make vLLM resolution consume placement inputs before plan assembly - keep serving internals backend-specific and remove speculative exports - clarify validation ownership, secret redaction, and digest semantics * fix: bind Slurm serving resolution inputs - Join placement and endpoint claims through one planner-owned context.\n- Normalize inconsistent resolved records to the serving error boundary.\n- Split resolver, record, and launch-policy tests by ownership. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * remove empty serving package initializer * fix serving contract validation boundaries Derive deployment and endpoint inputs from one resolved plan and centralize topology arithmetic. Reject plaintext secrets in persisted builder and vLLM argument payloads, normalize resolver errors, and require canonical argv tokens. * fix option argument validation gaps --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: compile deterministic Slurm run plans Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: enforce Slurm planning invariants Reject unsupported multi-shard semantics and managed output collisions at the effective configuration boundary. Normalize config errors and cover the corrected Big Iron field disposition. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: harden Slurm plan resolution Preserve partial early-shutdown intent and hide secret values in validation errors. Reject unsafe sharding semantics, require exact deployment coverage, and bind sourced payload identity. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: bind Slurm effective plan inputs Preserve all authored early-shutdown controls and reject direct effective invocation drift. Bind shard partition digests to deterministic persisted JSON bytes. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: align Slurm planning boundaries * fix: unify Slurm builder artifact identity * fix: validate Slurm compiler inputs * fix: complete Slurm plan resolution defaults * fix: address plan compiler review feedback * fix: preserve safe validation paths Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: preserve union validation paths Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: add Slurm service foundation Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: harden Slurm service contracts Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: bind Slurm service results Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: clarify Slurm service boundaries Separate run planning from attempt rendering so retries can reuse an immutable plan. Keep injected implementation seams private and align public service failures with the Data Designer error hierarchy. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: tighten Slurm service contracts * test: fix Slurm seam export guard * fix: define Slurm service boundary contracts * fix: expose Slurm service dependency contracts --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* fix: harden Slurm image publication Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: close Slurm image publication races - Hash SQSH bytes outside the global registry lock.\n- Preserve recovery state when commit synchronization fails.\n- Bind lifecycle artifacts to verified directory descriptors. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: add Slurm state writer Persist immutable run intent and shard manifests with restrictive descriptor-bound storage, locks, digest validation, and atomic publication. Validate monotonic attempt and readiness updates across concurrent writers, and cover crash recovery, unsafe filesystem state, tampering, and fresh-process reloads. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm state persistence Bind loaded records back to persisted plan and location invariants, harden lock and file handling against replacement and hardlink races, and avoid duplicate immutable context reads. Add adversarial, concurrency, interruption-recovery, import, and performance regression coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: recover interrupted state publication Repair a committed immutable record when an interruption leaves its package-owned temporary hard link behind. Add regression coverage for the exact post-publication interruption window. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: converge concurrent state recovery Treat a temporary link already removed by a recovery reader as a successful immutable publication. Cover the publisher-reader cleanup race directly. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover Slurm state reader errors Exercise invalid and missing shard, attempt, ordinal, and readiness inputs through the public state-writer API. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: retry concurrent state replacement Let descriptor-bound readers retry valid atomic replacements, including replacements that race interrupted-publication cleanup. Continue rejecting unsafe record types and bound the retry loop. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: compose Slurm state persistence Separate validated reads and descriptor-bound storage from the public mutation workflows. Close concurrent attempt publication, retry durability, exact-byte reading, and interrupted recovery races with focused regression coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: shorten state mutation workflows Keep error translation at public boundaries and move lock-held happy paths into small helpers. No state method now exceeds 30 lines. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: expose state collaborators within package Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: harden shard-local state boundaries Keep attempt and readiness mutations scoped to their target shard, move persisted plan validation into the state layer, and share descriptor-safe filesystem primitives across state and image storage. Make the schema-v1 no-migration policy explicit and freeze compatibility fixtures. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align state boundary invariants Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): add allocation-local client worker * fix(slurm): address client worker review feedback * fix(slurm): harden client worker boundaries * fix(slurm): harden client filesystem boundary

* feat: finalize Slurm shard winners Lease shard dataset workspaces under reviewed locks. Validate client and candidate records plus output file digests. Atomically publish one immutable winner and seal retries. Part of #869 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: unwind interrupted dataset locks Ensure an opened shard context closes when resume-lock acquisition is interrupted by any BaseException. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: rebind verified candidate files Reject candidate path replacement after digesting an opened output descriptor and before winner publication. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover nested candidate paths Exercise valid nested output verification and rejection of a symlinked intermediate directory. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: validate winner retry timestamps Reject non-UTC publication timestamps before the existing-winner idempotency path can ignore them. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: classify invalid winners as corruption Normalize semantically invalid persisted winner chains as corruption across load, finalize, attempt, and workspace APIs. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: compose shard winner finalization Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: compose finalization validation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden candidate publication boundaries Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: bind finalization publication Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: publish allocation results before success Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): wire M2 services and CLI Add package-owned execution, status, cancellation, and image service wiring. Resolve client dependencies and expose the M2 operations through the Slurm CLI. Part of #874 Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): reject swapped SQSH file types Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): clarify unavailable operations Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): resolve review findings --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* fix(slurm): close one-node M2 integration * fix(slurm): address integration review findings * fix(slurm): address smoke and review findings * docs(slurm): align status reconciliation help * fix(slurm): honor nested runtime mounts * fix(slurm): allow exclusive GPU allocations * fix(slurm): accept exclusive job GPU visibility * fix(slurm): load plans through nested mounts * fix(slurm): validate logical client paths * fix(slurm): resolve client artifacts by mount * fix(slurm): map serving model paths * fix(slurm): avoid idle GPU reaper * fix(slurm): isolate shared-node service ports * fix(slurm): validate allocation-local endpoints

* feat: reconcile persisted Slurm state Persist normalized scheduler observations and compose fresh-process run, shard, attempt, readiness, generation, and winner status. Preserve bounded accounting lag and immutable terminal evidence for status and benchmark refresh consumers. Part of #869 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): harden state reconciliation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): bound preemption reconciliation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): bound public preemption status Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat: add Slurm retry and deterministic collection (#915) * feat(slurm): add retry and deterministic collection Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): recover ambiguous submissions Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve collection snapshots Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): bind retries and collection ownership Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve scheduler path on retry Render retry allocations with the same profile-configured Slurm command path as initial generation attempts so cluster-local srun and scontrol binaries remain discoverable. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): isolate attempt observation clocks Apply persisted timestamp floors per scheduler identity so a clock-ahead sibling cannot expire another attempt's preemption-requeue window. Persist each active attempt with its own reconciled observation time. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): isolate plugin-aware client generation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): expose client worker boundary Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): compose client process boundary Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): refresh client overlay inventory Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): exercise plugin worker handoff Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* harden Slurm release artifacts Add sanitized credential and infrastructure scans for source, fixtures, runtime archives, logs, and wheels. Include Apache license text in the Slurm wheel and redact sensitive scheduler diagnostics. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix Slurm security audit gaps Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * harden adjacent secret redaction Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * make Slurm redaction fail closed Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * redact underscore secret options Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * harden Slurm security audit boundaries Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix control-obscured Slurm secrets Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix control-split secret redaction Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix whitespace-control secret suffixes Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * harden archive content classification Detect explicitly supplied ZIP and tar inputs from their content when filenames use opaque suffixes. Reject unsupported tar special members while continuing to allow ordinary directories. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * close Slurm security boundary gaps Handle serialized authorization headers and Unicode whitespace without exposing credential suffixes. Keep public-artifact scanning fail-closed for non-regular files and unreadable ZIP members, and narrow test sentinels to complete values. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * redact complete Authorization diagnostics Treat an Authorization field as credential-bearing through its line boundary even when its value is quoted. This keeps control-obscured suffixes from surviving the protected redaction pass. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): compose distributed runtime launch Layer multi-node topology and follower-failure handling onto the production allocation bootstrap introduced by #929. Preserve the one-node path while adding host-scoped steps, coordinated node workers, remote readiness probes, and distributed preflight coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * map distributed model paths Apply the resolved container-mount mapping to absolute model paths in both coordinated multi-node workers and remote-node serving commands. Cover differing host and container roots in the bootstrap manifest regression. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * adapt retry bootstrap coverage Pass the one-node allocation layout required by the distributed runtime manifest after composing the merged retry coverage onto the new shared base. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix distributed launch preflight Select the multi-process vLLM executor whenever pipeline or tensor parallelism spans more than one worker. Carry mapped absolute model paths through the node-worker contract so every assigned node verifies accessibility before launch. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): choose Parquet partitions at merge Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve collection intent on retry Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Avoid writing a duplicate Parquet export when the engine batches can be used directly as a bounded candidate. Keep the export path for schema drift or too many batch files. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Give each node-local vLLM replica its own tokenizer cache so concurrent startup cannot read or remove another replica’s partially downloaded vocabulary. Preserve an explicitly configured cache and clean generated caches after the replicas stop. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): merge runs by ID with default output Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): name default merge directory Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): align merge destination coverage Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): read only committed attempt results Ignore client-owned partial result files when an attempt has no bound candidate. Failed shards can then be reconciled and reported without treating their incomplete output as a finalized result. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * accept nonempty partial Slurm shards Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): analyze nonempty partial benchmarks Use actual winning rows for benchmark throughput and capacity estimates. Update the benchmark result contract and Slurm docs for partial winners. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): merge reordered nested shard schemas Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve saved collection plans Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): benchmark one array task per case Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): analyze saved benchmark manifests Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): validate benchmark versions lazily Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: clarify saved benchmark rates Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat(slurm): default benchmark records adaptively Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: fix Slurm example references Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: clarify Slurm topologies and shards Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Add opt-in shared-node GPU quotas and explicit GPU offsets for colocating distinct models, including multiple models on one GPU. Coordinate readiness before overlapping launches and support typed vLLM IO processor plugins. Signed-off-by: Andre Manoel <amanoel@nvidia.com>

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* fix(slurm): harden run lifecycle and diagnostics Validate built-in builder settings before allocation while deferring plugin and compute-local inputs. Protect image artifacts referenced by run plans and verify reused client overlays. Correct readiness, progress, benchmark, and multi-node behaviors found during branch review. Add regression coverage and update Slurm docs. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): address review findings Poll readiness across all started servers so a slow deployment cannot consume another deployment timeout. Protect images referenced by custom run IDs and retry transient batch metadata failures in progress reporting. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): narrow artifact locks and validate plans Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* refactor(slurm): simplify preflight and readiness tests Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): reuse readiness command helper Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): simplify run preparation ownership Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): tag submitted jobs for telemetry Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): honor user deployment type during submission Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat(telemetry): recognize internal Slurm deployments Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve deployment tags across job operations Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): align fallback tag across initial and retry jobs Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

#1080) * feat(slurm): support external model providers Allow hosted endpoints alongside managed models and run hosted-only recipes in CPU allocations. Resolve credentials at runtime and preserve native Data Designer model settings and asynchronous execution. Reuse shard, retry, resume, and collection contracts while preserving existing local-only plan serialization. Add configuration examples, documentation, and native client and allocation lifecycle coverage. Closes #1079 Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> * docs(slurm): describe hosted-only prerequisites Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> * fix(slurm): finalize preclaimed datasets and configure node memory Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> * fix(slurm): restrict generated datasets before winner publication Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> * refactor(slurm): reuse native model-provider settings Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> * fix(export): normalize batch column order before Parquet schema casts Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> * fix(slurm): validate hosted credentials before capturing provider settings Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> * docs(slurm): align hosted provider plan Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): keep provider payloads opaque Exclude native hosted-provider payloads from recursive Slurm secret\nreference discovery while retaining explicit API-key references.\n\nCover mixed and hosted-only plans with SecretRef-shaped request data,\nincluding missing environment fields and nested lists. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): defer retry credential validation Resolve credentials only when a new retry submission is required, under the coordinator locks and before prepared state is published. Preserve credential-free recovery of active and accepted retries, and cover missing keys, dry runs, and restored credential transport. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): reject model-free runs before submission Require a nonempty model alias set in resolved builder inputs and reject model-free inline or sourced recipes before staging or submission. Document the limitation and cover omitted, null, and empty model configs while retaining hosted-only CPU run coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs(slurm): clarify hosted provider settings Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Dhruv Nathawani <128275431+dhruvnathawani@users.noreply.github.com> Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> Co-authored-by: Nabin Mulepati <nmulepati@nvidia.com>

Apply the Python 3.11 minimum and current lint rules to Slurm. Update artifact-error tests to exercise the schema-aware readers retained during the rebase. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Select service ports excluded from automatic TCP source allocation and wait for proxy listener ownership before endpoint readiness probes. Preserve legacy sysctl compatibility and explicit occupied-port failures. Fixes #1097

* feat(slurm): support cross-node TP and EP Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): cover local PP with TP1 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): monitor health and drain endpoints Persist ongoing backend health evidence and fail managed attempts on service loss. Drain logical endpoints before stopping serving steps, and complete routing, timing, eviction, and timeout telemetry. Add health, shutdown, streaming, and real-client overload admission regressions. Live cluster acceptance remains tracked in #1103 and #870. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): add live vLLM overload acceptance Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): record live overload acceptance Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): reject success after health failure Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): signal endpoint tasks before draining Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): default pipeline parallelism to one Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): verify benchmark PP defaults Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): use a compatible benchmark base run Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

nabinchha force-pushed the feat/slurm-execution branch from f404976 to a498342 Compare October 15, 2026 21:09

nabinchha changed the title ~~feat(slurm): integrate Slurm execution for 1.0.0rc1~~ feat(slurm): integrate Slurm execution for 1.0.0 release Oct 16, 2026

github-actions Bot mentioned this pull request Oct 19, 2026

Agentic CI: Issue & PR Triage Tracker #562

Open

nabinchha force-pushed the feat/slurm-execution branch from 7bc52f4 to 9ba05f5 Compare October 22, 2026 20:47

dhruvnathawani mentioned this pull request Oct 23, 2026

feat(slurm): support hosted models through native DD provider settings #1080

Merged

12 tasks done

andreatnvidia and others added 24 commits October 27, 2026 11:55

feat: add Slurm image lifecycle jobs (#896)

d239c7d

feat: add one-node Slurm runtime (#909)

59a09a9

feat: add allocation-local Slurm client worker (#911)

9c79e27

* feat(slurm): add allocation-local client worker * fix(slurm): address client worker review feedback * fix(slurm): harden client worker boundaries * fix(slurm): harden client filesystem boundary

nabinchha and others added 19 commits October 27, 2026 11:56

fix(slurm): reuse final batch files for candidates (#1023)

bbd7a58

Avoid writing a duplicate Parquet export when the engine batches can be used directly as a bounded candidate. Keep the export path for schema drift or too many batch files. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

fix: recognize locked dependency licenses

8262492

fix(slurm): deduplicate runtime inventory paths

d5094b2

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

fix(slurm): sync checked examples with documentation

f568775

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

fix(slurm): speed up allocation artifact checks (#1071)

3127dfe

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

fix(slurm): remove vLLM release allowlist (#1086)

b75b9c7

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

fix(slurm): align rebased branch with main

672a4c1

Apply the Python 3.11 minimum and current lint rules to Slurm. Update artifact-error tests to exercise the schema-aware readers retained during the rebase. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

test(slurm): align JSON storage result fixture

bb54b50

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

nabinchha force-pushed the feat/slurm-execution branch from f109e32 to bb54b50 Compare October 27, 2026 18:03

This was referenced Oct 28, 2026

Support external model providers in Slurm runs #1079

Closed

Slurm: support Qwen3.8-capable vLLM runtimes #1085

Closed

Run Slurm generation without a client image using a versioned vLLM serving image #969

Closed

fix(slurm): avoid ephemeral port collisions (#1098)

28442e5

Select service ports excluded from automatic TCP source allocation and wait for proxy listener ownership before endpoint readiness probes. Preserve legacy sysctl compatibility and explicit occupied-port failures. Fixes #1097

andreatnvidia mentioned this pull request Oct 28, 2026

test(slurm): stabilize CLI truncation fixture #1101

Merged

2 tasks done

andreatnvidia and others added 5 commits October 28, 2026 13:49

test(slurm): stabilize CLI truncation fixture (#1101)

0a5b0e7

Merge branch 'main' into feat/slurm-execution

12a641f

Navigation Menu

Repository navigation

feat(slurm): integrate Slurm execution for 1.0.0 release - #1040

feat(slurm): integrate Slurm execution for 1.0.0 release#1040
nabinchha wants to merge 71 commits into
mainfrom
feat/slurm-execution

nabinchha commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nabinchha commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Summary

🔗 Related Issue

🔄 Changes

✨ Added

🔧 Changed

🐛 Fixed

🔍 Attention Areas

🧪 Testing

✅ Checklist

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nabinchha commented Oct 15, 2026 •

edited

Loading