[dhruvnathawani]

📋 Summary

Slurm recipes can now use hosted models configured through DD's existing model_providers.yaml, alongside managed vLLM deployments or in hosted-only CPU allocations. The Slurm run file describes only the models deployed locally; there is no parallel provider schema or invocation.model_providers setting.

Targets feat/slurm-execution. Live qualification combines head 774774487da6 with the independent vLLM version-gate fix in #1086 (3442c185abac). This PR remains draft for review.

🔗 Related Issue

Closes #1079. Implements the native-provider ownership and snapshot flow proposed in Nabin's comment.

🔄 Changes

• Resolve undeployed aliases through native DD settings and capture only the required providers as immutable native JSON in the run inputs. Validate missing/ambiguous providers, generated-name collisions, and API-key environment availability before capture/submission.
• Retain environment-variable references in the snapshot and forward values only to client steps, including retries. Initially support environment-backed API keys and non-secret static headers; reject plaintext keys and sensitive headers. Compute nodes use the captured settings rather than the submit host's home directory.
• Override endpoints only for locally deployed aliases. Preserve hosted model identities, inference settings, and the existing async execution, sharding, resume, retry, and collection behavior.
• Allow zero deployments: use a CPU allocation with --gres=none, without GPU discovery, serving images, containers, or proxies. Add per-node submission.memory for generation/collection.
• Fix native dataset handoff: let DD create its output directory exclusively, then restrict it before winner publication; preserve existing nonempty/linked paths.
• Fix Parquet export when async batches use different column orders: select the unified order before schema casting. The live baseline exposed this after successful generation/judging; a regression covers reordered columns, nested structs, and empty lists without changing row values.
• Update public examples and Fern documentation for normal DD provider configuration.

Async execution does not guarantee continuous GPU utilization. Local GPUs remain allocated during hosted requests; separate stages remain appropriate when hosted processing is substantial.

🧪 Testing

• Exact combined source with fix(slurm): remove vLLM release allowlist #1086: 2,035 Slurm tests passed.
• Native results/export suite: 49 passed; the new regression reproduced the live failure before the fix.
• make check-all, changed-file Ruff checks, and git diff --check passed.
• Fern: internal links across 101 pages and full validation passed (0 errors, one authentication-dependent warning).
• Native DD HTTP and service tests cover hosted/mixed routing, settings snapshots, unavailable/plaintext credentials, client-only secret scope, and candidate verification.
• Frozen baseline on the existing Qwen image generated 4/4 authored SQL fixture records, with 4/4 hosted judgments pass, then failed during Parquet consolidation. CPU replay with the export fix preserved all four saved records exactly; the original failed attempt remains preserved.
• Fresh native-settings hosted-only run: 2/2 records generated, judged, published, and collected. Generation and collection both completed with exit code 0 and zero GPUs.
• Fresh native-settings mixed run on the preserved Qwen image/checkpoint and TP4 configuration: 4/4 records generated, all four hosted judgments pass, and all four records published and collected. Generation completed with exit code 0; collection also completed with exit code 0 on a CPU allocation with zero GPUs.
• Paired fixture prompts, seed rows, judge code, inference settings, and serving parameters are unchanged after normalizing provider ownership, library version, and fixture paths. Persisted text-artifact scans found no hosted credential values in either final run.

The small fixture tests provider routing and publication. It does not qualify Gym SQL execution, production SQL data quality, sustained GPU utilization, or a full dataset/SFT rerun.

✅ Checklist

• Conventional commits and signed-off DCO.
• Internal contribution; external-contributor triage does not apply.
• Public documentation updated; package layering preserved.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[github-actions]

Fern preview: https://nvidia-preview-pr-1080.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

[greptile-apps]

Confidence Score: 5/5

[High impact] The changes since the previous review appear safe to merge.

Summary

This PR lets Slurm recipes combine local deployments with hosted models from DD’s normal provider settings.

• Slurm recipes can use hosted models alongside locally served models.
• Hosted-only recipes can run on CPU nodes without local deployments.
• Slurm restricts a fresh dataset directory before it publishes the winner.
• Parquet export aligns each batch with the unified column order.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Settings["Native provider settings"] --> Snapshot["Captured hosted providers"]
  Recipe["Recipe model aliases"] --> Plan["Slurm run plan"]
  Snapshot --> Plan
  Plan --> Client["Native DD client"]
  Plan --> Local["Optional local servers"]
  Client --> Hosted["Hosted endpoints"]
  Client --> Local
  Client --> Results["Saved dataset and collection"]

Loading

_{Reviews (4) · Last reviewed commit: "docs(slurm): clarify hosted provider set..." · Reviewed by Greptile}

[svcnvidia-nemo-ci]

Reviewed all 43 changed files and their directly affected provider, allocation, client, publication, and export paths. One retry-idempotency regression remains: recovering an already accepted retry now unnecessarily requires submission credentials. The inline finding includes a targeted correction and regression checks.

Static review only; tests were inspected but not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[svcnvidia-nemo-ci]

[P2] Defer credential validation until a new retry submission is needed

Failure: Submit a hosted-model retry with MODEL_API_KEY available, then repeat the same selection/resume mode from a fresh service with source_environment={} while that retry is pending or running. This unconditional call raises required secret environment 'MODEL_API_KEY' is unavailable before recovery is attempted. Expected: return the accepted job/attempt IDs without another submission or credentials already exported to that job. The coordinator explicitly returns active attempts before submitting, and the existing service regression asserts repeated requests return the same result. This also affects existing server/MCP/dependency secret references.

Correction: Resolve credentials only after the coordinator determines under its locks that a new submission is necessary, but before persisting PREPARED state. Merely skipping validation when preview_active() succeeds is unsafe: the active retry can finish before the coordinator refreshes state.

A targeted implementation is an optional lazy environment factory on SlurmRetryCoordinator, retaining existing explicit-mapping callers:

# Add Callable to collections.abc imports and a keyword-only constructor argument:
export_environment_factory: Callable[[], Mapping[str, str]] | None = None
# Store it in __init__:
self._export_environment_factory = export_environment_factory

# In retry(), inside the selection locks:
self._require_fresh_selection(status, selected)
if self._export_environment_factory is not None:
    self._export_environment = dict(self._export_environment_factory())
self._persist_prepared_retry(plan, timestamp)

Here, remove eager build_run_export_environment() evaluation and pass the factory instead:

coordinator = SlurmRetryCoordinator(
    self._workspace_root,
    run_id,
    self._launcher,
    self._export_environment,
    export_environment_factory=lambda: build_run_export_environment(
        resolved_plan, self._source_environment, operation
    ),
)

The invariant is that active/pending-submission recovery and dry runs never evaluate the factory; every genuinely new submission does so before publishing PREPARED state. The inspected coordinator returns active attempts before this insertion point and uses the resolved mapping in _submit(). Suggested code is unexecuted and requires author verification.

Preservation / acceptance — proposed checks: Extend the service idempotency setup with a hosted credential: accept retry job 43 using the key, recreate the service without it, and repeat the identical request while job 43 is active. Assert the same result and no third submission. After job 43 fails, requesting a genuinely new retry without the key must fail before creating a new retry record or submitting; restoring the key must export its value at runtime without persisting it. Retain the existing sparse-selection, pinned resume-mode, local-only idempotency, and dry-run-without-persistence assertions.

[nabinchha]

Fixed in 9b4ce45d.

Credential resolution now runs through a lazy environment factory inside the coordinator's selection locks, after freshness validation and before PREPARED state is persisted. Dry runs and recovery of active or accepted submissions bypass the factory; existing explicit-environment callers remain supported.

Regression coverage verifies recovery of both pending and running hosted retries from a fresh service without credentials, no additional submission, rejection of genuinely new retries without creating retry records when the key is missing, and successful runtime credential export after restoring the key without persisting its value. The lost-receipt recovery test also verifies the factory is never called during recovery.

Validation: all 2,040 Slurm tests passed, along with Ruff, formatting, diff checks, and commit hooks.

[nabinchha]

Reviewed the final head, including independent checks of provider configuration, credential transport, hosted-only and mixed runtime behavior, retry recovery, dataset publication, export, and documentation. The identified issues are fixed; no outstanding actionable findings remain.

Validation includes 2,052 passing Slurm tests on the current production code, changed-file lint/format checks, and Fern validation/internal links. Live cluster qualification was not repeated in this review.

---

This review was generated by an AI assistant.