Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
707c4fd
feat: add optional Slurm package (#856)
andreatnvidia Aug 18, 2026
57a779a
feat: define public generation invocation contract (#854)
andreatnvidia Aug 18, 2026
b0ffab1
feat: add lazy CLI extensions (#863)
andreatnvidia Aug 18, 2026
553187a
feat: define Slurm runtime and state record contracts (#878)
nabinchha Aug 20, 2026
fbff83f
feat: define Slurm config and plan contracts (#879)
andreatnvidia Aug 24, 2026
021cc2d
feat: integrate Slurm plan and runtime state (#889)
nabinchha Aug 24, 2026
92dc167
test: add deterministic Slurm fakes (#888)
andreatnvidia Aug 25, 2026
049ca8e
feat: add Slurm image registry foundation (#891)
nabinchha Aug 27, 2026
acd270f
feat: add Slurm command client and renderer (#892)
nabinchha Aug 27, 2026
c00aeb2
feat: resolve Slurm serving deployments (#890)
nabinchha Aug 28, 2026
f8a83c3
feat: compile deterministic Slurm run plans (#893)
andreatnvidia Aug 31, 2026
500f531
feat: add Slurm service foundation (#894)
andreatnvidia Aug 31, 2026
d239c7d
feat: add Slurm image lifecycle jobs (#896)
nabinchha Sep 2, 2026
baae0a9
fix: harden Slurm image publication (#907)
nabinchha Sep 2, 2026
ac72753
feat: add Slurm state writer (#908)
nabinchha Sep 3, 2026
59a09a9
feat: add one-node Slurm runtime (#909)
nabinchha Sep 3, 2026
9c79e27
feat: add allocation-local Slurm client worker (#911)
andreatnvidia Sep 3, 2026
fc5e96c
feat: finalize Slurm shard winners (#910)
nabinchha Sep 3, 2026
aa63100
feat(slurm): wire M2 services and CLI (#926)
andreatnvidia Sep 8, 2026
5e5b0dd
fix(slurm): close one-node M2 integration (#929)
andreatnvidia Sep 10, 2026
edbcbab
feat: reconcile persisted Slurm state (#913)
nabinchha Sep 10, 2026
306916e
fix(slurm): restore plugin activation boundary (#930)
nabinchha Sep 10, 2026
588e6da
fix: harden Slurm release artifacts (#916)
nabinchha Sep 10, 2026
d9a6433
feat(slurm): harden distributed allocation runtime (#914)
nabinchha Sep 10, 2026
fe86eec
feat(slurm): add profile setup and image lifecycle wiring (#933)
andreatnvidia Sep 11, 2026
a2f2c0d
feat(slurm): wire retry and merge commands (#931)
andreatnvidia Sep 11, 2026
d2bfc2e
feat(slurm): add deterministic benchmark workflows (#932)
andreatnvidia Sep 11, 2026
d1920a1
fix(slurm): use allocation-local scratch for runtime (#938)
andreatnvidia Sep 15, 2026
2859588
docs(slurm): add public workflow and release checks (#939)
andreatnvidia Sep 22, 2026
5523a29
fix(slurm): run collection with native pinned runtime (#965)
nabinchha Sep 28, 2026
2c5a970
feat(slurm): harden inference proxy
nabinchha Sep 29, 2026
bc86924
feat(slurm): run client natively with serving image (#970)
nabinchha Oct 5, 2026
4adb41f
test(slurm): repair runtime test fixtures (#980)
nabinchha Oct 5, 2026
6c6cbe6
fix(slurm): show CLI submission progress (#979)
nabinchha Oct 6, 2026
08b0dac
fix(slurm): support Enroot 3.5 OCI imports (#988)
nabinchha Oct 6, 2026
9d51b03
feat(slurm): improve CLI output and image logs (#992)
nabinchha Oct 7, 2026
966f304
fix(slurm): stabilize job and GPU replica startup (#993)
nabinchha Oct 7, 2026
2e6c125
feat(slurm): support multiple GPU partitions (#998)
nabinchha Oct 7, 2026
0e43897
fix(slurm): create profile parent directories (#999)
nabinchha Oct 7, 2026
1aa46f4
fix(slurm): expose vLLM logs and honor JSON output (#1000)
nabinchha Oct 8, 2026
e0b6320
fix(slurm): address retry, startup, and proxy failure paths (#1005)
andreatnvidia Oct 12, 2026
9074b1d
refactor(slurm): remove unused config flags and set client CPU defaul…
nabinchha Oct 12, 2026
33d52db
fix(slurm): leave host patterns empty by default (#1014)
nabinchha Oct 12, 2026
71c48cb
feat(slurm): improve image add progress and removal (#1016)
nabinchha Oct 12, 2026
d4685f3
fix(slurm): simplify image details output (#1018)
nabinchha Oct 13, 2026
8fda05d
fix(slurm): improve preparation and allocation diagnostics (#1019)
nabinchha Oct 13, 2026
7ef0b6d
fix(slurm): choose Parquet partitions at merge (#1021)
nabinchha Oct 14, 2026
bbd7a58
fix(slurm): reuse final batch files for candidates (#1023)
nabinchha Oct 14, 2026
7b28145
fix(slurm): isolate Harmony tokenizer caches (#1024)
nabinchha Oct 14, 2026
d59f497
feat(slurm): merge runs by ID with default output (#1025)
nabinchha Oct 14, 2026
37c1642
feat(slurm): accept nonempty partial shard outputs (#1026)
nabinchha Oct 14, 2026
730e483
fix(slurm): merge reordered nested shard schemas (#1028)
nabinchha Oct 14, 2026
f8499d5
fix(slurm): benchmark one array task per case (#1037)
nabinchha Oct 15, 2026
cf1217a
feat(slurm): colocate distinct models with GPU quotas (#1034)
andreatnvidia Oct 15, 2026
8262492
fix: recognize locked dependency licenses
nabinchha Oct 15, 2026
d5094b2
fix(slurm): deduplicate runtime inventory paths
andreatnvidia Oct 16, 2026
f568775
fix(slurm): sync checked examples with documentation
andreatnvidia Oct 16, 2026
f892066
fix(slurm): harden run lifecycle and diagnostics (#1044)
nabinchha Oct 16, 2026
d09b675
refactor(slurm): simplify preflight and run preparation (#1047)
nabinchha Oct 16, 2026
9e355b5
fix(slurm): tag submitted jobs for telemetry (#1060)
nabinchha Oct 21, 2026
3127dfe
fix(slurm): speed up allocation artifact checks (#1071)
andreatnvidia Oct 22, 2026
b75b9c7
fix(slurm): remove vLLM release allowlist (#1086)
nabinchha Oct 24, 2026
e6d3277
feat(slurm): support hosted models through native DD provider setting…
dhruvnathawani Oct 27, 2026
672a4c1
fix(slurm): align rebased branch with main
nabinchha Oct 27, 2026
bb54b50
test(slurm): align JSON storage result fixture
nabinchha Oct 27, 2026
28442e5
fix(slurm): avoid ephemeral port collisions (#1098)
andreatnvidia Oct 28, 2026
0a5b0e7
test(slurm): stabilize CLI truncation fixture (#1101)
andreatnvidia Oct 28, 2026
0ffd8cd
feat(slurm): support cross-node tensor and expert parallelism (#1112)
nabinchha Oct 29, 2026
14df42e
fix(slurm): complete proxy lifecycle and overload validation (#1104)
nabinchha Oct 30, 2026
12a641f
Merge branch 'main' into feat/slurm-execution
andreatnvidia Oct 30, 2026
678a382
fix(slurm): default pipeline parallelism to one (#1116)
nabinchha Oct 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
11 changes: 6 additions & 5 deletions .agents/recipes/dependencies/recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ permissions:

# Dependency Audit

Audit the dependency graph across all three packages. Write findings to
Audit the dependency graph across all four packages. Write findings to
`/tmp/audit-{{suite}}.md`.

Dependabot handles version bump PRs. This recipe focuses on what Dependabot
Expand Down Expand Up @@ -39,11 +39,12 @@ Read the `pyproject.toml` for each package:
cat packages/data-designer-config/pyproject.toml
cat packages/data-designer-engine/pyproject.toml
cat packages/data-designer/pyproject.toml
cat packages/data-designer-slurm/pyproject.toml
```

Note: engine and interface packages use `uv-dynamic-versioning` to inject
dependencies. Check both static declarations and the dynamic versioning
config.
Note: engine, interface, and Slurm packages use `uv-dynamic-versioning` to
inject dependencies. Check both static declarations and the dynamic
versioning config, including selected optional dependencies.

### 2. Direct dependency declaration gaps

Expand Down Expand Up @@ -79,7 +80,7 @@ intentionally deferred but still need to be declared as dependencies.

Check that shared dependencies use consistent version constraints:
```bash
# Extract dependency specs from all three pyproject.toml files
# Extract dependency specs from all four pyproject.toml files
grep -E "^\s+\"[a-zA-Z]" packages/*/pyproject.toml
```

Expand Down
13 changes: 8 additions & 5 deletions .agents/recipes/structure/recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,21 +19,24 @@ to `/tmp/audit-{{suite}}.md`.
Before starting, read the authoritative sources for the structural rules:

1. **`AGENTS.md`** (repo root) - "The Layering Is Structural" section defines
the three packages, their ownership, and the dependency direction rule.
the four packages, their ownership, and the dependency direction rule.
2. **`architecture/overview.md`** - system architecture diagram, package layout,
and the "no reverse imports" rule.

The canonical rules:

```
data-designer (interface) -> data-designer-engine -> data-designer-config
data-designer-slurm -> data-designer (interface) -> data-designer-engine -> data-designer-config
```

- `packages/data-designer-config/` must NOT import from `data_designer.engine`,
`data_designer.interface`, or `data_designer.cli`
`data_designer.interface`, `data_designer.cli`, or `data_designer.slurm`
- `packages/data-designer-engine/` must NOT import from
`data_designer.interface` or `data_designer.cli`
- `packages/data-designer/` CAN import from both engine and config
`data_designer.interface`, `data_designer.cli`, or `data_designer.slurm`
- `packages/data-designer/` CAN import from engine and config but must NOT
import from `data_designer.slurm`
- `packages/data-designer-slurm/` uses public interface/config APIs and must
NOT import engine internals

**What CI already enforces**: ruff rule `TID` catches relative imports. The
CI test matrix runs config, engine, and interface tests in isolation (separate
Expand Down
2 changes: 1 addition & 1 deletion .agents/recipes/test-health/recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ Write the report to `/tmp/audit-{{suite}}.md`:

| Check | Status | Detail |
|-------|--------|--------|
| Package imports | OK/FAIL | All three packages import cleanly |
| Package imports | OK/FAIL | All four packages import cleanly |
| Import timing | OK/FAIL | X.XXs (budget: 3s) |
| Registry completeness | OK/WARN | Column types resolve to config classes |

Expand Down
1 change: 1 addition & 0 deletions .github/workflows/agentic-ci-daily.yml
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ jobs:
if: matrix.suite == 'dependencies'
run: |
.venv/bin/python scripts/audit_package_dependencies.py \
--extra data-designer:slurm \
--output /tmp/dependency-inventory.json
jq '.packages[] | {package, missing, unresolved_modules}' \
/tmp/dependency-inventory.json
Expand Down
39 changes: 37 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,39 @@ jobs:
uv run --with pytest --with pytest-asyncio --with pytest-httpx --with pytest-env \
pytest packages/data-designer/tests

test-slurm-package:
name: ${{ matrix.name }}
needs: validate-dispatch
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- python-version: "3.11"
test-path: packages/data-designer-slurm/tests
name: Test Slurm package wheels

steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Install uv
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
version: "latest"
python-version: ${{ matrix.python-version }}
enable-cache: true

- name: Install development dependencies
run: make install-dev

- name: Run package tests
run: .venv/bin/pytest ${{ matrix.test-path }}

- name: Run built-wheel installation tests
if: matrix.python-version == '3.11'
run: make test-slurm-wheel-install

# ===========================================================================
# Combined Coverage Check
# Runs all tests together to verify overall coverage threshold
Expand Down Expand Up @@ -295,7 +328,7 @@ jobs:
test-summary:
name: Test (Python ${{ matrix.python-version }} on ${{ matrix.os }})
runs-on: ubuntu-latest
needs: [validate-dispatch, test-config, test-engine, test-interface]
needs: [validate-dispatch, test-config, test-engine, test-interface, test-slurm-package]
if: always()
strategy:
matrix:
Expand All @@ -308,12 +341,14 @@ jobs:
if [[ "${{ needs.validate-dispatch.result }}" != "success" ]] || \
[[ "${{ needs.test-config.result }}" != "success" ]] || \
[[ "${{ needs.test-engine.result }}" != "success" ]] || \
[[ "${{ needs.test-interface.result }}" != "success" ]]; then
[[ "${{ needs.test-interface.result }}" != "success" ]] || \
[[ "${{ needs.test-slurm-package.result }}" != "success" ]]; then
echo "One or more test jobs failed"
echo "validate-dispatch: ${{ needs.validate-dispatch.result }}"
echo "test-config: ${{ needs.test-config.result }}"
echo "test-engine: ${{ needs.test-engine.result }}"
echo "test-interface: ${{ needs.test-interface.result }}"
echo "test-slurm-package: ${{ needs.test-slurm-package.result }}"
exit 1
fi
echo "All test jobs passed successfully"
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,16 @@ If you are an agent helping a user **build a dataset**, use the [`data-designer`

## The Layering Is Structural

The `data_designer` namespace is split across three installable packages that merge at runtime via PEP 420 implicit namespace packages (no top-level `__init__.py`).
The `data_designer` namespace is split across four installable packages that merge at runtime via PEP 420 implicit namespace packages (no top-level `__init__.py`).

| Package | Path | Owns |
|---------|------|------|
| `data-designer-config` | `packages/data-designer-config/` | `data_designer.config` β€” column configs, model configs, sampler params, builder API, plugin system, lazy imports |
| `data-designer-engine` | `packages/data-designer-engine/` | `data_designer.engine` β€” column generators, dataset builders, DAG execution, model facade, validators, sampling |
| `data-designer` | `packages/data-designer/` | `data_designer.interface` β€” public `DataDesigner` class, results, errors; `data_designer.cli` β€” CLI entry point; `data_designer.integrations` |
| `data-designer-slurm` | `packages/data-designer-slurm/` | `data_designer.slurm` β€” optional Slurm execution |

**Dependency direction (left depends on right):** interface β†’ engine β†’ config. Never import against this flow.
**Import direction (left imports right):** Slurm β†’ interface β†’ engine β†’ config. The `data-designer[slurm]` extra creates a packaging-only reverse edge; no code may import against this flow.

## Core Concepts

Expand All @@ -34,7 +35,7 @@ The `data_designer` namespace is split across three installable packages that me

## Structural Invariants

- **Import direction** β€” interface β†’ engine β†’ config (left depends on right). No reverse imports.
- **Import direction** β€” Slurm β†’ interface β†’ engine β†’ config (left imports right). No reverse imports.
- **Fast imports** β€” heavy third-party libraries are lazy-loaded via `data_designer.lazy_heavy_imports`. See [STYLEGUIDE.md](STYLEGUIDE.md) for the pattern.
- **No relative imports** β€” absolute imports only, enforced by ruff rule `TID`.
- **Typed code** β€” all functions, methods, and class attributes require type annotations. Modern syntax: `list[str]`, `str | None`.
Expand Down
5 changes: 3 additions & 2 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,16 +109,17 @@ uv run ruff format --check # Check formatting

### Running Tests

`make test` runs all three package test suites in sequence (config, engine, interface). When iterating on a single package, run its tests directly:
`make test` runs all four package test suites in sequence (config, engine, interface, Slurm). When iterating on a single package, run its tests directly:

```bash
# Run all tests (config + engine + interface)
# Run all tests (config + engine + interface + Slurm)
make test

# Run a single package's tests
make test-config # data-designer-config
make test-engine # data-designer-engine
make test-interface # data-designer (interface)
make test-slurm # data-designer-slurm

# Run a specific test file
uv run pytest tests/config/test_sampler_constraints.py
Expand Down
Loading
Loading