diff --git a/assets/images/slurm/oneslurm/dark/oneslurm_first_job.png b/assets/images/slurm/oneslurm/dark/oneslurm_first_job.png new file mode 100644 index 000000000..d40e49332 Binary files /dev/null and b/assets/images/slurm/oneslurm/dark/oneslurm_first_job.png differ diff --git a/assets/images/slurm/oneslurm/dark/oneslurm_service_roles.png b/assets/images/slurm/oneslurm/dark/oneslurm_service_roles.png new file mode 100644 index 000000000..e56bead0b Binary files /dev/null and b/assets/images/slurm/oneslurm/dark/oneslurm_service_roles.png differ diff --git a/assets/images/slurm/oneslurm/light/oneslurm_first_job.png b/assets/images/slurm/oneslurm/light/oneslurm_first_job.png new file mode 100644 index 000000000..014ee6b7f Binary files /dev/null and b/assets/images/slurm/oneslurm/light/oneslurm_first_job.png differ diff --git a/assets/images/slurm/oneslurm/light/oneslurm_service_roles.png b/assets/images/slurm/oneslurm/light/oneslurm_service_roles.png new file mode 100644 index 000000000..b709ae33a Binary files /dev/null and b/assets/images/slurm/oneslurm/light/oneslurm_service_roles.png differ diff --git a/content/platform_services/application_appliances/overview.md b/content/platform_services/application_appliances/overview.md index 9f6ee64b8..4fef458dc 100644 --- a/content/platform_services/application_appliances/overview.md +++ b/content/platform_services/application_appliances/overview.md @@ -12,6 +12,6 @@ You can browse available Marketplace Appliances in the dedicated [OpenNebula Mar * [**Harbor Container Registry**](https://github.com/OpenNebula/one-apps/wiki/harbor_intro): Deploy a secure, self-managed image repository. * [**MinIO**](https://github.com/OpenNebula/one-apps/wiki/minio_intro): Object storage solution with an AWS S3-compatible API. * [**vLLM AI**](https://github.com/OpenNebula/one-apps/wiki/vllm_intro): Deploy high-performance LLM inferencing. -* [**Slurm**](https://github.com/OpenNebula/one-apps/wiki/slurm_intro): Fault-tolerant, highly scalable Cluster management and job scheduling system for HPC. +* [**Slurm**]({{% relref "platform_services/slurm/getting_started/overview" %}}): Fault-tolerant, highly scalable Cluster management and job scheduling system for HPC. * [**NVIDIA NIM Service**](https://github.com/OpenNebula/one-apps/wiki/nim_intro): GPU-accelerated inference microservices for pre-trained and customized AI models. * [**Rancher CAPI**](https://github.com/OpenNebula/one-apps/wiki/capi_intro): Out-of-the-box solution for managing Kubernetes Clusters using Rancher and OpenNebula. \ No newline at end of file diff --git a/content/platform_services/slurm/_index.md b/content/platform_services/slurm/_index.md index 28cf8fe58..4fb9684b5 100644 --- a/content/platform_services/slurm/_index.md +++ b/content/platform_services/slurm/_index.md @@ -2,12 +2,10 @@ title: "Managed Slurm" linkTitle: "Managed Slurm" date: "2026-05-12" -description: "" +description: "OneSlurm deploys Slurm Clusters on OpenNebula as a OneFlow service, with a controller and elastic workers." categories: pageintoc: "13" tags: type: docs weight: "2" --- - -Slurm is an open source, fault-tolerant, and highly scalable Cluster management and job scheduling system for large and small Linux Clusters. \ No newline at end of file diff --git a/content/platform_services/slurm/getting_started/_index.md b/content/platform_services/slurm/getting_started/_index.md new file mode 100644 index 000000000..1caca1bf6 --- /dev/null +++ b/content/platform_services/slurm/getting_started/_index.md @@ -0,0 +1,11 @@ +--- +title: "Getting Started with OneSlurm" +linkTitle: "Getting Started" +date: "2026-10-07" +description: "Learn what OneSlurm deploys, prepare your cloud and create your first Slurm Cluster." +categories: +pageintoc: "13" +tags: +type: docs +weight: "1" +--- diff --git a/content/platform_services/slurm/getting_started/core_concepts.md b/content/platform_services/slurm/getting_started/core_concepts.md new file mode 100644 index 000000000..b993badc0 --- /dev/null +++ b/content/platform_services/slurm/getting_started/core_concepts.md @@ -0,0 +1,64 @@ +--- +title: "Core Concepts" +linkTitle: "Core Concepts" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "4" +--- + +## Controller and Workers + +| | **Controller** | **Worker** | +|---|---|---| +| Slurm daemon | `slurmctld` | `slurmd` | +| Number of VMs | 1 | 1 or more | +| Hostname | `slurm-one-controller` | The VM name, for example `worker-0--service-233` | +| Role in the Cluster | Schedules jobs, keeps the configuration, hosts the optional local LDAP | Runs jobs | + +Users connect to the controller to compile code and submit jobs with `srun` or `sbatch`. + +## Configless Slurm + +The controller is the only VM that has the Slurm configuration files. The workers start `slurmd` with `--conf-server slurm-one-controller:6817` and download the configuration from the controller. A change on the controller reaches every worker, and new workers need no configuration. + +## Dynamic Nodes + +Workers join Slurm as dynamic nodes (`slurmd -Z`). Each worker reports to the controller what it has. + +* Its CPUs, from `nproc` +* Its memory, from `/proc/meminfo` +* Its NVIDIA GPUs, from `nvidia-smi`, as the `gpu` GRES + +The nodes are not listed in `slurm.conf`. + +## OneGate Coordination + +The controller and the workers share their data through OneGate, which keeps data on each VM of the service. + +1. The controller starts, configures Slurm and publishes `READY=YES` and the Munge key. +2. OneFlow waits for `READY=YES` before it creates the workers. This is the `ready_status_gate` option of the service. The VM templates also set `REPORT_READY=YES`, so one-context reports readiness too. +3. Each worker reads the controller IP, the Munge key and the LDAP settings from OneGate. +4. Each worker checks that port 6817 of the controller answers, then starts `slurmd`. +5. Each worker publishes its Slurm node name as `SLURM_NODENAME`. + +Refer to [Service Architecture]({{% relref "platform_services/slurm/references/service_architecture" %}}) for a list of every attribute that the VMs publish. + +## Munge Key + +Slurm uses [Munge](https://dun.github.io/munge/) to authenticate the messages between the daemons. All the VMs of a Cluster need the same key. + +* The controller creates the key at its first boot and keeps it. +* It publishes the key in base64 to OneGate as `SLURM_MUNGE_KEY`. +* Each worker installs the key and checks that Munge works before it starts `slurmd`. + +## Scaling and Cleanup + +When you remove workers, two mechanisms remove them from Slurm. + +* Each worker deletes its own node from Slurm when it shuts down normally. +* The controller checks every 60 seconds, starting 2 minutes after boot, for nodes that have no VM anymore and that Slurm sees as down. It deletes them, or drains them if they still have running jobs. + +Refer to [Cluster Lifecycle Management]({{% relref "platform_services/slurm/management/cluster_lifecycle_management" %}}) for more details. diff --git a/content/platform_services/slurm/getting_started/overview.md b/content/platform_services/slurm/getting_started/overview.md new file mode 100644 index 000000000..ce3a07e06 --- /dev/null +++ b/content/platform_services/slurm/getting_started/overview.md @@ -0,0 +1,68 @@ +--- +title: "Overview" +linkTitle: "Overview" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "1" +aliases: + - /platform_services/slurm/overview/ +--- + +[Slurm](https://slurm.schedmd.com/documentation.html) is an open source workload manager and job scheduler for AI and HPC workloads on Linux Clusters. It gives jobs access to compute nodes, starts and monitors them, and keeps pending jobs in a queue. + +OneSlurm deploys a Slurm Cluster on OpenNebula as a OneFlow service. The service has two roles: + +| **Role** | **Appliance** | **What it does** | +|---|---|---| +| `controller` | [Service Slurm Controller](https://marketplace.opennebula.io/appliance/db17c081-969f-45e0-9c8b-e0f7236c15aa) | Runs `slurmctld`, schedules jobs and owns the Slurm configuration. Users log in here to submit jobs | +| `worker` | [Service Slurm Worker](https://marketplace.opennebula.io/appliance/c03f8c40-39bb-42cd-a9d5-781f59846b57) | Runs `slurmd` and executes jobs. You can add or remove workers at any time | + +Both roles come together in the [Service OneSlurm](https://marketplace.opennebula.io/appliance/8ce164d5-3cce-42a7-b9a7-0e8133ef92c6) appliance of the OpenNebula Public Marketplace. The controller publishes the Cluster data through OneGate, and each worker reads it and joins the Cluster without manual steps. + +Optional features are set when you create the service: + +* Shared user accounts through a local or an external LDAP directory. +* Shared `/home` and `/scratch` from NFS exports. +* NVIDIA GPUs attached to the workers. +* InfiniBand for MPI jobs. + +## How Should I Read This Chapter + +1. Check [Requirements and Preparation]({{% relref "platform_services/slurm/getting_started/requirements" %}}). +2. Deploy a first Cluster with the [Quick Start Guide]({{% relref "platform_services/slurm/getting_started/quick_start" %}}). +3. Read [Core Concepts]({{% relref "platform_services/slurm/getting_started/core_concepts" %}}) to understand how the controller and the workers work together. + +Then use the Management pages for daily operation: + +* [Cluster Lifecycle Management]({{% relref "platform_services/slurm/management/cluster_lifecycle_management" %}}) +* [Cluster Configuration]({{% relref "platform_services/slurm/management/cluster_configuration" %}}) +* [Worker Profiles and Accelerators]({{% relref "platform_services/slurm/management/worker_profiles_and_accelerators" %}}) +* [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}) +* [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}}) +* [InfiniBand and High-Performance Networking]({{% relref "platform_services/slurm/management/infiniband" %}}) +* [Monitoring and Troubleshooting]({{% relref "platform_services/slurm/management/monitoring_and_troubleshooting" %}}) + +The References pages list the [Service Architecture]({{% relref "platform_services/slurm/references/service_architecture" %}}), every [Configuration Parameter]({{% relref "platform_services/slurm/references/configuration_parameters" %}}), the [CLI]({{% relref "platform_services/slurm/references/oneslurm_cli" %}}) commands and the [API]({{% relref "platform_services/slurm/references/oneslurm_api" %}}) calls. + +There is also a [OneSlurm tutorial]({{% relref "solutions/ai_factory_blueprints/direct_ai_execution/nvidia_slurm/" %}}) with NVIDIA GPUs in the AI Factory Blueprints. + +{{< alert title="Note" type="info" >}} +An **OpenNebula Cluster** groups Hosts, datastores and Virtual Networks. A **Slurm Cluster** is a group of VMs that run Slurm. A Slurm Cluster deployed by OneSlurm runs as a workload on top of OpenNebula resources, and you can run several Slurm Clusters in one OpenNebula Cluster. +{{< /alert >}} + +## Release Notes + +The [Service OneSlurm](https://marketplace.opennebula.io/appliance/8ce164d5-3cce-42a7-b9a7-0e8133ef92c6) page of the Marketplace shows the current version, the guest OS and the supported OpenNebula versions. The [one-apps release page](https://github.com/OpenNebula/one-apps/releases) lists the changes of each version. + +## Known Limitations + +| **Limitation** | **Effect** | +|---|---| +| No `slurmdbd` | No job accounting, job history or usage statistics through SlurmDB | +| Munge only | Munge is the only Slurm authentication method | +| One controller | The controller has no failover. If it stops, no new jobs start until it is back | +| Up to 100 workers | `slurm.conf` sets `MaxNodeCount=100` | +| LDAP without TLS | The LDAP clients connect with plain `ldap://`. See [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}) | diff --git a/content/platform_services/slurm/getting_started/quick_start.md b/content/platform_services/slurm/getting_started/quick_start.md new file mode 100644 index 000000000..4900f80bc --- /dev/null +++ b/content/platform_services/slurm/getting_started/quick_start.md @@ -0,0 +1,114 @@ +--- +title: "Quick Start" +linkTitle: "Quick Start" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "3" +--- + +This guide deploys a Slurm cluster with one controller and one worker, then runs a first job. You can create the service from Sunstone or from the CLI. Check the [Requirements]({{% relref "platform_services/slurm/getting_started/requirements" %}}) before proceeding. + +## Step 1: Import the Appliance + +Export `Service OneSlurm` from the OpenNebula Public Marketplace. This creates the two images, the two VM templates and the OneFlow service template. + +```shell +onemarketapp export 'Service OneSlurm' 'Service OneSlurm' --datastore default +IMAGE + ID: 34 + ID: 35 +VMTEMPLATE + ID: 62 + ID: 63 +SERVICE_TEMPLATE + ID: 64 +``` + +## Step 2: Adjust the Templates (Optional) + +The default parameters are enough for a test Cluster. For real workloads you may need more resources, for example, more CPUs or memory of the workers, or adding a GPU. + +```shell +onetemplate list # find the worker VM template +onetemplate update +oneflow-template update 'Service OneSlurm' +``` + +[Worker Profiles and Accelerators]({{% relref "platform_services/slurm/management/worker_profiles_and_accelerators" %}}) explains the common changes. + +## Step 3: Create the Service + +In Sunstone, go to **Templates -> Service Templates**, select **Service OneSlurm** and click the **Instantiate** button (the play icon). The wizard has four steps. + +1. **General**: Keep the name or type your own, then click **Next**. +2. **Networks**: Keep **Existing** and select an appropriate Virtual Network in the table, then click **Next**. +3. **Service Inputs**: Click **Next** to keep the defaults. LDAP, InfiniBand and NFS stay disabled. +4. **Charter**: Click **Finish**. + +### From the CLI + +```shell +oneflow-template instantiate 'Service OneSlurm' +``` + +The command asks for the inputs, then for the network. + +1. Press **Enter** on every input to keep the defaults. +2. For `Service`, choose `1` (existing network), then give the ID of your Virtual Network. + +```shell + * (Service) Service + TYPE Existing(1), Create(2), Reserve(3). Press enter for default. 1 + VN ID. 2 +ID: 233 +``` + +The command prints the ID of the new service. + +## Step 4: Wait for the Service + +OneFlow creates the controller first. It creates the worker only when the controller reports that it is ready, so the service needs a few minutes to reach `RUNNING`. + +In Sunstone, go to **Instances -> Services** and open the ONeSlurm service. The **Roles** tab shows the controller and the worker. Select both roles to see their VMs: + +{{< image path="/images/slurm/oneslurm/light/oneslurm_service_roles.png" + pathDark="/images/slurm/oneslurm/dark/oneslurm_service_roles.png" +alt="Roles tab of a running OneSlurm service with the controller and the worker" align="center" width="90%" mb="30px" >}} + +From the CLI: + +```shell +oneflow list + ID USER GROUP NAME STARTTIME STAT + 233 oneadmin oneadmin Service OneSlurm 10/07 14:42:30 RUNNING +onevm list -f NAME~'service_233' -l NAME,STAT +NAME STAT +worker_0_(service_233) runn +controller_0_(service_233) runn +``` + +## Step 5: Run a First Job + +1. Connect to the controller: + + ```shell + $ onevm ssh + ``` + +2. Run `sinfo`. The worker must be in the `idle` state. +3. Run a job on the worker with `srun -N1 hostname`. It prints the name of the worker: + +{{< image path="/images/slurm/oneslurm/light/oneslurm_first_job.png" + pathDark="/images/slurm/oneslurm/dark/oneslurm_first_job.png" +alt="Terminal on the controller with the output of sinfo and srun" align="center" width="70%" mb="30px" >}} + +## Next Steps + +Now that you have an operational OneSlurm Cluster, you can scale the Cluster and manage users or storage: + +* Add workers with `oneflow scale`, as demonstrated in [Cluster Lifecycle Management]({{% relref "platform_services/slurm/management/cluster_lifecycle_management" %}}). +* Add shared users with [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}). +* Add a shared `/home` with [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}}). diff --git a/content/platform_services/slurm/getting_started/requirements.md b/content/platform_services/slurm/getting_started/requirements.md new file mode 100644 index 000000000..624f2dc4f --- /dev/null +++ b/content/platform_services/slurm/getting_started/requirements.md @@ -0,0 +1,53 @@ +--- +title: "Requirements and Preparation" +linkTitle: "Requirements" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "2" +--- + +## Requirements + +| **Requirement** | **Details** | +|---|---| +| OpenNebula | 7.0, 7.2 or 7.4 | +| OneFlow | Running. OneSlurm is a OneFlow service | +| OneGate | Running and reachable from the controller and worker VMs | +| OneGate to OneFlow | OneGate can reach the OneFlow API | +| Virtual Network | One network for the service. The controller and the workers attach to it | +| Port 6817/TCP | Open from the workers to the controller, for `slurmctld` and configless Slurm | + +### OneGate Details + +* The controller publishes the Cluster data through [OneGate]({{% relref "product/operation_references/opennebula_services_configuration/onegate" %}}), and the workers read it. Without OneGate, no worker joins the Cluster. +* The VMs can reach OneGate directly or through the [transparent proxies]({{% relref "product/virtual_machines_operation/virtual_machines_networking/tproxy" %}}). +* OneGate must reach the active OneFlow server. The `:oneflow_server:` setting in `/etc/one/onegate-server.conf` sets it. + +To check it, run this command in any VM of the service. It must print the service, not `Service not found`. + +```shell +onegate service show +``` + +### VM Size + +The Marketplace templates use these defaults. They cover the Slurm daemons only, so size the workers for your jobs. + +| **Role** | **CPU** | **Memory** | **Disk** | +|---|---|---|---| +| Controller | 2 | 2 GB | 10 GB | +| Worker | 1 | 1 GB | 10 GB | + +## Optional Features + +Prepare these before you create the service, only for the features you want. + +| **Feature** | **What you need first** | **Guide** | +|---|---|---| +| Shared users | Nothing for the local LDAP on the controller. For an external LDAP, its URL and its domain or base DN | [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}) | +| Shared storage | NFS exports for `/home` and `/scratch`, reachable from all the VMs | [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}}) | +| GPUs | NVIDIA GPUs on the Hosts, with PCI passthrough configured | [Worker Profiles and Accelerators]({{% relref "platform_services/slurm/management/worker_profiles_and_accelerators" %}}) | +| InfiniBand | An InfiniBand HCA for each worker, and a subnet manager already running on the fabric | [InfiniBand]({{% relref "platform_services/slurm/management/infiniband" %}}) | diff --git a/content/platform_services/slurm/management/_index.md b/content/platform_services/slurm/management/_index.md new file mode 100644 index 000000000..46e575cd8 --- /dev/null +++ b/content/platform_services/slurm/management/_index.md @@ -0,0 +1,11 @@ +--- +title: "Slurm Cluster Management" +linkTitle: "Management" +date: "2026-10-07" +description: "Scale, configure, connect and troubleshoot your OneSlurm Clusters." +categories: +pageintoc: "13" +tags: +type: docs +weight: "2" +--- diff --git a/content/platform_services/slurm/management/cluster_configuration.md b/content/platform_services/slurm/management/cluster_configuration.md new file mode 100644 index 000000000..a86266e94 --- /dev/null +++ b/content/platform_services/slurm/management/cluster_configuration.md @@ -0,0 +1,56 @@ +--- +title: "Cluster Configuration" +linkTitle: "Cluster Configuration" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "2" +--- + +The controller writes the Slurm configuration. The workers download it from the controller at start (configless mode), so they have no configuration files of their own. + +## Files on the Controller + +| **File** | **Content** | +|---|---| +| `/etc/slurm/slurm.conf` | Cluster, scheduler, plugins and partition | +| `/etc/slurm/gres.conf` | `AutoDetect=nvidia`, to find NVIDIA GPUs | +| `/etc/slurm/cgroup.conf` | `CgroupAutomount=yes` and `ConstrainDevices=yes` | + +## Default `slurm.conf` + +| **Setting** | **Value** | **Meaning** | +|---|---|---| +| `ClusterName` | `one` | Name of the Slurm Cluster | +| `SlurmctldHost` | `slurm-one-controller` | Hostname of the controller | +| `AuthType` | `auth/munge` | Munge authentication | +| `SchedulerType` | `sched/backfill` | Backfill scheduling | +| `SelectType` | `select/cons_tres` | Allocates CPUs, memory and GPUs per job | +| `ProctrackType` | `proctrack/cgroup` | Tracks job processes with cgroups | +| `TaskPlugin` | `task/cgroup,task/affinity` | Limits tasks with cgroups and pins them to CPUs | +| `GresTypes` | `gpu` | GPUs as generic resources | +| `SlurmctldParameters` | `enable_configless` | Workers download the configuration | +| `MaxNodeCount` | `100` | Maximum number of dynamic nodes | +| `Nodeset` | `one Feature=one` | Groups all the workers | +| `PartitionName` | `all Nodes=ALL Default=yes` | One partition with all the nodes | + +With InfiniBand enabled, the controller also adds `MpiDefault=pmix` and `PropagateResourceLimitsExcept=MEMLOCK`. Refer to the [InfiniBand Documentation]({{% relref "platform_services/slurm/management/infiniband" %}}). + +## Changes to the Configuration + +{{< alert title="Important" type="warning" >}} +The controller rewrites `slurm.conf`, `gres.conf` and `cgroup.conf` every time it boots. Changes that you make by hand in these files are lost after a reboot of the controller. +{{< /alert >}} + +To change the Cluster, use the service inputs when you create it. The [Configuration Parameters]({{% relref "platform_services/slurm/references/configuration_parameters" %}}) page lists them all. + +| **You want** | **How** | +|---|---| +| Shared users | LDAP inputs, see [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}) | +| Shared `/home` and `/scratch` | NFS inputs, see [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}}) | +| MPI over InfiniBand | InfiniBand inputs, see [InfiniBand]({{% relref "platform_services/slurm/management/infiniband" %}}) | +| More CPU, memory or GPUs per worker | Worker VM template, see [Worker Profiles and Accelerators]({{% relref "platform_services/slurm/management/worker_profiles_and_accelerators" %}}) | + +The appliance does not configure accounting (`slurmdbd`), QoS or more partitions. diff --git a/content/platform_services/slurm/management/cluster_lifecycle_management.md b/content/platform_services/slurm/management/cluster_lifecycle_management.md new file mode 100644 index 000000000..23283e34c --- /dev/null +++ b/content/platform_services/slurm/management/cluster_lifecycle_management.md @@ -0,0 +1,75 @@ +--- +title: "Slurm Cluster Lifecycle Management" +linkTitle: "Cluster Lifecycle" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "1" +--- + +A OneSlurm Cluster is a OneFlow service. You create, scale and delete it with the `oneflow` commands. + +| **Task** | **Command** | +|---|---| +| Create a Cluster | `oneflow-template instantiate 'Service OneSlurm'` | +| List Clusters | `oneflow list` | +| Show the VMs of a Cluster | `oneflow show ` | +| Change the number of workers | `oneflow scale worker ` | +| Delete a Cluster | `oneflow delete ` | + +The [Quick Start Guide]({{% relref "platform_services/slurm/getting_started/quick_start" %}}) demonstrates how to create a Cluster. + +## Add Workers + +```shell +oneflow scale worker 4 +``` + +OneFlow creates the new worker VMs. Each new worker reads the Cluster data from OneGate and joins Slurm as a dynamic node, as described in [Core Concepts]({{% relref "platform_services/slurm/getting_started/core_concepts" %}}). Check the new nodes from the controller. + +```shell +sinfo +``` + +| **Limit** | **Value** | +|---|---| +| Recommended minimum number of workers | 1 | +| Maximum number of nodes in Slurm | 100 (`MaxNodeCount` in `slurm.conf`) | +| Cooldown after a scale operation | 30 seconds. During the cooldown you cannot scale or delete the service | + +## Remove Workers + +```shell +oneflow scale worker 1 +``` + +OneFlow deletes the extra worker VMs. Two mechanisms then remove the nodes from Slurm. + +| **Mechanism** | **Where it runs** | **When** | **What it does** | +|---|---|---|---| +| Self-drain | Each worker, `oneslurm-self-drain.service` | When the VM shuts down normally | Sets the node `DOWN` and deletes it from Slurm | +| Reconciler | Controller, `oneslurm-reconcile.timer` | 2 minutes after boot, then every 60 seconds | Removes nodes that have no VM anymore | + +The reconciler is a safety net for VMs that stop without a normal shutdown. It acts on a node only when all three following conditions are true: + +1. The node is registered in Slurm. +2. No worker VM of the service publishes this node name in OneGate. +3. Slurm sees the node as `DOWN` or `NOT_RESPONDING`. + +Then it deletes the node. If the node still has running jobs, it drains the node instead, with the reason `removed from OneFlow service`. + +If OneGate does not answer, or no worker has published its name yet, the reconciler changes nothing. Keep at least one worker in the service. + +{{< alert title="Note" type="info" >}} +Scaling down deletes worker VMs even when they run jobs. Drain the nodes you want to remove first and wait for their jobs to finish. `scontrol update NodeName= State=DRAIN Reason=` drains a node. +{{< /alert >}} + +## Delete the Cluster + +```shell +oneflow delete +``` + +This deletes all VMs of the Cluster. Data stored inside the VMs is lost, including the users of the local LDAP on the controller. Data on external NFS exports persists. diff --git a/content/platform_services/slurm/management/identity_management.md b/content/platform_services/slurm/management/identity_management.md new file mode 100644 index 000000000..09d9e4340 --- /dev/null +++ b/content/platform_services/slurm/management/identity_management.md @@ -0,0 +1,111 @@ +--- +title: "Identity Management" +linkTitle: "Identity" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "4" +--- + +A user needs the same account, with the same UID and GID, on the controller and on every worker. OneSlurm gets these accounts from an LDAP directory, through SSSD on each VM. + +## Identity Modes + +Choose one mode when you create the service. + +| **Mode** | **Inputs** | **What the appliance does** | +|---|---|---| +| Disabled (default) | Leave the LDAP inputs empty | No LDAP. SSSD is stopped. Only local Linux users exist | +| Local LDAP | `ONEAPP_LDAP_ENABLE=YES` | Installs OpenLDAP on the controller and connects the controller and the workers to it | +| External LDAP | `ONEAPP_LDAP_URL` and `ONEAPP_LDAP_DOMAIN` | Connects the controller and the workers to your LDAP server | + +With `ONEAPP_LDAP_ENABLE=YES`, the appliance ignores `ONEAPP_LDAP_URL`. + +The controller decides the mode. It publishes the LDAP settings in OneGate, and the workers use them. The workers ignore their own LDAP inputs. + +## LDAP Inputs + +| **Parameter** | **Default** | **Description** | +|---|---|---| +| `ONEAPP_LDAP_ENABLE` | `NO` | `YES` installs a local LDAP server on the controller | +| `ONEAPP_LDAP_DOMAIN` | `slurm.local` | LDAP domain or base DN | +| `ONEAPP_LDAP_ADMIN_USER` | empty, the appliance uses `admin` | Admin of the local LDAP, as a name or a DN | +| `ONEAPP_LDAP_ADMIN_PASSWORD` | empty | Password of the local LDAP admin. Empty means the admin can only work as root on the controller | +| `ONEAPP_LDAP_URL` | empty | URL of an external LDAP server, for example `ldap://10.0.0.5` | +| `ONEAPP_LDAP_BIND_USER` | empty | User to read the external LDAP, as a name or a DN. Empty means anonymous read | +| `ONEAPP_LDAP_BIND_PASSWORD` | empty | Password of the bind user | + +### Domain and Base DN + +The base DN is the top entry of the directory. OneSlurm builds it from `ONEAPP_LDAP_DOMAIN`. + +| **Domain input** | **s** | +|---|---| +| `slurm.local` | `dc=slurm,dc=local` | +| `dc=example,dc=org` | `dc=example,dc=org` | + +* A domain such as `slurm.local` becomes one `dc=` part for each word. +* A value with `=` is already a base DN and is used without changes. +* Users must be in `ou=People,` and groups in `ou=Groups,`. +* A user name without `=`, such as `admin`, becomes `cn=admin,`. + +## Local LDAP + +With `ONEAPP_LDAP_ENABLE=YES`, the controller creates the base DN, `ou=People` and `ou=Groups`. The directory is empty. Add users and groups as root on the controller. + +This example adds the user `alice` with UID and GID `10001`. + +```shell +$ ldapadd -Y EXTERNAL -H ldapi:/// -Q <') +EOF +``` + +Check the user on the controller and on a worker: + +```shell +getent passwd alice +srun -N1 -n1 getent passwd alice +``` + +If `/home` is an NFS export, create the home directory once, on the NFS server, owned by the user. With the default `root_squash` export option, root on the controller cannot create it. + +```shell +install -d -m 700 -o 10001 -g 10001 /srv/nfs/slurm/home/alice # on the NFS server +``` + +{{< alert title="Important" type="warning" >}} +The local LDAP lives inside the controller VM. Deleting the service deletes all its users. For production, it is recommended to use an external LDAP. +{{< /alert >}} + +## External LDAP + +Set `ONEAPP_LDAP_URL` and `ONEAPP_LDAP_DOMAIN`. Your directory must have users in `ou=People,` and groups in `ou=Groups,`, with POSIX attributes (`uidNumber`, `gidNumber`, `homeDirectory`). + +If the directory does not allow anonymous reads, also set `ONEAPP_LDAP_BIND_USER` and `ONEAPP_LDAP_BIND_PASSWORD`. + +## Security + +| **Topic** | **Behavior** | +|---|---| +| Encryption | SSSD connects with plain `ldap://`, without StartTLS, and does not check certificates. Keep the LDAP traffic on a private network | +| Bind password | The controller publishes `LDAP_BIND_PASSWORD` in OneGate, so the workers can read it. Any VM of the service can read it too | +| Loopback URL | Workers refuse an LDAP URL that points to `127.0.0.1` or `localhost` | diff --git a/content/platform_services/slurm/management/infiniband.md b/content/platform_services/slurm/management/infiniband.md new file mode 100644 index 000000000..223de4194 --- /dev/null +++ b/content/platform_services/slurm/management/infiniband.md @@ -0,0 +1,87 @@ +--- +title: "InfiniBand and High-Performance Networking" +linkTitle: "InfiniBand" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "6" +--- + +OneSlurm can configure IPoIB on the workers and set Slurm for MPI jobs over InfiniBand. It is disabled by default. + +## Before You Start + +Ensure that: + +* Each worker VM has an InfiniBand HCA, attached by PCI passthrough in the worker VM template. See [PCI Passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov" %}}). +* The InfiniBand fabric already has a subnet manager. OneSlurm does not configure it. + +## What the Images Include + +| **Image** | **Packages** | +|---|---| +| Worker | `rdma-core`, `infiniband-diags`, UCX from the Canonical DOCA PPA, Open MPI runtime | +| Controller | Open MPI development packages, to compile MPI programs | + +Both images set the `memlock` limit to `unlimited`, which RDMA needs. + +## Enable InfiniBand + +| **Parameter** | **Default** | **Description** | +|---|---|---| +| `ONEAPP_SLURM_INFINIBAND_ENABLE` | `NO` | `YES` configures IPoIB on the workers and MPI settings in `slurm.conf` | +| `ONEAPP_SLURM_IPOIB_SUBNET` | empty | IPv4 subnet for IPoIB, for example `10.20.0.0/24`. Without a prefix, `/24` is used. Required when InfiniBand is enabled | + +With InfiniBand enabled, the controller adds two lines to `slurm.conf`. + +| **Setting** | **Effect** | +|---|---| +| `MpiDefault=pmix` | `srun` and `sbatch` use PMIx without `--mpi=pmix` | +| `PropagateResourceLimitsExcept=MEMLOCK` | Jobs keep the `memlock` limit of the worker | + +## IPoIB Addresses + +Each worker builds its IPoIB address from the subnet and its Ethernet address. It copies the last octets of the Ethernet address, as many as the prefix leaves free. + +| **IPoIB subnet** | **Worker Ethernet IP** | **Worker IPoIB address** | +|---|---|---| +| `10.20.0.0/24` | `172.16.0.11` | `10.20.0.11/24` | +| `10.20.0.0/16` | `172.16.5.42` | `10.20.5.42/16` | +| `10.0.0.0/8` | `172.16.5.42` | `10.16.5.42/8` | + +* Only subnets `/8`, `/16` and `/24` are valid. +* The copied octets must be different on each worker. Otherwise two workers get the same IPoIB address. +* The copied octets cannot be all `0` or all `255`. +* The worker writes the address to `/etc/netplan/90-oneslurm-ipoib.yaml`. + +Slurm itself stays on the Ethernet network. Controller discovery, Munge, LDAP and `slurmd` do not use IPoIB. MPI and RDMA traffic can use InfiniBand. + +## Check InfiniBand + +On a worker: + +```shell +ibstat +ibv_devinfo +ip addr show ib0 +ucx_info -d | grep mlx5 +``` + +On the controller: + +```shell +mpicc --version +scontrol show config | grep -E 'MpiDefault|PropagateResourceLimitsExcept' +srun -N2 hostname +``` + +## Run an MPI Job + +Compile on the controller and run with `srun`: + +```shell +mpicc mpi_program.c -o mpi_program +srun -N2 ./mpi_program +``` diff --git a/content/platform_services/slurm/management/monitoring_and_troubleshooting.md b/content/platform_services/slurm/management/monitoring_and_troubleshooting.md new file mode 100644 index 000000000..4c1c6e9b9 --- /dev/null +++ b/content/platform_services/slurm/management/monitoring_and_troubleshooting.md @@ -0,0 +1,67 @@ +--- +title: "Monitoring and Troubleshooting" +linkTitle: "Troubleshooting" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "7" +--- + +## Check the Cluster + +Connect to the controller and use the Slurm commands on the command line: + +```shell +onevm ssh +``` + +| **Command** | **Shows** | +|---|---| +| `sinfo` | Partitions and the state of each node | +| `scontrol show nodes` | CPUs, memory, GPUs and state of each node | +| `squeue` | Pending and running jobs | +| `onegate service show` | The VMs of the service, as OneGate sees them | +| `systemctl status slurmctld` | The Slurm controller daemon | + +On a worker, `systemctl status slurmd` shows the Slurm node daemon. + +## Appliance Logs and Status + +Every VM of the service writes the same files: + +| **File** | **Content** | +|---|---| +| `/etc/one-appliance/status` | `bootstrap_success` when the appliance finished without errors | +| `/var/log/one-appliance/configure.log` | Log of the configuration at each boot | +| `/var/log/one-appliance/bootstrap.log` | Log of the last stage | +| `/var/log/one-appliance/install.log` | Log of the image build | + +The reconciler of the controller writes to the journal. + +```shell +journalctl -u oneslurm-reconcile +``` + +## Common Problems + +| **Symptom** | **Likely cause** | **Fix** | +|---|---|---| +| The service stays in `DEPLOYING` and no worker appears | The controller did not publish `READY=YES` | Read `configure.log` on the controller | +| `onegate service show` prints `Service not found` | OneGate cannot reach OneFlow | Set `:oneflow_server:` in `/etc/one/onegate-server.conf` | +| Worker log shows `Could not discover Slurm controller through OneGate` | The worker cannot read the controller data | Check OneGate from the worker with `onegate service show` | +| Worker log shows `Cannot connect to Slurm controller at :6817` | Port 6817 is closed between the worker and the controller | Open 6817/TCP in the security groups and firewalls | +| A node stays `DOWN` after a scale down | The worker VM stopped without a normal shutdown | Wait about one minute for the reconciler, then check `journalctl -u oneslurm-reconcile` | +| A node is `DRAIN` with the reason `removed from OneFlow service` | Its VM left the service while jobs were running | Cancel the jobs, or wait for them, then delete the node with `scontrol delete NodeName=` | +| `getent passwd ` prints nothing | SSSD does not reach LDAP, or the user is not in `ou=People` | Check `systemctl status sssd` and the LDAP inputs | +| `/home` is not mounted | The NFS export is wrong or not reachable | Check the input format `host:/export` and run `findmnt /home` | + +## Check the Optional Features + +| **Feature** | **Command on the controller** | +|---|---| +| LDAP users | `getent passwd `, `getent group ` and `srun -N1 -n1 getent passwd ` | +| NFS | `findmnt /home` and `srun -N1 -n1 findmnt /home` | +| GPUs | `srun -N1 -n1 --gres=gpu:1 nvidia-smi -L` | +| InfiniBand | `scontrol show config \| grep MpiDefault` | diff --git a/content/platform_services/slurm/management/shared_storage.md b/content/platform_services/slurm/management/shared_storage.md new file mode 100644 index 000000000..88c6781c2 --- /dev/null +++ b/content/platform_services/slurm/management/shared_storage.md @@ -0,0 +1,63 @@ +--- +title: "Shared Storage" +linkTitle: "Shared Storage" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "5" +--- + +A job can run on any worker, so users need the same files on the controller and on every worker. OneSlurm can mount two NFS exports in every VM of the Cluster. + +| **Parameter** | **Mount point** | **Example** | +|---|---|---| +| `ONEAPP_SLURM_NFS_HOME` | `/home` | `10.125.0.1:/srv/nfs/slurm/home` | +| `ONEAPP_SLURM_NFS_SCRATCH` | `/scratch` | `10.125.0.1:/srv/nfs/slurm/scratch` | + +Both are optional. An empty value skips the mount. + +## Before You Start + +The controller and worker images already include the NFS client. + +* Create the NFS server and the exports outside OneSlurm. +* The controller and all the workers must reach the NFS server. +* Do not attach another disk at `/home` or `/scratch` when you set the matching input. + +## How It Works + +The `net-12-mount-nfs` script runs in every VM when the network is configured. + +| **Item** | **Value** | +|---|---| +| Format of the input | `host:/export` | +| Filesystem type | `nfs4` | +| Mount options | `sec=sys,_netdev` | +| Persistence | Adds a line to `/etc/fstab` after a successful mount, only once | + +If a mount point is already mounted, the script skips it. + +## Check the Mounts + +On the controller. + +```shell +findmnt /home +TARGET SOURCE FSTYPE OPTIONS +/home 192.168.100.222:/export/home nfs4 rw,relatime,vers=4.2,...,sec=sys,... +``` + +On a worker, through Slurm. + +```shell +srun -N1 -n1 findmnt /home +srun -N1 -n1 findmnt /scratch +``` + +## User Directories + +* `/home` holds the home directory of each user. Create it after the user exists in LDAP, owned by the UID and GID of the user. See [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}). +* `/scratch` has no structure. Create a directory for each user, for example `/scratch/alice`, owned by the user. +* Create these directories on the NFS server. With the default `root_squash` export option, root on the Cluster VMs cannot write to the exports. diff --git a/content/platform_services/slurm/management/worker_profiles_and_accelerators.md b/content/platform_services/slurm/management/worker_profiles_and_accelerators.md new file mode 100644 index 000000000..9c47230eb --- /dev/null +++ b/content/platform_services/slurm/management/worker_profiles_and_accelerators.md @@ -0,0 +1,50 @@ +--- +title: "Worker Profiles and Accelerators" +linkTitle: "Worker Profiles" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "3" +--- + +CPU, memory, disk and GPUs are settings of the worker VM template, not service inputs. Change the template before you create the service or before you add workers. + +```shell +onetemplate update +``` + +Each worker reports its CPUs, memory and GPUs to Slurm when it joins, so Slurm always uses the real size of the VM. + +## CPU and Memory + +The Marketplace worker template has 1 CPU and 1 GB of memory. Set values for `CPU`, `VCPU` and `MEMORY` in the template appropriately for your jobs. For CPU pinning and NUMA, refer to the documentation about [NUMA and CPU Pinning]({{% relref "product/cluster_configuration/hosts_and_clusters/numa" %}}). + +## NVIDIA GPUs + +The worker image includes the NVIDIA driver packages (`nvidia-driver-595-server-open`). The worker template has no attached GPU by default. + +1. Add a GPU to the worker VM template, as a PCI device or a PCI profile of your Host. See [PCI Passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov" %}}). This example requests an NVIDIA H100 PCIe: + + ```text + PCI = [ + VENDOR = "10de", + DEVICE = "2331", + CLASS = "0302" ] + ``` + + The device ID and the PCI profiles depend on your Hosts. You can find the relevant parameters by running `lspci -nn | grep -i nvidia` on the Host's command line. + +2. Create the service or add workers. Each worker counts its GPUs with `nvidia-smi` and starts `slurmd` with `Gres=gpu:`. + +3. Check from the controller: + + ```shell + scontrol show nodes | grep Gres + srun -N1 -n1 --gres=gpu:1 nvidia-smi -L + ``` + +## InfiniBand + +Workers for MPI jobs need an InfiniBand HCA in the VM template. Refer to [InfiniBand and High-Performance Networking]({{% relref "platform_services/slurm/management/infiniband" %}}). diff --git a/content/platform_services/slurm/overview.md b/content/platform_services/slurm/overview.md deleted file mode 100644 index 51011ac50..000000000 --- a/content/platform_services/slurm/overview.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "Managed Slurm Overview" -linkTitle: "Overview" -weight: 2 -type: docs ---- - -The OpenNebula Managed Slurm Service enables the deployment of scalable Slurm Clusters using preconfigured OpenNebula appliances and OneFlow. - -Slurm is an open source, fault-tolerant, and highly scalable workload manager and job scheduling system for executing AI and HPC workloads on Linux Clusters. It allocates compute resources to workloads, starts and monitors jobs on the assigned nodes, and manages pending jobs through scheduling queues. Slurm operates without kernel modifications and is largely self-contained. - -OpenNebula provides multiple appliances for deploying a Slurm Cluster: - -* [**OneSlurm Service**](https://marketplace.opennebula.io/appliance/8ce164d5-3cce-42a7-b9a7-0e8133ef92c6): Slurm Cluster manager for KVM Hosts, orchestrated by OneFlow. -* [**Slurm Controller**](https://marketplace.opennebula.io/appliance/db17c081-969f-45e0-9c8b-e0f7236c15aa): Manages Cluster coordination and scheduling. -* [**Slurm Worker**](https://marketplace.opennebula.io/appliance/c03f8c40-39bb-42cd-a9d5-781f59846b57): Provides the compute capacity used to run jobs. - -The appliances are designed to operate together as roles within a OneFlow service. The controller publishes the information required to coordinate the Cluster through OneGate, while worker nodes automatically retrieve this information and join the Cluster. - -This architecture allows worker capacity to be added or removed through OneFlow, providing a foundation for elastic Slurm Clusters managed through OpenNebula. - -For deployment instructions, configuration options, and operational details, refer to the [Slurm Wiki Documentation](https://github.com/OpenNebula/one-apps/wiki/slurm_intro). There is also a [OneSlurm Tutorial]({{% relref "solutions/ai_factory_blueprints/direct_ai_execution/nvidia_slurm/" %}}) in the AI Factory Blueprints. \ No newline at end of file diff --git a/content/platform_services/slurm/references/_index.md b/content/platform_services/slurm/references/_index.md new file mode 100644 index 000000000..f0ec6c209 --- /dev/null +++ b/content/platform_services/slurm/references/_index.md @@ -0,0 +1,11 @@ +--- +title: "OneSlurm References" +linkTitle: "References" +date: "2026-10-07" +description: "Service architecture, configuration parameters, CLI and API references for OneSlurm." +categories: +pageintoc: "13" +tags: +type: docs +weight: "4" +--- diff --git a/content/platform_services/slurm/references/configuration_parameters.md b/content/platform_services/slurm/references/configuration_parameters.md new file mode 100644 index 000000000..4ec7ed0ed --- /dev/null +++ b/content/platform_services/slurm/references/configuration_parameters.md @@ -0,0 +1,68 @@ +--- +title: "Configuration Parameters" +linkTitle: "Configuration Parameters" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "2" +--- + +## Service Network + +| **Name** | **Mandatory** | **Description** | +|---|---|---| +| `Service` | Yes | Virtual Network for the controller and the workers | + +## Service Inputs + +All the inputs are optional. You set them when you create the service. + +### LDAP + +Used by the controller. The workers get the LDAP settings from the controller through OneGate. See [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}). + +| **Parameter** | **Default** | **Description** | +|---|---|---| +| `ONEAPP_LDAP_ENABLE` | `NO` | `YES` installs a local LDAP server on the controller | +| `ONEAPP_LDAP_DOMAIN` | `slurm.local` | LDAP domain or base DN | +| `ONEAPP_LDAP_ADMIN_USER` | `admin` | Admin of the local LDAP, as a name or a DN | +| `ONEAPP_LDAP_ADMIN_PASSWORD` | empty | Password of the local LDAP admin | +| `ONEAPP_LDAP_URL` | empty | URL of an external LDAP server. Ignored with local LDAP | +| `ONEAPP_LDAP_BIND_USER` | empty | User to read the external LDAP, as a name or a DN | +| `ONEAPP_LDAP_BIND_PASSWORD` | empty | Password of the bind user | + +The admin user input is empty in the service. When it stays empty, the appliance uses `admin`. + +### NFS + +Used by the controller and the workers. See [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}}). + +| **Parameter** | **Default** | **Description** | +|---|---|---| +| `ONEAPP_SLURM_NFS_HOME` | empty | NFS export mounted at `/home` | +| `ONEAPP_SLURM_NFS_SCRATCH` | empty | NFS export mounted at `/scratch` | + +Both use the format `host:/export`. + +### InfiniBand + +See [InfiniBand and High-Performance Networking]({{% relref "platform_services/slurm/management/infiniband" %}}). + +| **Parameter** | **Default** | **Used by** | **Description** | +|---|---|---|---| +| `ONEAPP_SLURM_INFINIBAND_ENABLE` | `NO` | Controller, workers | `YES` configures IPoIB and the MPI settings | +| `ONEAPP_SLURM_IPOIB_SUBNET` | empty | Workers | IPoIB subnet. Required when InfiniBand is enabled | + +The IPoIB subnet must use a `/8`, `/16` or `/24` prefix. + +## Image Build Variables + +These variables apply when you build the appliance images yourself from [one-apps](https://github.com/OpenNebula/one-apps). The Marketplace images use the defaults. + +| **Variable** | **Default** | **Image** | **Description** | +|---|---|---|---| +| `INSTALL_INFINIBAND` | `true` | Controller, worker | Installs the InfiniBand, UCX and Open MPI packages | +| `INSTALL_DRIVERS` | `true` | Worker | Installs the NVIDIA driver | +| `NVIDIA_DRIVER_BRANCH` | `595` | Worker | NVIDIA driver branch, for `nvidia-driver--server-open` | diff --git a/content/platform_services/slurm/references/oneslurm_api.md b/content/platform_services/slurm/references/oneslurm_api.md new file mode 100644 index 000000000..a0a4356e7 --- /dev/null +++ b/content/platform_services/slurm/references/oneslurm_api.md @@ -0,0 +1,104 @@ +--- +title: "OneSlurm API" +linkTitle: "API" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "4" +--- + +OneSlurm has no API of its own. A OneSlurm cluster is a OneFlow service, so you create, scale and delete it with the [OneFlow Server API]({{% relref "product/integration_references/system_interfaces/appflow_api" %}}). This page shows the calls relevant to OneSlurm. + +| **Item** | **Value** | +|---|---| +| Endpoint | `http://:2474` | +| Authentication | HTTP Basic, with an OpenNebula user and password | +| Format | JSON | + +The examples use these variables. + +```shell +FLOW=http://localhost:2474 +AUTH='oneadmin:' +``` + +## Find the Service Template + +```shell +curl -s -u "$AUTH" $FLOW/service_template +``` + +The response lists the service templates in `DOCUMENT_POOL.DOCUMENT`. Take the `ID` of the one named `Service OneSlurm`. + +## Create a Cluster + +```shell +curl -s -u "$AUTH" -X POST -H 'Content-Type: application/json' \ + $FLOW/service_template//action -d '{ + "action": { + "perform": "instantiate", + "params": { + "merge_template": { + "name": "oneslurm-api", + "networks_values": [ { "Service": { "id": "2" } } ], + "user_inputs_values": { + "ONEAPP_LDAP_ENABLE": "YES", + "ONEAPP_LDAP_DOMAIN": "slurm.local", + "ONEAPP_LDAP_ADMIN_USER": "", + "ONEAPP_LDAP_ADMIN_PASSWORD": "", + "ONEAPP_LDAP_URL": "", + "ONEAPP_LDAP_BIND_USER": "", + "ONEAPP_LDAP_BIND_PASSWORD": "", + "ONEAPP_SLURM_INFINIBAND_ENABLE": "NO", + "ONEAPP_SLURM_IPOIB_SUBNET": "", + "ONEAPP_SLURM_NFS_HOME": "", + "ONEAPP_SLURM_NFS_SCRATCH": "" + } + } + } + } +}' +``` + +* `networks_values` sets the `Service` network. Replace `2` with the ID of your Virtual Network. +* `user_inputs_values` must contain **all** the service inputs, also the empty ones. Otherwise the API answers `Verify that every User Input have its corresponding value defined`. +* The response is the new service. Its ID is in `DOCUMENT.ID`. + +Refer to [Configuration Parameters]({{% relref "platform_services/slurm/references/configuration_parameters" %}}) for a description of each input. + +## Check the Cluster + +```shell +curl -s -u "$AUTH" $FLOW/service/ +``` + +The state of the service is in `DOCUMENT.TEMPLATE.BODY.state`, and the roles with their VMs in `DOCUMENT.TEMPLATE.BODY.roles`. + +| `state` | **Meaning** | +|---|---| +| `1` | Deploying | +| `2` | Running | +| `8` | Scaling | +| `10` | Cooldown | + +## Change the Number of Workers + +```shell +$ curl -s -u "$AUTH" -X POST -H 'Content-Type: application/json' \ + $FLOW/service//scale \ + -d '{ "role_name": "worker", "cardinality": 2, "force": false }' +``` + +* The API answers `204` when it accepts the request. +* Always send `force`. Without it, the API answers `500`. +* The service goes through `Scaling` and `Cooldown` and returns to `Running`. The new workers join Slurm on their own. + +## Delete the Cluster + +```shell +curl -s -u "$AUTH" -X DELETE $FLOW/service/ +``` + +The API answers `204`. You cannot delete the service during the cooldown after a scale operation. diff --git a/content/platform_services/slurm/references/oneslurm_cli.md b/content/platform_services/slurm/references/oneslurm_cli.md new file mode 100644 index 000000000..a3bdd5dd6 --- /dev/null +++ b/content/platform_services/slurm/references/oneslurm_cli.md @@ -0,0 +1,43 @@ +--- +title: "OneSlurm CLI" +linkTitle: "CLI" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "3" +--- + +## OpenNebula Commands + +Run these on the OpenNebula Front-end. + +| **Command** | **Purpose** | +|---|---| +| `onemarketapp export 'Service OneSlurm' 'Service OneSlurm' --datastore default` | Import the appliance from the Marketplace | +| `onetemplate update ` | Change a controller or worker VM template | +| `oneflow-template update ` | Change the service template | +| `oneflow-template instantiate 'Service OneSlurm'` | Create a Cluster | +| `oneflow list` | List the Clusters | +| `oneflow show ` | Show the roles and VMs of a Cluster | +| `oneflow scale worker ` | Change the number of workers | +| `oneflow delete ` | Delete a Cluster | +| `onevm ssh ` | Connect to a VM of the Cluster | + +## Commands Inside the VMs + +| **Command** | **Where** | **Purpose** | +|---|---|---| +| `sinfo` | Controller | Partitions and node states | +| `scontrol show nodes` | Controller | Details of each node | +| `squeue` | Controller | Job queue | +| `srun -N1 hostname` | Controller | Run a test job | +| `scontrol update NodeName= State=DRAIN Reason=` | Controller | Drain a node before you remove it | +| `scontrol update NodeName= State=RESUME` | Controller | Put a drained or down node back in service | +| `scontrol delete NodeName=` | Controller | Delete a dynamic node | +| `journalctl -u oneslurm-reconcile` | Controller | Log of the reconciler | +| `onegate service show` | Any VM | The service, as OneGate sees it | +| `getent passwd ` | Any VM | Check an LDAP user | +| `findmnt /home` | Any VM | Check the NFS mount | +| `cat /etc/one-appliance/status` | Any VM | Result of the appliance configuration | diff --git a/content/platform_services/slurm/references/service_architecture.md b/content/platform_services/slurm/references/service_architecture.md new file mode 100644 index 000000000..754a81ba0 --- /dev/null +++ b/content/platform_services/slurm/references/service_architecture.md @@ -0,0 +1,86 @@ +--- +title: "Service Architecture" +linkTitle: "Service Architecture" +date: "2026-10-07" +description: "" +categories: +tags: +type: docs +weight: "1" +--- + +## Components + +| **Component** | **Where** | **Role** | +|---|---|---| +| OneFlow service `OneSlurm` | OpenNebula Front-end | Creates the roles in order and scales the workers | +| `controller` role | 1 VM | `slurmctld`, Munge, optional OpenLDAP, reconciler | +| `worker` role | 1 or more VMs, child of `controller` | `slurmd`, Munge, self-drain hook | +| OneGate | OpenNebula Front-end | Shares data between the VMs of the service | +| Service network | Virtual Network | Connects all the VMs. Chosen when the service is created | + +The service uses `deployment: straight` and `ready_status_gate: true`. OneFlow creates the controller first and the workers only after the controller reports `READY=YES`. + +## Data in OneGate + +| **Attribute** | **Published by** | **Read by** | **Content** | +|---|---|---|---| +| `READY` | Controller | OneFlow, workers | `YES` when the controller is configured | +| `SLURM_MUNGE_KEY` | Controller | Workers | Munge key in base64 | +| `LDAP_URL` | Controller | Workers | LDAP URL. Empty when LDAP is disabled | +| `LDAP_DOMAIN` | Controller | Workers | LDAP domain or base DN | +| `LDAP_ADMIN_USER` | Controller | | LDAP admin user, when set | +| `LDAP_BIND_USER` | Controller | Workers | LDAP bind user, when set | +| `LDAP_BIND_PASSWORD` | Controller | Workers | LDAP bind password, when set | +| `SLURM_NODENAME` | Each worker | Controller reconciler | Name of the Slurm node of this VM | + +With local LDAP, `LDAP_URL` is `ldap://`. + +## Network Ports + +The following table shows the default ports of each service. + +| **Port** | **Service** | **From** | **To** | +|---|---|---|---| +| 6817/TCP | `slurmctld` and configless configuration | Workers | Controller | +| 389/TCP | LDAP | All VMs | Local LDAP on the controller, or the external LDAP | +| 2049/TCP | NFSv4 | All VMs | NFS server, when NFS is used | +| 22/TCP | SSH | Users | Controller | + +## Files and Units + +| **Item** | **VM** | **Purpose** | +|---|---|---| +| `/etc/slurm/slurm.conf`, `gres.conf`, `cgroup.conf` | Controller | Slurm configuration, written at each boot | +| `/etc/munge/munge.key` | All | Munge key | +| `/etc/sssd/sssd.conf` | All, with LDAP | SSSD client for LDAP | +| `oneslurm-reconcile.timer` | Controller | Runs the reconciler 2 minutes after boot, then every 60 seconds | +| `/usr/local/sbin/oneslurm-reconcile-nodes` | Controller | Reconciler script | +| `oneslurm-self-drain.service` | Workers | Removes the node from Slurm at shutdown | +| `/etc/systemd/system/slurmd.service` | Workers | `slurmd` in configless mode with dynamic registration | +| `/etc/netplan/90-oneslurm-ipoib.yaml` | Workers, with InfiniBand | IPoIB address | + +## Boot Sequence + +In every VM, the NFS exports are mounted first, while the network is configured. Then the appliance configures the role. + +### Controller + +1. Sets the hostname `slurm-one-controller`. +2. Writes `slurm.conf`, `gres.conf` and `cgroup.conf`. +3. Creates the Munge key at the first boot and starts `slurmctld`. +4. Configures LDAP, local or external, and publishes the LDAP data in OneGate. +5. Publishes `SLURM_MUNGE_KEY` and `READY=YES` in OneGate. +6. Starts the reconciler timer. + +### Worker + +1. Waits until the controller publishes `READY=YES`, then reads its IP, the Munge key and the LDAP data. +2. Sets its hostname. The Marketplace template uses the VM name. Without `SET_HOSTNAME`, the hostname is `slurm-one-worker-`. +3. Checks that port 6817 of the controller answers. +4. Maps `slurm-one-controller` to the controller IP in `/etc/hosts`. +5. Configures IPoIB, when InfiniBand is enabled. +6. Installs the Munge key and starts `slurmd`. +7. Publishes `SLURM_NODENAME` in OneGate. +8. Installs the self-drain hook. +9. Configures the SSSD client, when the controller published LDAP data. diff --git a/content/solutions/engineering_blueprints/ee_poc/poc_success_criteria.md b/content/solutions/engineering_blueprints/ee_poc/poc_success_criteria.md index 92e6e7189..68d75e7c2 100644 --- a/content/solutions/engineering_blueprints/ee_poc/poc_success_criteria.md +++ b/content/solutions/engineering_blueprints/ee_poc/poc_success_criteria.md @@ -90,7 +90,7 @@ Each OpenNebula deployment is different but they all have to provide a basic set | **ID** | **Task** | **Validation description** | **Video tutorials** | **Documentation** | | ------ | ------------------------ | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | -| AI‑SC19 | Deploy a GPU-enabled Slurm Cluster | Provision a Slurm cluster with GPU-capable compute nodes and verify that Slurm detects and manages the GPU resources correctly. | | [Elastic Slurm Overview]({{% relref "platform_services/slurm/overview/" %}}) -| AI‑SC20 | Schedule a Slurm GPU Job | Submit a Slurm job that requests GPU resources and verify that it is scheduled onto an appropriate GPU node and completes successfully. | | [Elastic Slurm Overview]({{% relref "platform_services/slurm/overview/" %}}) -| AI‑SC21 | Run a Multi-GPU Slurm Job | Execute a workload that requests multiple GPUs and verify that all assigned GPUs are available to the job and operate as expected.| | [Elastic Slurm Overview]({{% relref "platform_services/slurm/overview/" %}}) -| AI‑SC22 | Validate Manual Elastic Slurm Capacity | Manually add compute capacity in response to queued workload demand and verify that the new nodes are provisioned, join the Slurm cluster, execute the workload, and can be manually released when no longer required. | | [Elastic Slurm Overview]({{% relref "platform_services/slurm/overview/" %}}) +| AI‑SC19 | Deploy a GPU-enabled Slurm Cluster | Provision a Slurm cluster with GPU-capable compute nodes and verify that Slurm detects and manages the GPU resources correctly. | | [Elastic Slurm Overview]({{% relref "platform_services/slurm/getting_started/overview/" %}}) +| AI‑SC20 | Schedule a Slurm GPU Job | Submit a Slurm job that requests GPU resources and verify that it is scheduled onto an appropriate GPU node and completes successfully. | | [Elastic Slurm Overview]({{% relref "platform_services/slurm/getting_started/overview/" %}}) +| AI‑SC21 | Run a Multi-GPU Slurm Job | Execute a workload that requests multiple GPUs and verify that all assigned GPUs are available to the job and operate as expected.| | [Elastic Slurm Overview]({{% relref "platform_services/slurm/getting_started/overview/" %}}) +| AI‑SC22 | Validate Manual Elastic Slurm Capacity | Manually add compute capacity in response to queued workload demand and verify that the new nodes are provisioned, join the Slurm cluster, execute the workload, and can be manually released when no longer required. | | [Elastic Slurm Overview]({{% relref "platform_services/slurm/getting_started/overview/" %}})