Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,6 @@ You can browse available Marketplace Appliances in the dedicated [OpenNebula Mar
* [**Harbor Container Registry**](https://github.com/OpenNebula/one-apps/wiki/harbor_intro): Deploy a secure, self-managed image repository.
* [**MinIO**](https://github.com/OpenNebula/one-apps/wiki/minio_intro): Object storage solution with an AWS S3-compatible API.
* [**vLLM AI**](https://github.com/OpenNebula/one-apps/wiki/vllm_intro): Deploy high-performance LLM inferencing.
* [**Slurm**](https://github.com/OpenNebula/one-apps/wiki/slurm_intro): Fault-tolerant, highly scalable Cluster management and job scheduling system for HPC.
* [**Slurm**]({{% relref "platform_services/slurm/getting_started/overview" %}}): Fault-tolerant, highly scalable Cluster management and job scheduling system for HPC.
* [**NVIDIA NIM Service**](https://github.com/OpenNebula/one-apps/wiki/nim_intro): GPU-accelerated inference microservices for pre-trained and customized AI models.
* [**Rancher CAPI**](https://github.com/OpenNebula/one-apps/wiki/capi_intro): Out-of-the-box solution for managing Kubernetes Clusters using Rancher and OpenNebula.
4 changes: 1 addition & 3 deletions content/platform_services/slurm/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,10 @@
title: "Managed Slurm"
linkTitle: "Managed Slurm"
date: "2026-05-12"
description: ""
description: "OneSlurm deploys Slurm Clusters on OpenNebula as a OneFlow service, with a controller and elastic workers."
categories:
pageintoc: "13"
tags:
type: docs
weight: "2"
---

Slurm is an open source, fault-tolerant, and highly scalable Cluster management and job scheduling system for large and small Linux Clusters.
11 changes: 11 additions & 0 deletions content/platform_services/slurm/getting_started/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: "Getting Started with OneSlurm"
linkTitle: "Getting Started"
date: "2026-10-07"
description: "Learn what OneSlurm deploys, prepare your cloud and create your first Slurm Cluster."
categories:
pageintoc: "13"
tags:
type: docs
weight: "1"
---
64 changes: 64 additions & 0 deletions content/platform_services/slurm/getting_started/core_concepts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
title: "Core Concepts"
linkTitle: "Core Concepts"
date: "2026-10-07"
description: ""
categories:
tags:
type: docs
weight: "4"
---

## Controller and Workers

| | **Controller** | **Worker** |
|---|---|---|
| Slurm daemon | `slurmctld` | `slurmd` |
| Number of VMs | 1 | 1 or more |
| Hostname | `slurm-one-controller` | The VM name, for example `worker-0--service-233` |
| Role in the Cluster | Schedules jobs, keeps the configuration, hosts the optional local LDAP | Runs jobs |

Users connect to the controller to compile code and submit jobs with `srun` or `sbatch`.

## Configless Slurm

The controller is the only VM that has the Slurm configuration files. The workers start `slurmd` with `--conf-server slurm-one-controller:6817` and download the configuration from the controller. A change on the controller reaches every worker, and new workers need no configuration.

## Dynamic Nodes

Workers join Slurm as dynamic nodes (`slurmd -Z`). Each worker reports to the controller what it has.

* Its CPUs, from `nproc`
* Its memory, from `/proc/meminfo`
* Its NVIDIA GPUs, from `nvidia-smi`, as the `gpu` GRES

The nodes are not listed in `slurm.conf`.

## OneGate Coordination

The controller and the workers share their data through OneGate, which keeps data on each VM of the service.

1. The controller starts, configures Slurm and publishes `READY=YES` and the Munge key.
2. OneFlow waits for `READY=YES` before it creates the workers. This is the `ready_status_gate` option of the service. The VM templates also set `REPORT_READY=YES`, so one-context reports readiness too.
3. Each worker reads the controller IP, the Munge key and the LDAP settings from OneGate.
4. Each worker checks that port 6817 of the controller answers, then starts `slurmd`.
5. Each worker publishes its Slurm node name as `SLURM_NODENAME`.

Refer to [Service Architecture]({{% relref "platform_services/slurm/references/service_architecture" %}}) for a list of every attribute that the VMs publish.

## Munge Key

Slurm uses [Munge](https://dun.github.io/munge/) to authenticate the messages between the daemons. All the VMs of a Cluster need the same key.

* The controller creates the key at its first boot and keeps it.
* It publishes the key in base64 to OneGate as `SLURM_MUNGE_KEY`.
* Each worker installs the key and checks that Munge works before it starts `slurmd`.

## Scaling and Cleanup

When you remove workers, two mechanisms remove them from Slurm.

* Each worker deletes its own node from Slurm when it shuts down normally.
* The controller checks every 60 seconds, starting 2 minutes after boot, for nodes that have no VM anymore and that Slurm sees as down. It deletes them, or drains them if they still have running jobs.

Refer to [Cluster Lifecycle Management]({{% relref "platform_services/slurm/management/cluster_lifecycle_management" %}}) for more details.
68 changes: 68 additions & 0 deletions content/platform_services/slurm/getting_started/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: "Overview"
linkTitle: "Overview"
date: "2026-10-07"
description: ""
categories:
tags:
type: docs
weight: "1"
aliases:
- /platform_services/slurm/overview/
---

[Slurm](https://slurm.schedmd.com/documentation.html) is an open source workload manager and job scheduler for AI and HPC workloads on Linux Clusters. It gives jobs access to compute nodes, starts and monitors them, and keeps pending jobs in a queue.

OneSlurm deploys a Slurm Cluster on OpenNebula as a OneFlow service. The service has two roles:

| **Role** | **Appliance** | **What it does** |
|---|---|---|
| `controller` | [Service Slurm Controller](https://marketplace.opennebula.io/appliance/db17c081-969f-45e0-9c8b-e0f7236c15aa) | Runs `slurmctld`, schedules jobs and owns the Slurm configuration. Users log in here to submit jobs |
| `worker` | [Service Slurm Worker](https://marketplace.opennebula.io/appliance/c03f8c40-39bb-42cd-a9d5-781f59846b57) | Runs `slurmd` and executes jobs. You can add or remove workers at any time |

Both roles come together in the [Service OneSlurm](https://marketplace.opennebula.io/appliance/8ce164d5-3cce-42a7-b9a7-0e8133ef92c6) appliance of the OpenNebula Public Marketplace. The controller publishes the Cluster data through OneGate, and each worker reads it and joins the Cluster without manual steps.

Optional features are set when you create the service:

* Shared user accounts through a local or an external LDAP directory.
* Shared `/home` and `/scratch` from NFS exports.
* NVIDIA GPUs attached to the workers.
* InfiniBand for MPI jobs.

## How Should I Read This Chapter

1. Check [Requirements and Preparation]({{% relref "platform_services/slurm/getting_started/requirements" %}}).
2. Deploy a first Cluster with the [Quick Start Guide]({{% relref "platform_services/slurm/getting_started/quick_start" %}}).
3. Read [Core Concepts]({{% relref "platform_services/slurm/getting_started/core_concepts" %}}) to understand how the controller and the workers work together.

Then use the Management pages for daily operation:

* [Cluster Lifecycle Management]({{% relref "platform_services/slurm/management/cluster_lifecycle_management" %}})
* [Cluster Configuration]({{% relref "platform_services/slurm/management/cluster_configuration" %}})
* [Worker Profiles and Accelerators]({{% relref "platform_services/slurm/management/worker_profiles_and_accelerators" %}})
* [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}})
* [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}})
* [InfiniBand and High-Performance Networking]({{% relref "platform_services/slurm/management/infiniband" %}})
* [Monitoring and Troubleshooting]({{% relref "platform_services/slurm/management/monitoring_and_troubleshooting" %}})

The References pages list the [Service Architecture]({{% relref "platform_services/slurm/references/service_architecture" %}}), every [Configuration Parameter]({{% relref "platform_services/slurm/references/configuration_parameters" %}}), the [CLI]({{% relref "platform_services/slurm/references/oneslurm_cli" %}}) commands and the [API]({{% relref "platform_services/slurm/references/oneslurm_api" %}}) calls.

There is also a [OneSlurm tutorial]({{% relref "solutions/ai_factory_blueprints/direct_ai_execution/nvidia_slurm/" %}}) with NVIDIA GPUs in the AI Factory Blueprints.

{{< alert title="Note" type="info" >}}
An **OpenNebula Cluster** groups Hosts, datastores and Virtual Networks. A **Slurm Cluster** is a group of VMs that run Slurm. A Slurm Cluster deployed by OneSlurm runs as a workload on top of OpenNebula resources, and you can run several Slurm Clusters in one OpenNebula Cluster.
{{< /alert >}}

## Release Notes

The [Service OneSlurm](https://marketplace.opennebula.io/appliance/8ce164d5-3cce-42a7-b9a7-0e8133ef92c6) page of the Marketplace shows the current version, the guest OS and the supported OpenNebula versions. The [one-apps release page](https://github.com/OpenNebula/one-apps/releases) lists the changes of each version.

## Known Limitations

| **Limitation** | **Effect** |
|---|---|
| No `slurmdbd` | No job accounting, job history or usage statistics through SlurmDB |
| Munge only | Munge is the only Slurm authentication method |
| One controller | The controller has no failover. If it stops, no new jobs start until it is back |
| Up to 100 workers | `slurm.conf` sets `MaxNodeCount=100` |
| LDAP without TLS | The LDAP clients connect with plain `ldap://`. See [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}) |
114 changes: 114 additions & 0 deletions content/platform_services/slurm/getting_started/quick_start.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
---
title: "Quick Start"
linkTitle: "Quick Start"
date: "2026-10-07"
description: ""
categories:
tags:
type: docs
weight: "3"
---

This guide deploys a Slurm cluster with one controller and one worker, then runs a first job. You can create the service from Sunstone or from the CLI. Check the [Requirements]({{% relref "platform_services/slurm/getting_started/requirements" %}}) before proceeding.

## Step 1: Import the Appliance

Export `Service OneSlurm` from the OpenNebula Public Marketplace. This creates the two images, the two VM templates and the OneFlow service template.

```shell
onemarketapp export 'Service OneSlurm' 'Service OneSlurm' --datastore default
IMAGE
ID: 34
ID: 35
VMTEMPLATE
ID: 62
ID: 63
SERVICE_TEMPLATE
ID: 64
```

## Step 2: Adjust the Templates (Optional)

The default parameters are enough for a test Cluster. For real workloads you may need more resources, for example, more CPUs or memory of the workers, or adding a GPU.

```shell
onetemplate list # find the worker VM template
onetemplate update <WORKER_TEMPLATE_ID>
oneflow-template update 'Service OneSlurm'
```

[Worker Profiles and Accelerators]({{% relref "platform_services/slurm/management/worker_profiles_and_accelerators" %}}) explains the common changes.

## Step 3: Create the Service

In Sunstone, go to **Templates -> Service Templates**, select **Service OneSlurm** and click the **Instantiate** button (the play icon). The wizard has four steps.

1. **General**: Keep the name or type your own, then click **Next**.
2. **Networks**: Keep **Existing** and select an appropriate Virtual Network in the table, then click **Next**.
3. **Service Inputs**: Click **Next** to keep the defaults. LDAP, InfiniBand and NFS stay disabled.
4. **Charter**: Click **Finish**.

### From the CLI

```shell
oneflow-template instantiate 'Service OneSlurm'
```

The command asks for the inputs, then for the network.

1. Press **Enter** on every input to keep the defaults.
2. For `Service`, choose `1` (existing network), then give the ID of your Virtual Network.

```shell
* (Service) Service
TYPE Existing(1), Create(2), Reserve(3). Press enter for default. 1
VN ID. 2
ID: 233
```

The command prints the ID of the new service.

## Step 4: Wait for the Service

OneFlow creates the controller first. It creates the worker only when the controller reports that it is ready, so the service needs a few minutes to reach `RUNNING`.

In Sunstone, go to **Instances -> Services** and open the ONeSlurm service. The **Roles** tab shows the controller and the worker. Select both roles to see their VMs:

{{< image path="/images/slurm/oneslurm/light/oneslurm_service_roles.png"
pathDark="/images/slurm/oneslurm/dark/oneslurm_service_roles.png"
alt="Roles tab of a running OneSlurm service with the controller and the worker" align="center" width="90%" mb="30px" >}}

From the CLI:

```shell
oneflow list
ID USER GROUP NAME STARTTIME STAT
233 oneadmin oneadmin Service OneSlurm 10/07 14:42:30 RUNNING
onevm list -f NAME~'service_233' -l NAME,STAT
NAME STAT
worker_0_(service_233) runn
controller_0_(service_233) runn
```

## Step 5: Run a First Job

1. Connect to the controller:

```shell
$ onevm ssh <CONTROLLER_VM_ID>
```

2. Run `sinfo`. The worker must be in the `idle` state.
3. Run a job on the worker with `srun -N1 hostname`. It prints the name of the worker:

{{< image path="/images/slurm/oneslurm/light/oneslurm_first_job.png"
pathDark="/images/slurm/oneslurm/dark/oneslurm_first_job.png"
alt="Terminal on the controller with the output of sinfo and srun" align="center" width="70%" mb="30px" >}}

## Next Steps

Now that you have an operational OneSlurm Cluster, you can scale the Cluster and manage users or storage:

* Add workers with `oneflow scale`, as demonstrated in [Cluster Lifecycle Management]({{% relref "platform_services/slurm/management/cluster_lifecycle_management" %}}).
* Add shared users with [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}).
* Add a shared `/home` with [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}}).
53 changes: 53 additions & 0 deletions content/platform_services/slurm/getting_started/requirements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
title: "Requirements and Preparation"
linkTitle: "Requirements"
date: "2026-10-07"
description: ""
categories:
tags:
type: docs
weight: "2"
---

## Requirements

| **Requirement** | **Details** |
|---|---|
| OpenNebula | 7.0, 7.2 or 7.4 |
| OneFlow | Running. OneSlurm is a OneFlow service |
| OneGate | Running and reachable from the controller and worker VMs |
| OneGate to OneFlow | OneGate can reach the OneFlow API |
| Virtual Network | One network for the service. The controller and the workers attach to it |
| Port 6817/TCP | Open from the workers to the controller, for `slurmctld` and configless Slurm |

### OneGate Details

* The controller publishes the Cluster data through [OneGate]({{% relref "product/operation_references/opennebula_services_configuration/onegate" %}}), and the workers read it. Without OneGate, no worker joins the Cluster.
* The VMs can reach OneGate directly or through the [transparent proxies]({{% relref "product/virtual_machines_operation/virtual_machines_networking/tproxy" %}}).
* OneGate must reach the active OneFlow server. The `:oneflow_server:` setting in `/etc/one/onegate-server.conf` sets it.

To check it, run this command in any VM of the service. It must print the service, not `Service <id> not found`.

```shell
onegate service show
```

### VM Size

The Marketplace templates use these defaults. They cover the Slurm daemons only, so size the workers for your jobs.

| **Role** | **CPU** | **Memory** | **Disk** |
|---|---|---|---|
| Controller | 2 | 2 GB | 10 GB |
| Worker | 1 | 1 GB | 10 GB |

## Optional Features

Prepare these before you create the service, only for the features you want.

| **Feature** | **What you need first** | **Guide** |
|---|---|---|
| Shared users | Nothing for the local LDAP on the controller. For an external LDAP, its URL and its domain or base DN | [Identity Management]({{% relref "platform_services/slurm/management/identity_management" %}}) |
| Shared storage | NFS exports for `/home` and `/scratch`, reachable from all the VMs | [Shared Storage]({{% relref "platform_services/slurm/management/shared_storage" %}}) |
| GPUs | NVIDIA GPUs on the Hosts, with PCI passthrough configured | [Worker Profiles and Accelerators]({{% relref "platform_services/slurm/management/worker_profiles_and_accelerators" %}}) |
| InfiniBand | An InfiniBand HCA for each worker, and a subnet manager already running on the fabric | [InfiniBand]({{% relref "platform_services/slurm/management/infiniband" %}}) |
11 changes: 11 additions & 0 deletions content/platform_services/slurm/management/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: "Slurm Cluster Management"
linkTitle: "Management"
date: "2026-10-07"
description: "Scale, configure, connect and troubleshoot your OneSlurm Clusters."
categories:
pageintoc: "13"
tags:
type: docs
weight: "2"
---
Loading
Loading