diff --git a/assets/images/open_ondemand/light/job_composer_slurm.png b/assets/images/open_ondemand/light/job_composer_slurm.png deleted file mode 100644 index 40daa1d51..000000000 Binary files a/assets/images/open_ondemand/light/job_composer_slurm.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/job_options_slurm.png b/assets/images/open_ondemand/light/job_options_slurm.png deleted file mode 100644 index 8cdd92504..000000000 Binary files a/assets/images/open_ondemand/light/job_options_slurm.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/jupyter_form.png b/assets/images/open_ondemand/light/jupyter_form.png deleted file mode 100644 index 2a181fdde..000000000 Binary files a/assets/images/open_ondemand/light/jupyter_form.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/jupyter_form_sizes.png b/assets/images/open_ondemand/light/jupyter_form_sizes.png deleted file mode 100644 index a175879c2..000000000 Binary files a/assets/images/open_ondemand/light/jupyter_form_sizes.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/jupyterlab.png b/assets/images/open_ondemand/light/jupyterlab.png deleted file mode 100644 index 9a27aea37..000000000 Binary files a/assets/images/open_ondemand/light/jupyterlab.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/portal_dashboard.png b/assets/images/open_ondemand/light/portal_dashboard.png deleted file mode 100644 index a1295d4da..000000000 Binary files a/assets/images/open_ondemand/light/portal_dashboard.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/portal_login.png b/assets/images/open_ondemand/light/portal_login.png deleted file mode 100644 index 5e213a58f..000000000 Binary files a/assets/images/open_ondemand/light/portal_login.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/session_card.png b/assets/images/open_ondemand/light/session_card.png deleted file mode 100644 index 75c2528ab..000000000 Binary files a/assets/images/open_ondemand/light/session_card.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_instantiate_general.png b/assets/images/open_ondemand/light/sunstone_instantiate_general.png deleted file mode 100644 index d44618ea4..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_instantiate_general.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_instantiate_inputs.png b/assets/images/open_ondemand/light/sunstone_instantiate_inputs.png deleted file mode 100644 index ef1b20232..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_instantiate_inputs.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_instantiate_networks.png b/assets/images/open_ondemand/light/sunstone_instantiate_networks.png deleted file mode 100644 index d4f63032c..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_instantiate_networks.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_portal_ood_url.png b/assets/images/open_ondemand/light/sunstone_portal_ood_url.png deleted file mode 100644 index 143d68de4..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_portal_ood_url.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_service_roles.png b/assets/images/open_ondemand/light/sunstone_service_roles.png deleted file mode 100644 index a4745f15f..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_service_roles.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_service_running.png b/assets/images/open_ondemand/light/sunstone_service_running.png deleted file mode 100644 index bffdfcf67..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_service_running.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_service_templates.png b/assets/images/open_ondemand/light/sunstone_service_templates.png deleted file mode 100644 index 96f5f8ec2..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_service_templates.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_vnet_addresses.png b/assets/images/open_ondemand/light/sunstone_vnet_addresses.png deleted file mode 100644 index 87803d499..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_vnet_addresses.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_vnet_conf.png b/assets/images/open_ondemand/light/sunstone_vnet_conf.png deleted file mode 100644 index 675418f23..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_vnet_conf.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_vnet_general.png b/assets/images/open_ondemand/light/sunstone_vnet_general.png deleted file mode 100644 index 2ffa98fcc..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_vnet_general.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_vnet_management_addresses.png b/assets/images/open_ondemand/light/sunstone_vnet_management_addresses.png deleted file mode 100644 index 4313287ee..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_vnet_management_addresses.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_vnet_management_conf.png b/assets/images/open_ondemand/light/sunstone_vnet_management_conf.png deleted file mode 100644 index 0ae4dd87e..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_vnet_management_conf.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_vnet_management_context.png b/assets/images/open_ondemand/light/sunstone_vnet_management_context.png deleted file mode 100644 index 6314bc470..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_vnet_management_context.png and /dev/null differ diff --git a/assets/images/open_ondemand/light/sunstone_vnet_management_general.png b/assets/images/open_ondemand/light/sunstone_vnet_management_general.png deleted file mode 100644 index 9f3dc25e9..000000000 Binary files a/assets/images/open_ondemand/light/sunstone_vnet_management_general.png and /dev/null differ diff --git a/assets/images/open_ondemand/open_od_architecture.svg b/assets/images/open_ondemand/open_od_architecture.svg deleted file mode 100644 index bda6b4363..000000000 --- a/assets/images/open_ondemand/open_od_architecture.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/assets/images/slurm/open_ondemand/dark/ood_wizard_home.png b/assets/images/slurm/open_ondemand/dark/ood_wizard_home.png new file mode 100644 index 000000000..6a033bbca Binary files /dev/null and b/assets/images/slurm/open_ondemand/dark/ood_wizard_home.png differ diff --git a/assets/images/slurm/open_ondemand/dark/ood_wizard_ldap.png b/assets/images/slurm/open_ondemand/dark/ood_wizard_ldap.png new file mode 100644 index 000000000..0483d0dc2 Binary files /dev/null and b/assets/images/slurm/open_ondemand/dark/ood_wizard_ldap.png differ diff --git a/assets/images/slurm/open_ondemand/dark/ood_wizard_network.png b/assets/images/slurm/open_ondemand/dark/ood_wizard_network.png new file mode 100644 index 000000000..c2505dbb3 Binary files /dev/null and b/assets/images/slurm/open_ondemand/dark/ood_wizard_network.png differ diff --git a/assets/images/slurm/open_ondemand/dark/ood_wizard_slurm.png b/assets/images/slurm/open_ondemand/dark/ood_wizard_slurm.png new file mode 100644 index 000000000..f161be4f8 Binary files /dev/null and b/assets/images/slurm/open_ondemand/dark/ood_wizard_slurm.png differ diff --git a/assets/images/slurm/open_ondemand/light/ood_wizard_home.png b/assets/images/slurm/open_ondemand/light/ood_wizard_home.png new file mode 100644 index 000000000..0319f3df4 Binary files /dev/null and b/assets/images/slurm/open_ondemand/light/ood_wizard_home.png differ diff --git a/assets/images/slurm/open_ondemand/light/ood_wizard_ldap.png b/assets/images/slurm/open_ondemand/light/ood_wizard_ldap.png new file mode 100644 index 000000000..cf7d82133 Binary files /dev/null and b/assets/images/slurm/open_ondemand/light/ood_wizard_ldap.png differ diff --git a/assets/images/slurm/open_ondemand/light/ood_wizard_network.png b/assets/images/slurm/open_ondemand/light/ood_wizard_network.png new file mode 100644 index 000000000..66cf71af5 Binary files /dev/null and b/assets/images/slurm/open_ondemand/light/ood_wizard_network.png differ diff --git a/assets/images/slurm/open_ondemand/light/ood_wizard_slurm.png b/assets/images/slurm/open_ondemand/light/ood_wizard_slurm.png new file mode 100644 index 000000000..321a94267 Binary files /dev/null and b/assets/images/slurm/open_ondemand/light/ood_wizard_slurm.png differ diff --git a/assets/images/slurm/open_ondemand/ood_active_jobs.png b/assets/images/slurm/open_ondemand/ood_active_jobs.png new file mode 100644 index 000000000..bdbc716c9 Binary files /dev/null and b/assets/images/slurm/open_ondemand/ood_active_jobs.png differ diff --git a/assets/images/slurm/open_ondemand/ood_files.png b/assets/images/slurm/open_ondemand/ood_files.png new file mode 100644 index 000000000..9cfc8035c Binary files /dev/null and b/assets/images/slurm/open_ondemand/ood_files.png differ diff --git a/assets/images/slurm/open_ondemand/ood_job_composer.png b/assets/images/slurm/open_ondemand/ood_job_composer.png new file mode 100644 index 000000000..802b194af Binary files /dev/null and b/assets/images/slurm/open_ondemand/ood_job_composer.png differ diff --git a/assets/images/slurm/open_ondemand/ood_login.png b/assets/images/slurm/open_ondemand/ood_login.png new file mode 100644 index 000000000..1176519b9 Binary files /dev/null and b/assets/images/slurm/open_ondemand/ood_login.png differ diff --git a/assets/images/slurm/open_ondemand/ood_shell.png b/assets/images/slurm/open_ondemand/ood_shell.png new file mode 100644 index 000000000..e7245c2b9 Binary files /dev/null and b/assets/images/slurm/open_ondemand/ood_shell.png differ diff --git a/content/platform_services/slurm/integrations/_index.md b/content/platform_services/slurm/integrations/_index.md new file mode 100644 index 000000000..a18c88f99 --- /dev/null +++ b/content/platform_services/slurm/integrations/_index.md @@ -0,0 +1,11 @@ +--- +title: "Slurm Integrations" +linkTitle: "Integrations" +date: "2026-10-06" +description: "Connect your Slurm clusters with other tools, such as the Open OnDemand web portal." +categories: +pageintoc: "13" +tags: +type: docs +weight: "3" +--- diff --git a/content/platform_services/slurm/integrations/open_ondemand.md b/content/platform_services/slurm/integrations/open_ondemand.md new file mode 100644 index 000000000..203370409 --- /dev/null +++ b/content/platform_services/slurm/integrations/open_ondemand.md @@ -0,0 +1,301 @@ +--- +title: "Open OnDemand" +linkTitle: "Open OnDemand" +date: "2026-10-06" +description: "" +categories: +tags: +type: docs +weight: "1" +aliases: + - /solutions/integration_blueprints/open_ondemand/ + - /solutions/integration_blueprints/open_ondemand/overview/ + - /solutions/integration_blueprints/open_ondemand/architecture/ + - /solutions/integration_blueprints/open_ondemand/quick_start/ + - /solutions/integration_blueprints/open_ondemand/configuration/ + - /solutions/integration_blueprints/open_ondemand/operations/ + - /solutions/integration_blueprints/open_ondemand/monitoring_and_troubleshooting/ +--- + + +[Open OnDemand](https://openondemand.org/) is a web portal for HPC users. The Open OnDemand appliance runs this portal in one VM, connected to your OneSlurm Clusters. Users sign in with their LDAP account and, from a web browser, they can: + +* Manage the files in their home +* Submit jobs and follow them +* Open a terminal on a Cluster + +The appliance is available in the [OpenNebula Public Marketplace](https://marketplace.opennebula.io/appliance/d3c8ea14-f6f0-455c-8a41-6129affa9a0e). It uses Ubuntu 26.04 LTS and Open OnDemand 4.2. + +## How It Works + +The portal does not run Slurm. It sends the Slurm commands of each user to the controller of the Cluster over SSH. + +The portal and the Clusters share two services: + +* The same LDAP directory, so users have the same account everywhere +* The same NFS export for `/home`, so users see the same files everywhere + +One portal can serve several OneSlurm Clusters. + +### What Happens at Boot + +1. The appliance checks the values of the wizard. +2. It mounts the NFS export on `/home`. +3. It reads the users from LDAP. +4. It creates a self-signed certificate for the IP address of the VM. +5. It adds each Slurm Cluster to the portal and saves the SSH host key of each controller. +6. It starts the portal and publishes its URL in OneGate. + +### What Happens at the First Login of a User + +* The portal creates an SSH key for the user in `~/.ssh/id_ed25519_portal`. +* It adds the key to `~/.ssh/authorized_keys`, accepted only from the IP addresses of the portal. +* The controllers read the same home directory over NFS, so the portal can run `sbatch`, `squeue` and `scancel` as that user. The terminal uses the same key. + +Each user works with their own Unix account, with the same UID as on the Clusters. Users cannot read the files or cancel the jobs of other users. + +## Requirements + +Before you deploy the portal, check that you have the elements in the following table. + +| **Requirement** | **What the portal needs** | +|-------------|-----------------------| +| OneSlurm Clusters | One or more Clusters in `RUNNING` state, and the IP address of each controller. | +| LDAP server | The LDAP server of the Clusters, with a `ldap://` URL. | +| NFS export | The export with the home directories of the users, for example `10.0.0.10:/export/home`. | +| User homes | One home directory for each user on that export, owned by the user, with mode `700`. | +| Virtual Network | A Virtual Network that reaches the controllers, the LDAP server and the NFS server. | +| [OneGate]({{% relref "product/operation_references/opennebula_services_configuration/onegate" %}}) | Reachable from that network. The portal publishes its URL and its errors there. | + +### LDAP Details + +* Use the same domain as `ONEAPP_LDAP_DOMAIN` in OneSlurm. The default is `slurm.local`. +* You can also use a base DN, such as `dc=example,dc=org`. +* Users must be `posixAccount` entries under `ou=People`. +* The portal has no bind user, so the LDAP server must allow anonymous search under `ou=People`. + +The LDAP server of OneSlurm meets all of these points. + +### NFS Details + +* Use the same export as `ONEAPP_SLURM_NFS_HOME` in OneSlurm. +* The NFS server must allow the IP address of the portal. +* The portal uses NFS version 4. The export can use `root_squash`. + +### Size of the VM + +The default VM has 2 vCPU, 4 GB of memory and a 10 GB disk. Each active user needs about 200 MB of memory (measured with six users). Add memory if more users work at the same time. + +## Deploy the Portal + +### Step 1. Import the Appliance + +In Sunstone: + +1. Go to **Storage -> Apps**. +2. Search for **Open OnDemand** and select it. +3. Click **Import** and choose the image datastore. + +Or, with the CLI on the Front-end: + +```shell +onemarketapp export "Open OnDemand" "Open OnDemand" --datastore default +``` + +This imports an image and a VM template, both called **Open OnDemand**. + +### Step 2. Start the Wizard + +1. Go to **Templates -> VM Templates**. +2. Select the **Open OnDemand** template and click **Instantiate**. +3. Enter a name for the VM and click **Next**. + +### Step 3. Fill in the User Inputs + +The wizard has three tabs, **LDAP**, **Home** and **Slurm**. + +#### LDAP Tab + +* In **LDAP URL**, enter the LDAP server of the Clusters, for example `ldap://10.0.0.20`. +* In **LDAP domain or base DN**, keep `slurm.local`, the default of OneSlurm, or enter the domain of your LDAP server. The base DN is the root of the directory. A domain such as `slurm.local` becomes the base DN `dc=slurm,dc=local`, and the portal looks for users under `ou=People` of that base DN. + +{{< image path="/images/slurm/open_ondemand/light/ood_wizard_ldap.png" + pathDark="/images/slurm/open_ondemand/dark/ood_wizard_ldap.png" +alt="LDAP tab of the Open OnDemand wizard" align="center" width="90%" mb="30px" >}} + +#### Home Tab + +* In **NFS export for /home**, enter the export with the home directory of the users, for example `10.0.0.10:/export/home`. + +{{< image path="/images/slurm/open_ondemand/light/ood_wizard_home.png" + pathDark="/images/slurm/open_ondemand/dark/ood_wizard_home.png" +alt="Home tab of the Open OnDemand wizard" align="center" width="90%" mb="30px" >}} + +#### Slurm Tab + +* In **Slurm Clusters**, enter one line for each Cluster as `name:IP`, for example `cpu:10.0.0.20`. The name is what users see in the portal and must be unique. The IP is the IPv4 address of the controller. Every Cluster must use the same LDAP server and the same `/home` as the portal. + +{{< image path="/images/slurm/open_ondemand/light/ood_wizard_slurm.png" + pathDark="/images/slurm/open_ondemand/dark/ood_wizard_slurm.png" +alt="Slurm tab of the Open OnDemand wizard" align="center" width="90%" mb="30px" >}} + +### Step 4. Attach the Network of the Clusters + +1. Click **Next**. +2. In **Advanced options**, open the **Network** tab. +3. Click **Attach NIC** and select the Virtual Network of the Clusters. +4. Click **Finish**. + +{{< image path="/images/slurm/open_ondemand/light/ood_wizard_network.png" + pathDark="/images/slurm/open_ondemand/dark/ood_wizard_network.png" +alt="Network tab of the Open OnDemand wizard with the NIC of the Clusters" align="center" width="90%" mb="30px" >}} + +The portal uses the IPv4 address of the first NIC. Users must be able to reach that address with their web browser. + +### Deploy with the CLI + +Pass the same values given in the above Sunstone examples with `--user-inputs`. Separate the Clusters with spaces: + +```shell +onetemplate instantiate "Open OnDemand" --name open-ondemand --nic \ + --user-inputs "ONEAPP_LDAP_SERVER_URL=ldap://10.0.0.20,ONEAPP_LDAP_SERVER_DOMAIN=slurm.local,ONEAPP_HOME_NFS_EXPORT=10.0.0.10:/export/home,ONEAPP_SLURM_CLUSTERS_LIST=cpu:10.0.0.20 gpu:10.0.0.30" +``` + +### Step 5. Check That the Portal Is Ready + +Open the VM in Sunstone and go to **Info -> Attributes**. With the CLI, run `onevm show `: + +| **Attribute** | **Meaning** | +|-----------|---------| +| `READY` = `YES` | The portal is ready. | +| `OOD_URL` | The address of the portal, for example `https://10.0.0.50/`. | +| `OOD_ERROR` | The configuration failed. The value says why, and the portal stays closed. | + +Examples of `OOD_ERROR`: + +* `Cannot mount 10.0.0.10:/export/nope on /home` +* `The controller 10.0.0.30 does not answer on the SSH port` + +| **Cause of the error** | **How to fix it** | +|--------------------|---------------| +| A service was down | Fix the service and reboot the VM. | +| A value was wrong | Change it as shown in [Change the Clusters](#change-the-clusters), then reboot the VM so that OneGate shows the new result. | + +## Use the Portal + +1. Open the address of `OOD_URL` in a web browser. +2. Accept the certificate warning. The certificate is self-signed. +3. Sign in with the LDAP user name and password. + +{{< image path="/images/slurm/open_ondemand/ood_login.png" +alt="Login page of Open OnDemand" align="center" width="80%" mb="20px" >}} + +### Tools for Users + +| **Menu** | **Tool** | **What users do** | +|------|------|---------------| +| Files | Home Directory | Browse, upload, download and delete files. | +| Jobs | Active Jobs | See their jobs on every Cluster and cancel them. | +| Jobs | Job Composer | Create a job from a script or a template and submit it to a Cluster. | +| Jobs | Project Manager | Group job scripts in project folders. | +| Clusters | `` Shell Access | Open a terminal on the controller of a Cluster. | +| Clusters | System Status | See the nodes, CPUs and jobs of each Cluster. | + +The portal has no remote desktop, because the OneSlurm workers have no VNC server. + +{{< image path="/images/slurm/open_ondemand/ood_files.png" +alt="Files app of Open OnDemand with the home of the user" align="center" width="90%" mb="20px" >}} + +* A file uploaded in **Files** is on every Cluster at once, because they share `/home`. +* **Open in Terminal** opens a terminal on the first Cluster of the list. + +{{< image path="/images/slurm/open_ondemand/ood_shell.png" +alt="Terminal on the controller of a Cluster, with a job submitted with sbatch" align="center" width="90%" mb="20px" >}} + +Jobs submitted from the terminal or from the Job Composer appear in **Active Jobs**. + +{{< image path="/images/slurm/open_ondemand/ood_active_jobs.png" +alt="Active Jobs with a running job on the cpu Cluster" align="center" width="90%" mb="20px" >}} + +{{< image path="/images/slurm/open_ondemand/ood_job_composer.png" +alt="Job Composer with jobs on two Clusters" align="center" width="90%" mb="20px" >}} + +## Add Users + +1. Create the user in the LDAP server of the Clusters. +2. Create the home directory on the NFS export, owned by the user, with mode `700`. + +The portal needs no change. If the user signs in before the home directory exists, the portal shows **Home directory not found**. Create the home directory, then the user clicks **Restart Web Server** on that page. + +## Change the Clusters + +You can add or remove Clusters on a running portal. + +Before you add a Cluster: + +* Check that its controller is running. The portal saves its SSH host key at this step. +* If the Cluster is on another network, attach that network to the VM first. + +Then: + +1. In Sunstone, open the VM. +2. Go to **Configuration -> Update configuration -> Context -> Context Custom Variables**. +3. Edit `ONEAPP_SLURM_CLUSTERS_LIST`. Use one Cluster per line, or separate them with spaces. +4. Save the change. + +The appliance configures the portal again in about three minutes. Removed Clusters disappear from the portal, and new Clusters appear in its menus. + +{{< alert title="Note" type="info" >}} +After **Update configuration**, OneGate does not answer the VM until it reboots, so `OOD_URL` and `OOD_ERROR` keep their old values. Read the result in `/var/log/one-appliance/configure.log` in the VM. This is a known issue of the `one-context` package 7.4.0. +{{< /alert >}} + +## Security + +### Network Ports + +Limit the traffic of the portal, for example with a [security group]({{% relref "product/virtual_machines_operation/virtual_machines_networking/security_groups" %}}). + +| **Direction** | **Port** | **Use** | +|-----------|------|-----| +| Inbound | TCP 443 | Users open the portal. | +| Inbound | TCP 80 | Redirects to 443. | +| Inbound | TCP 22 | Administrators only. | +| Outbound | TCP 22 | SSH to the controllers. | +| Outbound | TCP 389 | LDAP server. | +| Outbound | TCP 2049 | NFS server. | +| Outbound | OneGate port | OneGate. | + +Do not open port 5558. The login service of the portal (Dex) shows metrics there without authentication, and Open OnDemand does not allow changing it. + +### LDAP Without TLS + +The portal connects to LDAP with `ldap://`, like OneSlurm. Passwords travel without encryption between the portal and the LDAP server. Keep the portal, the Clusters and the LDAP server on a private network. + +### SSH to the Portal VM + +Only `root` can open an SSH session on the portal VM, with the key of the OpenNebula user that created it. LDAP users cannot use SSH or copy files to the portal VM. They use the portal or a Cluster controller. + +## Troubleshooting + +| **Problem** | **What to do** | +|---------|------------| +| The VM shows `OOD_ERROR` | Fix the value or the service named in the message, then reboot the VM. Full log in `/var/log/one-appliance/configure.log`. | +| The browser does not open the portal | Check that the VM shows `READY=YES` and that your network reaches the IP of `OOD_URL` on port 443. Logs: `journalctl -u apache2 -u ondemand-dex`. | +| Login fails with the right password | The user is not under `ou=People`, or LDAP does not answer. Test in the VM with `ldapsearch -x -H -b ou=People, uid=`. For `slurm.local`, the base DN is `dc=slurm,dc=local`. | +| Home directory not found | Create the home directory of the user, then click **Restart Web Server** on that page. | +| Active Jobs is empty, or `Permission denied` in Submit or the terminal | Click **Help -> Restart Web Server**. This adds the key of the user to `~/.ssh/authorized_keys` again. Logs: `journalctl -t ood-prehook`. | +| System Status shows an error page | A controller does not answer. That Cluster shows no jobs, and the Job Composer and the terminal show an SSH error. Everything works again when the controller is back. | +| Pages load forever with no message | The NFS server does not answer. The portal works again by itself when NFS is back. Meanwhile, use the VM console in Sunstone, because SSH to the VM can also hang. | + +## Known Limitations + +| **Area** | **Limitation** | **What to do** | +|------|------------|------------| +| Files | Uploads are limited to 256 MiB. | Copy bigger files with `scp` or `rsync` to a Cluster controller, which has the same home. | +| Terminal | The terminal closes when the page reloads. | Run long work under `tmux` or `screen` on the controller. | +| Job Composer | If the first visit is a direct link to `/pun/sys/myjobs/`, it creates an empty database and shows error 500 from then on. | Open it the first time from the **Jobs** menu. To fix it, delete `~/ondemand/data/sys/myjobs/production.sqlite3` of that user (only if the file has no tables) and open the Job Composer from the menu. | +| Job Composer | **New Job -> From Template** creates a job without a script. | Open **Job Options**, select the script and click **Save**. | +| Job states | OneSlurm has no Slurm accounting, so a cancelled job shows `Completed` in Active Jobs and `Failed` in the Job Composer. | Check the real state with `scontrol show job ` or in the output file of the job. | +| Project Manager | Deleting a project only removes it from the list. The files stay. | Delete the files in **Files**. | +| Two portals | With two portals on the same homes, the Job Composer fails with error 500 when a user opens it on both at the same time. | Use one portal for each NFS home export. | +| New Clusters | A new Cluster needs a running controller. If it does not answer, the configuration stops with `OOD_ERROR`. | Start the controller first. Clusters the portal already knows can be stopped for maintenance. | diff --git a/content/solutions/integration_blueprints/open_ondemand/_index.md b/content/solutions/integration_blueprints/open_ondemand/_index.md deleted file mode 100644 index c1448b550..000000000 --- a/content/solutions/integration_blueprints/open_ondemand/_index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: "Open OnDemand" -linkTitle: "Open OnDemand" -date: "2026-09-15" -description: "" -categories: -pageintoc: "14" -tags: -type: docs -weight: "1" ---- - -[Open OnDemand](https://openondemand.org/) is a web portal for HPC users. They sign in from a browser and open JupyterLab, RStudio, VS Code, a desktop or a shell, and submit batch jobs. The Open OnDemand Service runs it on OpenNebula as a OneFlow service from the Community Marketplace, with its own Slurm cluster inside. Every session is a Slurm job on a pool of compute VMs that grows and shrinks with the queue. diff --git a/content/solutions/integration_blueprints/open_ondemand/architecture.md b/content/solutions/integration_blueprints/open_ondemand/architecture.md deleted file mode 100644 index 5de8b8ffd..000000000 --- a/content/solutions/integration_blueprints/open_ondemand/architecture.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: "Service Architecture" -linkTitle: "Service Architecture" -date: "2026-09-15" -description: -categories: -tags: -weight: "2" -type: docs ---- - -The Open OnDemand Service has three roles and uses two Virtual Networks. Each user session runs as a Slurm job on a pool of compute VMs that grows and shrinks. - -## Roles - -| **Role** | **Cardinality** | **What it runs** | -|---|---|---| -| `storage` | 1 | NFS server for the shared home directories, the state of the Slurm controller and the shared software directory. Squid cache for the EESSI catalog, unless the service uses a proxy of the site | -| `portal` | 1 | Open OnDemand, its LDAP directory, Dex, the Slurm controller `slurmctld`, the accounting daemon `slurmdbd` with MariaDB, the Prometheus metrics exporter | -| `worker` | 1 to 6, elastic | `slurmd`, joined to the Cluster as a dynamic node, and the user sessions as Slurm jobs | -{.w-100} - -{{< image path="/images/open_ondemand/open_od_architecture.svg" -alt="The storage, portal and worker roles between the management and compute networks, with the flows between them" align="center" width="100%" mb="20px" border="false" shadow="false" >}} - -OneFlow starts `storage` first. It starts `portal` when the storage role is ready, and `worker` when both are ready. Each role reports `READY` through OneGate only when it is actually serving. The service therefore reaches `RUNNING` when a user can sign in and open a session. - -## Networks - -Every VM has two network interfaces: - -* **Management network**: Facilitates communication with OneGate and the Internet. The portal publishes its web interface here, and the storage role downloads the software catalog through it. -* **Compute network**: Reserved for the service. It carries NFS, LDAP, the Slurm traffic between the controller and the nodes, along with the proxied sessions and the software cache. - -No address is fixed in advance. OneFlow provides each role the compute addresses of the roles it depends on. The storage role queries OneGate which VM is the portal. - -A worker joins the Slurm Cluster of its own accord at boot. A reconciler on the portal removes the node of any VM that has left the service. - -The portal uses the whole `/24` subnet containing its compute address as the worker range, so keep that network reserved for the service. For larger subnets, configure `ONEAPP_POOL_RANGE` as described in [Configuration]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/#advanced-attributes" %}}). - -## How a Session Runs - -1. The user signs in. Dex checks the credentials against the LDAP directory, or against an external OpenID Connect provider. Open OnDemand then starts a web server for that user, running as their Unix account. -2. The user launches an application and picks cores, memory and session hours. Cores and memory are limited to the capacity of the largest worker. The portal submits the session script with `sbatch` as the user. It requests one node, the cores and memory chosen, and GPUs if the pool has any. -3. Slurm starts the script on a worker with sufficient cores and memory free. If no adequate worker is available, the job remains in the queue and the card shows it as queued until a worker becomes available or OneFlow adds one. The job runs as the user under `slurmd`, with the home directory and the EESSI catalog mounted. Slurm uses cgroup v2 to limit the job to the cores and memory it requested. No other session shares its cores, and any process that uses more memory than requested is terminated. -4. The application listens on a specific port of the worker and publishes the compute address of the worker. The portal proxies the browser to it. Desktops run under a TurboVNC server on the worker and reach the browser through noVNC on the portal. -5. Deleting the session cancels the job. Reaching the session hours ends it. In both cases Slurm stops every process of the job, so no trace of the session stays on the worker. - -A job is never transferred to another worker, since the browser connection points to the worker that started it. Every job is recorded in the accounting database. **Active Jobs** lists the sessions, and the **Job Composer** submits batch jobs of your own to the same Cluster. - -## How the Worker Pool Grows and Shrinks - -Every worker publishes the following attributes to OneGate every 30 seconds: - -| **Attribute** | **Meaning** | -|---|---| -| `SLURM_PENDING` | Jobs waiting for a worker of this role, the same figure on every worker | -| `ACTIVE_SESSIONS` | Jobs currently running on this worker | -| `IDLE_SECONDS` | Seconds since the last job on this worker ended | -| `OLDEST_IDLE` | `1` when the oldest worker of the role is drained and empty, so OneFlow may remove it | -| `SLURM_IDLE_NODES`, `SLURM_ALLOC_NODES` | Nodes of the Cluster without a job, and nodes with one | -| `HEALTHY` | `1` when the home mount, the software catalog, munge and `slurmd` are all ready | -| `SESSION_USERS` | The user and the start time of each job on this worker | -| `SLURM_NODENAME` | The node name of this worker, published once at boot. The reconciler on the portal matches it against the Cluster | -{.w-100} - -OneFlow reads the figures of every worker and applies the rules of the worker role. The numbers are from the marketplace template. - -### Scaling Up - -OneFlow adds one worker when `SLURM_PENDING` is above 0 in two consecutive reads, and then waits 120 seconds before it reads again, so one pending job adds one VM and that VM has time to boot. OneFlow reads the figures every `autoscaler_interval` seconds, 90 by default in `/etc/one/oneflow-server.conf` on the Front-end. With the 30-second interval that the [Requirements](#requirements) recommend, a pending job normally receives its worker within about a minute and a half, and each further pending job a few minutes later. A job that no worker could ever run does not count. A GPU request on a pool without GPUs is refused at submit time, and a job that asks for more cores than any worker has waits with the reason `PartitionConfig`, which the workers leave out of `SLURM_PENDING`. - -### Scaling Down - -OneFlow always removes the oldest VM of a role, without draining it, so the appliance drains first. When the oldest worker has had no job for `ONEAPP_WORKER_IDLE_SECONDS` (ten minutes by default) and the queue is empty, it drains its own Slurm node. While that node is drained and empty, every worker publishes `OLDEST_IDLE` as `1`, and OneFlow removes the VM when the average remains above 0.99 in two reads 60 seconds apart. The node leaves the Cluster when the VM shuts down. - -Three guards apply: - -* A worker never drains while the role is at `min_vms`. -* A drained node returns to service when a job appears, or when OneFlow has not removed it after `ONEAPP_WORKER_DRAIN_SECONDS` (ten minutes by default). -* A reconciler on the portal runs every 30 seconds and deletes any node whose VM has left the service and holds no job. - -On the testbed a scale down removed exactly the drained VM, and the job on the other worker kept running. The pool changes one VM at a time between 1 and `max_vms`. 6 is set as default in the Open OnDemand marketplace template. - -## Requirements - -* OpenNebula 6.10 or later with [OneFlow]({{% relref "product/operation_references/opennebula_services_configuration/oneflow/" %}}) and [OneGate]({{% relref "product/operation_references/opennebula_services_configuration/onegate/" %}}) enabled, and the OneGate endpoint reachable from the service networks. -* The two Virtual Networks defined [above](#networks). The compute network is reserved for the service and no larger than a `/24` subnet, unless you set `ONEAPP_POOL_RANGE`. If a firewall sits between the networks, the roles need NFS (2049) and the software cache (3128) on the storage VM, LDAP (389) and the Slurm controller (6817) on the portal, `slurmd` (6818) and the session ports on the workers, and OneGate (5030 by default) from every VM. -* Outbound HTTP from the storage role to the EESSI CernVM-FS servers. -* Capacity for three VMs plus the workers you expect. The marketplace template gives every VM 2 vCPU and 4 GB of memory, and 8 GB to the portal, which runs the Slurm controller and MariaDB along with Open OnDemand. The portal used 710 MB of its 7941 MB with everything running on the testbed. The VM template passes the CPU of the Host through to the VMs (`CPU_MODEL` set to `host-passthrough`), so EESSI loads the software built for that CPU family. Every worker of a role has the same size, and a session is prohibited from requesting more resources than a single worker is allocated. -* For a faster scale-up, set `:autoscaler_interval: 30` in `/etc/one/oneflow-server.conf` on the Front-end and restart `opennebula-flow`. OneFlow then reads the figures of the workers every 30 seconds instead of 90. This is a setting of the Front-end, made once for every service that runs on it. It cannot be set through the appliance. - -## The Munge Key - -Munge is how the portal and the workers manage credentials. Every Slurm message carries a token signed with one secret key that all VMs share. The portal creates the key at first boot (1024 random bytes) and keeps it on the storage export, so a replaced portal uses the same key. It also publishes the key, base64 encoded, as `SLURM_MUNGE_KEY` in the user template of the portal VM through OneGate, and every worker reads it from there at boot. - -Anyone who can read the template of the portal VM in OpenNebula can read the key, and with the key they can interact with the Cluster as if they were any given user. The Managed Slurm service has the same exposure. Therefore it is important to restrict who can inspect the VMs of the service. Inside the VMs the key and the OneGate token are readable by root only. Therefore a session user cannot read them. diff --git a/content/solutions/integration_blueprints/open_ondemand/configuration.md b/content/solutions/integration_blueprints/open_ondemand/configuration.md deleted file mode 100644 index 20ebd504b..000000000 --- a/content/solutions/integration_blueprints/open_ondemand/configuration.md +++ /dev/null @@ -1,202 +0,0 @@ ---- -title: "Configuration" -linkTitle: "Configuration" -date: "2026-09-15" -description: -categories: -tags: -weight: "4" -type: docs ---- - -This section describes the inputs of the Open OnDemand Service. It also covers role sizing, the worker pool, worker sizes, the Slurm Cluster, using an external Slurm Cluster, using an external identity provider and user management. - -## Service Inputs - -All inputs are `ONEAPP_*` context variables. OneFlow places every input in the context of every VM of the service, where root can read it. This includes `ONEAPP_PORTAL_CERTIFICATE_KEY` and `ONEAPP_AUTH_OIDC_CLIENT_SECRET`. Every input is optional, and the Sunstone wizard groups them into four tabs, **Portal**, **Users and login**, **Home directories** and **Software catalogue**. A feature with an `_ENABLED` switch shows its other inputs only while the switch is set to `YES`, and ignores them while it is set to `NO`. If a switch is set to `YES` and a required field is empty, the role stops at boot, and the message names the field. Refer to [A Role Does Not Reach RUNNING]({{% relref "solutions/integration_blueprints/open_ondemand/monitoring_and_troubleshooting/#a-role-does-not-reach-running" %}}) in the troubleshooting documentation. - -| **Tab** | **Input** | **Default** | **Description** | -|---|---|---|---| -| Portal | `ONEAPP_PORTAL_HOST_NAME` | empty | Public Host name of the portal, resolving to its management address. When empty, the portal answers on that address. | -| Portal | `ONEAPP_PORTAL_LETSENCRYPT_ENABLED` | `NO` | Request a Let's Encrypt certificate for the Host name. The name has to resolve to the portal. Ports 80 and 443 have to be reachable from the Internet when the portal boots. On failure, the portal keeps its self-signed certificate. | -| Portal | `ONEAPP_PORTAL_CERTIFICATE_ENABLED` | `NO` | Use a certificate of your own instead of the self-signed one. | -| Portal | `ONEAPP_PORTAL_CERTIFICATE_CHAIN`, `ONEAPP_PORTAL_CERTIFICATE_KEY` | empty | PEM certificate chain and private key, both required when the switch is on. | -| Users and login | `ONEAPP_AUTH_LOCAL_USERS` | `demo1:demo1pass` | Initial users, `user:password` separated by spaces, created in the directory of the portal at first boot. A uid may follow, `user:password:uid`, to match accounts that exist elsewhere. The others get the next free number from 10001. | -| Users and login | `ONEAPP_AUTH_OIDC_ENABLED` | `NO` | Also allow sign in through an OpenID Connect provider, see [External Identity Provider](#external-identity-provider). | -| Users and login | `ONEAPP_AUTH_OIDC_ISSUER`, `ONEAPP_AUTH_OIDC_CLIENT_ID`, `ONEAPP_AUTH_OIDC_CLIENT_SECRET` | empty | Issuer URL, client id and client secret registered at the provider. The issuer and the client id are required when the switch is on. The secret may stay empty if the provider allows public clients. | -| Users and login | `ONEAPP_AUTH_OIDC_NAME` | `Institutional login` | Name of the provider on the login page. | -| Home directories | `ONEAPP_HOME_NFS_ENABLED` | `NO` | Use an NFS server of your own for the home directories instead of the storage role, which then keeps only the software cache. | -| Home directories | `ONEAPP_HOME_NFS_SERVER` | empty | Address of that NFS server. Required when the switch is on. | -| Home directories | `ONEAPP_HOME_NFS_EXPORT` | `/export/home` | Path of the home export on that server. Ignored while the switch is off, the storage role then exports `/export/home`. | -| Software catalogue | `ONEAPP_SOFTWARE_PROXY_ENABLED` | `NO` | Use a CernVM-FS proxy of your own instead of the cache on the storage role. The portal and the workers then fetch the EESSI catalogue through that proxy, and the storage role runs no cache. | -| Software catalogue | `ONEAPP_SOFTWARE_PROXY_URL` | empty | URL of that proxy, for example `http://proxy.example.org:3128`. Required when the switch is on, and reachable from the compute network. | -{.w-100} - -You can also set the Host name, the OpenID Connect inputs and the external Slurm attributes on a running portal with `onevm updateconf`. The portal reconfigures itself in under a minute. The portal keeps the certificate it already has for its Host name. A certificate switch changed later therefore takes effect only together with a new Host name. To apply it under the same Host name, remove `/etc/ood/ssl/.crt` and `.key` on the portal. Without a Host name, the files are named after the address. The next reconfiguration issues the certificate again. - -## Advanced Attributes - -These attributes never appear in the wizard. Set them in the `template_contents` of a role in the service template, or in the `CONTEXT` of a standalone portal VM. A value set in a role reaches that role only. The table names the role that reads each attribute: - -| **Attribute** | **Default** | **Description** | -|---|---|---| -| `ONEAPP_WORKER_IDLE_SECONDS` | `600` | Seconds the oldest worker waits without a job before it drains its Slurm node and leaves the pool. The worker role reads it. | -| `ONEAPP_WORKER_DRAIN_SECONDS` | `600` | Seconds a drained worker waits for OneFlow to remove it before it returns its node to service. The worker role reads it. | -| `ONEAPP_POOL_RANGE` | derived | Address range the portal accepts workers from, as `first-last`. It also limits the number of Slurm nodes. Inside a service, the portal derives it from its compute interface, the whole `/24` subnet including its address. Set it only for a compute network larger than a `/24` subnet, or for a portal outside a OneFlow service. The range has to stay inside one `/24` subnet, because a session publishes the worker address that starts with the first three octets of the range. On a larger compute network, reserve a `/24` subnet slice of it for the workers and give that slice here. The portal role reads it. | -| `ONEAPP_SLURM_STATE_EXPORT` | `/export/slurm` | Export of the storage role that keeps the state of the Slurm controller, the munge key and the accounting dumps. The portal role mounts it. | -| `ONEAPP_SLURM_DEF_MEM_PER_CPU` | `1024` | Memory in MB a job gets per core when it requests none. The portal role reads it. | -| `ONEAPP_SLURM_CONTROLLER_ENABLED` | `NO` | Add a second Slurm Cluster of the site for batch jobs, see [An External Slurm Cluster](#an-external-slurm-Cluster). The portal role reads it. | -| `ONEAPP_SLURM_CONTROLLER_HOST` | empty | Address or Host name of that controller. Required when the switch is on. The portal role reads it. | -| `ONEAPP_SLURM_TITLE` | `External Slurm` | Name of that Cluster in the Job Composer and Active Jobs. The portal role reads it. | -{.w-100} - -## Sizing the Roles - -The marketplace template gives every VM 2 vCPU and 4 GB of memory, and 8 GB for the portal. Every session is a Slurm job that reserves the cores and the memory its form requests, and a worker takes sessions until its cores or its memory are fully reserved. Size the worker role for the sessions one VM should hold. OneFlow always adds workers of the size set in the worker role, never a bigger one, and the forms offer at most the resources of the largest worker. To offer bigger sessions, raise the memory or the CPU parameter of the worker role, or add a second worker role of another size. Refer to [Worker Sizes](#worker-sizes). - -The portal runs the Slurm controller, its accounting daemon and MariaDB beside Open OnDemand, and uses about 700 MB of memory when idle. To change a size, edit the service template before instantiating it and set `CPU`, `VCPU` and `MEMORY` in the `template_contents` of the role. - -## Scaling the Worker Pool - -The worker pool scales automatically. Refer to [How the Worker Pool Grows and Shrinks]({{% relref "solutions/integration_blueprints/open_ondemand/architecture/#how-the-worker-pool-grows-and-shrinks" %}}) for more details. The workers publish `SLURM_PENDING`, the number of jobs waiting for a worker of their role, and OneFlow adds one VM when the value stays above 0 for two periods of 30 seconds. The oldest worker drains its Slurm node once it has had no job for `ONEAPP_WORKER_IDLE_SECONDS` and nothing is pending, publishes `OLDEST_IDLE=1` while the node is drained and empty, and OneFlow removes it after two periods of 60 seconds. `ONEAPP_WORKER_DRAIN_SECONDS` undoes a drain when OneFlow does not remove the worker in time, and the last worker of a role never drains. Both are [Advanced Attributes](#advanced-attributes) of the worker role. - -For a pool larger than six VMs, raise `max_vms` of the worker role in the service template. OneFlow refuses a scale command during the cooldown after each scale operation, 300 seconds after a policy acts and 120 seconds after a manual scale. To set the size manually while the service is `RUNNING`: - -```shell -oneflow scale worker -``` - -## Worker Sizes - -Every role whose name starts with `worker` is a pool of session VMs. To offer a larger VM, copy the `worker` role in the service template under a new name, such as `worker_large`. Give it the CPU and memory required, and the same parents and elasticity policies. Each worker registers its Slurm node with the name of its role as a feature. A session that chooses a size is submitted with `--constraint=` and runs only on a worker of that role. A job with no constraint takes any worker. The application forms show a **Worker size** field only when several worker roles exist. - -{{< image path="/images/open_ondemand/light/jupyter_form_sizes.png" -alt="The Jupyter form offering two worker sizes" align="center" width="90%" mb="20px" >}} - -Each size scales automatically and keeps at least one VM, because OneFlow scales a role from the metrics of its VMs. The workers count only the pending jobs their role can serve. A GPU size is a `worker_gpu` role whose `template_contents` also carries the PCI device of the Host, as the [NVIDIA GPU passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough/" %}}) page describes. A worker that boots with `/dev/nvidia0` registers its node with the GPU as a Slurm resource. The forms then show a **GPUs** field, and a session that requests a GPU is submitted with `--gres=gpu:`. The image ships no driver, so the site installs it on that role. - -{{< alert title="Warning" type="warning" >}} -The GPU role has not been tested on a GPU. Run `nvidia-smi` inside a session before offering the size to users. -{{< /alert >}} - -## Batch Jobs with Slurm - -The service runs its own Slurm Cluster, always on. The portal role runs `slurmctld`, `slurmdbd` and MariaDB beside Open OnDemand, and keeps the controller state on the `/export/slurm` export of the storage role. Every worker joins the partition `main` as a dynamic node, with the name of its role as a feature. Every interactive session is a Slurm job, so batch jobs and sessions share one queue and no two of them share a core. The portal offers the Cluster as `Slurm` in the Job Composer and Active Jobs. `sbatch`, `squeue`, `scancel`, `sinfo`, `sacct` and `scontrol` run on the portal itself, so the Shell app submits a job as the user: - -```shell -sbatch -c 1 --mem=512M -t 5 --wrap hostname -squeue -sacct -j -``` - -The job runs on a worker with the cores and the memory it requested. A job without `--mem` gets `ONEAPP_SLURM_DEF_MEM_PER_CPU` MB per core. The partition gives a job one hour when it sets no time limit, and allows up to twelve hours. A job that awaits a worker grows the pool. Refer to [Scaling the Worker Pool](#scaling-the-worker-pool). A job does not grow the pool if no worker could run it. A GPU request on a pool without one is refused upon submission with `Requested node configuration is not available`. A request for more cores than the largest worker waits with reason `PartitionConfig`. Accounting runs through `slurmdbd` on the portal, so `sacct` keeps the history of every job and session. The portal dumps the database to the storage export every 30 minutes, and a new portal restores the newest dump when it boots. - -{{< image path="/images/open_ondemand/light/job_composer_slurm.png" -alt="A job submitted from the Job Composer and completed on the Slurm Cluster" align="center" width="90%" mb="20px" >}} - -### MPI Jobs on Several Workers - -A batch job can use several workers at once through MPI. The EESSI catalogue provides [OpenMPI](https://www.open-mpi.org/), and the image ships the PMIx library that `srun` uses to start processes. A job script loads the module and starts the program with `srun --mpi=pmix` or with `mpirun`: - -```shell -#!/bin/bash -l -#SBATCH -N 2 --ntasks-per-node=1 -c 1 --mem=512M -t 10 -module load OpenMPI/5.0.8-GCC-14.3.0 -srun --mpi=pmix ./hello -``` - -On the testbed, a two-node program compiled with `mpicc` from EESSI ran on both workers with `srun --mpi=pmix` and with `mpirun`. The interactive applications use one worker each. - -### An External Slurm Cluster - -The service runs its own Slurm Cluster and does not require an existing Slurm Cluster. If your site already runs a second Slurm Cluster, such as the [Managed Slurm]({{% relref "platform_services/slurm/" %}}) service, you can attach it for batch jobs. It has to share the users and the home directories with the portal. The portal offers it in the Job Composer and Active Jobs under the name in `ONEAPP_SLURM_TITLE`. The interactive applications keep running on the service's own Cluster. `ONEAPP_SLURM_CONTROLLER_ENABLED` and `ONEAPP_SLURM_CONTROLLER_HOST` are [Advanced Attributes](#advanced-attributes) of the portal role. A running portal accepts them with `onevm updateconf`. - -1. Note the compute addresses of the portal and storage VMs of the running service, the second address of each in `onevm list --list ID,NAME,IP`. -2. Instantiate `OneSlurm` on the same compute network with these values for four of its service inputs. They are not environment variables. In Sunstone they are fields of the instantiate wizard of the OneSlurm service template, and from the command line they go in the JSON file that `oneflow-template instantiate` takes, which has to list every input of the template, the others at their defaults: - - | Service input | Value | - |---|---| - | `ONEAPP_LDAP_ENABLE` | `NO` | - | `ONEAPP_LDAP_DOMAIN` | `ood.local` | - | `ONEAPP_LDAP_URL` | `ldap://` | - | `ONEAPP_SLURM_NFS_HOME` | `:/export/home` | - {.w-100} - - ```shell - oneflow-template instantiate 'OneSlurm' oneslurm.json - ``` - - ```json - { - "networks_values": [ { "Service": { "id": "" } } ], - "custom_attrs_values": { - "ONEAPP_LDAP_ENABLE": "NO", - "ONEAPP_LDAP_DOMAIN": "ood.local", - "ONEAPP_LDAP_URL": "ldap://", - "ONEAPP_SLURM_NFS_HOME": ":/export/home" - } - } - ``` - -3. Once the OneSlurm service is `RUNNING`, declare the Cluster on the portal with the compute address of the controller: - - ```shell - onevm updateconf --append <` lists the recipes that EasyBuild ships, more than a thousand. A recipe of your own is a text file like the example, with the name, the version, the source URL, the toolchain and the dependencies of the package. Use a toolchain that EESSI provides, for example `GCCcore/14.3.0` or `foss/2025a`, so the build reuses the compiler and the libraries of the catalogue. The shared directory belongs to the service and is deleted with it, so keep the recipes and build them again on a new service. - -## External Identity Provider - -When `ONEAPP_AUTH_OIDC_ENABLED` is `YES`, the login page offers the provider in `ONEAPP_AUTH_OIDC_ISSUER`, `ONEAPP_AUTH_OIDC_CLIENT_ID` and `ONEAPP_AUTH_OIDC_CLIENT_SECRET` beside the local directory. It appears under the name in `ONEAPP_AUTH_OIDC_NAME`. Register `https:///dex/callback` as the redirect URI at the provider. The account name is the `preferred_username` claim. When the provider sends no such claim, it is the part of the email before the @ sign. The session runs as a Unix user with a home directory. A user who signs in this way therefore needs an entry in the LDAP directory under that name. - -## Managing Users - -Users live in the LDAP directory of the portal role. The portal generates the administrator password at first boot and keeps it in `/etc/one-ondemand/ldap-admin.pass`, readable by root only. To add a new user, run the following on the portal VM command line, with the appropriate details replaced: - -```shell -ldapadd -x -D cn=admin,dc=ood,dc=local -y /etc/one-ondemand/ldap-admin.pass <`. The user can sign in immediately, and the workers resolve users against the same directory. diff --git a/content/solutions/integration_blueprints/open_ondemand/monitoring_and_troubleshooting.md b/content/solutions/integration_blueprints/open_ondemand/monitoring_and_troubleshooting.md deleted file mode 100644 index 78274c255..000000000 --- a/content/solutions/integration_blueprints/open_ondemand/monitoring_and_troubleshooting.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: "Monitoring and Troubleshooting" -linkTitle: "Monitoring and Troubleshooting" -date: "2026-09-15" -description: -categories: -tags: -weight: "6" -type: docs ---- - -The Open OnDemand Service reports its state through OneFlow, OneGate and a Prometheus endpoint on the portal. - -## Metrics - -The portal serves Prometheus metrics on port 9101 of its management address, at `http://:9101/metrics`: - -| **Metric** | **Meaning** | -|---|---| -| `ood_worker_active_sessions` | Slurm jobs running on the worker, sessions included (labels `vm_id`, `name`, `address`) | -| `ood_worker_idle_seconds` | Seconds since the last job on the worker ended | -| `ood_worker_oldest_idle` | `1` when the oldest worker of the role is drained and empty, so OneFlow may remove it | -| `ood_slurm_pending` | Jobs waiting for a worker of this role, as seen by the worker | -| `ood_slurm_idle_nodes`, `ood_slurm_alloc_nodes` | Idle nodes of the Cluster, and nodes with a job, as seen by the worker | -| `ood_worker_healthy` | `1` when the worker passed its last check of the home mount, the software catalogue, munge and slurmd | -| `ood_role_cardinality` | VMs in each role | -| `ood_portal_puns` | Per-user web servers running on the portal, one per signed in user | -| `ood_service_state` | The OneFlow state of the service, as its numeric code (`2` is RUNNING) | -| `ood_exporter_scrape_ok` | `1` when the exporter's last read of the service through OneGate succeeded | -{.w-100} - -The same values are in the user template of each worker VM, in Sunstone or in `onevm show `. The template also holds `SESSION_USERS`, the users with a job on the worker. `SLURM_NODENAME` in the same template is the name of its Slurm node. A worker whose home mount, software catalogue, munge or slurmd fails publishes `HEALTHY=0` until it recovers. On the portal, `sinfo` lists every worker as a node with its role as a feature. `squeue` lists the sessions and the batch jobs. - -## Logs - -| **Location** | **Description** | -|-----------|--------------| -| `/var/log/ood-appliance-configure.log` | What each role did at boot, on every VM | -| `/etc/one-appliance/status` | `bootstrap_success` when the role is ready | -| `/var/log/ondemand-nginx//error.log` | The user's web server, on the portal | -| `/var/log/apache2/` | The front Apache and the login flow, on the portal | -| `/var/log/slurm/slurmctld.log` | The Slurm controller, on the portal | -| `/var/log/slurm/slurmd.log` | The Slurm node daemon, on every worker | -| `~/ondemand/data/sys/dashboard/batch_connect/sys//output//output.log` | What a session printed, in the user's home | - -On the workers, `journalctl -t ood-slurm-elastic` shows what the elastic loop published and what the health check found. On the portal, `journalctl -t ood-slurm-reconcile` shows the nodes the reconciler removed. - -## A Role Does Not Reach RUNNING - -`oneflow show ` names the role. When a check stops the role, the VM carries the reason in its `ERROR` attribute. Sunstone shows it in a red banner on the VM, and `onevm show | grep ERROR` prints it. Fix the value and instantiate the service again. When the attribute is missing, read the last lines of `/var/log/ood-appliance-configure.log` on the VM. - -The following list presents some common causes: - -* **A switch enabled with its field empty**: `ONEAPP_HOME_NFS_ENABLED`, `ONEAPP_SOFTWARE_PROXY_ENABLED`, `ONEAPP_SLURM_CONTROLLER_ENABLED`, `ONEAPP_AUTH_OIDC_ENABLED` and `ONEAPP_PORTAL_CERTIFICATE_ENABLED` each require their fields, and the message names the missing one. The wizard cannot check this, because Sunstone validates a required field even while the switch hides it. -* **A wrong entry in the initial users**: A duplicate user name or uid, a missing password, a uid under 1000 or a name with capital letters stops the portal. The message quotes the entry. -* **OneGate not reachable**: A role that cannot reach OneGate never declares itself ready. The appliance tries `ONEGATE_ENDPOINT` from the VM context first, then port 5030 on the VM's default gateway. `/var/log/ood-appliance-configure.log` records which one answered. Check that one of the two is reachable from the management network. -* **The storage role cannot reach the EESSI servers**: It needs outbound HTTP on ports 80 and 8000. -* **The portal cannot mount the home**: The storage role grants the portal root on the export through OneGate, within about 20 seconds of the portal VM existing. If the portal still waits after three minutes, check OneGate. -* **The portal cannot mount the Slurm state**: The storage role exports `/export/slurm` to the portal address in the same way. A portal still waiting after three minutes stops with `could not mount` and the export in the message. -* **A worker never receives the munge key**: The portal publishes it as `SLURM_MUNGE_KEY` through OneGate. A worker that has waited 180 seconds for it stops with `the portal has not published SLURM_MUNGE_KEY after 180s`, so check the portal first. -* **The service stays in `DEPLOYING_NETS`**: OneFlow never leaves that state when the VM template of a role no longer exists, for example after the template was deleted and the service template was not updated. `oneflow show` reports nothing, and the exception is only in `journalctl -u opennebula-flow` on the Front-end. Point the service template at an existing VM template and instantiate again. -* **The wrong address range**: The portal takes the worker range from its compute interface, the /24 around its address. A compute network larger than a /24 needs `ONEAPP_POOL_RANGE` set to a /24 slice of it. A standalone portal needs `ONEAPP_POOL_RANGE` to match the network. The range also limits the number of Slurm nodes. A range that is not written as `first-last` stops the portal with `ONEAPP_POOL_RANGE must be "first-last"` in `/var/log/ood-appliance-configure.log`. - -## A Session Does Not Start - -1. Check that every role is `RUNNING`. -2. On the portal, `squeue -u ` shows the job of the session. `PENDING` with reason `Resources` means no worker has the cores or the memory free. If the pool is growing, see [The Pool Does Not Scale](#the-pool-does-not-scale). `PartitionConfig` means no worker of the pool could ever run it, so lower the cores or the memory in the form. -3. `sinfo -R` lists the down or drained nodes with the reason. A node `drained` with reason `one-ondemand scale-down` is about to be removed. A node `down*` has stopped answering the controller. Refer to [A Worker Is Missing from sinfo](#a-worker-is-missing-from-sinfo). -4. `sacct -j ` shows how the job ended. `NODE_FAIL` means its worker died, and `OUT_OF_MEMORY` means the session used more memory than it requested. -5. Read `output.log` in the session directory. `module load` errors mean a problem with the software catalogue, and `Permission denied` on the home means a problem with the export. `/var/log/slurm/slurmd.log` on the worker records why a job could not start there. -6. On the worker, `runuser -u -- ls /cvmfs/software.eessi.io/versions` shows whether that user can reach the catalogue. - -## Users Cannot Sign In - -On the portal, run `systemctl status slapd ondemand-dex` and `ldapsearch -x -H ldap://localhost -b dc=ood,dc=local uid=`. A user who signs in but cannot open a session has no Unix account on the worker. There, `getent passwd ` has to answer through `sssd`. `sssd` asks the portal on port 389. - -## The Pool Does Not Scale - -Check that the workers publish their attributes with `onevm show`, and read `/var/log/one/oneflow.log` on the Front-end. OneFlow writes a `[AE] Checking policies for service: ` line for every service every `autoscaler_interval` seconds, 90 by default, and `oneflow show --json` keeps the time of the last evaluation in `last_eval` of each policy. - -When those lines stop, or `last_eval` stays empty while the workers publish `SLURM_PENDING` above 0 or `OLDEST_IDLE` at 1, the evaluation thread of OneFlow has died. `systemctl restart opennebula-flow` on the Front-end starts it again without touching the service or its VMs. In OpenNebula 7.4.0 the thread dies within a minute of a Front-end reboot when a service is `RUNNING` and its VMs are still in `POWEROFF`, because the monitoring pool that `oned` returns has no record for them and `role.rb` stops on the empty entry. Nothing reaches `oneflow.log`. The only trace is a `terminated with exception` line in `journalctl -u opennebula-flow`. After a Front-end reboot, resume the VMs and then restart `opennebula-flow`, and check that the `[AE]` line appears again. - -The pool grows only while the workers publish `SLURM_PENDING` above 0, and only up to `max_vms`. `SLURM_PENDING` counts the jobs waiting for cores or memory that a worker of the role could give them; a job waiting with reason `PartitionConfig` asks for more cores than the largest worker has and never counts. - -The pool shrinks only when the oldest worker publishes `OLDEST_IDLE=1`, plus the five-minute cooldown of the scale down policy. The worker publishes it once it has drained its node, after `ONEAPP_WORKER_IDLE_SECONDS` without a job and nothing pending, and the node then shows as `drained` with reason `one-ondemand scale-down` in `sinfo -R`. A single worker never drains, and a drain that OneFlow does not act on within `ONEAPP_WORKER_DRAIN_SECONDS` is undone. - -## A Worker Is Missing from sinfo - -A worker whose node is missing from `sinfo` on the portal has not registered with the controller. On the worker, `munge -n | unmunge` has to succeed. Success proves it holds the key the portal published as `SLURM_MUNGE_KEY`. `Invalid credential` means the worker holds another key. `munged` loads a reinstalled `/etc/munge/munge.key` only after `systemctl restart munge`. `journalctl -u slurmd` shows why `slurmd` did not start or did not reach the controller. `ood-portal` in `/etc/hosts` has to resolve to the compute address of the portal, and port 6817 there has to answer. - -`scontrol delete` and the reconciler both delete a node from the controller. A deleted node never returns on its own. The elastic loop of the worker therefore restarts `slurmd` when its node is absent from `sinfo` while munge and the controller answer. It does this at most once every five minutes, and the node registers again within seconds. A worker that reboots returns to `idle` on its own in under a minute. A worker that starts before its portal, for example when the VMs are resumed in the wrong order after a Host outage, cannot fetch its configuration at first. Its `slurmd` unit retries every 20 seconds until the controller answers, and the elastic loop also restarts a `slurmd` that is down. - -## The VMs Are in POWEROFF After a Host Reboot - -OpenNebula does not resume the VMs on its own, and OneFlow shows the service in `WARNING`. Resume the storage VM, then the portal, then the workers. The service returns to `RUNNING` by itself, with its users, home directories and Slurm queue intact. Then restart `opennebula-flow` on the Front-end, because its evaluation thread dies during the reboot. Refer to [The Pool Does Not Scale](#the-pool-does-not-scale). A worker resumed before the portal joins the Cluster once the portal answers, because its `slurmd` keeps retrying. diff --git a/content/solutions/integration_blueprints/open_ondemand/operations.md b/content/solutions/integration_blueprints/open_ondemand/operations.md deleted file mode 100644 index 0726c43a3..000000000 --- a/content/solutions/integration_blueprints/open_ondemand/operations.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: "Operations" -linkTitle: "Operations" -date: "2026-09-15" -description: -categories: -tags: -weight: "5" -type: docs ---- - -This section covers the lifecycle of a running Open OnDemand Service. It explains how to keep the home directories across deployments, upgrade the appliance, and remove the service. - -## Keeping the Home Directories - -By default the shared home is on the root disk of the storage VM and is deleted with the service. There are two options to preserve the directory and its contents. - -### A Persistent Disk on the Storage Role - -Create a persistent Datablock and add it to the `template_contents` of the storage role in the service template. List it next to the disk of the appliance image, because a `DISK` entry there replaces the whole disk list of the VM template: - -```shell -oneimage create --name ood-home --type DATABLOCK --size 51200 --persistent --datastore default -``` - -```json -"DISK": [ - { "IMAGE_ID": "" }, - { "IMAGE_ID": "" } -] -``` - -At first boot the storage role formats a blank second disk, labels it `ood-home` and keeps the home directories on it. A disk that already has the label is mounted as it is. If you delete the service and instantiate it again with the same disk, every home directory is restored. A disk with any other filesystem is not touched, and the role stops with an error. - -### An Existing NFS Server - -At instantiation, enable **NFS server of your own** in the **Home directories** tab (`ONEAPP_HOME_NFS_ENABLED`). Set `ONEAPP_HOME_NFS_SERVER` and `ONEAPP_HOME_NFS_EXPORT`. The portal creates each home directory on first login. The server must therefore export the path with `no_root_squash` for the address the portal uses to reach it. That is the compute address of the portal when the server is on the compute network, and its management address otherwise. The server can keep `root_squash` for the workers. The storage role still runs the software cache. - -## Upgrading - -A new version of the appliance is a new image and new templates. A running service keeps the old ones. Download the new version and instantiate a new service with the same inputs. Move the home directories as described above. Attach the same persistent disk, or use the same export. You can also save them with `onevm disk-saveas` first. Then point the DNS name of the portal to the new portal VM and delete the old service. Users are created again from `ONEAPP_AUTH_LOCAL_USERS`. Add the others again on the new portal. - -## Removing the Service - -From **Instances -> Services** in Sunstone, or from the Front-end command line: - -```shell -oneflow delete -``` - -This terminates the VMs of every role and their non-persistent disks. A persistent home disk is released and keeps its content. An external export is not touched. The imported image and templates stay until you delete them. - -## Limitations - -* The portal and the storage role are one VM each, with no high availability. A reboot of the portal keeps the running and the pending jobs. This works because the queue, the nodes and the munge key are on the storage export. The accounting database is dumped there every 30 minutes for a replaced portal. Measured on the testbed: munge, MariaDB, `slurmdbd`, `slurmctld` and the timers were active well under a minute after the reboot, and both jobs were still in the queue. A portal that OneFlow replaces with a new VM has not been tested. The workers follow the new address from the service document, and the key and the state are restored from the export. Both are untested. -* The state of the Slurm controller is on NFS, so the storage VM must stay reachable from the portal. During a 33-second pause of the NFS server, `sinfo` still answered and the controller stayed active. The running job was not affected. A longer outage has not been measured. -* Every session and every Job Composer job runs on the cluster of the service. A Slurm Cluster of the site can be attached as a second target for batch jobs. See [Configuration]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/#batch-jobs-with-slurm" %}}). -* GPU workers are prepared but untested. diff --git a/content/solutions/integration_blueprints/open_ondemand/overview.md b/content/solutions/integration_blueprints/open_ondemand/overview.md deleted file mode 100644 index 2e6aca922..000000000 --- a/content/solutions/integration_blueprints/open_ondemand/overview.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -title: "Overview" -linkTitle: "Overview" -date: "2026-09-15" -description: -categories: -tags: -weight: "1" -type: docs ---- - -The Open OnDemand Service deploys an [Open OnDemand](https://openondemand.org/) portal on OpenNebula. A user can sign in from a browser and open JupyterLab, RStudio, Octave, a C++ notebook, VS Code or an Xfce desktop on a compute VM. Scientific software comes from the [EESSI](https://www.eessi.io/) catalogue, and the home directory is the same in every session. - -The service is a OneFlow service built from a single appliance image. Three roles boot from that same image, and every session is a job of the service's own Slurm Cluster. OneFlow adds a compute VM when a job is waiting for a free VM, and removes the oldest worker after it has been idle for a period. The service brings its own Slurm Cluster and needs no other Cluster. It suits teaching environments and notebook access for research groups. - -## How Should I Read this Chapter - -Start with the [Quick Start Guide]({{% relref "solutions/integration_blueprints/open_ondemand/quick_start/" %}}) to deploy the Open OnDemand service from the Community Marketplace and open a notebook. The [Service Architecture]({{% relref "solutions/integration_blueprints/open_ondemand/architecture/" %}}) explains the roles, the networks and how a session runs. Then, consult the following guides for further details: - -* [Configuration]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/" %}}): Covers inputs, scaling, worker sizes, Slurm, external identity providers and users. -* [Operations]({{% relref "solutions/integration_blueprints/open_ondemand/operations/" %}}): Covers manual scaling, persistent home directories, upgrades and removal. -* [Monitoring and Troubleshooting]({{% relref "solutions/integration_blueprints/open_ondemand/monitoring_and_troubleshooting/" %}}): Covers metrics, logs, and what to check when something fails. - -## Usage Model in a Multi-Tenant Cloud - -The service suits a cloud where several groups share one OpenNebula Front-end and each group works inside its own Virtual Data Center (VDC). Three actors take part, and only two of them need an OpenNebula account. - -* **Infrastructure Admin**: Manages the OpenNebula cloud and downloads the Open OnDemand Service from the Community Marketplace, which makes the image, the VM template and the service template available to every group. The admin also prepares what belongs to the site, including the two Virtual Networks, the quotas of each VDC, the `autoscaler_interval` setting of the Front-end and, for infrastructure including GPUs, the PCI passthrough configuration of the Hosts. -* **Tenant Operator**: Belongs to an OpenNebula group that owns a VDC. Instantiates the service from Sunstone inside that VDC and chooses the networks and the inputs. Creates the accounts of the end users in Open OnDemand, with `ONEAPP_AUTH_LOCAL_USERS` or through an OpenID Connect provider of the group. Watches the service, scales the worker role by hand when needed, and removes the service when the group no longer needs it. The VMs of the service count against the quotas of the VDC. -* **OnDemand End User**: A member of the research team that the group serves. Signs in to the portal from a browser with the account the Tenant Operator created, opens JupyterLab, RStudio, VS Code or a desktop, and submits batch jobs. The OnDemand end user does not require an OpenNebula account and never interacts with Sunstone. Every session runs as a Slurm job under that user, with the cores and the memory requested with the job. - -The unit of isolation between groups is the Open OnDemand Service instance, not the portal inside it. Each instance has one list of user accounts, one Slurm Cluster and one shared home export, in which each user has a home directory of their own. Its users share the portal, the workers and the quotas, separated from each other only as Unix users and Slurm jobs. Two groups that must not share users, data or queue should run two instances of the service, each in its own VDC, and OpenNebula keeps them isolated. - -## What the Service Manages - -Every VM boots from the same image. `ONEAPP_ROLE`, set for each role in the service template, decides what the VM does: - -* **storage**: A single VM. It exports the shared home directory, the state of the Slurm controller and the shared software directory over NFS, and caches the EESSI software catalogue. -* **portal**: A single VM. It runs Open OnDemand, the LDAP directory, the Dex login service, and the Slurm controller with its accounting database. -* **worker**: One to six VMs, scaled by OneFlow. Each VM joins the Cluster as a Slurm worker node and runs the user sessions as jobs. Each job gets only the cores and the memory it requested. - -The service also creates the initial users at first boot and keeps an accounting record of every session. It gives the portal a TLS certificate, which is self-signed by default. For a public name, the certificate comes from [Let's Encrypt](https://letsencrypt.org/), or you can use one of your own. - -## Slurm Components - -The service uses Slurm 23.11 as its job scheduler. A job is a request for cores, memory and time, together with the command to run. Every interactive session and every batch job from the Job Composer is a Slurm job. These are the components and where they run. - -| **Component** | **Runs on** | **Function** | -|---|---|---| -| `slurmctld`, the controller | portal | Keeps the list of nodes with their free cores and memory, keeps the job queue, and assigns each job to a node that has the requested resources. A job that fits on no node waits in the queue. | -| `slurmd`, the node daemon | every worker | Registers the worker as a node at boot, starts the jobs the controller assigns, as the requesting user, and limits each job to its cores and memory with cgroup v2. | -| `slurmdbd` and MariaDB, the accounting | portal | Record every job with its user, its node and its start and end times. `sacct` reads the record. | -| munge, the authentication | every role | Signs every message between the Slurm daemons and commands with a key shared by all the VMs of the service. A VM without the key cannot join the Cluster or submit jobs. The portal generates the key at first boot and publishes it to the workers through OneGate. | -| OpenMPI and PMIx, multi-node jobs | workers, from EESSI | Let a batch job run processes on several workers at once. The job requests the nodes with `sbatch -N ` and starts the processes with `srun --mpi=pmix` or `mpirun`. Interactive sessions use one worker each. | -{.w-100} - -## Related Components - -* [**OneFlow**]({{% relref "product/operation_references/opennebula_services_configuration/oneflow/" %}}): Starts the roles in order and applies the elasticity policies of the worker role. -* [**OneGate**]({{% relref "product/operation_references/opennebula_services_configuration/onegate/" %}}): The channel between the VMs and OpenNebula. Roles report readiness through it, and the portal publishes the Cluster's munge key. The workers publish the queue figures that OneFlow uses to scale. OneGate must be reachable from the service networks. -* [**Open OnDemand**](https://osc.github.io/ood-documentation/latest/): The portal software, from the Ohio Supercomputer Center. The service uses its `slurm` adapter for its own Cluster and for an optional second Slurm Cluster. -* [**Slurm**](https://slurm.schedmd.com/): The workload manager, version 23.11 from Ubuntu 24.04. The portal runs the controller and the accounting daemon. Every worker joins as a dynamic node. Every session is a job, and cgroup v2 limits it to its cores and memory. -* [**EESSI**](https://www.eessi.io/docs/): A shared scientific software catalogue distributed over CernVM-FS. The image ships the CernVM-FS client and a site cache. The software itself is fetched on demand. -* [**Managed Slurm**]({{% relref "platform_services/slurm/" %}}): The OneSlurm service is optional. The Open OnDemand Service does not use it. A site that already runs one can attach it as a second Cluster for batch jobs, with the same users and home directory, as described in [Configuration]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/#an-external-slurm-cluster" %}}). - -## Supported Versions - -The appliance ships Open OnDemand 4.2 on Ubuntu 24.04 LTS, with Slurm 23.11.4, munge 0.5.15 and MariaDB 10.11 from the Ubuntu archive. It also ships EESSI 2025.06, Apptainer 1.5, TurboVNC 3.3 and Xfce 4.18. It runs on OpenNebula 6.10 and later with OneFlow and OneGate enabled. Open OnDemand is MIT licensed and the appliance code is Apache 2.0. diff --git a/content/solutions/integration_blueprints/open_ondemand/quick_start.md b/content/solutions/integration_blueprints/open_ondemand/quick_start.md deleted file mode 100644 index 524b97b54..000000000 --- a/content/solutions/integration_blueprints/open_ondemand/quick_start.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -title: "Quick Start" -linkTitle: "Quick Start" -date: "2026-09-15" -description: -categories: -tags: -weight: "3" -type: docs ---- - -This guide demonstrates the deployment of an Open OnDemand Service through the Sunstone Web UI, with one portal, one storage VM and one worker, exposing a JupyterLab notebook to a user through the service. Check the [Requirements]({{% relref "solutions/integration_blueprints/open_ondemand/architecture/#requirements" %}}) before beginning this guide. You will need OneFlow and OneGate enabled, along with two Virtual Networks, one with Internet access and one reserved for the service. - -Quick-start workflow: - -* **Create the Networks**: Create a management network with Internet access (or choose an existing one), and a compute network reserved for the service. -* **Download the Appliance**: Import the image, the VM template and the service template from the Community Marketplace. -* **Instantiate the Service**: Select the two Virtual Networks and keep the default inputs. -* **Wait for the Service**: It reaches the `RUNNING` state within about four minutes. -* **Open the Portal and Launch a Notebook**: Sign in with the initial user and open JupyterLab as a Slurm job on the worker VM. - -## Before You Start - -The service requires two Virtual Networks. The service wizard requests both of the following networks during instantiation: - -* **Management Network**: The network your VMs already use to reach the Internet. The portal publishes its web interface on it, and OneGate must be reachable from it. -* **Compute Network**: A private network for intercommunication among the VMs of this service and nothing else. It carries the shared home directory, the user directory, the software cache and the Slurm traffic between the portal and the workers. - -Keep the compute network private and small, a `/24` subnet or smaller. Its traffic is not encrypted. The workers join the Slurm Cluster through it and the portal treats every address of that `/24` subnet as a possible worker. A larger network needs `ONEAPP_POOL_RANGE`, described in [Configuration]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/#advanced-attributes" %}}). - -Create both networks from the **Networks -> Virtual Networks** view with **+ Create Virtual Network** and select **From scratch**. Most installations already have a management network, so skip the first one if yours does. From the Front-end command line, `onevnet create` creates the same networks from a template file. See [Managing Virtual Networks]({{% relref "product/cluster_configuration/networking_system/manage_vnets/" %}}). - -## Step 1: Create the Networks - -### Create the Management Network - -Your cloud may already have a public network suitable for the management network. If this is the case, skip the following steps and proceed to [Create the Compute Network](#create-the-compute-network) - -1. **General**: Give it a name, for example `ood-management`. Select the Cluster of the Hosts that will run the service. Click **Next**. - -{{< image path="/images/open_ondemand/light/sunstone_vnet_management_general.png" -alt="Create Virtual Network wizard, General step of the management network" align="center" width="90%" mb="20px" >}} - -2. **Configuration**: Choose the network mode of your Hosts. On a single Host, select **Bridged** and **Custom name for bridge**. Enter the name of the bridge that reaches the Internet, for instance `br0`. - -{{< image path="/images/open_ondemand/light/sunstone_vnet_management_conf.png" -alt="Create Virtual Network wizard, Configuration tab of the management network" align="center" width="90%" mb="20px" >}} - -3. **Addresses**: Click **+ Add Address Range**. Set the first free address of the network and the number of addresses, for instance `192.168.100.50` and `200`. Click **Accept**. - -{{< image path="/images/open_ondemand/light/sunstone_vnet_management_addresses.png" -alt="Create Virtual Network wizard, Addresses tab of the management network" align="center" width="90%" mb="20px" >}} - -4. **Context**: Set the network address, the network mask, the gateway and the DNS server, so the VMs reach the Internet. Use for instance `192.168.100.0`, `255.255.255.0`, `192.168.100.1` and `1.1.1.1`. Click **Finish**. - -{{< image path="/images/open_ondemand/light/sunstone_vnet_management_context.png" -alt="Create Virtual Network wizard, Context tab of the management network" align="center" width="90%" mb="20px" >}} - -### Create the Compute Network - -1. **General**: Give it a name, for instance `ood-compute`, and select the same Cluster. Click **Next**. - -{{< image path="/images/open_ondemand/light/sunstone_vnet_general.png" -alt="Create Virtual Network wizard, General step of the compute network" align="center" width="90%" mb="20px" >}} - -2. **Configuration**: Any mode where the VMs of the Cluster reach each other works. On a single Host, select **Bridged** and **Custom name for bridge**. Enter a new bridge name, for instance `br1`. Across several Hosts, use the mode and physical device of your other private networks, for instance VXLAN. - -{{< image path="/images/open_ondemand/light/sunstone_vnet_conf.png" -alt="Create Virtual Network wizard, Configuration tab of the compute network" align="center" width="90%" mb="20px" >}} - -3. **Addresses**: Click **+ Add Address Range**. Set a first address and a size inside one `/24` subnet, for instance `172.22.0.50` and `200`. Click **Accept**. - -{{< image path="/images/open_ondemand/light/sunstone_vnet_addresses.png" -alt="Create Virtual Network wizard, Addresses tab of the compute network" align="center" width="90%" mb="20px" >}} - -4. **Context**: Leave it empty. This network has no gateway, because its traffic never leaves the Hosts. Click **Finish**. - -## Step 2. Download the Appliance - -Register the [Community Marketplace](https://github.com/OpenNebula/marketplace-community/wiki/marketplace_start) once if your installation does not already have it. Then, from **Storage -> Apps**, search for and download `Open OnDemand Service` into an available image datastore with sufficient space. The download imports three objects: an image that the three roles share, the VM template that boots that image, and the service template `Open OnDemand Service`. Alternatively, you can register the marketplace from the Front-end command line using the following command: - -```shell -onemarketapp export 'Open OnDemand Service' 'Open OnDemand Service' --datastore default -``` - -The image is about 1.6 GB, so wait until `oneimage list` reports the image in the `rdy` state. - -## Step 3. Instantiate the Service - -From **Templates -> Service Templates**, select `Open OnDemand Service`. Click the **Instantiate** icon (the play button icon). - -{{< image path="/images/open_ondemand/light/sunstone_service_templates.png" -alt="The Open OnDemand Service template in Sunstone" align="center" width="90%" mb="20px" >}} - -The wizard has four steps. In **General**, keep the name and one instance. - -{{< image path="/images/open_ondemand/light/sunstone_instantiate_general.png" -alt="Instantiate wizard, General step" align="center" width="90%" mb="20px" >}} - -In **Networks**, click the **Management** entry on the left and select its relevant Virtual Network in the table. The Management Network is the network with Internet access described [earlier](#before-you-start), where the portal publishes its web interface. Then click the **Compute** entry and select the Compute Network you created [earlier](#create-the-compute-network). - -{{< image path="/images/open_ondemand/light/sunstone_instantiate_networks.png" -alt="Instantiate wizard, Networks step" align="center" width="90%" mb="20px" >}} - -In **Service Inputs**, nothing needs to change for this quick-start guide. Every input is optional. The default values give a portal with a self-signed certificate on its management address, and one local user with username `demo1` and password `demo1pass`. The home directories and the software cache stay on the storage role, and the Slurm Cluster of the service needs no input. The inputs sit in four tabs, one per topic. Each section has a help icon with its explanation. Every optional feature is a switch that shows its fields only when it is on. The **Portal** tab looks like this: - -{{< image path="/images/open_ondemand/light/sunstone_instantiate_inputs.png" -alt="Instantiate wizard, Service Inputs step, Portal tab" align="center" width="90%" mb="20px" >}} - -The **Portal** tab holds the public name and the TLS certificate of the web portal. For this guide, leave the name empty and both toggles off: - -| **Option** | **What it does** | -|---|---| -| **Host name**
`ONEAPP_PORTAL_HOST_NAME` | The public name at which users open the portal. If empty, the portal answers on its management address. You can set the name of an already running portal later, see [Configuration]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/#service-inputs" %}}). | -| **Let's Encrypt**
`ONEAPP_PORTAL_LETSENCRYPT_ENABLED` | Requests a certificate for the Host name when the portal boots. The name must resolve to the portal, and ports 80 and 443 must be reachable from the Internet. When off, the portal starts with a self-signed certificate. | -| **Your own certificate**
`ONEAPP_PORTAL_CERTIFICATE_ENABLED` | Replaces the self-signed certificate with a PEM chain and a PEM key. Paste them into the two boxes revealed when this option is chosen. | -{.w-100} - -The **Users and login** tab decides who can sign in. For this guide, keep the default user and the option off: - -| **Option** | **What it does** | -|---|---| -| **Local users**
`ONEAPP_AUTH_LOCAL_USERS` | The accounts the portal creates in its directory at first boot, as `user:password` separated by spaces. The default is `demo1:demo1pass`. A uid may follow, as `user:password:uid`, to match accounts that exist elsewhere. | -| **OpenID Connect**
`ONEAPP_AUTH_OIDC_ENABLED` | Adds an institutional login next to the local one, for instance MyAccessID or Keycloak. The switch reveals the issuer URL, the client id, the client secret and the name shown on the login page. | -{.w-100} - -The **Home directories** tab decides where the files of the users live. For this guide, leave the option off: - -| **Option** | **What it does** | -|---|---| -| **NFS server of your own**
`ONEAPP_HOME_NFS_ENABLED` | Mounts the home directories from an NFS server you already run, instead of the storage role. The storage role then only keeps the software cache. The switch reveals the address of the server and the export path. | -{.w-100} - -The **Software catalogue** tab decides where the EESSI files come from. For this guide, leave the option off: - -| **Option** | **What it does** | -|---|---| -| **CernVM-FS proxy of your own**
`ONEAPP_SOFTWARE_PROXY_ENABLED` | Points the portal and the workers at a CernVM-FS proxy the site already runs, instead of the cache on the storage role. The switch reveals the URL of the proxy. | -{.w-100} - -In **Charter**, leave the list empty and click **Finish**. From the Front-end command line, `oneflow-template instantiate 'Open OnDemand Service'` requests the same values interactively. - -## Step 4: Wait for the Service - -The roles start in order, `storage` first and `worker` last. Each role declares itself ready through OneGate only when it is serving. From **Instances -> Services**, the service normally moves from `DEPLOYING` to `RUNNING` within about four minutes. - -{{< image path="/images/open_ondemand/light/sunstone_service_running.png" -alt="The Open OnDemand Service running in Sunstone" align="center" width="90%" mb="20px" >}} - -Click on the service in the list and select the **Roles** tab to see the three roles. Select a role to list its VMs. - -{{< image path="/images/open_ondemand/light/sunstone_service_roles.png" -alt="The three roles of the running service" align="center" width="90%" mb="20px" >}} - -A role that remains in `DEPLOYING` has not declared itself ready. Refer to [Monitoring and Troubleshooting]({{% relref "solutions/integration_blueprints/open_ondemand/monitoring_and_troubleshooting/#a-role-does-not-reach-running" %}}). - -## Step 5. Open the Portal - -The portal publishes its address as `OOD_URL` in the user template of the portal VM. In the **Roles** tab of the service, tick the `portal` role and click its VM in the table. Then read the value in the **Template** tab, under **User Template**, next to `READY`. - -{{< image path="/images/open_ondemand/light/sunstone_portal_ood_url.png" -alt="The user template of the portal VM in Sunstone, with OOD_URL and READY" align="center" width="90%" mb="20px" >}} - -The same value is available from the Front-end command line: - -```shell -onevm show | grep OOD_URL -``` - -```default -OOD_URL="https://192.168.100.165/" -``` - -The value is `https://` followed by the management address of the portal VM, or by `ONEAPP_PORTAL_HOST_NAME` if you set a Host name. With the default self-signed certificate, the browser asks you to accept it. - -The address belongs to the management network. If your computer reaches that network, open the address as it is. If it does not, follow [the instructions below](#reach-the-portal-from-outside). - -### Reach the Portal from Outside - -There are three things that give the portal a public name: - -* A public address that forwards to the portal VM -* A DNS name for that address -* The DNS name as the portal Host name - -In the following example, the Front-end of a single-host cloud owns the public address `51.159.138.137`. The portal VM is at `192.168.100.184` on the management network. - -1. **A DNS Name**: Point a name of your domain at the public address, or use [sslip.io](https://sslip.io). This public DNS service resolves any name with an IP address written within it, so `ood.51-159-138-137.sslip.io` resolves to `51.159.138.137` without the need to register the name. - -2. **Port Forwarding to the Portal**: On a cloud provider, use a floating address or a port forwarding rule of the provider to the management address of the portal VM. On a Linux Host that owns the public address, two NAT rules forward ports 443 and 80. Port 80 only matters for Let's Encrypt. Save the rules so they survive a reboot: - - ```shell - iptables -t nat -A PREROUTING -i ens2 -p tcp --dport 443 -j DNAT --to-destination 192.168.100.184:443 - iptables -t nat -A PREROUTING -i ens2 -p tcp --dport 80 -j DNAT --to-destination 192.168.100.184:80 - iptables-save > /etc/iptables/rules.v4 - ``` - - `ens2` is the interface that carries the public address. The VMs of the management network need a route back through that Host. A network with the Host as gateway already has that route. - -3. **The Name as Host Name of the Portal**: Set it at instantiation, in the **Portal** tab, or on a running portal. A running portal reconfigures itself in under a minute and issues a self-signed certificate for the name: - - ```shell - onevm updateconf --append <}} - -The home directory is created on first login. The **Interactive Apps** menu lists the six interactive applications, and the dashboard pins five of them. - -{{< image path="/images/open_ondemand/light/portal_dashboard.png" -alt="Open OnDemand dashboard" align="center" width="90%" mb="20px" >}} - -## Launch a Notebook - -From **Interactive Apps -> Jupyter Notebook**, keep one core, 2 GB of memory and one hour, or raise them. The limits are what the largest worker has. Click **Launch**. A **GPUs** field appears only when a worker has a GPU. A **Worker size** field appears only when the service has several worker roles. Refer to [Worker Sizes]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/#worker-sizes" %}}). - -{{< image path="/images/open_ondemand/light/jupyter_form.png" -alt="Jupyter launch form" align="center" width="90%" mb="20px" >}} - -The session is a job of the Slurm Cluster of the service. Its card under **My Interactive Sessions** shows the Cluster, the job id and the worker it runs on. An example is `Runs on: Slurm, job 1 on ood-worker-128`. A session that finds no free cores shows `Queued` on its card. It waits until a session ends or OneFlow adds a worker. With the Front-end setting recommended in the [Requirements]({{% relref "solutions/integration_blueprints/open_ondemand/architecture/#requirements" %}}) this normally takes a couple of minutes. The first session on a fresh deployment takes longer to start, while the site cache fetches the Python module. When the card turns `Running`, click **Open the Jupyter notebook**. - -{{< image path="/images/open_ondemand/light/session_card.png" -alt="Session card" align="center" width="90%" mb="20px" >}} - -JupyterLab opens through the portal proxy, with the Python and Octave kernels from EESSI. - -{{< image path="/images/open_ondemand/light/jupyterlab.png" -alt="JupyterLab running on a compute VM" align="center" width="90%" mb="20px" >}} - -Files saved in the notebook go to the user's home directory on the storage role. They appear in **Files -> Home Directory** and in every later session. Deleting the session from **My Interactive Sessions** cancels its job on the worker. - -## Next Steps - -* [Service Architecture]({{% relref "solutions/integration_blueprints/open_ondemand/architecture/" %}}) describes what was deployed and how a session runs. -* [Configuration]({{% relref "solutions/integration_blueprints/open_ondemand/configuration/" %}}) covers the Host name and certificate, users, worker sizes, Slurm and external identity providers. -* [Operations]({{% relref "solutions/integration_blueprints/open_ondemand/operations/" %}}) covers keeping the home directories across deployments, upgrading and removing the service.