Skip to content

pyxis: allow unprivileged user namespaces so enroot works on Ubuntu 23.10+ - #1392

Open
100milliongold wants to merge 1 commit into
NVIDIA:masterfrom
xiilab:fix/enroot-apparmor-userns
Open

pyxis: allow unprivileged user namespaces so enroot works on Ubuntu 23.10+#1392
100milliongold wants to merge 1 commit into
NVIDIA:masterfrom
xiilab:fix/enroot-apparmor-userns

Conversation

@100milliongold

Copy link
Copy Markdown
Contributor

Problem

Ubuntu 23.10 and later set kernel.apparmor_restrict_unprivileged_userns=1 by default. Under that default, enroot-nsenter fails as soon as pyxis starts a container:

enroot-nsenter: failed to create user namespace: Permission denied

The nvidia.enroot galaxy role does not handle this, and DeepOps lists Ubuntu 24.04 LTS as a supported OS, so container jobs do not work out of the box on a supported platform.

Observed on DGX OS 7.5.0 (Ubuntu-based), Slurm 26.05.1, pyxis 0.11.1.

Fix

Add a sysctl task in roles/pyxis for compute nodes. The task is gated on the presence of /proc/sys/kernel/apparmor_restrict_unprivileged_userns rather than on ansible_distribution_version, so it is a no-op on kernels built without AppArmor userns restrictions and does not need updating for future releases.

Note on hardening

This relaxes a system-wide hardening default: unprivileged user namespaces are what AppArmor is restricting here. A narrower alternative is an AppArmor profile scoped to enroot-nsenter (as packaged by some distributions for other userns consumers). If maintainers prefer that approach, I am happy to rework the patch.

The sysctl is written to /etc/sysctl.d/60-enroot-userns.conf so it is visible and revertible, rather than being applied only at runtime.

…3.10+

Ubuntu 23.10 and later ship with
kernel.apparmor_restrict_unprivileged_userns=1. Under that default,
enroot-nsenter fails when pyxis starts a container:

  enroot-nsenter: failed to create user namespace: Permission denied

The nvidia.enroot galaxy role does not handle this, and DeepOps lists
Ubuntu 24.04 LTS as a supported OS, so container jobs are broken out of
the box on a supported platform.

Add a sysctl task on compute nodes, gated on the presence of the knob in
/proc rather than on the distribution version, so it is a no-op on kernels
built without AppArmor userns restrictions.

Observed on DGX OS 7.5.0 (Ubuntu-based) with Slurm 26.05.1 and pyxis
0.11.1: srun --container-image=... failed with the message above until
the sysctl was set.

Signed-off-by: Jea-Eok-Kim <je.kim@xiilab.com>
@100milliongold
100milliongold marked this pull request as ready for review September 4, 2026 00:13

@dholt dholt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace the unconditional host-wide sysctl change with a narrowly scoped AppArmor policy for enroot-nsenter and provide evidence from a real Pyxis/Enroot container launch on an affected supported OS. If a global sysctl fallback is still needed for unusual installations, make it an explicit, documented, default-off administrator choice rather than changing the security default whenever the kernel knob exists.


Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants