Fix pystemd.run waits and pystemd.futures on systemd 259 / Python 3.14 by aleivag · Pull Request #126 · systemd/pystemd

aleivag · 2026-10-29T19:34:28Z

pystemd.run:

wait=True / wait_for_activation=True could block forever: systemd only emits unit PropertiesChanged signals while some client has called Manager.Subscribe(), and run() only eavesdrops via BecomeMonitor. On a desktop something else is usually subscribed; on a minimal host (e.g. a Fedora 44 cloud VM) nobody is. Subscribe before starting the unit.
Drain every message sd-bus has buffered on each wakeup. One socket read can bring several signals, and select() does not fire again for the ones already read, so processing one message per wakeup could miss the state change we are waiting for.

pystemd.futures:

Python 3.14 made forkserver the default start method on Linux, which broke TransientUnitPoolExecutor ("SemLock created in a fork context is being shared with a process in a spawn context"). Workers now subclass ForkProcess, which enter_unit relies on anyway.
Start the anchor unit with Type=exec. With Type=simple the unit is reported running right after fork, while systemd >= 255's executor is still root, so enter_unit copied uid 0 and User= was ignored.
The pool's anchor process is now tail --pid=<parent> instead of sleep infinity (as TransientUnitProcess already does), so the unit does not leak when the parent dies without reaching exit.

Tested on Fedora 44 (systemd 259, kernel 6.19, aarch64, Python 3.14): futures.run with User=nobody + PrivateNetwork, CPUQuota and MemoryMax; examples/future_cpucap_{method,process,pool}.py; unit tests pass.

pystemd.run: - wait=True / wait_for_activation=True could block forever: systemd only emits unit PropertiesChanged signals while some client has called Manager.Subscribe(), and run() only eavesdrops via BecomeMonitor. On a desktop something else is usually subscribed; on a minimal host (e.g. a Fedora 44 cloud VM) nobody is. Subscribe before starting the unit. - Drain every message sd-bus has buffered on each wakeup. One socket read can bring several signals, and select() does not fire again for the ones already read, so processing one message per wakeup could miss the state change we are waiting for. pystemd.futures: - Python 3.14 made forkserver the default start method on Linux, which broke TransientUnitPoolExecutor ("SemLock created in a fork context is being shared with a process in a spawn context"). Workers now subclass ForkProcess, which enter_unit relies on anyway. - Start the anchor unit with Type=exec. With Type=simple the unit is reported running right after fork, while systemd >= 255's executor is still root, so enter_unit copied uid 0 and User= was ignored. - The pool's anchor process is now `tail --pid=<parent>` instead of `sleep infinity` (as TransientUnitProcess already does), so the unit does not leak when the parent dies without reaching __exit__. Tested on Fedora 44 (systemd 259, kernel 6.19, aarch64, Python 3.14): futures.run with User=nobody + PrivateNetwork, CPUQuota and MemoryMax; examples/future_cpucap_{method,process,pool}.py; unit tests pass.

Without them pystemd.run(extra={...}) raises KeyError for these properties. Types are what systemd 259 exposes on org.freedesktop.systemd1.Service: - PrivatePIDs: "s" ("yes"/"no"; systemd >= 257), not a boolean - ProtectHostnameEx: "(ss)", e.g. ("private", "myhostname") - SystemCallErrorNumber: "i", e.g. errno.EPERM so SystemCallFilter= denies with an error instead of killing the process Tested on Fedora 44 (systemd 259): a transient unit with PrivatePIDs="yes", ProtectHostnameEx=("private", "pycon-unit"), SystemCallFilter=(False, ["socket"]) and SystemCallErrorNumber=EPERM runs as PID 1 with its own hostname, and socket() fails with EPERM.

wait_for_activation=True only stopped waiting when it saw SubState "running" for the start job. A Type=notify or Type=exec unit that fails during startup never gets there (it goes straight to "failed"), so pystemd.run() blocked forever. Also stop once the start job is over and the unit is in an exit substate (exited/failed/dead), the same condition wait=True uses; with raise_on_fail the caller then gets PystemdRunError as usual. Tested on systemd 259: Type=notify `sh -c "exit 3"` and Type=exec `/nonexistent` now return in ~20ms as failed/failed (both hung before); a Type=notify unit that sends READY=1 still waits for it.

Needed to use the file descriptor store from pystemd.run(extra={...}) (otherwise KeyError). Types as exposed by systemd 259 on org.freedesktop.systemd1.Service: FileDescriptorStoreMax "u", FileDescriptorStorePreserve "s" ("no" / "yes" / "restart"). Tested: a Type=notify service with FileDescriptorStoreMax=1 stores a memfd (FDSTORE=1, FDNAME=db) and gets it back via LISTEN_FDS across four restarts; Service.DumpFileDescriptorStore() lists it.

- notify(unset_environment, *states, fds=None, **kwstates): when `fds` is given, call sd_pid_notify_with_fds() so the fds travel with the message; without it (or with an empty list), sd_notify() exactly as before. `fds` is keyword-only, and there is no FDS= notify state for it to shadow. It takes a single fd or a list of them; each fd is an int or an object with fileno(). fds=0 sends fd 0 (it is not "no fds"), and str/bytes are rejected with TypeError: bytes would otherwise iterate as ints and silently send the wrong fds. e.g. notify(False, fds=[memfd], fdstore=1, fdname="db") hands the memfd to systemd's file descriptor store. - listen_fds_with_names(unset_environment=0): sd_listen_fds_with_names(), returned as {name: [fd, ...]}; this is how a restarted service gets its stored fds back (and how named socket activation fds are found). Names are not unique (a socket unit with several ListenStream= passes all its fds under one name; FDNAME= can repeat), hence the lists. Tested on systemd 259: a Type=notify service with FileDescriptorStoreMax=1 stores a memfd holding a SQLite database with notify(fds=...) and gets it back with listen_fds_with_names()["db"][0] across four restarts; fds=<int> lands in the store too, and two fds stored under the same FDNAME both come back (LISTEN_FDNAMES=same:same).

Helpers for systemd's file descriptor store, on top of notify(fds=...): - store_fd(name, fds, unset_environment=False, **kwstates): hand a single fd or a list of fds to the store under `name`; the same as notify(unset_environment, fds=fds, fdstore=1, fdname=name, **kwstates). - store_fds({name: fds, ...}, unset_environment=False, **kwstates): several names at once. FDNAME= covers a whole notify message, so this sends one message per name (not atomic); unset_environment and the extra states (e.g. ready=1) only go with the last message, since unsetting $NOTIFY_SOCKET earlier would make the remaining sends no-ops. Returns the lowest notify() result. unset_environment can be passed positionally (like notify()) or as a keyword, for readability: store_fds({...}, unset_environment=True). Both validate names first (1-255 printable ASCII characters, no ':', as LISTEN_FDNAMES is ':'-separated) and raise ValueError: systemd ignores an invalid FDNAME= silently, so the fds would just never be stored. Together with listen_fds_with_names() -> {name: [fd, ...]}, storing and getting fds back is a dict both ways. Tested on systemd 259: store_fds({"a": fd, "b": [fd1, fd2], "c": fd}, unset_environment=True, ready=1, status="stored") stores all four fds under three names and activates the unit with the status set; store_fd() is used by a memfd, chat-server (listening + client sockets), pty and pidfd example that all get their fds back across restarts.

pystemd/utils.py becomes pystemd/utils/, one module per kind of helper: - utils/x2.py: x2char_star, str2cmd, strlist2cmd, x2cmdlist (the old file, moved, so its history follows) - utils/unit.py: random_unit_name - utils/typing.py: T, unwrap utils/__init__.py re-exports all of them, so existing imports such as `from pystemd.utils import x2char_star` and `pystemd.utils.unwrap(...)` keep working unchanged (including mock.patch("pystemd.utils.random_unit_name") in the tests). setup.py lists packages explicitly, so pystemd.utils is added there. No behaviour change; this makes room for new helper modules (e.g. namespaces) without growing a single catch-all file.

join_namespaces(pid, namespaces=None) moves the calling process into the namespaces of `pid` (like `nsenter --target pid --all`) and returns the names it joined. `namespaces` is an iterable of names ("net", "mnt", ...; NAMESPACES lists them), all by default. - Namespaces already shared with `pid` are skipped (compared by inode under /proc/*/ns/), so asking for "all" works when we share e.g. the user namespace, which setns() would otherwise reject with EINVAL. - First tries a single setns() on a pidfd (Python >= 3.12, Linux >= 5.8), which joins everything atomically; otherwise (Python 3.11, older kernels) joins one namespace at a time in nsenter's order, closing each fd. - Failures raise OSError instead of being ignored. pystemd.futures.enter_unit() used to open every /proc/<pid>/ns/* file, call setns() on it, ignore the result and never close the fd. It now calls join_namespaces(main_pid): no fd leak (one per namespace, in every worker), and a namespace that can't be joined (e.g. PrivateNetwork=) is an error rather than a worker that silently runs outside the sandbox. Tested on Linux 6.19 / Python 3.14: joining a PrivateNetwork= + ProtectHostnameEx= unit gives ['uts', 'net', 'mnt'] with only `lo` and the unit's hostname, through both the pidfd path and the forced fallback, with no fds leaked; futures.run() with User=nobody + PrivateNetwork, CPUQuota and MemoryMax behaves as before.

pystemd.run(cmd, delegate=True) sets Delegate= on the transient unit, a shortcut for extra={"Delegate": True}. Delegation is what lets Service.AttachProcesses move processes into the unit. None (the default) sends nothing and keeps systemd's default; extra= still overrides it. Adds tests/test_run.py, the first unit tests for pystemd.run, with the D-Bus side mocked.

Needed to set them from pystemd.run(extra={...}) (otherwise KeyError). Types as exposed by systemd 259 on org.freedesktop.systemd1.Service: OOMPolicy "s", OOMScoreAdjust "i", ManagedOOMSwap "s", ManagedOOMMemoryPressure "s", ManagedOOMMemoryPressureLimit "u", ManagedOOMMemoryPressureDurationUSec "t", ManagedOOMPreference "s". Tested: stress-ng --vm under MemoryMax=50M with OOMPolicy=continue and OOMScoreAdjust=500 keeps running while memory.events oom_kill climbs; with the default policy (stop) the unit fails with result oom-kill.

peturingi

Good job.
All of my suggestions, except the one where ValueError is thrown for None while the type-hint specifies (thereby an interface contract) that None is valid, are mere personal preferences -- feel free to ignore them.

peturingi · 2026-11-02T01:46:07Z

@@ -0,0 +1,29 @@
+#
+# Copyright (c) 2017-present, Facebook, Inc.


Personal annoyance only: unless you work for Facebook, is there a reason to assign them copyright?

peturingi · 2026-11-02T01:46:54Z

+from __future__ import annotations
+
+import os
+from typing import Iterable


Suggested change

from typing import Iterable

from collections.abc import Iterable

peturingi · 2026-11-02T01:47:49Z

+
+# CLONE_NEW* flags, defined here because os only has them from Python 3.12.
+# The order is the one nsenter(1) joins them in: user first, mount late.
+NAMESPACES = {


Suggested change

NAMESPACES = {

NAMESPACES: Final[Mapping[str, int]] = {

Above, also from collections.abc import Mapping

peturingi · 2026-11-02T01:48:19Z

+    """
+    Move the calling process into the namespaces of process `pid`, like


Suggested change

"""

Move the calling process into the namespaces of process `pid`, like

"""Move the calling process into the namespaces of process `pid`, like

peturingi · 2026-11-02T01:52:11Z

+}
+
+
+def join_namespaces(pid: int, namespaces: Iterable[str] | None = None) -> list[str]:


Never use collections.abc.Iterable/Sequence with [str] unless you intend to accept any of:
{'foo', 'bar'}, ('foo', bar') (and their friends) .. along with 'foobar'
A str is itself an Iterable[str].

peturingi · 2026-11-02T02:29:30Z

+        with self.assertRaises(ValueError):
+            join_namespaces(os.getpid(), ["net", "nope"])
+
+    def test_names(self):


Suggested change

def test_names(self):

def test_names(self) -> None:

peturingi · 2026-11-02T02:30:14Z

+
+    def test_names(self):
+        self.assertEqual(
+            set(NAMESPACES),


Suggested change

set(NAMESPACES),

NAMESPACES.keys(),

peturingi · 2026-11-02T02:30:27Z

+            {"user", "cgroup", "ipc", "uts", "net", "pid", "mnt", "time"},
+        )
+
+    def test_reexported(self):


Suggested change

def test_reexported(self):

def test_reexported(self) -> None:

peturingi · 2026-11-02T02:32:38Z

+    def unit_properties(self, SDManager: MagicMock, **kwargs: object) -> dict:
+        run_module.run(["/bin/true"], user_mode=False, **kwargs)
+        manager = SDManager.return_value.__enter__.return_value
+        (_name, _mode, properties), _ = manager.Manager.StartTransientUnit.call_args


Suggested change

(_name, _mode, properties), _ = manager.Manager.StartTransientUnit.call_args

*_, properties, _ = manager.Manager.StartTransientUnit.call_args

or

Suggested change

(_name, _mode, properties), _ = manager.Manager.StartTransientUnit.call_args

return manager.Manager.StartTransientUnit.call_args[2]

peturingi · 2026-11-02T02:34:55Z

@@ -66,7 +66,13 @@
    version="0.15.3",


Suggested change

version="0.15.3",

version="0.15.4",

aleivag added 4 commits October 29, 2026 12:32

aleivag force-pushed the fix-futures-systemd259-py314 branch 3 times, most recently from 1509dfe to aa8eaa4 Compare October 29, 2026 22:32

aleivag force-pushed the fix-futures-systemd259-py314 branch 2 times, most recently from 92cb60f to 1963192 Compare October 29, 2026 23:35

aleivag force-pushed the fix-futures-systemd259-py314 branch from 1963192 to 3744d5a Compare October 29, 2026 23:37

aleivag added 2 commits October 29, 2026 17:01

aleivag force-pushed the fix-futures-systemd259-py314 branch from 2db3f13 to c80ed8e Compare October 30, 2026 00:01

aleivag added 2 commits October 30, 2026 13:29

peturingi approved these changes Nov 1, 2026

View reviewed changes

peturingi reviewed Nov 2, 2026

View reviewed changes

Navigation Menu

Repository navigation

Fix pystemd.run waits and pystemd.futures on systemd 259 / Python 3.14 - #126

Fix pystemd.run waits and pystemd.futures on systemd 259 / Python 3.14#126
aleivag wants to merge 10 commits into
systemd:mainfrom
aleivag:fix-futures-systemd259-py314

aleivag commented Oct 29, 2026

Uh oh!

peturingi left a comment

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

peturingi Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,29 @@
		#
		# Copyright (c) 2017-present, Facebook, Inc.

	from typing import Iterable
	from collections.abc import Iterable

		"""
		Move the calling process into the namespaces of process `pid`, like

	"""
	Move the calling process into the namespaces of process `pid`, like
	"""Move the calling process into the namespaces of process `pid`, like

		}


		def join_namespaces(pid: int, namespaces: Iterable[str] \| None = None) -> list[str]:

	def test_reexported(self):
	def test_reexported(self) -> None:

	(_name, _mode, properties), _ = manager.Manager.StartTransientUnit.call_args
	*_, properties, _ = manager.Manager.StartTransientUnit.call_args

	(_name, _mode, properties), _ = manager.Manager.StartTransientUnit.call_args
	return manager.Manager.StartTransientUnit.call_args[2]

Conversation

aleivag commented Oct 29, 2026

Uh oh!

peturingi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants