Skip to content

fix: make a daimon organization able to boot - #9

Open
apresmoi wants to merge 10 commits into
mainfrom
fix/daimon-broker-registration-dir-mode
Open

fix: make a daimon organization able to boot#9
apresmoi wants to merge 10 commits into
mainfrom
fix/daimon-broker-registration-dir-mode

Conversation

@apresmoi

Copy link
Copy Markdown
Collaborator

Everything needed to make a runtime: daimon organization actually boot. Found by
deploying Clank & Slop — a real 16-agent org — for the first time, and fixing each
failure as it surfaced. Stacked on #8.

Two compiler bounds the org exceeded

workspaceBundleArtifacts.ts capped a declared kind: bundle tar at 64 MiB and at
10,000 entries. Clank & Slop declares a 395 MB relief grid (1 entry), a 284 MB
node_modules closure (11,074 entries) and a 74 MB one. Neither bound is documented in
specs/SPEC.md or specs/CONTAINERS.md, and integrity comes from the declared sha256:
plus validateWorkspaceBundleTar, not from size or entry count. Raised to 512 MiB and a
named MAX_WORKSPACE_BUNDLE_ENTRIES = 65_536, and split the entry-count message away from
the genuine truncation message they shared — that shared message cost real debugging time.

A capability family: the root provisioning phase assumed privileges it is denied

Daimon organizations run --cap-drop=ALL plus CHOWN, SETUID, SETGID, DAC_READ_SEARCH.
That set is normative: the entrypoint asserts the broker launcher's CapBnd is exactly
00000000000000c1. But the rendered root phase needed two more capabilities it does not
have, and each failed differently:

  • CAP_DAC_OVERRIDEmkdirSync('/etc/daimon-engine-broker', 0o555) then writing two
    files into it (EACCES), and on the next start rm -rf on the same 0555 directory.
  • CAP_FOWNERchown 2100:2100 <path> followed by chmod of the same path (EPERM).

Fixed by ordering, not by widening: set modes while root still owns the inode and chown
last; write before tightening; loosen a directory you own before removing it. Final
ownership and modes are unchanged everywhere.

Two capabilities genuinely were missing and are added:

  • CAP_SETPCAP — required to lower a bounding set. Without it
    setpriv --bounding-set=… silently no-ops and exits 0, so the launcher stayed at
    CapBnd …c5 while the entrypoint asserted …c1 and the container restarted forever.
    Measured: without it c5 → c5; with it 1c5 → c1, exactly the asserted value. The
    renderer's own relay line already asks for +setpcap, so this was always intended.
  • CAP_KILL — the entrypoint supervises children it deliberately dropped to uid
    2100/2200+. Root without CAP_KILL cannot signal a different uid, so kill -0 "$child"
    reported a live backend as dead, the probe fell into its "exited before readiness"
    branch, and wait blocked forever on a running process — a silent hang with
    completely empty docker logs, because the explanatory echo comes after the wait.
    Measured: kill -0 on a uid-2100 child returns rc=1 without the capability, rc=0 with it.

Both additions leave the container strictly more confined than the broken state: the
launcher now ends at c1 instead of c5.

The wake fuse had nowhere it could write

Daimon's fuse keeps a durable, fail-closed admission ledger and defaults its directory to
the turn-usage directory — provisioned 0750 2100:2100 for the grok broker. The fuse
runs in the organization runtime, uid 2000, because it must record an admission before
an engine is spawned, so it could not even scandir it: EACCES, and Daimon died before
readiness. WakeFuse.open() deliberately does not mkdir, so an unprovisioned fuse fails
closed rather than running unbounded.

This adds a daimon-wake-fuse exclusive-reattach mount at /var/lib/spawnfile/daimon/wake-fuse,
provisioned 2000:2000 mode 0700, for every daimon organization — not gated on grok/agy,
because the fuse is universal — and sets DAIMON_WAKE_FUSE_DIRECTORY in the runtime's exec
environment so an operator does not have to know the knob exists. /var/lib/spawnfile/daimon/usage
is untouched and a test guards that it stays 2100:2100 0750.

Tests

Every change is mutation-locked: the guarantee was deleted, the suite observed going red,
and restored. Full suite 3,368 passing.

Not fixed here, recorded instead

runProject.ts mounts persistent volumes with volume-nocopy, which is exactly the flag
that stops Docker seeding a fresh volume from the image — where the .spawnfile-volume-init
bootstrap markers that the ownership guard demands as its fresh-volume preimage are written.
A first deployment therefore cannot bootstrap. Worked around by seeding the volumes from the
org image once; the real fix is a design choice (an explicit seeding step, or dropping
volume-nocopy for volumes the image is the authority for) and is not invented here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant