fix: make a daimon organization able to boot - #9
Open
apresmoi wants to merge 10 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything needed to make a
runtime: daimonorganization actually boot. Found bydeploying Clank & Slop — a real 16-agent org — for the first time, and fixing each
failure as it surfaced. Stacked on #8.
Two compiler bounds the org exceeded
workspaceBundleArtifacts.tscapped a declaredkind: bundletar at 64 MiB and at10,000 entries. Clank & Slop declares a 395 MB relief grid (1 entry), a 284 MB
node_modules closure (11,074 entries) and a 74 MB one. Neither bound is documented in
specs/SPEC.mdorspecs/CONTAINERS.md, and integrity comes from the declaredsha256:plus
validateWorkspaceBundleTar, not from size or entry count. Raised to 512 MiB and anamed
MAX_WORKSPACE_BUNDLE_ENTRIES = 65_536, and split the entry-count message away fromthe genuine truncation message they shared — that shared message cost real debugging time.
A capability family: the root provisioning phase assumed privileges it is denied
Daimon organizations run
--cap-drop=ALLplusCHOWN, SETUID, SETGID, DAC_READ_SEARCH.That set is normative: the entrypoint asserts the broker launcher's
CapBndis exactly00000000000000c1. But the rendered root phase needed two more capabilities it does nothave, and each failed differently:
mkdirSync('/etc/daimon-engine-broker', 0o555)then writing twofiles into it (
EACCES), and on the next startrm -rfon the same 0555 directory.chown 2100:2100 <path>followed bychmodof the same path (EPERM).Fixed by ordering, not by widening: set modes while root still owns the inode and chown
last; write before tightening; loosen a directory you own before removing it. Final
ownership and modes are unchanged everywhere.
Two capabilities genuinely were missing and are added:
CAP_SETPCAP— required to lower a bounding set. Without itsetpriv --bounding-set=…silently no-ops and exits 0, so the launcher stayed atCapBnd …c5while the entrypoint asserted…c1and the container restarted forever.Measured: without it
c5 → c5; with it1c5 → c1, exactly the asserted value. Therenderer's own relay line already asks for
+setpcap, so this was always intended.CAP_KILL— the entrypoint supervises children it deliberately dropped to uid2100/2200+. Root without CAP_KILL cannot signal a different uid, so
kill -0 "$child"reported a live backend as dead, the probe fell into its "exited before readiness"
branch, and
waitblocked forever on a running process — a silent hang withcompletely empty
docker logs, because the explanatoryechocomes after thewait.Measured:
kill -0on a uid-2100 child returnsrc=1without the capability,rc=0with it.Both additions leave the container strictly more confined than the broken state: the
launcher now ends at
c1instead ofc5.The wake fuse had nowhere it could write
Daimon's fuse keeps a durable, fail-closed admission ledger and defaults its directory to
the turn-usage directory — provisioned
0750 2100:2100for the grok broker. The fuseruns in the organization runtime, uid 2000, because it must record an admission before
an engine is spawned, so it could not even
scandirit:EACCES, and Daimon died beforereadiness.
WakeFuse.open()deliberately does notmkdir, so an unprovisioned fuse failsclosed rather than running unbounded.
This adds a
daimon-wake-fuseexclusive-reattach mount at/var/lib/spawnfile/daimon/wake-fuse,provisioned 2000:2000 mode 0700, for every daimon organization — not gated on grok/agy,
because the fuse is universal — and sets
DAIMON_WAKE_FUSE_DIRECTORYin the runtime's execenvironment so an operator does not have to know the knob exists.
/var/lib/spawnfile/daimon/usageis untouched and a test guards that it stays
2100:2100 0750.Tests
Every change is mutation-locked: the guarantee was deleted, the suite observed going red,
and restored. Full suite 3,368 passing.
Not fixed here, recorded instead
runProject.tsmounts persistent volumes withvolume-nocopy, which is exactly the flagthat stops Docker seeding a fresh volume from the image — where the
.spawnfile-volume-initbootstrap markers that the ownership guard demands as its fresh-volume preimage are written.
A first deployment therefore cannot bootstrap. Worked around by seeding the volumes from the
org image once; the real fix is a design choice (an explicit seeding step, or dropping
volume-nocopyfor volumes the image is the authority for) and is not invented here.