fix: green deploy blockers (decisions crash-loop, wsx container names, worktree imports) by rennf93 · Pull Request #1325 · rennf93/roboco

rennf93 · 2026-11-01T06:18:36Z

Follow-up to trail #54 (change 176): the 2026-11-01 green deploy on the NAS left nginx answering 502 on every route. Three fixes, as approved.

Why the deploy stopped

roboco-decisions crash-looped with PermissionError: [Errno 13] Permission denied: '/app/server.py'. The image runs as uid 10001 (884b18c), COPY keeps the build folder's modes, and the NAS copy of the repo is owner-only (0600 on every file). deploy-nas.sh waits for every core container to be healthy before it runs alembic and switches front/active-upstreams.conf, so it stopped with nginx still on the stopped blue stack. A GitHub checkout has 0644/0755, which is why CI never saw it.

Changes

Non-root images set the mode of every file they copy in (161b2534). COPY --chmod in decisions, video-renderer (its build also failed outright: RUN node ensure-browser.mjs after USER 1000:1000), docker-gate and agent-base hooks. New guard tests/unit/test_dockerfile_copy_modes.py fails on any bare build-context COPY into a final stage whose runtime user is not root (it follows FROM roboco-<base> to inherit the user). It flags the four Dockerfiles on the old code.
Decisions keeps llama.cpp's default physical batch (687a4dc7, reverts b936126). Clef has no KV cache, so with batch = context a 39k-character input held ~10 NAS cores for over 9 minutes, could not be cancelled, and every queued request timed out with 503. With the default 512-token batch, an oversize input is refused in 0.1 s and a 500-character input returns 200 in about 50 s.
roboco-wsx-* containers name their owner and purpose (9b1f5cb6): roboco-wsx-<agent>-<purpose>-<8 hex>, for example roboco-wsx-be-dev-1-codegen-3f9a01c2. Callers pass WorkspaceCommand(argv, purpose); the purpose is the gate step, hook-<type>, entire-<step>, or the workspace probe.
A task worktree imports its own source (c87f666d). Each worktree's .venv symlinks to the clone root's shared venv, whose editable install named the clone root, so be-dev-1's make codegen imported an Aug 7 roboco (Role has no attribute DEVOPS) three times and could not reach i_am_done. _link_shared_venv now writes a stdlib-only hook (roboco/utils/worktree_imports.py) and a .pth into the shared venv; an interpreter started through a worktree's .venv points clone-root sys.path entries at the worktree. Started from the clone root, nothing changes. Doctrine note in .claude/rules/workspaces-and-git.md.
Map rows re-anchored (b30d59ea); the live Docker contract passes a WorkspaceCommand (36c3baeb); the import hook is split so xenon ranks its module A (16765f28).
Slave's red quality gate (d9afea60): after the uv.lock upgrade (1c16d78, pydantic 2.14, mypy 2.4) FieldInfo.annotation is TypeForm[Any] | None and mypy rejected issubclass(model, BaseModel) in tests/unit/test_security_middleware.py. It now checks isinstance(model, type) first, as the same file already does elsewhere.

Verification

Built from the NAS's 0600 folder: the decisions image was healthy in 25 s; video-renderer builds (exit 0).
Real uv run in a worktree imports the worktree's package; uv sync keeps the hook. tests/unit/services/test_workspace_worktree_imports.py runs a real interpreter.
Full suite without Postgres: 16402 passed, 3836 skipped. tests/unit/api/test_cloud_auth.py::test_login_route_parses_oauth2_form_not_query_params was deselected locally only because it connects to the host's 127.0.0.1:5432; CI runs it against its own Postgres. pytest --co: 20279 collected, no errors. Docs map: 0 errors, 0 warnings. Every other make quality step (ruff, reflow, mypy, xenon, vulture, bandit, deptry, compose-sync) passes locally.

Open for the CEO

roboco/services/decisions/client.py sends Clef up to 40,000 characters (_CLEF_STATE_BUDGET_CHARS). At about 5 tokens/s on the NAS CPU that does not fit the proxy's 120 s timeout; it needs a product decision, not part of this PR.

🤖 Generated with Claude Code

`roboco-wsx-<12 random hex>` said nothing about what a container was: during the 2026-11-01 green deploy a dozen of them looked like strays. Containers are now `roboco-wsx-<owner>-<purpose>-<8 hex>`, e.g. `roboco-wsx-be-dev-1-codegen-3f9a1c2e`: - owner: the agent slug of /data/workspaces/<project>/<team>/<agent>/..., `private` for the orchestrator-only /data/private scratch clones, else `workspace`; - purpose: what runs, carried with the argv in a new `WorkspaceCommand` (`codegen` and every other gate step by its configured name, `dev-deps`, `toolchain-smoke`, `dep-update-probe`, `hook-<type>`, `entire-<step>`). Both segments keep only [a-z0-9_.-] and at most 40 characters; the Docker gate's `roboco-wsx-` pattern accepts every generated name (tested against gate.NAME). The git-argv source scanner treats a WorkspaceCommand like a direct runner argument: it only ever runs in the throwaway container. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… root's Every per-task worktree's `.venv` is a symlink to the clone root's shared venv, and that venv's editable install (`uv sync` at the clone root) names the clone root's checkout. Any interpreter started in a worktree therefore imported the clone root's code, parked on whatever branch it last held. Live, 2026-11-01: be-dev-1's `make codegen` in `.worktrees/f5d66db6` imported `roboco` from the clone root at 8ee98c2 (2026-08-07) and died on `Role has no attribute DEVOPS`, refusing its `i_am_done` three times. Agents' own `uv run` scripts in worktrees loaded the same stale code. `_link_shared_venv` now also writes a startup hook into the shared venv: roboco/utils/worktree_imports.py (stdlib only) as `_roboco_worktree_imports.py`, imported by `zz_roboco_worktree_imports.pth`, which sorts after the editable install's own `.pth`. An interpreter started through `<clone>/.worktrees/<id>/.venv` keeps that path as `sys.prefix` (verified under `uv run`), so the hook knows its worktree and points every `sys.path` entry inside the clone root (not `.venv`, `.worktrees`, `.uv-python`) at the same place in the worktree. From the clone root nothing changes. It is (re)written on every link call, i.e. at every spawn, so a venv rebuilt after a lockfile change gets it back; `uv sync` leaves it in place. Tests run a real interpreter: a clone with an editable-style `.pth`, a worktree, `scripts/show.py` importing a package present in both; before the hook it prints "clone root", after it "worktree", and the clone root still gets its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…hey copy in The 2026-11-01 green deploy never switched nginx: deploy-nas.sh waits for roboco-decisions to be healthy before it migrates and flips traffic, and the sidecar crash-looped on `PermissionError: [Errno 13] Permission denied: '/app/server.py'`. COPY keeps each file's mode from the build folder; the NAS deploy copy is owner-only (all 1410 files under docker/ roboco/ panel/ scripts/ are 0600), and since 884b18c the decisions proxy runs as uid 10001. nginx stayed on the stopped blue color: 502 on every request. CI builds from a checkout (0644/0755) and could not see it. - decisions: server.py `--chmod=0644`, entrypoint.sh `--chmod=0755`. Built from the NAS folder itself, the image is healthy in 25 s. - video-renderer (uid 1000): package.json, the lockfiles, ensure-browser.mjs, server.js and render.js `--chmod=0644`; `RUN node ensure-browser.mjs` runs after `USER 1000:1000`, so the build itself failed from the NAS folder. - docker-gate: gate.py `--chmod=0644` (the image's own user is 65534; compose runs it as root, a plain `docker run` would not). - agent-base: the hook scripts and umask-exec.sh `--chmod=0755` instead of a later glob chmod, the npm pins file `--chmod=0644`. tests/unit/test_dockerfile_copy_modes.py fails any Dockerfile whose final stage runs as a non-root user (following FROM roboco-* to its base) and copies a build-context file without --chmod, --chown or a later chmod of its destination; on the previous Dockerfiles it flags decisions, video-renderer, docker-gate and agent-base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

b936126 sized --batch-size/--ubatch-size to each backend's context so inputs over 512 tokens would reach Clef. Clef keeps no KV cache, so an input is one physical batch, and on the NAS CPU Clef reads ~5 tokens/s, measured 2026-11-01 against the shipped image: 219 tokens 48 s, 366 tokens 76 s, 598 tokens 118 s (the proxy's upstream timeout is 120 s). A ~6k-token state then held ~10 cores for over nine minutes, could not be cancelled, and every decision queued behind it timed out with 503. With the default 512-token batch the same oversize input is refused in 0.1 s ("input (5990 tokens) is too large to process") and the client takes it as no verdict (fail-open), while a 500-char state still answers (50 s). Matching the client's 40,000-char Clef budget to what this hardware can read in time is a separate decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-11-01T06:19:00Z

Thanks for opening your first pull request on RoboCo!

Quick checklist before review (most of these are enforced by CI, but worth a glance):

make quality — ruff format check, ruff check, mypy, pytest (≥80% coverage), and the rest of the gate
Panel changes pass pnpm lint and pnpm exec tsc --noEmit (run from panel/)
No # noqa / # type: ignore shortcuts; pre-existing violations in touched files are fixed
Added an entry under ## [Unreleased] in CHANGELOG.md
Signed the CLA (the bot will prompt you on this PR)
Signed your commits — master requires verified signatures (SSH signing setup)
Updated any affected docs under docs/

See CONTRIBUTING.md for the full workflow and the Code of Conduct for the community standards we follow.

Welcome aboard — a maintainer will review shortly.

github-actions · 2026-11-01T06:20:59Z

DevOps image toolchain smoke results

DevOps toolchain smoke check (agent-devops)

Docker version 28.4.0, build d8eb465
Docker Compose version v2.39.4
clientVersion:
buildDate: "2025-08-13T14:23:59Z"
v3.18.6+gb76a950
Terraform v1.13.3
Haskell Dockerfile Linter 2.12.0
version: 0.9.0
yamllint 1.37.1
DEVOPS_TOOLCHAIN_SMOKE_OK

9b1f5cb changed run_in_workspace_container to take a WorkspaceCommand; the opt-in live Docker contract still passed a bare argv, which mypy rejects in CI's quality gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ks A xenon --max-modules A ranked roboco/utils/worktree_imports.py B: its one function carried the whole rewrite (complexity 7). Finding the worktree a prefix serves and moving one entry into it are now their own functions; behaviour is unchanged (an empty or relative entry still stays as it is, since it is never inside the absolute clone root). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lass pydantic 2.14 (uv.lock upgrade 1c16d78) types FieldInfo.annotation as TypeForm[Any] | None, and mypy 2.4 rejects passing it to issubclass, which turned slave's quality gate red. Check isinstance(model, type) first, as the same file's payload builder already does: a non-class annotation would also make issubclass raise at runtime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-11-01T06:23:35Z

Agent Image Smoke (Playwright) results

Playwright image size delta (real `docker inspect` sizes)

image	before	after	delta
agent-qa-fe	967MB	967MB	+0MB
agent-ux	915MB	915MB	+0MB
agent-pm	915MB	915MB	+0MB

Headless chromium launch smoke check

PLAYWRIGHT_SMOKE_OK
PLAYWRIGHT_SMOKE_OK
PLAYWRIGHT_SMOKE_OK

Playwright MCP server smoke check

Version 0.0.78
RESOLVED_HEADLESS_SHELL: /app/.playwright/chromium_headless_shell-1248/chrome-headless-shell-linux64/chrome-headless-shell
MCP_EXECUTABLE_LAUNCH_OK
Version 0.0.78
RESOLVED_HEADLESS_SHELL: /app/.playwright/chromium_headless_shell-1248/chrome-headless-shell-linux64/chrome-headless-shell
MCP_EXECUTABLE_LAUNCH_OK
Version 0.0.78
RESOLVED_HEADLESS_SHELL: /app/.playwright/chromium_headless_shell-1248/chrome-headless-shell-linux64/chrome-headless-shell
MCP_EXECUTABLE_LAUNCH_OK

Headless browser verification screenshot (ux-qa image)

Screenshot of a live panel page (/login), taken from inside agent-ux:after. See the ux-qa-panel-screenshot build artifact.

github-actions · 2026-11-01T06:25:24Z

DevOps image toolchain smoke results

DevOps toolchain smoke check (agent-devops)

Docker version 28.4.0, build d8eb465
Docker Compose version v2.39.4
clientVersion:
buildDate: "2025-08-13T14:23:59Z"
v3.18.6+gb76a950
Terraform v1.13.3
Haskell Dockerfile Linter 2.12.0
version: 0.9.0
yamllint 1.37.1
DEVOPS_TOOLCHAIN_SMOKE_OK

github-actions · 2026-11-01T06:27:47Z

Agent Image Smoke (Playwright) results

Playwright image size delta (real `docker inspect` sizes)

image	before	after	delta
agent-qa-fe	967MB	967MB	+0MB
agent-ux	915MB	915MB	+0MB
agent-pm	915MB	915MB	+0MB

Headless chromium launch smoke check

PLAYWRIGHT_SMOKE_OK
PLAYWRIGHT_SMOKE_OK
PLAYWRIGHT_SMOKE_OK

Playwright MCP server smoke check

Version 0.0.78
RESOLVED_HEADLESS_SHELL: /app/.playwright/chromium_headless_shell-1248/chrome-headless-shell-linux64/chrome-headless-shell
MCP_EXECUTABLE_LAUNCH_OK
Version 0.0.78
RESOLVED_HEADLESS_SHELL: /app/.playwright/chromium_headless_shell-1248/chrome-headless-shell-linux64/chrome-headless-shell
MCP_EXECUTABLE_LAUNCH_OK
Version 0.0.78
RESOLVED_HEADLESS_SHELL: /app/.playwright/chromium_headless_shell-1248/chrome-headless-shell-linux64/chrome-headless-shell
MCP_EXECUTABLE_LAUNCH_OK

Headless browser verification screenshot (ux-qa image)

Screenshot of a live panel page (/login), taken from inside agent-ux:after. See the ux-qa-panel-screenshot build artifact.

rennf93 and others added 5 commits November 1, 2026 06:41

docs(map): re-anchor the rows the green-deploy fixes shifted

b30d59e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-project-automation Bot added this to RoboCo Kanban Nov 1, 2026

github-project-automation Bot moved this to Backlog in RoboCo Kanban Nov 1, 2026

rennf93 changed the title ~~bug/claude/green-deploy-blockers~~ fix: green deploy blockers (decisions crash-loop, wsx container names, worktree imports) Nov 1, 2026

rennf93 changed the base branch from master to slave November 1, 2026 06:19

rennf93 marked this pull request as ready for review November 1, 2026 06:19

rennf93 and others added 3 commits November 1, 2026 07:23

github-actions Bot removed the dependencies pyproject.toml / uv.lock or panel package manifests label Nov 1, 2026

rennf93 merged commit 53cfe66 into slave Nov 1, 2026
26 of 29 checks passed

github-project-automation Bot moved this from Backlog to Done in RoboCo Kanban Nov 1, 2026

github-actions Bot locked and limited conversation to collaborators Nov 1, 2026

rennf93 deleted the bug/claude/green-deploy-blockers branch November 1, 2026 12:40

Navigation Menu

Repository navigation

fix: green deploy blockers (decisions crash-loop, wsx container names, worktree imports) - #1325

fix: green deploy blockers (decisions crash-loop, wsx container names, worktree imports)#1325
rennf93 merged 8 commits into
slavefrom
bug/claude/green-deploy-blockers

rennf93 commented Nov 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 1, 2026

Uh oh!

github-actions Bot commented Nov 1, 2026

Uh oh!

github-actions Bot commented Nov 1, 2026

Uh oh!

github-actions Bot commented Nov 1, 2026

Uh oh!

github-actions Bot commented Nov 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

rennf93 commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why the deploy stopped

Changes

Verification

Open for the CEO

Uh oh!

github-actions Bot commented Nov 1, 2026

Uh oh!

github-actions Bot commented Nov 1, 2026

DevOps image toolchain smoke results

DevOps toolchain smoke check (agent-devops)

Uh oh!

github-actions Bot commented Nov 1, 2026

Agent Image Smoke (Playwright) results

Playwright image size delta (real docker inspect sizes)

Headless chromium launch smoke check

Playwright MCP server smoke check

Headless browser verification screenshot (ux-qa image)

Uh oh!

github-actions Bot commented Nov 1, 2026

DevOps image toolchain smoke results

DevOps toolchain smoke check (agent-devops)

Uh oh!

github-actions Bot commented Nov 1, 2026

Agent Image Smoke (Playwright) results

Playwright image size delta (real docker inspect sizes)

Headless chromium launch smoke check

Playwright MCP server smoke check

Headless browser verification screenshot (ux-qa image)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rennf93 commented Nov 1, 2026 •

edited

Loading

Playwright image size delta (real `docker inspect` sizes)

Playwright image size delta (real `docker inspect` sizes)