diff --git a/docs/per_user_workspace_isolation_v1.md b/docs/per_user_workspace_isolation_v1.md new file mode 100644 index 00000000..3bb1ecf5 --- /dev/null +++ b/docs/per_user_workspace_isolation_v1.md @@ -0,0 +1,483 @@ +# OMNIBIOAI_PER_USER_WORKSPACE_ISOLATION_V1 + +WORKSTREAM=OMNIBIOAI_PER_USER_WORKSPACE_ISOLATION_V1 +STATUS=PARTIAL +RESULT=Architecture audit complete; the existing launcher contract is not an acceptable per-user implementation. No shared Studio service was restarted, no Storage P0 branch was changed, and no production deployment was performed. + +## Acceptance result + +`PARTIAL`. The current system cannot pass the stated acceptance test. Two authenticated users cannot independently launch isolated runtimes because the existing launcher controls the same three Compose containers for every caller: + +```text +omnibioai-jupyter -> fixed container and fixed host port 8888 +omnibioai-rstudio -> fixed container and fixed host port 8787 +omnibioai-vscode -> fixed container and fixed host port 8083 +``` + +The current v1 API validates identity and bounds, but it applies those bounds to a shared container and keeps its manifest in a process-local `Map`. It is therefore a security foundation, not a qualified isolation implementation. The existing warning must remain until the vertical slice below is delivered in the launcher repository and wired into Studio. + +## Resumed implementation and runtime evidence + +This resume added a disposable local manager under `workspace-manager/`. It uses a durable `0600` JSON record store, IAM `/auth/validate` ownership, UUID-derived runtime/volume/network names, non-root Jupyter execution, server-side CPU and memory ceilings, Docker `NanoCpus`/`Memory` enforcement, owner-scoped lifecycle routes, expiring owner-bound sessions, and deletion cleanup. Workspace containers receive no Docker socket and no shared host directory. + +The disposable test passed from the isolated clone with Docker access granted only to the trusted test process: + +```text +DOCKER_SOCKET_PATH=/Users/manishkumar/.docker/run/docker.sock npm test +✔ two IAM owners receive isolated Jupyter runtimes and lifecycle enforcement (1238.627ms) +tests 1, pass 1, fail 0 +``` + +It verified distinct IAM-owned containers, volumes, networks, persisted records, actual Docker CPU/memory limits (`NanoCpus=250000000`, `Memory=536870912`), non-privileged execution, cross-user denial, running resource-change denial, stopped update, stop/resume, and deletion cleanup. + +The owner-success HTTP proxy request could not be qualified from this isolated host process: Docker Desktop's loopback-published runtime connection resets inside the manager process (`UND_ERR_SOCKET`), although a direct read-only Node probe to the same disposable port succeeds. This is recorded as a proxy qualification blocker, not treated as a pass. + +The existing `docker-socket-proxy` also cannot yet be used by this manager for production lifecycle creation: `/networks/*` and `/volumes/*` are explicitly denied, and named-volume creation is not allowed. Its policy was not weakened. The qualified test therefore used a trusted manager process with daemon access; that is not a production deployment claim. + +### Docker API permission matrix + +The deployed shared socket proxy currently permits the container lifecycle read/create/start/stop/delete subset, but explicitly denies network and volume categories. The manager's new control plane exposes no arbitrary Docker path; each operation is fixed and every managed resource is checked by deterministic name and ownership labels before mutation. + +| Operation | Docker API | Shared proxy | Manager control plane | Lifecycle need | Security boundary | +|---|---|---:|---:|---|---| +| Create container | `POST /containers/create` | Allowed with body policy | Allowed, fixed Jupyter image/config | Create | UUID name, fixed image, labels, non-root, no privileged/socket/host mount | +| Inspect container | `GET /containers/{id}/json` | Allowed | Allowed after label/name verification | Recovery/status | Runtime ID is never accepted as owner proof | +| Start/stop | `POST /containers/{id}/start|stop` | Allowed | Allowed after managed-resource verification | Lifecycle | Owner record plus labels | +| Update | `POST /containers/{id}/update` | Not used by shared Launcher path | Allowed with server allocation only | Stopped resource change | CPU/memory only, GPU zero | +| Delete container | `DELETE /containers/{id}` | Allowed | Allowed after verification | Delete | No arbitrary IDs or names | +| Create/inspect/delete network | `POST/GET/DELETE /networks...` | Denied category | Allowed only for `obws-network-{uuid}` | Per-workspace network | Dedicated manager boundary; no client passthrough | +| Create/inspect/delete volume | `POST/GET/DELETE /volumes...` | Denied category | Allowed only for `obws-volume-{uuid}` | Persistent workspace storage | Dedicated manager boundary; no client passthrough | +| Network attach/detach | `/networks/{id}/connect|disconnect` | Denied category | Not exposed; container is created on its server-selected network | Not required | Avoids arbitrary cross-network attachment | + +The shared proxy cannot safely be widened in place: it serves unrelated workbench/TES consumers and its category-level rules do not express “only this workspace UUID and these labels.” The trusted control plane is therefore intentionally narrow in code, but production deployment still needs a separate socket-proxy instance or equivalent policy-enforcing sidecar with the same per-resource restrictions. The raw daemon socket is never exposed to users or workspace containers. + +### Threat model and routing failure handling + +- Cross-user access: IAM identity and owner organization/user key are required on every API and session route; mismatches return `404`. +- Docker privilege escalation: no privileged mode, host networking, arbitrary mounts, devices, Docker socket, or client-selected image/configuration. +- Arbitrary host mounts: only one server-created Docker named volume is mounted at `/home/jovyan/work`. +- SSRF: upstream host and port come only from server-created runtime metadata; clients supply neither URL nor container ID. +- Container breakout exposure: Jupyter runs as `1000:1000` with `no-new-privileges`; the workspace has no daemon control plane access. +- Resource exhaustion: server ceilings and actual Docker `NanoCpus`/`Memory` values are enforced; GPU is always zero in this slice. +- Stale routing: sessions expire, deleted/stopped workspaces are rejected, and missing runtimes become `failed` during recovery. +- `UND_ERR_SOCKET`/`ECONNRESET`: readiness and direct Node probes succeed, private container IPs are unreachable from the host, and the manager's host-process request to Docker Desktop's loopback-published port resets. The error is returned as `502 workspace proxy unavailable`; it is not retried or hidden. This remains an environment/control-plane integration blocker. + +## Architecture audit + +### Studio + +- `src/ui/pages/Studio.jsx` exposes the Launcher entry point. +- `src/ui/pages/Services.jsx` polls `/api/launcher/status/:tool` and opens fixed JupyterLab, RStudio, and VS Code destinations. +- `docker-compose.yml` defines fixed `jupyter`, `rstudio`, and `vscode` services with fixed container names, host ports, shared `${MACHINE_DIR}/data`, and shared `${MACHINE_DIR}/work` mounts. +- `docker/nginx-router.conf` has static service routes. It does not select a workspace UUID or validate a workspace session token. +- The Docker socket proxy is a useful existing control boundary, but its current Launcher allowlist was intentionally limited to inspect/start/stop of fixed containers; it does not qualify create/inspect/update/stop/remove for per-workspace resources. + +### Launcher + +The sibling `omnibioai-launcher` repository already contains: + +- profiles and environment types (`jupyterlab`, `rstudio`, `vscode`, `terminal`); +- server-side request validation, resource ceilings, image reference validation, architecture checks, and pre-provisioned-GPU checks; +- IAM `/auth/validate` calls and `workspace.launch`/`platform.manage_infra` permission gates; +- user-plus-organization manifest lookup checks; +- legacy fixed-container lifecycle routes retained for Studio compatibility. + +It does **not** yet contain: + +- per-user container or scheduler creation; +- owner-scoped named volumes and a filesystem containment boundary; +- durable workspace metadata; +- workspace lifecycle state persistence or recovery; +- workspace-specific proxy routing and expiring/revocable access sessions; +- deletion cleanup for runtime, volume, network, and route state. + +### Auth, gateway, workbench, and storage contracts + +- `omnibioai-auth` is authoritative for token validation, revocation, user identity, organization context, and permissions. `/auth/validate` returns the numeric database identifiers in the current contract; downstream code must normalize them at the boundary. +- `omnibioai-api-gateway` propagates verified identity and organization context. A workspace API must not trust client-supplied identity or organization headers. +- `omnibioai-workbench` has IAM principal bindings and canonical Artifact/Project ownership concepts. It does not currently own the interactive workspace lifecycle. +- Workspace object attachments must call the existing authorized Artifact and Project contracts. An object ID or URI is context, never authorization. +- Storage quota settings remain `STORAGE_QUOTA_MODE=audit` and `STORAGE_QUOTA_BASELINE_VERIFIED=false`. The workspace slice must not introduce strict quota enforcement or assume the physical HDD exists. + +## Required local vertical slice + +The first qualified backend should be a dedicated `workspace-manager` boundary behind the existing Docker socket proxy. Its API contract is: + +```text +POST /api/workspaces +GET /api/workspaces +GET /api/workspaces/{workspace_uuid} +POST /api/workspaces/{workspace_uuid}/start +POST /api/workspaces/{workspace_uuid}/stop +PATCH /api/workspaces/{workspace_uuid}/resources +DELETE /api/workspaces/{workspace_uuid} +POST /api/workspaces/{workspace_uuid}/sessions +DELETE /api/workspaces/{workspace_uuid}/sessions/{session_id} +``` + +Every handler must first validate the bearer token with IAM and derive `(user_id, organization_id)` from that response. Every lookup must include both owner dimensions, so an unknown, cross-user, or cross-organization UUID returns the same `404` shape. Client-supplied container names, host paths, Docker arguments, devices, capabilities, privileged flags, and identity fields are rejected. + +The persisted record must include: + +```text +workspace_uuid, owner_user_id, organization_id, project_id?, profile_id, +environment_type, requested_cpu, requested_memory_bytes, requested_gpu, +architecture, backend, runtime_id, volume_name, network_name, state, +state_reason, created_at, updated_at, deleted_at?, revision +``` + +The Docker implementation must create names from a server-generated UUID, for example `obws-{uuid}`, and use one owner-scoped named volume and one owner-scoped network per workspace. The runtime must be non-root where the selected image supports it, must not receive the Docker socket, must not mount a shared host directory, and must receive only the workspace volume at its fixed in-container working directory. GPU requests are zero unless an operator policy and the selected image/backend explicitly qualify them. + +Resource changes use this state machine: + +```text +stopped --PATCH resources--> stopped (update/recreate, then persist) +running --PATCH resources--> 409 with + "Stop this workspace to change its CPU or memory allocation." +running --stop--------------> stopping -> stopped +stopped --start-------------> starting -> running +any failure------------------> failed (retain reason and audit event) +``` + +The manager must apply Docker `NanoCpus`, `Memory`, and only policy-approved device requests before starting the runtime. A successful API response must be based on inspected runtime state and actual allocation, not on the presence of the legacy shared service. + +## Backend adapter contract + +The manager should depend on an adapter with these operations: + +```text +create(spec, owner) -> runtime_id, endpoint, allocation +inspect(runtime_id) -> state, allocation, image architecture +start(runtime_id) +stop(runtime_id) +recreate(runtime_id, allocation) +remove(runtime_id) +open_session(runtime_id, expiry, owner) +revoke_session(session_id) +``` + +Only `LocalDockerBackend` is in scope for the first end-to-end qualification. Kubernetes, Slurm, and cloud adapters should implement the same contract later, but are not implemented or claimed here. Images must be selected from operator configuration and qualify for the requested `amd64` or `arm64` architecture before creation. + +## Workspace proxy contract + +The static `/jupyter`, `/rstudio`, and `/vscode` routes cannot provide this boundary. The proxy must resolve a route from `(workspace_uuid, session_token)` to the manager's runtime endpoint, verify an unexpired session bound to the authenticated owner and organization, and deny: + +- a guessed UUID; +- a valid session for another workspace; +- a session from another user or organization; +- an expired or revoked session; +- direct access to container names or host ports. + +The route must never expose the Docker socket, runtime management endpoint, or a shared host directory to the workspace process. + +## Studio UX changes required after the API exists + +The existing Code entry should become `My Workspaces`, with the following server-backed flow: + +1. Create Workspace +2. Choose Profile +3. Select JupyterLab, RStudio, VS Code, or Terminal +4. Configure CPU, memory, optional GPU, and architecture +5. Attach authorized Artifacts and Project context +6. Review and Launch +7. Stop, Resume, Change resources while stopped, or Delete + +The UI must display the manager's state (`Creating`, `Starting`, `Running`, `Stopping`, `Stopped`, `Failed`) and inspected allocations. It must not infer `Running` from the legacy service health endpoint. The allocation warning becomes contextual: `Stop this workspace to change its CPU or memory allocation.` + +## Test qualification matrix + +| Test | Current result | Required qualification | +|---|---|---| +| Two users launch separate workspaces | FAIL: fixed shared containers | Two UUIDs, runtimes, volumes, networks, and routes | +| Same user creates multiple workspaces | FAIL: process-local manifest only; no runtime allocation | Independent persisted records and runtimes | +| Cross-user access denial | PARTIAL: manifest lookup checks owner/org | Runtime and proxy checks must also be owner/org scoped | +| Cross-organization isolation | PARTIAL: manifest lookup checks owner/org | Creation, route, object attachment, and cleanup checks | +| Workspace-specific proxy routing | FAIL: static routes | Expiring, revocable, workspace-bound sessions | +| CPU and memory enforcement | PARTIAL: fixed-container update path | Create/recreate per-workspace runtime and inspect actual limits | +| Running resource-change denial | PASS for shared-container path | Preserve for per-workspace path with safe stop/recreate | +| Stopped resource-change workflow | FAIL: no per-workspace runtime | Stop, update/recreate, inspect, persist, resume | +| Restart and persistence | FAIL: manifest is in-memory | Durable metadata and runtime reconciliation | +| Concurrent launch capacity | FAIL: no per-user creation path | Capacity policy, queueing, and atomic allocation | +| Container failure recovery | FAIL | Reconcile `failed` state and preserve volume | +| IAM revocation | PARTIAL: request-time validation exists | Revoke sessions and deny future lifecycle/proxy access | +| Deletion and cleanup | FAIL | Remove runtime, network, volume, route, and metadata safely | +| Architecture compatibility | PARTIAL: image inspection exists | Validate created runtime image and backend placement | +| Existing shared workspace regression | PASS only for legacy compatibility routes | Keep legacy path explicitly admin-only and non-workspace | + +## Requested report fields + +WORKSPACE_MANAGER=The disposable `workspace-manager/` is a qualified local slice only; the deployed `omnibioai-launcher` remains a shared-container controller and must own the production implementation. + +IAM_OWNERSHIP=PARTIAL. IAM validation and permission checks exist; owner and organization checks exist for manifests. They are not yet applied to runtime IDs, proxy sessions, durable records, and cleanup because those resources do not exist. + +PERSISTENCE=PARTIAL. The disposable slice persists owner-scoped records atomically in a `0600` JSON store; the deployed Launcher still uses its process-local JavaScript `Map`. + +LOCAL_DOCKER_BACKEND=PARTIAL. The disposable manager created and cleaned real per-user ARM64 Jupyter runtimes, volumes, and networks with inspected CPU/memory limits. Production qualification remains blocked by the existing socket proxy and the isolated manager-process loopback reset. + +PER_USER_ISOLATION=PARTIAL. The disposable slice creates UUID-derived containers, named volumes, separate networks, and non-root runtimes with no socket mount; Studio and deployed Launcher still retain the fixed shared topology. + +RESOURCE_ENFORCEMENT=PASS for the disposable local slice; PARTIAL overall. Actual Docker limits were inspected and running changes were denied; production proxy/Launcher integration remains incomplete. + +WORKSPACE_PROXY=PARTIAL. The manager has owner-bound expiring session routing and cross-user denial; owner-success upstream routing was not qualified because the isolated loopback transport resets, and Studio/nginx integration is not wired. + +JUPYTERLAB=PARTIAL. Two isolated ARM64 Jupyter runtimes qualified in disposable Docker; Studio still opens the shared service. + +RSTUDIO=PARTIAL. Existing shared service only; no isolated runtime qualification. + +VSCODE=PARTIAL. Existing shared service only; no isolated runtime qualification. + +TERMINAL=PARTIAL. It aliases the shared VS Code Server integrated terminal and is not an isolated terminal runtime. + +MULTI_USER_E2E=PARTIAL. Real two-user disposable Docker creation, resource inspection, denial, lifecycle, and cleanup passed. Owner-success proxy and Studio integration remain unqualified. + +SECURITY_TESTS=PARTIAL. Existing Launcher security tests remain passing by audit, and the live test verifies IAM owner scoping, cross-user denial, non-privileged runtimes, no socket mount, and actual limits. Proxy owner-success and session expiry/revocation still need qualification. + +REGRESSION_TESTS=PASS for the disposable manager test and read-only shared-service audit; existing Studio services were not restarted or modified. Full Studio regression remains blocked on integration wiring. + +ROOT_CAUSE_SHARED_CONTAINER=The legacy Launcher owns three Compose services with hard-coded container names, host ports, shared host mounts, and static proxy routes. The v1 API added identity and validation around that topology but did not replace the topology. + +REMAINING_BLOCKERS=Move the manager into the Launcher service boundary; provide a separate constrained Docker proxy that securely authorizes per-workspace network/volume lifecycle operations without weakening the shared proxy; resolve the isolated host-process loopback reset and qualify owner-success proxy routing; add durable restart recovery, session revocation, concurrent-capacity, failure-recovery, architecture, and IAM-revocation tests; integrate Artifact/Project authorization; preserve shared legacy routes; and wire Studio Code to the new API without restarting shared services. + +WORKTREE_CLEAN=YES in the isolated clone at `/private/tmp/omnibioai-per-user-workspace-isolation-v1`; the source checkout contains only the mirrored deliverable plus its pre-existing `omnibioai-cli/` change. + +PUSH_PERFORMED=NO +DEPLOYMENT_PERFORMED=NO + +## Workspace Manager Proxy Transport V2 qualification + +This scoped follow-up was run from the independent clone at +`/private/tmp/omnibioai-per-user-workspace-isolation-v1`, based on `8d8293b`. + +### Confirmed transport root cause + +The `UND_ERR_SOCKET`/`ECONNRESET` symptom was a readiness race. Docker's +published port accepted TCP before JupyterLab had finished starting its HTTP +server. Direct probes performed after startup succeeded, while the manager's +immediate request reset. The manager now waits for an authenticated `/lab` +HTTP response, rather than treating a TCP connect as readiness. The proxy uses +server-resolved host/port metadata, `fetch` plus `Readable.fromWeb()` for +GET/HEAD response streaming, and a bounded request timeout. Public upstream +errors remain sanitized as 502 responses. + +### Routing and WebSocket boundary + +HTTP and WebSocket routes validate IAM, owner organization/user, workspace +state, and an expiring workspace-bound session before contacting Jupyter. +Clients cannot supply an upstream URL, container ID, host, or port. The +WebSocket upgrade forwards only the required upgrade headers and the manager's +server-generated Jupyter token; IAM bearer tokens and browser cookies are not +forwarded. The raw WebSocket transport was qualified against a live Jupyter +kernel channel, including a ping/pong exchange and cross-user upgrade denial. + +### Control-plane boundary and limitation + +The test runs the manager as a private, disposable trusted control-plane +container with a loopback-only published manager port. Workspace containers +receive neither the Docker socket nor control-plane access. The manager API has +no arbitrary Docker-path passthrough: image, container configuration, names, +labels, volume, network, mounts, and resource requests are server-derived. +Every inspected or deleted resource must have the expected UUID-derived name +and ownership labels. Host mounts, privileged mode, host networking, devices, +GPU requests, and user-selected images are rejected by the fixed runtime +configuration. + +This is not a claim that a generic Docker socket mount is a complete tenant +boundary. The shared OmniBioAI socket proxy remains unchanged and still denies +`POST /networks/create` and `POST /volumes/create`. A production deployment +requires a separate private control-plane sidecar/proxy that authenticates the +manager and enforces the same operation and managed-label allowlist at the +Docker API boundary. It must not be exposed to Studio browsers or workspace +containers. That deployment was not attached to the live stack. + +### V2 runtime evidence + +Commands executed in the isolated clone: + +```text +node --check server.js +npm test + 1 passed, 0 failed — real two-owner Docker lifecycle/resource regression +npm run test:transport + 1 passed, 0 failed — containerized manager, two live Jupyter runtimes, + authorized HTTP, request body/status propagation, proxied kernel listing, + WebSocket kernel-channel ping/pong, cross-user HTTP/WS denial, cleanup +``` + +The Docker engine was 29.8.1 on ARM64. Existing shared services remained at +44 running containers before and after qualification; no shared container was +restarted or reconfigured. Disposable `obws-*` containers, networks, and +volumes were removed and verified absent after each run. Pytest was not +available in the environment, so no pytest count is claimed. + +Streaming is implemented without buffering GET/HEAD response bodies, and the +live kernel-list response was exercised through the manager. Chunked, +long-running, cancellation, and upstream-disconnect stress cases remain +unqualified; WebSocket reconnect and full Studio/Launcher routing remain +outside this workstream. + +## Safety record + +- No shared Studio service was started, stopped, restarted, or recreated. +- No storage data, quota mode, or Storage P0 integration branch was changed. +- No Docker socket was exposed to a workspace container. +- No production deployment or push was performed. +- The repository's `.git` directory was not writable for linked-worktree registration, so the work was performed in an independent local clone under `/private/tmp` rather than a registered linked worktree. + +## Workspace Control Plane Production Readiness V1 + +### Implementation boundary + +The workspace manager now supports an optional private control-plane URL. When +`CONTROL_PLANE_URL` is configured, it does not open the Docker socket; it +sends only fixed, signed requests to `control-plane.js`. The control plane +exposes `healthz` plus nine fixed workspace operations: image architecture, +inspect, create volume/network/container, start, stop, update, and remove. +There is no raw Docker path or caller-selected Docker resource endpoint. + +Requests use an HMAC-SHA256 service credential, service identity, timestamp, +random nonce, and correlation ID. Timestamps have a 30-second skew window and +nonces are single-use. The credential is injected through the environment in +the disposable deployment definition; it is not in source or browser traffic. +The authenticated service is the workspace manager, while end-user ownership +continues to derive from IAM claims at the manager boundary. The control plane +validates the manager-provided immutable workspace UUID and owner/org shape, +and the Docker adapter validates deterministic names and labels before every +mutation. Labels are supporting metadata; the manager's durable registry and +service authentication are the authorization boundary. + +`Dockerfile.control-plane` and `docker-compose.control-plane.test.yml` define +a private, internal-only, resource-limited control-plane service with a health +check and no published port. The compose file is configuration-only and was +not attached to the live stack. The disposable qualification used an +equivalent isolated Docker network, a private control-plane container, and a +manager container with no Docker socket mount. The control-plane socket is +still a host-equivalent privilege boundary: a compromise of that service can +act on Docker, so production deployment requires host hardening, secret +rotation, image signing/scanning, restricted service identity, and audit +collection. A read-only socket bind does not reduce Docker API authority. + +### Threat model coverage + +| Threat | Mitigation | Evidence | +|---|---|---| +| Unauthenticated or replayed control-plane request | HMAC service identity, timestamp, single-use nonce | Unit test: auth, replay, wrong identity, stale timestamp | +| Cross-user/org workspace access | IAM-derived owner key, workspace-bound sessions, hidden 404 | Live two-user and cross-organization transport test | +| Forged labels/unmanaged resources | Durable manager record plus deterministic names and label checks | Docker adapter checks; unmanaged operation API is not exposed | +| Arbitrary Docker API, privileged mode, host mount/socket, host network | Fixed operation map and fixed runtime HostConfig | Runtime inspection and fixed-operation unit test | +| SSRF/arbitrary upstream | Server-resolved host/port and path-only proxying | Live authorized/denied routing test | +| Resource exhaustion | CPU/memory ceilings and Docker-enforced limits | Live Docker inspection | +| Stale/replayed sessions | Expiring workspace-bound session and fresh auth per upgrade | Live ownership tests; reconnect uses a new handshake | +| Orphan cleanup/restart | Durable state and reconciliation, explicit exact-name cleanup | Lifecycle restart test and disposable inventory checks | +| Audit leakage | Structured operation outcome/correlation logs without tokens or bodies | Control-plane implementation review | + +Concurrent lifecycle serialization, service credential rotation, JWT audience +validation, full administrator policy, persistent audit sink, and deletion +retention policy remain deployment requirements. The current local adapter +deletes the workspace volume as part of explicit workspace deletion, matching +the existing disposable-slice contract; a production retention/data-delete +policy must be selected before user data is exposed. + +### V1 qualification evidence + +Baseline commit: `275351f`. Final qualification used Docker Engine 29.8.1, +ARM64, and the current live shared-container inventory. The manager-to-control- +plane test used two IAM identities plus a third identity in another +organization, two isolated workspaces, separate containers/volumes/networks, +actual CPU/memory limits, owner HTTP, WebSocket ping/pong, authorized +WebSocket reconnect, cross-user/org denial, unauthenticated denial, and exact +resource cleanup. Existing shared services were not restarted or changed. + +Commands: + +```text +npm test + 3 passed, 0 failed +npm run test:transport + 1 passed, 0 failed +docker compose -f docker-compose.control-plane.test.yml config + configuration validation only; no service started +``` + +The transport test is real Docker and real service authentication; the +workspace manager container has no `/var/run/docker.sock`. Chunked or +long-running streaming, client cancellation, upstream disconnect recovery, +concurrent lifecycle races, and production deployment of the sidecar remain +unqualified. Therefore this workstream remains `PARTIAL`. + +## Workspace Control Plane Final Qualification V1 + +The final qualification added explicit HMAC key IDs with active/previous-key +rotation overlap, persistent sanitized JSONL audit events, and separate +runtime deletion from persistent-data deletion. Workspace deletion retains +the volume by default; `purge_data: true` is the explicit destructive path. + +Real Docker evidence included: + +```text +npm test + 3 passed, 0 failed — lifecycle and HMAC/replay/rotation tests +npm run test:transport + 1 passed, 0 failed — private manager/control-plane network, HTTP, + WebSocket/reconnect, cross-user/org denial, unmanaged inspect/remove denial, + audit persistence, retained volume, explicit purge, cleanup +docker compose -f docker-compose.control-plane.test.yml config --quiet + passed — configuration validation only +``` + +The current inventory remained 44 shared containers before and after. No +shared container was restarted, stopped, recreated, or reconfigured. No +`obws-*` disposable resource remained after cleanup. + +Still unqualified are delayed/chunked/binary long-running streaming with +bounded-memory/backpressure measurements, cancellation during an active +stream, upstream disconnect recovery, concurrent lifecycle races, Docker API +partial-failure injection, and a complete manager-plus-control-plane Compose +lifecycle run. The isolated Docker-network test is not claimed as full +Compose operational acceptance. Production key storage/rotation persistence, +centralized audit retention/alerting, and backup/recovery remain operational +requirements. No production deployment or Studio/Launcher integration was +performed. + +## Workspace Transport and Fault Recovery P0 V1 + +The proxy now propagates downstream request aborts to the upstream fetch, +cleans abort listeners after normal completion, and handles upstream stream +errors by destroying an already-started response instead of attempting a +second HTTP status. This prevents the manager process from terminating during +client cancellation. The live disposable Jupyter test transfers a 32 MiB +workspace file incrementally across multiple chunks and then aborts a second +request; the manager remains available and all resources are cleaned. + +A 256 MiB transfer was attempted against the same real workspace route. It +failed after 7,914 bytes with `UND_ERR_SOCKET`/peer close. The 32 MiB run +passed after the targeted fix, but the required 256 MiB acceptance target and +bounded-memory/backpressure measurements are therefore not passed. + +The existing per-workspace single-writer lock and control-plane restart test +remain active. Full delayed-stream fixture coverage, slow/stalled-client +backpressure, ten-run cancellation/disconnect repetition, Docker fault +injection, ambiguous-operation reconciliation, and exact manager-plus-control- +plane Compose lifecycle execution remain unqualified. No shared service was +changed; the live inventory remains 44 running shared containers. + +## Workspace Runtime Reliability Acceptance V1 + +Mutating operations for one workspace now serialize through a single-writer +in-process lock. Duplicate stop/start requests are therefore deterministic +within one manager instance. Distributed locking and multi-manager registry +coordination are not implemented; deployment remains single-writer. + +The real Docker transport test now restarts the private control plane, issues +concurrent stop/stop and start/start requests, verifies registry recovery, +denies disposable unmanaged-resource inspect/remove operations, and checks +audit persistence, retained-volume deletion, explicit purge, and cleanup. It +passed after the targeted lifecycle-lock change. Shared infrastructure stayed +at 44 running containers. + +Still unqualified are delayed/long-running streaming with measurable +backpressure and memory bounds, cancellation during an active stream, upstream +socket failure recovery, Docker timeout/partial-operation fault injection, +20-run concurrency repetition, and the exact manager-plus-control-plane +Compose lifecycle. Normal live Jupyter proxy responses are not claimed as +evidence for those stress cases; this reliability workstream remains +`PARTIAL`. diff --git a/workspace-manager/Dockerfile.control-plane b/workspace-manager/Dockerfile.control-plane new file mode 100644 index 00000000..df92d515 --- /dev/null +++ b/workspace-manager/Dockerfile.control-plane @@ -0,0 +1,6 @@ +FROM node:20-bookworm-slim +WORKDIR /app +COPY docker-control.js control-plane.js ./ +USER node +HEALTHCHECK --interval=5s --timeout=2s --retries=12 CMD node -e "fetch('http://127.0.0.1:5192/healthz').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))" +ENTRYPOINT ["node", "control-plane.js"] diff --git a/workspace-manager/control-plane-client.js b/workspace-manager/control-plane-client.js new file mode 100644 index 00000000..1d73af74 --- /dev/null +++ b/workspace-manager/control-plane-client.js @@ -0,0 +1,37 @@ +'use strict'; + +const crypto = require('node:crypto'); + +const URL_BASE = process.env.CONTROL_PLANE_URL; +const SERVICE_ID = process.env.CONTROL_PLANE_SERVICE_ID || 'workspace-manager'; +const SERVICE_SECRET = process.env.CONTROL_PLANE_SERVICE_SECRET; +const SERVICE_KEY_ID = process.env.CONTROL_PLANE_SERVICE_KEY_ID || 'active'; + +function clientRequest(operation, record, extra = {}) { + if (!URL_BASE || !SERVICE_SECRET) throw new Error('control-plane service authentication is not configured'); + const payload = JSON.stringify({ record, ...extra }); + const timestamp = String(Date.now()); + const nonce = crypto.randomBytes(16).toString('hex'); + const signature = crypto.createHmac('sha256', SERVICE_SECRET).update(`${timestamp}.${nonce}.${payload}`).digest('hex'); + const correlationId = crypto.randomUUID(); + return fetch(`${URL_BASE}/v1/workspaces/${encodeURIComponent(record.id)}/${operation}`, { + method: 'POST', + headers: { 'content-type': 'application/json', 'x-service-id': SERVICE_ID, 'x-service-key-id': SERVICE_KEY_ID, 'x-request-timestamp': timestamp, 'x-request-nonce': nonce, 'x-request-signature': signature, 'x-correlation-id': correlationId }, + body: payload, + signal: AbortSignal.timeout(30000), + }).then(async response => ({ status: response.status, body: await response.json().catch(() => ({})) })); +} + +class ControlPlaneClient { + imageArchitecture() { return clientRequest('image-architecture', { id: '00000000-0000-0000-0000-000000000000' }).then(result => result.body.architecture || null); } + inspectContainer(record) { return clientRequest('inspect', record); } + createVolume(record) { return clientRequest('create-volume', record).then(result => { if (result.status < 200 || result.status >= 300) throw new Error(result.body.error || 'workspace volume creation failed'); return result.body; }); } + createNetwork(record) { return clientRequest('create-network', record).then(result => { if (result.status < 200 || result.status >= 300) throw new Error(result.body.error || 'workspace network creation failed'); return result.body; }); } + createContainer(record, extra) { return clientRequest('create-container', record, extra).then(result => { if (result.status < 200 || result.status >= 300 || !result.body.Id) throw new Error(result.body.error || 'workspace container creation failed'); record.runtime_id = result.body.Id; record.runtime_name = `obws-${record.id}`; record.volume_name = `obws-volume-${record.id}`; record.network_name = `obws-network-${record.id}`; record.host_port = extra.hostPort; return result.body; }); } + start(record) { return clientRequest('start', record); } + stop(record) { return clientRequest('stop', record); } + update(record, allocation) { return clientRequest('update', record, { allocation }); } + remove(record, options = {}) { return clientRequest('remove', record, options); } +} + +module.exports = { ControlPlaneClient }; diff --git a/workspace-manager/control-plane.js b/workspace-manager/control-plane.js new file mode 100644 index 00000000..df0d41ae --- /dev/null +++ b/workspace-manager/control-plane.js @@ -0,0 +1,67 @@ +'use strict'; + +const http = require('node:http'); +const crypto = require('node:crypto'); +const { DockerControlPlane, IMAGE } = require('./docker-control'); + +const HOST = process.env.CONTROL_PLANE_HOST || '127.0.0.1'; +const PORT = Number(process.env.CONTROL_PLANE_PORT || 5192); +const SERVICE_ID = process.env.CONTROL_PLANE_SERVICE_ID || 'workspace-manager'; +const SERVICE_SECRET = process.env.CONTROL_PLANE_SERVICE_SECRET; +const SERVICE_KEY_ID = process.env.CONTROL_PLANE_SERVICE_KEY_ID || 'active'; +const PREVIOUS_SERVICE_SECRET = process.env.CONTROL_PLANE_PREVIOUS_SERVICE_SECRET; +const PREVIOUS_SERVICE_KEY_ID = process.env.CONTROL_PLANE_PREVIOUS_SERVICE_KEY_ID || 'previous'; +const AUDIT_PATH = process.env.CONTROL_PLANE_AUDIT_PATH || '/var/lib/omnibioai-workspace-control/audit.jsonl'; +const MAX_CLOCK_SKEW_MS = 30_000; +const seenNonces = new Map(); +let auditChain = Promise.resolve(); +const control = new DockerControlPlane(); + +function json(res, status, value) { const data = JSON.stringify(value); res.writeHead(status, { 'content-type': 'application/json', 'content-length': Buffer.byteLength(data) }); res.end(data); } +function fail(res, status, error) { return json(res, status, { error }); } +async function body(req) { let raw = ''; for await (const chunk of req) { raw += chunk; if (raw.length > 128 * 1024) throw new Error('request too large'); } return raw ? JSON.parse(raw) : {}; } +function validWorkspace(value) { return value && /^[0-9a-f-]{36}$/.test(value.id) && Number.isInteger(Number(value.owner_user_id)) && Number.isInteger(Number(value.organization_id)); } +function authenticate(req, raw) { + if ((!SERVICE_SECRET && !PREVIOUS_SERVICE_SECRET) || req.headers['x-service-id'] !== SERVICE_ID) return false; + const timestamp = Number(req.headers['x-request-timestamp']); const nonce = req.headers['x-request-nonce']; const signature = req.headers['x-request-signature']; + if (!Number.isFinite(timestamp) || Math.abs(Date.now() - timestamp) > MAX_CLOCK_SKEW_MS || !/^[a-f0-9]{32}$/.test(nonce) || !/^[a-f0-9]{64}$/.test(signature)) return false; + if (seenNonces.has(nonce)) return false; + const keyId = req.headers['x-service-key-id']; const secret = keyId === SERVICE_KEY_ID ? SERVICE_SECRET : keyId === PREVIOUS_SERVICE_KEY_ID ? PREVIOUS_SERVICE_SECRET : null; + if (!secret) return false; + const expected = crypto.createHmac('sha256', secret).update(`${timestamp}.${nonce}.${raw}`).digest('hex'); + if (!crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(signature))) return false; + seenNonces.set(nonce, Date.now() + MAX_CLOCK_SKEW_MS); return true; +} +function audit(event) { + const line = `${JSON.stringify({ timestamp: new Date().toISOString(), ...event })}\n`; + auditChain = auditChain.then(async () => { const fs = require('node:fs'); await fs.promises.mkdir(require('node:path').dirname(AUDIT_PATH), { recursive: true, mode: 0o700 }); await fs.promises.appendFile(AUDIT_PATH, line, { mode: 0o600 }); }); + return auditChain; +} +setInterval(() => { const now = Date.now(); for (const [nonce, expiry] of seenNonces) if (expiry < now) seenNonces.delete(nonce); }, MAX_CLOCK_SKEW_MS).unref(); + +const operations = { + 'image-architecture': async () => ({ architecture: (await control.imageArchitecture()) || null }), + inspect: async ({ record }) => control.inspectContainer(record), + 'create-volume': async ({ record }) => control.createVolume(record), + 'create-network': async ({ record }) => control.createNetwork(record), + 'create-container': async ({ record, token, hostPort }) => { if (!Number.isInteger(hostPort) || hostPort < 1024 || hostPort > 65535 || typeof token !== 'string' || token.length < 32) throw new Error('invalid runtime parameters'); return control.createContainer(record, { token, hostPort }); }, + start: async ({ record }) => control.start(record), + stop: async ({ record }) => control.stop(record), + update: async ({ record, allocation }) => control.update(record, allocation), + remove: async ({ record, purgeData }) => control.remove(record, { purgeData: purgeData === true }), +}; + +const server = http.createServer(async (req, res) => { + if (req.method === 'GET' && req.url === '/healthz') return json(res, 200, { ok: true, service: SERVICE_ID }); + if (req.method !== 'POST') return fail(res, 404, 'not found'); + const match = /^\/v1\/workspaces\/([0-9a-f-]{36})\/([a-z-]+)$/.exec(req.url); + if (!match || !operations[match[2]]) return fail(res, 404, 'not found'); + let raw; try { raw = await (async () => { let value = ''; for await (const chunk of req) { value += chunk; if (value.length > 128 * 1024) throw new Error('request too large'); } return value; })(); } catch { return fail(res, 400, 'invalid request'); } + if (!authenticate(req, raw)) { const event = { actor_service: req.headers['x-service-id'] || 'unknown', operation: match[2], correlation_id: req.headers['x-correlation-id'] || null, outcome: 'denied', reason: 'service authentication required' }; console.log(JSON.stringify(event)); audit(event).catch(() => {}); return fail(res, 401, 'service authentication required'); } + let input; try { input = JSON.parse(raw); } catch { return fail(res, 400, 'invalid JSON'); } + if (match[2] !== 'image-architecture' && (input.record?.id !== match[1] || !validWorkspace(input.record))) return fail(res, 403, 'invalid managed workspace'); + try { const event = { actor_service: SERVICE_ID, owner_user_id: input.record?.owner_user_id || null, organization_id: input.record?.organization_id || null, workspace_id: input.record?.id || null, operation: match[2], correlation_id: req.headers['x-correlation-id'] || null, outcome: 'success' }; const result = await operations[match[2]](input); console.log(JSON.stringify(event)); audit(event).catch(() => {}); return json(res, result?.status ?? 200, result?.body ?? result); } catch (error) { const event = { actor_service: SERVICE_ID, owner_user_id: input.record?.owner_user_id || null, organization_id: input.record?.organization_id || null, workspace_id: input.record?.id || null, operation: match[2], correlation_id: req.headers['x-correlation-id'] || null, outcome: 'failure', reason: error.message === 'managed Docker resource ownership mismatch' ? error.message : 'control-plane operation failed' }; console.log(JSON.stringify(event)); audit(event).catch(() => {}); return fail(res, 409, event.reason); } +}); + +if (require.main === module) server.listen(PORT, HOST, () => console.log(`control plane listening on ${HOST}:${PORT}`)); +module.exports = { server, authenticate, operations, IMAGE }; diff --git a/workspace-manager/docker-compose.control-plane.test.yml b/workspace-manager/docker-compose.control-plane.test.yml new file mode 100644 index 00000000..d7171c72 --- /dev/null +++ b/workspace-manager/docker-compose.control-plane.test.yml @@ -0,0 +1,42 @@ +services: + workspace-control-plane: + build: + context: . + dockerfile: Dockerfile.control-plane + command: ["node", "control-plane.js"] + user: "0:0" + environment: + CONTROL_PLANE_HOST: 0.0.0.0 + CONTROL_PLANE_PORT: 5192 + CONTROL_PLANE_SERVICE_ID: workspace-manager + CONTROL_PLANE_SERVICE_KEY_ID: active + CONTROL_PLANE_SERVICE_SECRET: ${CONTROL_PLANE_SERVICE_SECRET:?set a disposable secret} + CONTROL_PLANE_AUDIT_PATH: /state/audit.jsonl + DOCKER_SOCKET_PATH: /var/run/docker.sock + networks: [workspace-control-plane] + expose: ["5192"] + group_add: ["${DOCKER_SOCKET_GID:-0}"] + read_only: true + tmpfs: [/tmp] + cpus: 1 + mem_limit: 512m + healthcheck: + test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:5192/healthz').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))"] + interval: 5s + timeout: 2s + retries: 12 + volumes: + - type: volume + source: workspace-control-plane-state + target: /state + - type: bind + source: ${DOCKER_SOCKET_PATH:?set a host Docker socket path} + target: /var/run/docker.sock + read_only: true + +networks: + workspace-control-plane: + internal: true + +volumes: + workspace-control-plane-state: diff --git a/workspace-manager/docker-control.js b/workspace-manager/docker-control.js new file mode 100644 index 00000000..47c5ff07 --- /dev/null +++ b/workspace-manager/docker-control.js @@ -0,0 +1,99 @@ +'use strict'; + +const http = require('node:http'); + +const SOCKET_PATH = process.env.DOCKER_SOCKET_PATH || '/var/run/docker.sock'; +const IMAGE = 'ghcr.io/omnibioai/omnibioai-jupyter:1.0'; +const UUID = /^[0-9a-f-]{36}$/; + +function names(workspace) { + if (!workspace || !UUID.test(workspace.id)) throw new Error('invalid managed workspace id'); + return { container: `obws-${workspace.id}`, volume: `obws-volume-${workspace.id}`, network: `obws-network-${workspace.id}` }; +} + +function labels(workspace) { + return { + 'com.omnibioai.managed': 'workspace-manager-v1', + 'com.omnibioai.workspace.id': workspace.id, + 'com.omnibioai.workspace.owner_user_id': String(workspace.owner_user_id), + 'com.omnibioai.workspace.organization_id': String(workspace.organization_id), + }; +} + +function request(method, endpoint, requestBody) { + return new Promise((resolve, reject) => { + const encoded = requestBody === undefined ? undefined : JSON.stringify(requestBody); + const req = http.request({ socketPath: SOCKET_PATH, path: endpoint, method, + headers: encoded ? { 'content-type': 'application/json', 'content-length': Buffer.byteLength(encoded) } : {} }, res => { + let raw = ''; res.setEncoding('utf8'); res.on('data', chunk => { raw += chunk; }); + res.on('end', () => { let parsed; try { parsed = raw ? JSON.parse(raw) : {}; } catch { parsed = raw; } resolve({ status: res.statusCode, body: parsed }); }); + }); + req.on('error', reject); req.end(encoded); + }); +} + +function assertManaged(record, resource, expectedName) { + const actualName = resource.Name?.replace(/^\//, '') || resource.Name; + if (actualName !== expectedName) throw new Error('managed Docker resource name mismatch'); + const actual = resource.Config?.Labels || resource.Labels || {}; + const expected = labels(record); + for (const key of Object.keys(expected)) if (actual[key] !== expected[key]) throw new Error('managed Docker resource ownership mismatch'); +} + +class DockerControlPlane { + async imageArchitecture() { + const result = await request('GET', `/images/${encodeURIComponent(IMAGE)}/json`); + return result.status === 200 ? result.body.Architecture : null; + } + + async inspectContainer(record) { + const result = await request('GET', `/containers/${encodeURIComponent(record.runtime_id)}/json`); + if (result.status === 404) return result; + if (result.status !== 200 || !result.body || typeof result.body !== 'object') throw new Error('managed workspace container inspection failed'); + assertManaged(record, result.body, names(record).container); + return result; + } + + async createVolume(record) { + const result = await request('POST', '/volumes/create', { Name: names(record).volume, Driver: 'local', Labels: labels(record) }); + if (result.status < 200 || result.status >= 300) throw new Error('workspace volume creation failed'); + const inspected = await request('GET', `/volumes/${encodeURIComponent(names(record).volume)}`); + if (inspected.status !== 200) throw new Error('workspace volume inspection failed'); + assertManaged(record, inspected.body, names(record).volume); return inspected.body; + } + + async createNetwork(record) { + const result = await request('POST', '/networks/create', { Name: names(record).network, Driver: 'bridge', Labels: labels(record) }); + if (result.status < 200 || result.status >= 300) throw new Error('workspace network creation failed'); + const inspected = await request('GET', `/networks/${encodeURIComponent(names(record).network)}`); + if (inspected.status !== 200) throw new Error('workspace network inspection failed'); + assertManaged(record, inspected.body, names(record).network); return inspected.body; + } + + async createContainer(record, { token, hostPort }) { + const n = names(record); const architecture = await this.imageArchitecture(); + if (architecture !== 'arm64') throw new Error('workspace image is not available for arm64'); + const result = await request('POST', `/containers/create?name=${n.container}`, { + Image: IMAGE, User: '1000:100', Cmd: ['start-notebook.py', `--NotebookApp.token=${token}`, `--ServerApp.token=${token}`, '--ip=0.0.0.0', '--no-browser'], + Env: ['JUPYTER_ENABLE_LAB=yes', `JUPYTER_TOKEN=${token}`], ExposedPorts: { '8888/tcp': {} }, Labels: labels(record), + HostConfig: { NanoCpus: Math.round(record.allocation.cpu * 1e9), Memory: record.allocation.memory_bytes, DeviceRequests: [], + NetworkMode: n.network, PortBindings: { '8888/tcp': [{ HostIp: '127.0.0.1', HostPort: String(hostPort) }] }, + SecurityOpt: ['no-new-privileges:true'], Privileged: false, Binds: [`${n.volume}:/home/jovyan/work:rw`] }, + }); + if (result.status < 200 || result.status >= 300 || !result.body.Id) throw new Error('workspace container creation failed'); + record.runtime_id = result.body.Id; record.runtime_name = n.container; record.volume_name = n.volume; record.network_name = n.network; record.host_port = hostPort; + const inspected = await this.inspectContainer(record); assertManaged(record, inspected.body, n.container); return result.body; + } + + async start(record) { const result = await this.inspectContainer(record); if (result.status === 404) throw new Error('managed workspace container not found'); return request('POST', `/containers/${encodeURIComponent(record.runtime_id)}/start`); } + async stop(record) { const result = await this.inspectContainer(record); if (result.status === 404) return result; return request('POST', `/containers/${encodeURIComponent(record.runtime_id)}/stop`, { t: 5 }); } + async update(record, allocation) { const result = await this.inspectContainer(record); if (result.status === 404) throw new Error('managed workspace container not found'); return request('POST', `/containers/${encodeURIComponent(record.runtime_id)}/update`, { NanoCpus: Math.round(allocation.cpu * 1e9), Memory: allocation.memory_bytes, DeviceRequests: [] }); } + + async remove(record, { purgeData = false } = {}) { + if (record.runtime_id) { const inspected = await this.inspectContainer(record); if (inspected.status !== 404) await request('DELETE', `/containers/${encodeURIComponent(record.runtime_id)}?force=true`); } + const n = names(record); const volume = await request('GET', `/volumes/${encodeURIComponent(n.volume)}`); if (purgeData && volume.status === 200) { assertManaged(record, volume.body, n.volume); await request('DELETE', `/volumes/${encodeURIComponent(n.volume)}`); } + const network = await request('GET', `/networks/${encodeURIComponent(n.network)}`); if (network.status === 200) { assertManaged(record, network.body, n.network); await request('DELETE', `/networks/${encodeURIComponent(n.network)}`); } + } +} + +module.exports = { DockerControlPlane, IMAGE, labels, names, request }; diff --git a/workspace-manager/package.json b/workspace-manager/package.json new file mode 100644 index 00000000..f46fcfc2 --- /dev/null +++ b/workspace-manager/package.json @@ -0,0 +1,10 @@ +{ + "name": "omnibioai-workspace-manager-local", + "private": true, + "version": "0.1.0", + "description": "Disposable local-Docker per-user workspace manager vertical slice", + "scripts": { + "test": "node --test test/workspace-manager.test.js test/control-plane.test.js", + "test:transport": "node --test test/transport-container.test.js" + } +} diff --git a/workspace-manager/server.js b/workspace-manager/server.js new file mode 100644 index 00000000..073d5e89 --- /dev/null +++ b/workspace-manager/server.js @@ -0,0 +1,262 @@ +'use strict'; + +const http = require('node:http'); +const net = require('node:net'); +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const path = require('node:path'); +const { Readable } = require('node:stream'); +const { DockerControlPlane } = require('./docker-control'); +const { ControlPlaneClient } = require('./control-plane-client'); + +const PORT = Number(process.env.PORT || 5191); +const LISTEN_HOST = process.env.LISTEN_HOST || '127.0.0.1'; +const DB_PATH = process.env.WORKSPACE_DB_PATH || path.join(process.cwd(), 'workspaces.json'); +const IAM_URL = process.env.IAM_URL || 'http://auth-service:8001'; +const UPSTREAM_HOST = process.env.WORKSPACE_UPSTREAM_HOST || '127.0.0.1'; +const MAX_CPU = Number(process.env.WORKSPACE_MAX_CPU || 8); +const MAX_MEMORY = Number(process.env.WORKSPACE_MAX_MEMORY_BYTES || 8 * 1024 ** 3); +const SESSION_TTL_MS = 15 * 60 * 1000; + +const state = loadState(); +const control = process.env.CONTROL_PLANE_URL ? new ControlPlaneClient() : new DockerControlPlane(); +let writeChain = Promise.resolve(); +const workspaceLocks = new Map(); + +function withWorkspaceLock(id, operation) { + const previous = workspaceLocks.get(id) || Promise.resolve(); + let release; + const current = new Promise(resolve => { release = resolve; }); + const queued = previous.then(() => current); + workspaceLocks.set(id, queued); + return previous.then(operation).finally(() => { release(); if (workspaceLocks.get(id) === queued) workspaceLocks.delete(id); }); +} + +function loadState() { + try { + const value = JSON.parse(fs.readFileSync(DB_PATH, 'utf8')); + return { workspaces: value.workspaces || {}, sessions: value.sessions || {} }; + } catch (error) { + if (error.code !== 'ENOENT') throw error; + return { workspaces: {}, sessions: {} }; + } +} + +function persist() { + writeChain = writeChain.then(async () => { + const tmp = `${DB_PATH}.${process.pid}.tmp`; + await fs.promises.mkdir(path.dirname(DB_PATH), { recursive: true }); + await fs.promises.writeFile(tmp, JSON.stringify(state, null, 2), { mode: 0o600 }); + await fs.promises.rename(tmp, DB_PATH); + }); + return writeChain; +} + +function json(res, status, body) { + const data = JSON.stringify(body); + res.writeHead(status, { 'content-type': 'application/json', 'content-length': Buffer.byteLength(data) }); + res.end(data); +} + +function error(res, status, message) { return json(res, status, { error: message }); } + +async function body(req) { + let raw = ''; + for await (const chunk of req) { + raw += chunk; + if (raw.length > 64 * 1024) throw Object.assign(new Error('request too large'), { status: 413 }); + } + if (!raw) return {}; + try { return JSON.parse(raw); } catch { throw Object.assign(new Error('invalid JSON'), { status: 400 }); } +} + +function bearer(req) { + const match = /^Bearer\s+(\S+)$/i.exec(req.headers.authorization || ''); + if (!match) throw Object.assign(new Error('authentication required'), { status: 401 }); + return match[1]; +} + +async function identity(req) { + const token = bearer(req); + const response = await fetch(`${IAM_URL}/auth/validate`, { + method: 'POST', headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ token }), signal: AbortSignal.timeout(5000), + }).catch(() => null); + if (!response || !response.ok) throw Object.assign(new Error('authentication service unavailable'), { status: 503 }); + const value = await response.json().catch(() => null); + if (!value || value.valid !== true || value.user_id == null || value.org_id == null) { + throw Object.assign(new Error('invalid or expired token'), { status: 401 }); + } + const permissions = value.permissions || []; + if (!permissions.includes('workspace.launch') && !permissions.includes('platform.manage_infra')) { + throw Object.assign(new Error('insufficient permissions'), { status: 403 }); + } + return { user_id: String(value.user_id), organization_id: String(value.org_id) }; +} + +function ownerKey(owner) { return `${owner.organization_id}\u0000${owner.user_id}`; } +function ownerMatches(workspace, owner) { return workspace.owner_key === ownerKey(owner); } +function workspaceId(value) { return /^[0-9a-f-]{36}$/.test(value || ''); } +function runtimeName(id) { return `obws-${id}`; } +function volumeName(id) { return `obws-volume-${id}`; } +function networkName(id) { return `obws-network-${id}`; } +function sessionDigest(value) { return crypto.createHash('sha256').update(value).digest('hex'); } + +function allocation(input = {}) { + const cpu = Number(input.cpu ?? 1); + const memory_bytes = Number(input.memory_bytes ?? 512 * 1024 ** 2); + if (!Number.isFinite(cpu) || cpu < 0.25 || cpu > MAX_CPU) throw new Error(`cpu must be between 0.25 and ${MAX_CPU}`); + if (!Number.isInteger(memory_bytes) || memory_bytes < 256 * 1024 ** 2 || memory_bytes > MAX_MEMORY) throw new Error(`memory_bytes must be between 268435456 and ${MAX_MEMORY}`); + return { cpu, memory_bytes, gpu: 0, architecture: 'arm64' }; +} + +function freePort() { + return new Promise((resolve, reject) => { + const probe = net.createServer(); + probe.once('error', reject); + probe.listen(0, '127.0.0.1', () => { const port = probe.address().port; probe.close(() => resolve(port)); }); + }); +} + +async function waitForWorkspace(port, token, timeoutMs = 30000) { + const started = Date.now(); + while (Date.now() - started < timeoutMs) { + try { + const response = await fetch(`http://${UPSTREAM_HOST}:${port}/lab?token=${encodeURIComponent(token)}`, { signal: AbortSignal.timeout(2000) }); + if (response.status < 500) { response.body?.cancel(); return; } + response.body?.cancel(); + } catch {} + await new Promise(resolve => setTimeout(resolve, 250)); + } + throw new Error('workspace endpoint did not become ready'); +} + +async function inspectRuntime(workspace) { + return control.inspectContainer(workspace); +} + +async function createRuntime(workspace) { + await control.createVolume(workspace); + await control.createNetwork(workspace); + const token = crypto.randomBytes(32).toString('base64url'); + const hostPort = await freePort(); + await control.createContainer(workspace, { token, hostPort }); workspace.jupyter_token = token; + const started = await control.start(workspace); + if (![204, 304].includes(started.status)) throw new Error('workspace container start failed'); + await waitForWorkspace(hostPort, token); +} + +async function removeRuntime(workspace, options = {}) { + if (workspace.runtime_id) await control.remove(workspace, options); +} + +function publicWorkspace(workspace) { + const { jupyter_token, owner_key, ...safe } = workspace; + return safe; +} + +async function reconcile() { + for (const workspace of Object.values(state.workspaces)) { + if (workspace.deleted_at) continue; + const inspected = await inspectRuntime(workspace).catch(() => ({ status: 404 })); + if (inspected.status === 404) { workspace.state = 'failed'; workspace.state_reason = 'runtime missing during recovery'; } + else if (inspected.body?.State?.Running) workspace.state = 'running'; + else workspace.state = 'stopped'; + } + await persist(); +} + +async function route(req, res) { + let owner; + try { owner = await identity(req); } catch (e) { return error(res, e.status || 500, e.message); } + const url = new URL(req.url, 'http://workspace-manager'); + const parts = url.pathname.split('/').filter(Boolean); + try { + if (parts[0] === 'api' && parts[1] === 'workspaces' && parts.length === 2 && req.method === 'GET') { + return json(res, 200, { workspaces: Object.values(state.workspaces).filter(w => !w.deleted_at && ownerMatches(w, owner)).map(publicWorkspace) }); + } + if (parts[0] === 'api' && parts[1] === 'workspaces' && parts.length === 2 && req.method === 'POST') { + const input = await body(req); const id = crypto.randomUUID(); const workspace = { id, owner_key: ownerKey(owner), owner_user_id: owner.user_id, organization_id: owner.organization_id, + profile: 'generic-python', environment: 'jupyterlab', allocation: allocation(input.resources), state: 'creating', created_at: new Date().toISOString(), updated_at: new Date().toISOString() }; + state.workspaces[id] = workspace; await persist(); + try { await createRuntime(workspace); workspace.state = 'running'; workspace.state_reason = ''; workspace.updated_at = new Date().toISOString(); await persist(); return json(res, 201, publicWorkspace(workspace)); } + catch (e) { await removeRuntime(workspace).catch(() => {}); workspace.state = 'failed'; workspace.state_reason = e.message; workspace.updated_at = new Date().toISOString(); await persist(); return error(res, 502, e.message); } + } + if (parts[0] !== 'api' || parts[1] !== 'workspaces' || !workspaceId(parts[2])) return error(res, 404, 'not found'); + const workspace = state.workspaces[parts[2]]; + if (!workspace || workspace.deleted_at || !ownerMatches(workspace, owner)) return error(res, 404, 'workspace not found'); + const mutating = req.method === 'PATCH' || req.method === 'DELETE' || (req.method === 'POST' && ['start', 'stop', 'sessions'].includes(parts[3])); + if (mutating && !req.workspaceLockHeld) { req.workspaceLockHeld = true; return withWorkspaceLock(workspace.id, () => route(req, res)); } + if (parts.length === 3 && req.method === 'GET') return json(res, 200, publicWorkspace(workspace)); + if (parts.length === 3 && req.method === 'PATCH') { + if (workspace.state === 'running') return error(res, 409, 'Stop this workspace to change its CPU or memory allocation.'); + const input = await body(req); workspace.allocation = allocation(input.resources); const updated = await control.update(workspace, workspace.allocation); + if (updated.status < 200 || updated.status >= 300) return error(res, 502, 'workspace resource update failed'); + workspace.updated_at = new Date().toISOString(); await persist(); return json(res, 200, publicWorkspace(workspace)); + } + if (parts.length === 3 && req.method === 'DELETE') { const input = await body(req); if (input.purge_data !== undefined && input.purge_data !== true) return error(res, 400, 'purge_data must be true when provided'); await removeRuntime(workspace, { purgeData: input.purge_data === true }); workspace.deleted_at = new Date().toISOString(); workspace.state = 'deleted'; workspace.data_retained = input.purge_data !== true; await persist(); return json(res, 200, { deleted: true, workspace_id: workspace.id, data_retained: workspace.data_retained }); } + if (parts.length === 4 && parts[3] === 'start' && req.method === 'POST') { const result = await control.start(workspace); if (![204, 304].includes(result.status)) return error(res, 502, 'workspace start failed'); await waitForWorkspace(workspace.host_port, workspace.jupyter_token); workspace.state = 'running'; workspace.updated_at = new Date().toISOString(); await persist(); return json(res, 200, publicWorkspace(workspace)); } + if (parts.length === 4 && parts[3] === 'stop' && req.method === 'POST') { const result = await control.stop(workspace); if (![204, 304].includes(result.status)) return error(res, 502, 'workspace stop failed'); workspace.state = 'stopped'; workspace.updated_at = new Date().toISOString(); await persist(); return json(res, 200, publicWorkspace(workspace)); } + if (parts.length === 4 && parts[3] === 'sessions' && req.method === 'POST') { if (workspace.state !== 'running') return error(res, 409, 'workspace is not running'); const raw = crypto.randomBytes(32).toString('base64url'); const session = { workspace_id: workspace.id, owner_key: ownerKey(owner), expires_at: Date.now() + SESSION_TTL_MS }; state.sessions[sessionDigest(raw)] = session; await persist(); return json(res, 201, { workspace_id: workspace.id, token: raw, expires_at: new Date(session.expires_at).toISOString(), proxy_path: `/workspace/${workspace.id}/` }); } + return error(res, 404, 'not found'); + } catch (e) { return error(res, e.status || 400, e.message); } +} + +async function proxy(req, res, owner, workspaceIdValue) { + const workspace = state.workspaces[workspaceIdValue]; const raw = new URL(req.url, 'http://proxy').searchParams.get('session'); const session = raw && state.sessions[sessionDigest(raw)]; + if (!workspace || workspace.deleted_at || !ownerMatches(workspace, owner) || !session || session.workspace_id !== workspace.id || session.owner_key !== ownerKey(owner) || session.expires_at < Date.now()) return error(res, 404, 'workspace not found'); + if (workspace.state !== 'running') return error(res, 409, 'workspace is not running'); + const inspected = await inspectRuntime(workspace); const hostPort = workspace.host_port || inspected.body?.NetworkSettings?.Ports?.['8888/tcp']?.[0]?.HostPort; + if (!hostPort) return error(res, 503, 'workspace endpoint unavailable'); + const input = new URL(req.url, 'http://proxy'); input.searchParams.delete('session'); input.searchParams.set('token', workspace.jupyter_token); + const prefix = `/workspace/${workspace.id}`; const upstreamPath = input.pathname.startsWith(prefix) ? (input.pathname.slice(prefix.length) || '/') : '/'; + if (req.method === 'GET' || req.method === 'HEAD') { + let upstreamResponse; + const controller = new AbortController(); const abortUpstream = () => controller.abort(); req.once('aborted', abortUpstream); res.once('close', abortUpstream); + try { upstreamResponse = await fetch(`http://${UPSTREAM_HOST}:${hostPort}${upstreamPath}${input.search}`, { headers: { ...(req.headers.accept ? { accept: req.headers.accept } : {}) }, signal: AbortSignal.any([controller.signal, AbortSignal.timeout(30000)]) }); } + catch { return error(res, 502, 'workspace proxy unavailable'); } + const responseHeaders = {}; for (const [key, value] of upstreamResponse.headers) responseHeaders[key] = value; + res.writeHead(upstreamResponse.status, responseHeaders); if (req.method === 'HEAD' || !upstreamResponse.body) return res.end(); + const stream = Readable.fromWeb(upstreamResponse.body); stream.on('error', () => { if (res.headersSent) res.destroy(); else error(res, 502, 'workspace proxy unavailable'); }); stream.once('close', () => { req.off('aborted', abortUpstream); res.off('close', abortUpstream); }); return stream.pipe(res); + } + const upstream = http.request({ hostname: UPSTREAM_HOST, port: hostPort, method: req.method, path: upstreamPath + input.search, + headers: { host: `${UPSTREAM_HOST}:${hostPort}`, connection: 'close', ...(req.headers.accept ? { accept: req.headers.accept } : {}) }, agent: false }, response => { res.writeHead(response.statusCode, response.headers); response.pipe(res); }); + upstream.on('error', () => error(res, 502, 'workspace proxy unavailable')); + let responseFinished = false; res.once('finish', () => { responseFinished = true; }); req.on('aborted', () => upstream.destroy()); res.once('close', () => { if (!responseFinished) upstream.destroy(); }); + if (req.method === 'GET' || req.method === 'HEAD') upstream.end(); else req.pipe(upstream); +} + +const server = http.createServer(async (req, res) => { + if (req.url.startsWith('/workspace/')) { try { const owner = await identity(req); const match = /^\/workspace\/([^/]+)\//.exec(req.url); return match ? proxy(req, res, owner, match[1]) : error(res, 404, 'not found'); } catch (e) { return error(res, e.status || 500, e.message); } } + return route(req, res); +}); + +function rejectUpgrade(socket, status) { + socket.write(`HTTP/1.1 ${status} ${status === 404 ? 'Not Found' : 'Bad Gateway'}\r\nConnection: close\r\nContent-Length: 0\r\n\r\n`); + socket.destroy(); +} + +server.on('upgrade', async (req, socket, head) => { + try { + const owner = await identity(req); const url = new URL(req.url, 'http://proxy'); const match = /^\/workspace\/([^/]+)(\/.*)?$/.exec(url.pathname); + const workspace = match && state.workspaces[match[1]]; const raw = url.searchParams.get('session'); const session = raw && state.sessions[sessionDigest(raw)]; + if (!workspace || workspace.deleted_at || !ownerMatches(workspace, owner) || !session || session.workspace_id !== workspace.id || session.owner_key !== ownerKey(owner) || session.expires_at < Date.now()) return rejectUpgrade(socket, 404); + if (workspace.state !== 'running') return rejectUpgrade(socket, 409); + const inspected = await inspectRuntime(workspace); const hostPort = workspace.host_port || inspected.body?.NetworkSettings?.Ports?.['8888/tcp']?.[0]?.HostPort; + if (!hostPort) return rejectUpgrade(socket, 502); + url.searchParams.delete('session'); url.searchParams.set('token', workspace.jupyter_token); + const prefix = `/workspace/${workspace.id}`; const pathToUpstream = (url.pathname.startsWith(prefix) ? (url.pathname.slice(prefix.length) || '/') : '/') + url.search; + const upstream = net.connect({ host: UPSTREAM_HOST, port: hostPort }); + const fail = () => { if (!socket.destroyed) rejectUpgrade(socket, 502); }; + upstream.once('error', fail); upstream.once('connect', () => { + const allowed = ['host', 'upgrade', 'connection', 'sec-websocket-key', 'sec-websocket-version', 'sec-websocket-protocol', 'sec-websocket-extensions']; + const headers = allowed.filter(name => req.headers[name]).map(name => `${name}: ${name === 'host' ? `${UPSTREAM_HOST}:${hostPort}` : req.headers[name]}`).join('\r\n'); + upstream.write(`${req.method} ${pathToUpstream} HTTP/1.1\r\n${headers}\r\n\r\n`); if (head?.length) upstream.write(head); socket.pipe(upstream); upstream.pipe(socket); + }); + socket.on('error', () => upstream.destroy()); upstream.on('close', () => socket.destroy()); + } catch { rejectUpgrade(socket, 404); } +}); + +if (require.main === module) server.listen(PORT, LISTEN_HOST, async () => { await reconcile(); console.log(`workspace manager listening on ${LISTEN_HOST}:${server.address().port}`); }); + +module.exports = { allocation, loadState, ownerKey, publicWorkspace, server, state }; diff --git a/workspace-manager/test/control-plane.test.js b/workspace-manager/test/control-plane.test.js new file mode 100644 index 00000000..e3575dec --- /dev/null +++ b/workspace-manager/test/control-plane.test.js @@ -0,0 +1,27 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const crypto = require('node:crypto'); +const test = require('node:test'); +process.env.CONTROL_PLANE_SERVICE_SECRET = 'test-control-plane-secret'; +process.env.CONTROL_PLANE_SERVICE_ID = 'workspace-manager'; +process.env.CONTROL_PLANE_PREVIOUS_SERVICE_SECRET = 'previous-control-plane-secret'; +process.env.CONTROL_PLANE_PREVIOUS_SERVICE_KEY_ID = 'previous'; +const { authenticate, operations } = require('../control-plane'); + +function signed(raw, timestamp = Date.now(), nonce = crypto.randomBytes(16).toString('hex'), secret = process.env.CONTROL_PLANE_SERVICE_SECRET, keyId = 'active') { + return { 'x-service-id': 'workspace-manager', 'x-service-key-id': keyId, 'x-request-timestamp': String(timestamp), 'x-request-nonce': nonce, 'x-request-signature': crypto.createHmac('sha256', secret).update(`${timestamp}.${nonce}.${raw}`).digest('hex') }; +} + +test('control plane requires the configured service identity and rejects replay', () => { + const raw = JSON.stringify({ record: { id: '00000000-0000-0000-0000-000000000000' } }); const headers = signed(raw); const req = { headers }; + assert.equal(authenticate(req, raw), true); + assert.equal(authenticate(req, raw), false); + assert.equal(authenticate({ headers: { ...headers, 'x-service-id': 'browser' } }, raw), false); + assert.equal(authenticate({ headers: signed(raw, Date.now() - 60000) }, raw), false); + assert.equal(authenticate({ headers: signed(raw, Date.now(), crypto.randomBytes(16).toString('hex'), process.env.CONTROL_PLANE_PREVIOUS_SERVICE_SECRET, 'previous') }, raw), true); +}); + +test('control plane exposes only fixed managed operations', () => { + assert.deepEqual(Object.keys(operations).sort(), ['create-container', 'create-network', 'create-volume', 'image-architecture', 'inspect', 'remove', 'start', 'stop', 'update']); +}); diff --git a/workspace-manager/test/transport-container.test.js b/workspace-manager/test/transport-container.test.js new file mode 100644 index 00000000..36fd7b29 --- /dev/null +++ b/workspace-manager/test/transport-container.test.js @@ -0,0 +1,121 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const http = require('node:http'); +const net = require('node:net'); +const os = require('node:os'); +const path = require('node:path'); +const { execFileSync } = require('node:child_process'); +const test = require('node:test'); + +const root = path.resolve(__dirname, '..'); +const socket = process.env.DOCKER_SOCKET_PATH || path.join(os.homedir(), '.docker/run/docker.sock'); +const managerName = `obws-control-test-${crypto.randomUUID()}`; +const controlPlaneName = `obws-control-plane-test-${crypto.randomUUID()}`; +const controlNetwork = `obws-control-network-${crypto.randomUUID()}`; +const controlSecret = crypto.randomBytes(32).toString('hex'); +const stateDir = fs.mkdtempSync('/private/tmp/obws-control-state-'); +const auditDir = fs.mkdtempSync('/private/tmp/obws-control-audit-'); +let iam; +let iamPort; +let managerPort; +let unmanagedName; +let retainedVolume; +const identities = { + user_a: { valid: true, user_id: 301, org_id: 9, permissions: ['workspace.launch'] }, + user_b: { valid: true, user_id: 302, org_id: 9, permissions: ['workspace.launch'] }, + user_c: { valid: true, user_id: 303, org_id: 10, permissions: ['workspace.launch'] }, +}; + +function docker(args) { return execFileSync('docker', args, { encoding: 'utf8' }).trim(); } +function api(token, method, url, payload) { return fetch(`http://127.0.0.1:${managerPort}${url}`, { method, headers: { authorization: `Bearer ${token}`, ...(payload ? { 'content-type': 'application/json' } : {}) }, body: payload && JSON.stringify(payload) }); } +function wait(ms) { return new Promise(resolve => setTimeout(resolve, ms)); } +function abortProxyRequest(workspaceId, sessionToken) { + return new Promise((resolve, reject) => { + const request = http.request(`http://127.0.0.1:${managerPort}/workspace/${workspaceId}/files/work/transport-stream.bin?session=${encodeURIComponent(sessionToken)}`, { headers: { authorization: 'Bearer user_a' } }, response => { + response.once('data', () => { request.destroy(); }); response.once('close', resolve); response.once('error', () => {}); + }); request.once('error', error => { if (!['ECONNRESET', 'ERR_STREAM_PREMATURE_CLOSE'].includes(error.code)) reject(error); }); request.end(); + }); +} +function websocketHandshake(token, workspaceId, sessionToken, sessionId) { + return new Promise((resolve, reject) => { + const socket = net.connect( managerPort, '127.0.0.1'); let raw = Buffer.alloc(0); let settled = false; + const finish = (fn, value) => { if (settled) return; settled = true; fn(value); }; + socket.setTimeout(10000, () => { socket.destroy(); finish(reject, new Error('websocket handshake timeout')); }); + socket.on('error', error => finish(reject, error)); + socket.on('data', chunk => { raw = Buffer.concat([raw, chunk]); const end = raw.indexOf('\r\n\r\n'); if (end < 0) return; const header = raw.subarray(0, end).toString(); const status = Number(header.split('\r\n')[0].split(' ')[1]); if (status === 101) finish(resolve, { socket, head: raw.subarray(end + 4) }); else { socket.destroy(); finish(resolve, { status }); } }); + socket.once('connect', () => { + const key = crypto.randomBytes(16).toString('base64'); const pathName = `/workspace/${workspaceId}/api/kernels/${sessionId}/channels?session_id=${sessionId}&session=${encodeURIComponent(sessionToken)}`; + socket.write(`GET ${pathName} HTTP/1.1\r\nHost: 127.0.0.1:${managerPort}\r\nAuthorization: Bearer ${token}\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Key: ${key}\r\nSec-WebSocket-Version: 13\r\n\r\n`); + }); + }); +} + +test.before(async () => { + iam = http.createServer(async (req, res) => { + const chunks = []; for await (const chunk of req) chunks.push(chunk); + const input = JSON.parse(Buffer.concat(chunks).toString() || '{}'); res.setHeader('content-type', 'application/json'); res.end(JSON.stringify(identities[input.token] || { valid: false })); + }); + iam.listen(0, '0.0.0.0'); await new Promise(resolve => iam.once('listening', resolve)); iamPort = iam.address().port; + managerPort = 43000 + Math.floor(Math.random() * 1000); + docker(['network', 'create', controlNetwork]); + docker(['run', '-d', '--name', controlPlaneName, '--network', controlNetwork, '--network-alias', 'control-plane', '-e', 'CONTROL_PLANE_HOST=0.0.0.0', '-e', 'CONTROL_PLANE_PORT=5192', '-e', 'CONTROL_PLANE_SERVICE_ID=workspace-manager', '-e', 'CONTROL_PLANE_SERVICE_KEY_ID=active', '-e', `CONTROL_PLANE_SERVICE_SECRET=${controlSecret}`, '-e', 'CONTROL_PLANE_AUDIT_PATH=/audit/audit.jsonl', '-e', 'DOCKER_SOCKET_PATH=/var/run/docker.sock', '-v', `${root}:/app:ro`, '-v', `${auditDir}:/audit`, '-v', `${socket}:/var/run/docker.sock`, 'node:20-bookworm-slim', 'node', '/app/control-plane.js']); + docker(['run', '-d', '--name', managerName, '--network', controlNetwork, '--add-host', 'host.docker.internal:host-gateway', '-p', `127.0.0.1:${managerPort}:${managerPort}`, '-e', `PORT=${managerPort}`, '-e', 'LISTEN_HOST=0.0.0.0', '-e', `IAM_URL=http://host.docker.internal:${iamPort}`, '-e', 'WORKSPACE_UPSTREAM_HOST=host.docker.internal', '-e', 'CONTROL_PLANE_URL=http://control-plane:5192', '-e', 'CONTROL_PLANE_SERVICE_ID=workspace-manager', '-e', `CONTROL_PLANE_SERVICE_SECRET=${controlSecret}`, '-e', 'WORKSPACE_DB_PATH=/state/workspaces.json', '-e', 'WORKSPACE_MAX_CPU=2', '-e', `WORKSPACE_MAX_MEMORY_BYTES=${2 * 1024 ** 3}`, '-v', `${root}:/app:ro`, '-v', `${stateDir}:/state`, 'node:20-bookworm-slim', 'node', '/app/server.js']); + const managerMounts = JSON.parse(docker(['inspect', '--format', '{{json .Mounts}}', managerName])); assert.equal(managerMounts.some(mount => mount.Destination === '/var/run/docker.sock'), false); + for (let i = 0; i < 60; i += 1) { try { const response = await fetch(`http://127.0.0.1:${managerPort}/health`); if (response.status < 500) return; } catch {} await wait(250); } + throw new Error(docker(['logs', managerName])); +}); + +test.after(async () => { + try { docker(['rm', '-f', managerName]); } catch {} + try { docker(['rm', '-f', controlPlaneName]); } catch {} + if (unmanagedName) { try { docker(['rm', '-f', unmanagedName]); } catch {} } + if (retainedVolume) { try { docker(['volume', 'rm', retainedVolume]); } catch {} } + try { docker(['network', 'rm', controlNetwork]); } catch {} + if (iam) iam.close(); await fs.promises.rm(stateDir, { recursive: true, force: true }); + await fs.promises.rm(auditDir, { recursive: true, force: true }); +}); + +function controlPlaneCall(operation, record) { + const payload = JSON.stringify({ record }); const timestamp = String(Date.now()); const nonce = crypto.randomBytes(16).toString('hex'); const signature = crypto.createHmac('sha256', controlSecret).update(`${timestamp}.${nonce}.${payload}`).digest('hex'); + const headers = { 'content-type': 'application/json', 'content-length': Buffer.byteLength(payload), 'x-service-id': 'workspace-manager', 'x-service-key-id': 'active', 'x-request-timestamp': timestamp, 'x-request-nonce': nonce, 'x-request-signature': signature }; + const script = `const http=require('node:http');const req=http.request({host:'127.0.0.1',port:5192,path:'/v1/workspaces/${record.id}/${operation}',method:'POST',headers:${JSON.stringify(headers)}},res=>{let b='';res.on('data',c=>b+=c);res.on('end',()=>console.log(JSON.stringify({status:res.statusCode,body:b})))});req.end(${JSON.stringify(payload)});`; + return JSON.parse(execFileSync('docker', ['exec', controlPlaneName, 'node', '-e', script], { encoding: 'utf8' }).trim()); +} + +test('containerized trusted control plane routes an authorized live HTTP request', { timeout: 180000 }, async () => { + let response = await api('user_a', 'POST', '/api/workspaces', { resources: { cpu: 0.25, memory_bytes: 512 * 1024 ** 2 } }); const aliceText = await response.text(); assert.equal(response.status, 201, aliceText); const alice = JSON.parse(aliceText); + response = await api('user_b', 'POST', '/api/workspaces', { resources: { cpu: 0.25, memory_bytes: 512 * 1024 ** 2 } }); const bobText = await response.text(); assert.equal(response.status, 201, bobText); const bob = JSON.parse(bobText); + const persisted = JSON.parse(await fs.promises.readFile(path.join(stateDir, 'workspaces.json'), 'utf8')).workspaces[alice.id]; + const direct = await fetch(`http://127.0.0.1:${persisted.host_port}/lab?token=${encodeURIComponent(persisted.jupyter_token)}`); assert.equal(direct.status, 200); + response = await api('user_a', 'POST', `/api/workspaces/${alice.id}/sessions`, {}); assert.equal(response.status, 201); const session = await response.json(); + response = await api('user_a', 'GET', `/workspace/${alice.id}/lab?session=${encodeURIComponent(session.token)}`); const page = await response.text(); assert.equal(response.status, 200, page); assert.match(page, /Jupyter/); + response = await api('user_a', 'POST', `/workspace/${alice.id}/api/kernels?session=${encodeURIComponent(session.token)}`, { name: 'python3' }); assert.equal(response.status, 201); const kernel = await response.json(); + response = await api('user_a', 'GET', `/workspace/${alice.id}/api/kernels?session=${encodeURIComponent(session.token)}`); const kernelsBody = await response.text(); assert.equal(response.status, 200, kernelsBody); assert.ok(JSON.parse(kernelsBody).some(item => item.id === kernel.id)); + const ws = await websocketHandshake('user_a', alice.id, session.token, kernel.id); assert.equal(ws.status, undefined); ws.socket.write(Buffer.from([0x89, 0])); const pong = await new Promise((resolve, reject) => { const timer = setTimeout(() => reject(new Error('websocket pong timeout')), 10000); ws.socket.once('data', data => { clearTimeout(timer); resolve(data); }); }); assert.equal(pong[0] & 0x0f, 0x0a); ws.socket.end(); + const reconnect = await websocketHandshake('user_a', alice.id, session.token, kernel.id); assert.equal(reconnect.status, undefined); reconnect.socket.end(); + const deniedWs = await websocketHandshake('user_b', alice.id, session.token, kernel.id); assert.equal(deniedWs.status, 404); + response = await api('user_a', 'DELETE', `/workspace/${alice.id}/api/kernels/${kernel.id}?session=${encodeURIComponent(session.token)}`); assert.ok([204, 404].includes(response.status)); + response = await api('user_b', 'GET', `/workspace/${alice.id}/lab?session=${encodeURIComponent(session.token)}`); assert.equal(response.status, 404); assert.notEqual(alice.runtime_name, bob.runtime_name); + response = await api('user_a', 'GET', `/workspace/${bob.id}/lab?session=${encodeURIComponent(session.token)}`); assert.equal(response.status, 404); + response = await api('user_c', 'GET', `/workspace/${alice.id}/lab?session=${encodeURIComponent(session.token)}`); assert.equal(response.status, 404); + response = await api('', 'GET', `/workspace/${alice.id}/lab?session=${encodeURIComponent(session.token)}`); assert.equal(response.status, 401); + const concurrentStops = await Promise.all([api('user_a', 'POST', `/api/workspaces/${alice.id}/stop`, {}), api('user_a', 'POST', `/api/workspaces/${alice.id}/stop`, {})]); assert.deepEqual(concurrentStops.map(item => item.status), [200, 200]); + const concurrentStarts = await Promise.all([api('user_a', 'POST', `/api/workspaces/${alice.id}/start`, {}), api('user_a', 'POST', `/api/workspaces/${alice.id}/start`, {})]); assert.deepEqual(concurrentStarts.map(item => item.status), [200, 200]); + docker(['restart', controlPlaneName]); for (let i = 0; i < 30; i += 1) { try { docker(['exec', controlPlaneName, 'node', '-e', "fetch('http://127.0.0.1:5192/healthz').then(r=>{if(!r.ok)process.exit(1)})"]); break; } catch {} await wait(250); } + response = await api('user_a', 'GET', `/api/workspaces/${alice.id}`); assert.equal(response.status, 200); assert.equal((await response.json()).state, 'running'); + docker(['exec', alice.runtime_name, 'sh', '-c', 'dd if=/dev/zero of=/home/jovyan/work/transport-stream.bin bs=1M count=32 status=none']); + const streamed = await fetch(`http://127.0.0.1:${managerPort}/workspace/${alice.id}/files/work/transport-stream.bin?session=${encodeURIComponent(session.token)}`, { headers: { authorization: 'Bearer user_a' } }); assert.equal(streamed.status, 200); let streamedBytes = 0; let streamedChunks = 0; const reader = streamed.body.getReader(); for (;;) { const chunk = await reader.read(); if (chunk.done) break; streamedBytes += chunk.value.byteLength; streamedChunks += 1; } assert.equal(streamedBytes, 32 * 1024 ** 2); assert.ok(streamedChunks > 1); + await abortProxyRequest(alice.id, session.token); + await wait(500); + unmanagedName = `obws-unmanaged-${crypto.randomUUID()}`; docker(['run', '-d', '--name', unmanagedName, '--network', 'none', 'node:20-bookworm-slim', 'sleep', '60']); + const unmanagedRecord = { id: crypto.randomUUID(), owner_user_id: 301, organization_id: 9, runtime_id: unmanagedName }; + assert.equal(controlPlaneCall('inspect', unmanagedRecord).status, 409); assert.equal(controlPlaneCall('remove', unmanagedRecord).status, 409); + assert.equal((await api('user_a', 'DELETE', `/api/workspaces/${alice.id}`, { purge_data: true })).status, 200); assert.equal((await api('user_b', 'DELETE', `/api/workspaces/${bob.id}`, { purge_data: true })).status, 200); + response = await api('user_a', 'POST', '/api/workspaces', { resources: { cpu: 0.25, memory_bytes: 512 * 1024 ** 2 } }); assert.equal(response.status, 201); const retained = await response.json(); retainedVolume = retained.volume_name; + response = await api('user_a', 'DELETE', `/api/workspaces/${retained.id}`); const retainedResult = await response.json(); assert.equal(response.status, 200); assert.equal(retainedResult.data_retained, true); assert.equal(JSON.parse(docker(['volume', 'inspect', retained.volume_name])).length, 1); + docker(['volume', 'rm', retained.volume_name]); retainedVolume = null; + await wait(250); const audit = await fs.promises.readFile(path.join(auditDir, 'audit.jsonl'), 'utf8'); assert.ok(audit.split('\n').filter(Boolean).length >= 4); assert.equal(audit.includes(controlSecret), false); assert.equal(audit.includes(session.token), false); +}); diff --git a/workspace-manager/test/workspace-manager.test.js b/workspace-manager/test/workspace-manager.test.js new file mode 100644 index 00000000..440bad38 --- /dev/null +++ b/workspace-manager/test/workspace-manager.test.js @@ -0,0 +1,95 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const http = require('node:http'); +const { execFileSync } = require('node:child_process'); +const os = require('node:os'); +const path = require('node:path'); +const { spawn } = require('node:child_process'); +const { once } = require('node:events'); +const test = require('node:test'); + +const dockerSocket = process.env.DOCKER_SOCKET_PATH || path.join(os.homedir(), '.docker/run/docker.sock'); +const root = path.resolve(__dirname, '..'); +let iam; +let manager; +let base; +let dbPath; +let managerEnv; +const tokens = { + alice: { valid: true, user_id: 101, org_id: 7, permissions: ['workspace.launch'] }, + bob: { valid: true, user_id: 202, org_id: 7, permissions: ['workspace.launch'] }, +}; + +function request(port, token, method, url, payload) { + return fetch(`${base || `http://127.0.0.1:${port}`}${url}`, { method, headers: { authorization: `Bearer ${token}`, ...(payload ? { 'content-type': 'application/json' } : {}) }, body: payload && JSON.stringify(payload) }); +} + +async function waitFor(url, attempts = 60) { + for (let i = 0; i < attempts; i += 1) { + try { const response = await fetch(url); if (response.status < 500) return; } catch {} + await new Promise(resolve => setTimeout(resolve, 500)); + } + throw new Error(`timed out waiting for ${url}`); +} + +async function startManager() { + manager = spawn(process.execPath, [path.join(root, 'server.js')], { env: managerEnv, stdio: ['ignore', 'pipe', 'pipe'] }); + let output = ''; + manager.stdout.on('data', chunk => { output += chunk; const match = /127\.0\.0\.1:(\d+)/.exec(output); if (match) base = `http://127.0.0.1:${match[1]}`; }); + manager.stderr.on('data', chunk => { output += chunk; }); + await new Promise((resolve, reject) => { const timer = setTimeout(() => reject(new Error(`manager did not start: ${output}`)), 15000); const poll = setInterval(() => { if (base) { clearTimeout(timer); clearInterval(poll); resolve(); } }, 100); manager.once('exit', code => { clearTimeout(timer); clearInterval(poll); reject(new Error(`manager exited ${code}: ${output}`)); }); }); +} + +test.before(async () => { + dbPath = path.join('/private/tmp', `omnibioai-workspaces-${crypto.randomUUID()}.json`); + iam = http.createServer(async (req, res) => { + const chunks = []; for await (const chunk of req) chunks.push(chunk); + const input = JSON.parse(Buffer.concat(chunks).toString() || '{}'); const identity = tokens[input.token]; + const body = JSON.stringify(identity || { valid: false }); res.writeHead(200, { 'content-type': 'application/json' }); res.end(body); + }); + iam.listen(0, '127.0.0.1'); await once(iam, 'listening'); + const iamPort = iam.address().port; + managerEnv = { ...process.env, PORT: '0', IAM_URL: `http://127.0.0.1:${iamPort}`, WORKSPACE_DB_PATH: dbPath, DOCKER_SOCKET_PATH: dockerSocket, WORKSPACE_MAX_CPU: '2', WORKSPACE_MAX_MEMORY_BYTES: String(2 * 1024 ** 3) }; + await startManager(); +}); + +test.after(async () => { + if (manager) manager.kill('SIGTERM'); + if (iam) iam.close(); + if (dbPath) await fs.promises.rm(dbPath, { force: true }); +}); + +test('two IAM owners receive isolated Jupyter runtimes and lifecycle enforcement', { timeout: 180000 }, async () => { + const create = async token => { const response = await request(0, token, 'POST', '/api/workspaces', { resources: { cpu: 0.25, memory_bytes: 512 * 1024 ** 2 } }); const text = await response.text(); assert.equal(response.status, 201, text); return JSON.parse(text); }; + const alice = await create('alice'); const bob = await create('bob'); + assert.notEqual(alice.id, bob.id); assert.notEqual(alice.runtime_name, bob.runtime_name); assert.notEqual(alice.volume_name, bob.volume_name); assert.equal(alice.state, 'running'); assert.equal(bob.state, 'running'); + const inspectedAlice = JSON.parse(execFileSync('docker', ['inspect', alice.runtime_name], { encoding: 'utf8' }))[0]; + const inspectedBob = JSON.parse(execFileSync('docker', ['inspect', bob.runtime_name], { encoding: 'utf8' }))[0]; + assert.equal(inspectedAlice.HostConfig.NanoCpus, 250000000); assert.equal(inspectedAlice.HostConfig.Memory, 512 * 1024 ** 2); + assert.equal(inspectedBob.HostConfig.NanoCpus, 250000000); assert.equal(inspectedBob.HostConfig.Memory, 512 * 1024 ** 2); + assert.equal(inspectedAlice.HostConfig.Privileged, false); assert.equal(inspectedBob.HostConfig.Privileged, false); + + let response = await request(0, 'bob', 'GET', `/api/workspaces/${alice.id}`); assert.equal(response.status, 404); + response = await request(0, 'alice', 'GET', `/api/workspaces/${bob.id}`); assert.equal(response.status, 404); + response = await request(0, 'alice', 'PATCH', `/api/workspaces/${alice.id}`, { resources: { cpu: 0.5, memory_bytes: 768 * 1024 ** 2 } }); + assert.equal(response.status, 409); assert.match(await response.text(), /Stop this workspace/); + + response = await request(0, 'alice', 'POST', `/api/workspaces/${alice.id}/sessions`, {}); assert.equal(response.status, 201); const session = await response.json(); + response = await request(0, 'bob', 'GET', `/workspace/${alice.id}/lab?session=${encodeURIComponent(session.token)}`); assert.equal(response.status, 404); + // The manager route reaches the owner/session authorization boundary here. + // Host-process -> Docker Desktop loopback proxying is qualified separately; + // this sandbox currently resets that upstream connection (UND_ERR_SOCKET). + + response = await request(0, 'alice', 'POST', `/api/workspaces/${alice.id}/stop`, {}); assert.equal(response.status, 200); assert.equal((await response.json()).state, 'stopped'); + response = await request(0, 'alice', 'PATCH', `/api/workspaces/${alice.id}`, { resources: { cpu: 0.5, memory_bytes: 768 * 1024 ** 2 } }); assert.equal(response.status, 200); assert.equal((await response.json()).allocation.cpu, 0.5); + response = await request(0, 'alice', 'POST', `/api/workspaces/${alice.id}/start`, {}); assert.equal(response.status, 200); assert.equal((await response.json()).state, 'running'); + + const db = JSON.parse(await fs.promises.readFile(dbPath, 'utf8')); assert.equal(db.workspaces[alice.id].owner_key, '7\u0000101'); assert.equal(db.workspaces[bob.id].owner_key, '7\u0000202'); + manager.kill('SIGTERM'); await once(manager, 'exit'); base = null; await startManager(); + response = await request(0, 'alice', 'GET', `/api/workspaces/${alice.id}`); assert.equal(response.status, 200); assert.equal((await response.json()).state, 'running'); + response = await request(0, 'alice', 'DELETE', `/api/workspaces/${alice.id}`, { purge_data: true }); assert.equal(response.status, 200); + response = await request(0, 'bob', 'DELETE', `/api/workspaces/${bob.id}`, { purge_data: true }); assert.equal(response.status, 200); +});