feat: add production-ready runtime safety baseline

config/production.example.yaml

@@ -0,0 +1,29 @@
+loop:
+  mode: closed
+  max_iterations: 12
+  max_repairs: 4
+  max_consecutive_no_progress: 2
+  token_budget: 500000
+  cost_budget: 25.00
+  wall_clock_budget_seconds: 7200
+fleet:
+  max_depth: 2
+  max_fanout: 4
+  max_parallel_mutators: 2
+permissions:
+  default: READ_ONLY
+  require_approval: [REMOTE_MUTATION, HIGH_IMPACT]
+models:
+  discoverer: {primary: cheap, fallbacks: [strong]}
+  planner: {primary: cheap, fallbacks: [strong]}
+  executor: {primary: strong, fallbacks: []}
+  verifier: {primary: strong, fallbacks: []}
+persistence:
+  state_backend: postgresql
+  idempotency_backend: postgresql
+  artifacts: object-storage
+  audit: append-only
+observability:
+  traces: otlp
+  metrics: enabled
+  secret_redaction: true

docs/ARCHITECTURE-PRODUCTION.md

@@ -0,0 +1,23 @@
+# ZLoop Production Architecture
+
+## Control plane
+Mission API -> Orchestrator -> Policy Engine + Budget Governor + Model Router.
+
+## Data plane
+Isolated Executors -> Connector Adapters -> Independent Verifiers -> Repair Planner.
+
+## Persistence
+PostgreSQL for canonical loop/checkpoint/idempotency/approval state; durable queue for worker leases/retries/cancellation; object storage for artifacts; append-only audit store for mutation events.
+
+## Security invariants
+- deny mutation by default;
+- tenant/actor/action/target binding on approvals;
+- no client-side provider secrets;
+- executor cannot directly mark `SHIPPED`;
+- stale revision/SHA blocks mutation;
+- credentials are scoped per adapter and never persisted in loop memory;
+- external side effects require durable idempotency keys;
+- `INCONCLUSIVE` is never success.
+
+## Fleet constraints
+Children receive strict subsets of parent scope and remaining budget. Depth, fan-out and concurrency caps are mandatory. Mutating children use isolated workspaces. Final acceptance belongs to an independent verifier.

docs/PRODUCTION-READINESS.md

@@ -0,0 +1,44 @@
+# ZLoop Production Readiness
+
+ZLoop is considered **production-grade-ready** only when every gate below has implementation evidence, automated tests, and operational ownership. This document distinguishes implemented foundations from deployment-dependent controls.
+
+## Implemented foundation
+
+- bounded loop lifecycle and terminal states;
+- independent verifier/reviewer contracts;
+- durable SQLite checkpoint and idempotency ledger;
+- durable SQLite reference queue with leases, lease-expiry reclaim, retry, cancellation and deadlines;
+- checkpoint serialization/restoration helpers;
+- legal state-transition validation;
+- provider-neutral model routing contract and bounded fallback;
+- hard global and per-profile token/cost limits with fail-closed behavior;
+- structured-output validation;
+- deny-by-default permission gate;
+- verifier registry with fail-closed `INCONCLUSIVE` behavior;
+- evidence fingerprints for repeat/no-progress detection;
+- bounded fleet depth/fan-out governor;
+- permission-bound command runner with timeout and reduced environment;
+- GitHub PR safety boundary with stale-SHA, approval and check gates;
+- audit event contract with tenant field;
+- metrics/tracing-compatible span primitives;
+- secret-redaction helper;
+- CI, CodeQL, Dependency Review, Dependabot, concurrency and timeouts.
+
+## Mandatory deployment gates before production mutation
+
+1. PostgreSQL durable state/idempotency implementation and migration tests.
+2. Production queue/worker backend with distributed leases and crash-recovery integration tests; SQLite is the local/reference backend only.
+3. OS/container sandbox with CPU, memory, process and network limits; the reference command runner is not a security sandbox.
+4. Capability-scoped real connector adapters with tenant/actor/action/target binding.
+5. External approval authority for `REMOTE_MUTATION` and `HIGH_IMPACT` operations.
+6. Durable append-only audit sink and retention policy.
+7. OpenTelemetry exporter configuration, SLO dashboards and alerting.
+8. Secrets-manager integration and production redaction tests.
+9. Branch protection with required CI/security/review checks.
+10. SBOM, provenance/attestation, signed release process and rollback runbook.
+11. PostgreSQL backups, restore drills, RPO/RTO and disaster-recovery ownership.
+12. Load, concurrency and chaos tests against the actual deployment topology.
+
+## Release criteria
+
+No production release may be marked ready unless all required checks have explicit passing evidence, no blocking review finding remains, mutating connectors are approval-bound, idempotency survives restarts, state recovery is crash-tested, budgets fail closed, `INCONCLUSIVE` never ships, secrets are absent from artifacts/logs, and rollback/handoff procedures are tested.

docs/RELEASE-GATES.md

@@ -0,0 +1,30 @@
+# Release Gates
+
+## Automated
+- unit tests
+- compile/static syntax checks
+- CodeQL/security analysis
+- dependency review
+- secret-pattern scan
+- package/build validation
+- persistence migration tests when schemas change
+
+## Engineering
+- maker/checker separation
+- blocking findings resolved
+- acceptance criteria mapped to evidence
+- backward compatibility assessed
+- rollback path documented
+- mutations covered by permission + idempotency tests
+
+## Production-only
+- required branch checks configured
+- release provenance verified
+- secrets-manager integration verified
+- audit retention configured
+- dashboards/alerts enabled
+- backup restore drill passed
+- capacity/load test passed
+- operator runbook reviewed
+
+No missing or `INCONCLUSIVE` mandatory gate may be treated as PASS.

schemas/audit-event.schema.json

@@ -0,0 +1,17 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "ZLoop Audit Event",
+  "type": "object",
+  "required": ["actor", "action", "target", "permission", "result", "created_at"],
+  "properties": {
+    "actor": {"type": "string", "minLength": 1},
+    "tenant_id": {"type": ["string", "null"]},
+    "action": {"type": "string", "minLength": 1},
+    "target": {"type": "string", "minLength": 1},
+    "permission": {"enum": ["READ_ONLY", "LOCAL_MUTATION", "REMOTE_MUTATION", "HIGH_IMPACT"]},
+    "result": {"type": "string", "minLength": 1},
+    "idempotency_key": {"type": ["string", "null"]},
+    "created_at": {"type": "number"}
+  },
+  "additionalProperties": false
+}

src/execution.py

@@ -0,0 +1,36 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Mapping, Sequence
+import os
+import subprocess
+
+from .runtime_contracts import Permission, PermissionGate
+
+
+@dataclass(frozen=True)
+class CommandResult:
+    returncode: int
+    stdout: str
+    stderr: str
+
+
+class CommandRunner:
+    def __init__(self, workspace: str, permission_gate: PermissionGate, timeout_seconds: int = 120) -> None:
+        self.workspace = Path(workspace).resolve()
+        self.permission_gate = permission_gate
+        self.timeout_seconds = timeout_seconds
+
+    def run(self, command: Sequence[str], *, env: Mapping[str, str] | None = None) -> CommandResult:
+        self.permission_gate.require(Permission.LOCAL_MUTATION)
+        if not self.workspace.exists() or not self.workspace.is_dir():
+            raise ValueError("workspace must exist")
+        safe_env = {"PATH": os.environ.get("PATH", ""), "LANG": "C.UTF-8"}
+        if env:
+            safe_env.update(env)
+        completed = subprocess.run(
+            list(command), cwd=self.workspace, env=safe_env,
+            text=True, capture_output=True, timeout=self.timeout_seconds, check=False,
+        )
+        return CommandResult(completed.returncode, completed.stdout, completed.stderr)

src/github_loop.py

@@ -0,0 +1,39 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol, Sequence
+
+from .runtime_contracts import Permission, PermissionGate
+
+
+@dataclass(frozen=True)
+class PullRequestSnapshot:
+    number: int
+    head_sha: str
+    base_branch: str
+    required_checks: Sequence[str]
+
+
+class GitHubAdapter(Protocol):
+    def get_pull_request(self, number: int) -> PullRequestSnapshot: ...
+    def push_branch(self, branch: str, expected_head_sha: str) -> str: ...
+    def merge_pull_request(self, number: int, expected_head_sha: str) -> str: ...
+
+
+class SafeGitHubLoop:
+    def __init__(self, adapter: GitHubAdapter, permissions: PermissionGate) -> None:
+        self.adapter = adapter
+        self.permissions = permissions
+
+    def assert_fresh(self, number: int, expected_head_sha: str) -> PullRequestSnapshot:
+        snapshot = self.adapter.get_pull_request(number)
+        if snapshot.head_sha != expected_head_sha:
+            raise RuntimeError("stale pull request head SHA")
+        return snapshot
+
+    def merge(self, number: int, expected_head_sha: str, checks_passed: bool, approved: bool) -> str:
+        self.permissions.require(Permission.HIGH_IMPACT)
+        self.assert_fresh(number, expected_head_sha)
+        if not checks_passed or not approved:
+            raise RuntimeError("merge gate not satisfied")
+        return self.adapter.merge_pull_request(number, expected_head_sha)

src/job_queue.py

@@ -0,0 +1,132 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+import json
+import sqlite3
+import time
+import uuid
+
+
+@dataclass(frozen=True)
+class Job:
+    job_id: str
+    payload: dict[str, Any]
+    status: str
+    attempts: int
+    lease_owner: str | None
+    lease_until: float | None
+    deadline: float | None
+
+
+class SQLiteJobQueue:
+    """Durable local queue with leases, reclaim, retry, cancellation and deadlines."""
+
+    def __init__(self, path: str) -> None:
+        self.path = path
+        Path(path).parent.mkdir(parents=True, exist_ok=True)
+        with self._connect() as conn:
+            conn.execute(
+                """
+                CREATE TABLE IF NOT EXISTS jobs (
+                    job_id TEXT PRIMARY KEY,
+                    payload_json TEXT NOT NULL,
+                    status TEXT NOT NULL,
+                    attempts INTEGER NOT NULL DEFAULT 0,
+                    available_at REAL NOT NULL,
+                    lease_owner TEXT,
+                    lease_until REAL,
+                    deadline REAL,
+                    updated_at REAL NOT NULL
+                )
+                """
+            )
+
+    def _connect(self) -> sqlite3.Connection:
+        conn = sqlite3.connect(self.path, timeout=30, isolation_level=None)
+        conn.row_factory = sqlite3.Row
+        conn.execute("PRAGMA journal_mode=WAL")
+        conn.execute("PRAGMA synchronous=FULL")
+        return conn
+
+    def enqueue(self, payload: dict[str, Any], *, job_id: str | None = None, deadline: float | None = None) -> str:
+        jid = job_id or str(uuid.uuid4())
+        now = time.time()
+        with self._connect() as conn:
+            conn.execute(
+                "INSERT INTO jobs(job_id,payload_json,status,attempts,available_at,deadline,updated_at) VALUES(?,?,?,0,?,?,?)",
+                (jid, json.dumps(payload, sort_keys=True), "READY", now, deadline, now),
+            )
+        return jid
+
+    def lease(self, owner: str, *, lease_seconds: int = 60) -> Job | None:
+        now = time.time()
+        lease_until = now + lease_seconds
+        with self._connect() as conn:
+            conn.execute("BEGIN IMMEDIATE")
+            conn.execute(
+                "UPDATE jobs SET status='READY', lease_owner=NULL, lease_until=NULL, updated_at=? WHERE status='LEASED' AND lease_until <= ?",
+                (now, now),
+            )
+            conn.execute(
+                "UPDATE jobs SET status='EXPIRED', updated_at=? WHERE status IN ('READY','LEASED') AND deadline IS NOT NULL AND deadline <= ?",
+                (now, now),
+            )
+            row = conn.execute(
+                "SELECT * FROM jobs WHERE status='READY' AND available_at <= ? ORDER BY available_at, job_id LIMIT 1",
+                (now,),
+            ).fetchone()
+            if row is None:
+                conn.execute("COMMIT")
+                return None
+            conn.execute(
+                "UPDATE jobs SET status='LEASED', attempts=attempts+1, lease_owner=?, lease_until=?, updated_at=? WHERE job_id=?",
+                (owner, lease_until, now, row["job_id"]),
+            )
+            conn.execute("COMMIT")
+        return self.get(row["job_id"])
+
+    def ack(self, job_id: str, owner: str) -> None:
+        now = time.time()
+        with self._connect() as conn:
+            cur = conn.execute(
+                "UPDATE jobs SET status='SUCCEEDED', lease_owner=NULL, lease_until=NULL, updated_at=? WHERE job_id=? AND status='LEASED' AND lease_owner=?",
+                (now, job_id, owner),
+            )
+            if cur.rowcount != 1:
+                raise RuntimeError("ack rejected: lease ownership mismatch")
+
+    def retry(self, job_id: str, owner: str, *, delay_seconds: float = 0) -> None:
+        now = time.time()
+        with self._connect() as conn:
+            cur = conn.execute(
+                "UPDATE jobs SET status='READY', available_at=?, lease_owner=NULL, lease_until=NULL, updated_at=? WHERE job_id=? AND status='LEASED' AND lease_owner=?",
+                (now + delay_seconds, now, job_id, owner),
+            )
+            if cur.rowcount != 1:
+                raise RuntimeError("retry rejected: lease ownership mismatch")
+
+    def cancel(self, job_id: str) -> None:
+        now = time.time()
+        with self._connect() as conn:
+            cur = conn.execute(
+                "UPDATE jobs SET status='CANCELLED', lease_owner=NULL, lease_until=NULL, updated_at=? WHERE job_id=? AND status NOT IN ('SUCCEEDED','CANCELLED','EXPIRED')",
+                (now, job_id),
+            )
+            if cur.rowcount != 1:
+                current = self.get(job_id)
+                if current is None:
+                    raise KeyError(job_id)
+                if current.status == "SUCCEEDED":
+                    raise RuntimeError("cannot cancel succeeded job")
+
+    def get(self, job_id: str) -> Job | None:
+        with self._connect() as conn:
+            row = conn.execute("SELECT * FROM jobs WHERE job_id=?", (job_id,)).fetchone()
+        if row is None:
+            return None
+        return Job(
+            job_id=row["job_id"], payload=json.loads(row["payload_json"]), status=row["status"],
+            attempts=int(row["attempts"]), lease_owner=row["lease_owner"], lease_until=row["lease_until"], deadline=row["deadline"],
+        )

src/observability.py

@@ -0,0 +1,33 @@
+from __future__ import annotations
+
+from collections import defaultdict
+from contextlib import contextmanager
+from dataclasses import dataclass
+from typing import Dict, Iterator
+import time
+
+
+@dataclass(frozen=True)
+class SpanRecord:
+    name: str
+    duration_seconds: float
+    success: bool
+
+
+class Metrics:
+    def __init__(self) -> None:
+        self.counters: Dict[str, int] = defaultdict(int)
+        self.timings: list[SpanRecord] = []
+
+    def increment(self, name: str, value: int = 1) -> None:
+        self.counters[name] += value
+
+    @contextmanager
+    def span(self, name: str) -> Iterator[None]:
+        started = time.perf_counter()
+        success = False
+        try:
+            yield
+            success = True
+        finally:
+            self.timings.append(SpanRecord(name, time.perf_counter() - started, success))