ZaumZeug - ZaZu

A harness SDK and factory for configurable multi-agent AI workflows.

ZaumZeug helps developers scaffold, configure, compile, and deploy multi-agent harnesses — not a single chatbot relabeled as an agent, but a workflow plus a collection of named agents, each with its own model and tool configuration. You edit YAML, add Python only where custom logic is needed, and ship a production-ready bundle to any host without touching Haystack, DSPy, or GEPA internals.

Status: Phase 1 complete — epics E2 (config/validation), E3 (runtime/serve/run), and E1 onboarding through E1-S07 (three-OS CI matrix) are done. E1-S08 (zaumzeug install-project) is deferred until PyPI package publish (open #3 resolved: clone+install/ for v1; CLI after publish). Phase 2 complete for E4 / E6 / E5 (2026-07-28 / 2026-07-29). Phase 4 pull-forward: E8-S07 (workflow run reliability — runtime.retry, stage INFO logs, output.format: json) done (2026-08-01). Phase 3: E7-S01 (MCP) through E7-S08 (web client deployment layouts) done (2026-08-21). Phase 4: E8-S01–E8-S06 (pipeline exposure, production auth, trace ratchet, optional A2A, Dockerfile template, DeepEval Mode B observe) done (2026-08-21 / 2026-08-26) — E8 complete. Post–Phase 4: E6-S08 (human quality rating → compile — zaumzeug rate / run --rate) done (2026-08-26); E4-S09 (workflow path Mermaid docs — docs/workflow-paths.md on compile) done (2026-08-26). Cross-platform CI: .github/workflows/ci-matrix.yml. Track progress in docs/stories/.

Formula: Harness = Workflow + Agents + Machinery

---

Why ZaumZeug?

ZaumZeug is not trying to be the most flexible agent orchestrator or the deepest provider SDK. It is a factory for turning YAML-defined multi-agent workflows into compile-optimized, mock-testable, deployable production harnesses — Haystack runtime, GEPA-guided improvement loops, and enforced dev/deploy boundaries.

Building on Haystack, DSPy, and GEPA today means wiring compile-time optimization and runtime serving yourself — and it's easy to mix them incorrectly. ZaumZeug provides a single SDK with enforced golden rules, a CLI workflow, and a clear boundary between development and production.

Most agent SDKs treat tracing, human feedback, and prompt tuning as separate tools or dashboards. ZaumZeug ships a closed improvement loop on the dev machine: run or serve → capture a trace → rate output and set gold with zaumzeug rate → merge into val.jsonl / train.jsonl → GEPA recompile → redeploy. The rate path uses stdlib only — no DeepEval required — which matters when you are building a new harness or switching LLM versions.

Problem	ZaumZeug approach
Complex Haystack + DSPy + GEPA integration	One SDK with compile/runtime separation built in
Unclear dev vs production artifacts	usr-harnesses/ for development, deploy/ for shipping
Model provider lock-in	Per-agent agents-models.yaml + stage-scoped model routing
Different clients need different protocols	One Haystack core, multiple serve surfaces (REST, MCP, ACP, optional A2A)
Prompt iteration is ad hoc	Journey C: trace → human rate → dataset merge → GEPA compile → redeploy (E6-S08)

How ZaumZeug is different

• Factory, not one-off framework — scaffold → configure → compile → deploy → serve; the unit of shipping is deploy/<name>/, not your dev repo
• Compile ≠ runtime, enforced — DSPy/GEPA offline; Haystack Agent on the hot path; never both loops in one request
• Workflow stages, not one mega-agent — serial stages (parallel later), each running one agent loop, chained by workflow.yaml
• Guides + sensors — GEPA-optimized prompts (guides) and metric feedback (sensors) baked into the lifecycle
• Human rating → compile — zaumzeug rate / run --rate scores a captured turn, writes gold to data/ratings/, merges into val/train, and optionally recompiles — stdlib on the rate path; DeepEval optional for automated eval tiers
• Mock-first dev — zaumzeug doctor, dry-run, and mock model profile for zero-API-key local dev and CI

What we do better

Shipped Phases 1–4 — not roadmap promises:

Area	ZaumZeug advantage
Improvement loop	Trace capture → human rate → val/train merge → GEPA recompile → redeploy; one CLI journey (Journey C, E6-S08) — most SDKs stop at tracing or eval dashboards
Compile → ship	DSPy/GEPA offline, Haystack online — enforced separation; optimized artifacts ship inside deploy/
Production packaging	Self-contained deploy/ bundle; GEPA and DeepEval never required on production servers
Config-first multi-agent	YAML workflow + per-agent models/tools; Mermaid workflow-diagram.md + workflow-paths.md regenerated at compile
Client surfaces	REST (OpenAI-compatible), MCP, ACP, and optional A2A from one Haystack core — no rewrite per IDE or app
Quality gates	Mock-first dev, doctor, optional DeepEval tiers, deploy eval gate before ship
Ops clarity	usr-harnesses/ vs deploy/ is a product rule, not a team convention

What others do better today

Framework	Their strength	ZaumZeug gap (for now)
LangGraph	Stateful graphs, checkpointing, runtime human-in-the-loop, branching	Serial workflows only (parallel deferred); no graph checkpoint/resume
OpenAI Agents SDK / Claude Agent SDK	Provider-native tracing, guardrails, sandbox execution	Sandbox deferred; less provider-native magic
CrewAI	Role-based crews, rapid prototyping, large community	Less persona ergonomics; more deploy discipline
Strands Compose / AgentCore	YAML multi-agent, swarm/graph modes, managed hosting	Fewer orchestration modes (parallel/swarm deferred); no managed microVM hosting
Raw Haystack	RAG pipelines, retrieval integrations	ZaumZeug wraps Haystack — use raw Haystack for RAG-only
Pydantic AI	Typed structured outputs, test models	Pydantic for config, not agent programming model
Hermes Agent	Personal agent with memory, skills, messaging, cron	Factory for domain harnesses — not a personal assistant

Honest caveat: LangGraph, OpenAI SDK, CrewAI, Strands, and Hermes have large ecosystems and battle stories. ZaumZeug has shipped Phases 1–4 — compile/deploy, MCP/ACP/A2A, trace → rate → compile — but remains narrower by design (factory for domain harnesses, not a personal assistant or graph debugger). Comparisons above reflect remaining gaps, not overall usefulness.

Full analysis: Competitive landscape

---

Core concepts

Harness — A named project containing a workflow definition, one or more agents, and the machinery to compile and serve them.

Workflow — A serial (or later, parallel) pipeline of stages. Each stage is either an agent (LLM-driven) or a pipeline (deterministic Haystack components).

Agents — Named, independently configured LLM agents. Each agent has its own agents-models.yaml for model selection and tool bindings.

Compile vs runtime — DSPy and GEPA run at compile time to optimize prompts and modules. Haystack runs at runtime to serve requests. ZaumZeug enforces this separation so production servers never run optimization workloads.

Improvement loop (Journey C) — Captured turns land in data/traces/; you rate output with zaumzeug rate or run --rate, set gold, merge into val/train, and optionally recompile (E6-S08). Automated eval (zaumzeug eval) and production trace capture (eval.trace_capture) feed the same loop.

Dev vs deploy — You author and experiment in usr-harnesses/<name>/. zaumzeug deploy produces a self-contained deploy/<name>/ bundle ready for any target host.

---

Creating a harness

Rough steps to go from zero to a working harness:

1. Install — OS wrapper or Python entrypoint (see installation):
   • Windows (PowerShell): .\install\client-install.ps1
   • macOS / Linux: ./install/client-install.sh
   • Any OS: python install/client_install.py
   • Install also scaffolds usr-harnesses/example/, config/models.yaml, .env.example, and related project files (idempotent; never overwrites an existing .env).
2. Scaffold more harnesses — zaumzeug init <name> creates additional usr-harnesses/<name>/ trees; add --with-eval for an optional DeepEval pytest scaffold under eval/. After install you can already zaumzeug serve example with the mock model (no API keys).
3. Define the workflow — edit workflow.yaml: set the entry agent and ordered stages (agent or pipeline steps).
4. Configure agents — under agents/<name>/, set prompts and tools in agent.yaml and pick models in agents-models.yaml.
5. Add code only if needed — custom tools, pipelines, or DSPy modules when YAML is not enough.
6. Validate — zaumzeug doctor <name> checks config, model refs, and environment before you run anything.

See Multi-agent workflow for file layout and YAML examples.

---

Development workflow

Once scaffolded, you iterate in usr-harnesses/<name>/:

1. Edit — change workflow, agent prompts, tools, or models in YAML; touch Python only for custom logic.
2. Validate — zaumzeug doctor <name> after each meaningful change.
3. Run locally — zaumzeug run <name> executes all workflow stages in order (--json for machine-readable output — E8-S07); zaumzeug serve <name> opens interactive chat with the entry agent.
4. Compile — zaumzeug compile <name> runs DSPy/GEPA offline, writes optimized artifacts under each agent, and regenerates docs/workflow-diagram.md + docs/workflow-paths.md (E4-S08 / E4-S09).
5. Test — pytest for SDK contributors; harness developers can add optional eval/ tests with DeepEval (pip install "zaumzeug[eval]" — E6-S01; scaffold via zaumzeug init --with-eval or the example harness — E6-S04; run deepeval test run usr-harnesses/<name>/eval/). Score a captured failure with zaumzeug eval --trace (E6-S05). Capture serve traces into datasets with eval.trace_capture / ZAUMZEUG_TRACE_CAPTURE (E8-S03). Merge failed eval rows into train via the GEPA ↔ DeepEval bridge (E6-S07). Rate output and set gold with zaumzeug rate / run --rate (E6-S08). Black-box a running serve with zaumzeug eval --base-url (E6-S06, staging/debug). Default dev uses the mock model profile — no API keys required.
6. Deploy — zaumzeug deploy <name> packages usr-harnesses/<name>/ into deploy/<name>/ for production. Ops ships that folder: copy → target_install → create .env on the target → zaumzeug serve <name> --deploy (or HARNESS_ROOT=deploy in containers / systemd). Optional container image: zaumzeug deploy <name> --dockerfile (or deploy.dockerfile: true) writes a Dockerfile into the bundle (E8-S05). Linux is the primary production target (E5-S08); see Deploy and ship.

Production note: Runtime deps on the target are updated by redeploying (zaumzeug deploy → copy bundle → target_install), not by upgrading packages in place. Target installs pip install zaumzeug only — no DSPy, GEPA, or DeepEval by default. Python 3.11+ must already be on the host. .env is created on the target and is never inside the bundle. By default deploy/ is gitignored (local-only bundles); teams may force-add deploy/<name>/ for audited release tags or upload the bundle as a CI artifact — see Git policy.

Never edit deploy/ by hand — it is regenerated on each deploy. See Testing for mock vs live test tiers.

---

Technology stack

Layer	Library	Role
Runtime	Haystack + Hayhooks	Agent loop, tools, RAG, REST/MCP serve
Specification	DSPy	Signatures and modules (compile-time only)
Optimization	GEPA	Offline prompt and module optimization
Evaluation	DeepEval (optional [eval] extra)	Dev regression — install with pip install "zaumzeug[eval]" (E6-S01); zaumzeug eval Done (E6-S02); deploy eval gate Done (E6-S03 / E5-S06); eval/ pytest scaffold Done (E6-S04); trace → test case Done (E6-S05); Mode C HTTP --base-url Done (E6-S06); GEPA ↔ DeepEval bridge Done (E6-S07 — failed eval → train, shared val aliases); human rate → compile Done (E6-S08 — zaumzeug rate / run --rate, no DeepEval required); production trace → dataset ratchet Done (E8-S03 — eval.trace_capture / ZAUMZEUG_TRACE_CAPTURE); Mode B @observe / Confident AI Done (E8-S06 — opt-in eval.confident_ai.enabled)

Model routing is configuration-driven via models.yaml — swap providers without changing agent code. See dependencies.md for the full list of external tools and libraries, including Python version notes.

---

CLI

Quick reference for all shipped subcommands — full flags in CLI reference.

Scaffold & validate

zaumzeug init <name> [--force] [--with-eval]   # New harness under usr-harnesses/ (E1-S04; --with-eval → eval/ scaffold E6-S04)
zaumzeug doctor [harness] [--deploy] [--verbose] [--json]   # Preflight env + config (E2-S04)

Doctor validates harness.yaml, workflow.yaml (entry, stages, agent refs, agent folder names), merged model registries, and runtime paths. It warns (non-blocking) when: MCP tools are configured but uv is missing; workflow.yaml is newer than docs/workflow-diagram.md; or docs/workflow-paths.md is missing or stale vs workflow.yaml / workflow-listed agents/*/agent.yaml. Optional eval: / deploy: sections are validated (no Phase warn after E6-S02 / E6-S03). In --deploy mode, doctor validates manifest.json, pinned artifact runs, and reuses core config/runtime checks against the deploy bundle (dev-only warnings are skipped).

Run & serve

zaumzeug run <harness> [--input "…"] [--from STAGE] [--deploy] [--json] [--verbose]   # Serial workflow (E3-S08; --json E8-S07)
zaumzeug serve <harness> [--dry-run] [--deploy] [--json] [--verbose] [--port N]   # Hayhooks REST + optional Chainlit /chat (E3-S02; E7-S04)
HARNESS_ROOT=deploy zaumzeug serve|run <harness>   # Production root without --deploy (E5-S04)

Live serve binds 127.0.0.1:{port} (default 1416); pipelines in serve.expose get POST /{id}/run (E8-S01). Opt-in trace capture: eval.trace_capture or ZAUMZEUG_TRACE_CAPTURE=1 (E8-S03).

Improve (Journey C)

zaumzeug rate <harness> (--trace PATH | --unrated) [--apply] [--compile] [--force]   # Human rating → gold → optional GEPA (E6-S08)
zaumzeug run <harness> --input "…" --rate [--apply] [--compile]   # Run then rate last agent stage (E6-S08)
zaumzeug eval <harness> [--framework builtin|deepeval] [--agent NAME] [--trace PATH] [--base-url]   # Regression + trace score + HTTP black-box (E6-S02/S05/S06)
deepeval test run usr-harnesses/<name>/eval/   # Optional pytest scaffold (E6-S04; needs [eval])

zaumzeug rate scores a captured turn in data/traces/, writes data/ratings/, merges gold into val/train on keep_val / keep_train, and optionally recompiles — stdlib only, no DeepEval required. See Human quality rating → recompile.

Compile & deploy

zaumzeug compile <harness> [--agent NAME] [--dry-run | --workflow-diagram-only] [--verbose]   # GEPA + workflow docs (E4-S02/S08/S09)
zaumzeug deploy <harness> [--eval-gate skip|warn|block] [--dockerfile] [--verbose]   # Production bundle (E5-S01; gate E6-S03; Dockerfile E8-S05)

Every compile (except --dry-run) regenerates docs/workflow-diagram.md and docs/workflow-paths.md. Deploy runs doctor + eval gate, then writes deploy/<name>/.

Client protocols

zaumzeug mcp <harness> [--deploy] [--port N] [--verbose]   # MCP at http://127.0.0.1:{mcp_port}/mcp (E7-S01; deploy E7-S06)
zaumzeug acp <harness> [--deploy] [--verbose]                # ACP stdio JSON-RPC — Zed / JetBrains (E7-S05)
zaumzeug a2a <harness> [--deploy] [--port N] [--verbose]     # Optional A2A agent card (E8-S04; needs serve.a2a + hayhooks[a2a])
# Cursor: .cursor/mcp.json → MCP URL (install --cursor-mcp or copy example; E7-S03)

Deferred: zaumzeug install-project (PyPI scaffold — E1-S08) · zaumzeug replay (PRD §9).

---

Repository layout

├── docs/                  # PRD, architecture, epic stories (planning)
├── user-docs/             # End-user guides (getting started, workflow, testing)
├── install/               # Client and target install scripts + templates
├── scripts/               # Doc maintenance utilities
├── src/zaumzeug/          # SDK source (`cli`, `config`, `doctor`, `runtime`, …)
├── usr-harnesses/         # Developer workspace for harness authoring
└── deploy/                # Production bundles (generated by zaumzeug deploy)

---

Documentation

Document	Description
Product Requirements (PRD)	Goals, personas, epics, acceptance criteria, phases
Architecture	Technical design — stack, data flows, config, deploy
Epic stories	Implementation backlog broken into epics and stories
User guides	How to build and run harnesses — install, workflow, compile, deploy/ship, production auth, ACP, A2A, testing
Dependencies	External tools, libraries, and protocols
Competitive landscape	How ZaumZeug compares to other harness SDKs

---

Who is this for?

• Harness developers — Scaffold a harness, configure models and tools, run → rate → compile, deploy
• SDK maintainers — Extend the framework, examples, CI, and docs; deploy git policy (ignore vs release commit vs CI artifact) in CONTRIBUTING.md
• Ops / platform — Copy deploy/<name>/ → target_install → .env → zaumzeug serve --deploy or zaumzeug mcp --deploy or zaumzeug a2a --deploy (or HARNESS_ROOT=deploy). Put Traefik in front for TLS + auth (Production auth). Linux-first production (VM / bare metal / Docker/K8s); Win/macOS best-effort. Full guide: Deploy and ship.
• Client integrators — Call a harness via REST, MCP, or ACP from an app or IDE

---

Contributing

Issues and discussions are welcome. See docs/stories/ for the implementation roadmap and current progress. For local SDK work, use Python 3.11–3.13 and see Python version and local install if pip install -e ".[dev]" fails.

---

License

GNU General Public License v3.0