diff --git a/.agents/skills/bpmn-to-agentic-workflow/SKILL.md b/.agents/skills/bpmn-to-agentic-workflow/SKILL.md index 6f106f7..daa3254 100644 --- a/.agents/skills/bpmn-to-agentic-workflow/SKILL.md +++ b/.agents/skills/bpmn-to-agentic-workflow/SKILL.md @@ -94,6 +94,8 @@ Report in the user's language: - **Install** — nothing was written into a real `.claude/`; the user copies it: `cp -R generated//.claude/. /.claude/` (merge `settings.json` `hooks` by hand if one exists). Details in `README.md`. +- **Run cost** — the generated payload records every run's token usage; after a run the user gets + the cost per BPMN element with `/bpmn2agent-cost` (needs the lanecraft plugin). - **Runs only on invocation** — a generated Workflow script never runs automatically, not even in this pipeline; the user starts it via the Workflow tool. - **Claude-only** — hooks, the Workflow script and generated agents exist only in Claude Code; diff --git a/.agents/skills/bpmn2agent-cost/SKILL.md b/.agents/skills/bpmn2agent-cost/SKILL.md new file mode 100644 index 0000000..1fa838a --- /dev/null +++ b/.agents/skills/bpmn2agent-cost/SKILL.md @@ -0,0 +1,61 @@ +--- +name: bpmn2agent-cost +description: Reports what a run of a generated workflow cost and which BPMN elements, lanes and phases the cost belongs to, from the session transcripts or the workflow's cost ledger, and reconciles the parts against the session total. Use after a generated workflow ran ("what did that run cost", "which step is expensive") or to benchmark repeated runs. Deterministic, read-only; needs the workflow's cost map. +--- + +# Run cost per BPMN element + +Tokens come from the session transcripts, prices from `scripts/prices.json`, the element lookup from +the generated `.claude/hooks/-cost-map.json`. Nothing here is estimated by the model: +report the numbers the script prints and never recompute or round them yourself. + +## 1. Find the inputs + +- **Cost map:** `.claude/hooks/-cost-map.json` in the project where the workflow runs. In the + repo that generated the workflow: `generated//.claude/hooks/-cost-map.json` + (legacy layout: `generated//hooks/…`). Missing → the workflow was generated before cost + tracking: re-run `bpmn2agent-generate`, which writes it. Several workflows → `AskUserQuestion` which one. +- **Run data**, in this order: + 1. `.claude/runs//ledger.jsonl`, written by the workflow's cost-ledger hook; survives + the cleanup of old transcripts. No session named → the newest. + 2. A session transcript `~/.claude/projects//.jsonl` (subagents sit next to it); + use it when the workflow has no ledger yet or to cross-check one. + +## 2. Run the report + +```bash +node ${CLAUDE_SKILL_DIR}/scripts/cost-report.mjs --map \ + --ledger .claude/runs//ledger.jsonl # or: --session +``` + +Optional: `--session-id ` (ledger with several runs), `--claude-json ` (result of +`claude -p --output-format json`, its `total_cost_usd` is compared), `--out `. It writes +`cost.json` and `cost.md` next to the ledger and prints the checks. Exit `1` means a reconciliation +check failed: say which one and do not present the cost as reliable. + +## 3. Read the result to the user + +- Lead with the total and the top elements; point at `cost.md` for the tables. +- **nicht zugeordnet** over 5 % means the label convention or the cost map is off. Name where the + requests came from (the report lists them); do not distribute the cost over elements. +- **Hilfsaufrufe** are in the session total but in no transcript (web-search helper model and the + like). A warning that a model's tokens are missing means the session was resumed or cleared. +- MCP calls (Gemini/NotebookLM and others) are counted but carry no price: they are billed outside + Claude. +- Prices are those of `prices.json`; the report names its version. A failing "Preis reproduziert + cost-state" check means the table is stale: update it from the current price list, then re-run. + +## 4. Repeated runs + +`scripts/cost-bench.mjs` runs the same input N times with `claude -p` (each in a fresh copy of a +template project) and reports median, min/max and IQR per element, lane and phase; see its header for +the options. Every run spends real money: print the plan first (no `--confirm`), tell the user the +number of runs, and start only after they agree. It refuses a workflow with human checkpoints +(`claude -p` cannot answer them) unless the prompt carries the answers (`--allow-checkpoints`). +`--drop-first` leaves out the first run, whose cache writes make it dearer. Under 3 runs the spread +says little. + +## What this skill never does + +Edit the workflow, the spec or the price table without being asked, or turn missing data into an +estimate. diff --git a/.agents/skills/bpmn2agent-cost/scripts/cost-bench.mjs b/.agents/skills/bpmn2agent-cost/scripts/cost-bench.mjs new file mode 100644 index 0000000..c64e46e --- /dev/null +++ b/.agents/skills/bpmn2agent-cost/scripts/cost-bench.mjs @@ -0,0 +1,162 @@ +#!/usr/bin/env node +// Runs the same input through a generated workflow N times with `claude -p` and reports the cost +// per BPMN element as median and spread. Measuring is deterministic, the runs are not: one run says +// little, the spread over several says what a step really costs. +// +// Usage: +// node cost-bench.mjs --map --prompt "" --template --confirm [options] +// Options: +// --prompt-file read the prompt from a file instead +// --template project directory (with the installed workflow); each run gets a fresh copy. +// Without it the runs share --cwd (default .) and see each other's results. +// --runs default 5 +// --drop-first leave the first run out of the statistics (cache writes make it dearer) +// --max-usd stop starting new runs once the runs so far cost more than x +// --claude-args "<..>" extra arguments for claude, split on spaces (e.g. "--permission-mode acceptEdits") +// --claude-bin default: claude (a .mjs path is run with node) +// --allow-checkpoints the prompt already answers every human checkpoint of the workflow +// --out default: ./cost-bench ; writes bench.json and bench.md +// --confirm required: every run spends real money. Without it only the plan is printed. +// +// A workflow with human checkpoints on its path cannot run headless; it is refused unless +// --allow-checkpoints says the prompt carries the answers. + +import { spawnSync } from 'node:child_process' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { fileURLToPath } from 'node:url' +import { buildReport } from './cost-report.mjs' +import { loadPrices, readSession } from './read-usage.mjs' + +export function quantile(sorted, q) { + if (!sorted.length) return 0 + const pos = (sorted.length - 1) * q + const lo = Math.floor(pos) + const hi = Math.ceil(pos) + return sorted[lo] + (sorted[hi] - sorted[lo]) * (pos - lo) +} + +export function stats(values) { + const s = [...values].sort((a, b) => a - b) + return { n: s.length, median: quantile(s, 0.5), min: s[0] ?? 0, max: s.at(-1) ?? 0, iqr: quantile(s, 0.75) - quantile(s, 0.25) } +} + +/** reports: buildReport results of the kept runs -> per-unit statistics (a run without the unit counts as 0). */ +export function aggregate(reports) { + const units = new Map() + for (const rep of reports) for (const u of rep.byUnit) { + const key = `${u.type}:${u.id}` + if (!units.has(key)) units.set(key, { type: u.type, id: u.id, label: u.label, lane: u.lane, perRun: new Array(reports.length).fill(0) }) + } + reports.forEach((rep, i) => { + for (const u of rep.byUnit) units.get(`${u.type}:${u.id}`).perRun[i] = u.costUsd + }) + const rows = [...units.values()].map((u) => ({ ...u, ...stats(u.perRun) })).sort((a, b) => b.median - a.median) + return { + runs: reports.length, + total: stats(reports.map((r) => r.totals.sessionUsd)), + lanes: aggregateBy(reports, (r) => r.byLane), + phases: aggregateBy(reports, (r) => r.byPhase), + units: rows, + } +} + +function aggregateBy(reports, pick) { + const names = new Set(reports.flatMap((r) => pick(r).map((x) => x.name))) + return [...names].map((name) => ({ + name, ...stats(reports.map((r) => pick(r).find((x) => x.name === name)?.costUsd ?? 0)), + })).sort((a, b) => b.median - a.median) +} + +const usd = (n) => n.toFixed(4) + +export function renderBenchMarkdown(workflow, agg, runInfo) { + const t = (head, rows) => [`| ${head.join(' | ')} |`, `| ${head.map(() => '---').join(' | ')} |`, ...rows.map((r) => `| ${r.join(' | ')} |`), ''] + const L = [`# Kostenstreuung ${workflow}`, '', `${agg.runs} Läufe${runInfo.dropped ? ` (erster Lauf verworfen)` : ''}, Preise \`prices.json\` ${runInfo.pricesVersion}.`, ''] + L.push(`**Gesamt je Lauf:** Median ${usd(agg.total.median)} USD, Spanne ${usd(agg.total.min)}–${usd(agg.total.max)} USD, IQR ${usd(agg.total.iqr)} USD.`, '') + if (agg.runs < 3) L.push('Hinweis: unter 3 Läufen sind Median und IQR nicht aussagekräftig.', '') + const row = (x) => [x.name ?? x.label, usd(x.median), usd(x.min), usd(x.max), usd(x.iqr), x.n] + L.push('## Je Lane', '', ...t(['Lane', 'Median', 'Min', 'Max', 'IQR', 'Läufe'], agg.lanes.map(row))) + L.push('## Je Phase', '', ...t(['Phase', 'Median', 'Min', 'Max', 'IQR', 'Läufe'], agg.phases.map(row))) + L.push('## Je Element', '', ...t(['Element', 'Label', 'Median', 'Min', 'Max', 'IQR'], agg.units.map((u) => [u.type === 'element' ? `\`${u.id}\`` : u.type, u.label.replace(/\|/g, '/'), usd(u.median), usd(u.min), usd(u.max), usd(u.iqr)]))) + L.push('## Läufe', '', ...t(['Lauf', 'Session', 'USD', 'Abgleich'], runInfo.runs.map((r, i) => [i + 1, r.sessionId, usd(r.totalUsd), r.ok ? 'ok' : 'FEHLER']))) + return L.join('\n') +} + +function findTranscript(sessionId) { + const root = path.join(process.env.CLAUDE_CONFIG_DIR ?? path.join(os.homedir(), '.claude'), 'projects') + for (const d of fs.existsSync(root) ? fs.readdirSync(root) : []) { + const f = path.join(root, d, `${sessionId}.jsonl`) + if (fs.existsSync(f)) return f + } + return null +} + +function parseArgs(argv) { + const flags = new Set(['--confirm', '--drop-first', '--allow-checkpoints']) + const o = {} + for (let i = 0; i < argv.length; i++) { + const a = argv[i] + if (!a.startsWith('--')) continue + o[a.slice(2)] = flags.has(a) ? true : argv[++i] + } + return o +} + +function main() { + const o = parseArgs(process.argv.slice(2)) + const prompt = o['prompt-file'] ? fs.readFileSync(o['prompt-file'], 'utf8') : o.prompt + if (!o.map || !prompt) { + console.error('usage: cost-bench.mjs --map --prompt "" --template --confirm [--runs 5] [--drop-first] [--max-usd x]') + process.exit(2) + } + const map = JSON.parse(fs.readFileSync(o.map, 'utf8')) + const runs = Number(o.runs ?? 5) + if ((map.humanCheckpoints ?? []).length && !o['allow-checkpoints']) { + console.error(`${map.workflow} has human checkpoints (${map.humanCheckpoints.map((c) => c.id).join(', ')}); claude -p cannot answer them.\n` + + 'Put the answers into the prompt and pass --allow-checkpoints, or benchmark a workflow without checkpoints on its path.') + process.exit(2) + } + console.log(`plan: ${runs} run(s) of ${map.workflow} with claude -p${o.template ? `, each in a fresh copy of ${o.template}` : `, all in ${o.cwd ?? '.'} (they share state)`}${o['max-usd'] ? `, stop above ${o['max-usd']} USD` : ''}.`) + if (!o.confirm) { + console.log('Every run spends real money. Re-run with --confirm to start.') + process.exit(0) + } + const prices = loadPrices() + const bin = o['claude-bin'] ?? 'claude' + const extra = (o['claude-args'] ?? '').split(/\s+/).filter(Boolean) + const results = [] + let spent = 0 + for (let i = 0; i < runs; i++) { + if (o['max-usd'] && spent > Number(o['max-usd'])) { console.log(`budget ${o['max-usd']} USD exceeded after ${i} run(s), stopping`); break } + let cwd = path.resolve(o.cwd ?? '.') + if (o.template) { + cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'cost-bench-')) + fs.cpSync(o.template, cwd, { recursive: true }) + } + const cmd = bin.endsWith('.mjs') ? ['node', [bin, '-p', prompt, '--output-format', 'json', ...extra]] : [bin, ['-p', prompt, '--output-format', 'json', ...extra]] + const r = spawnSync(cmd[0], cmd[1], { cwd, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 }) + if (r.status !== 0) { console.error(`run ${i + 1} failed (exit ${r.status}): ${(r.stderr || r.stdout).slice(0, 400)}`); process.exit(1) } + const result = JSON.parse(r.stdout.trim().split('\n').at(-1)) + const transcript = findTranscript(result.session_id) + if (!transcript) { console.error(`run ${i + 1}: no transcript found for session ${result.session_id}`); process.exit(1) } + const s = readSession(transcript) + const report = buildReport({ ...s, map, prices, claudeTotal: result.total_cost_usd }) + spent += report.totals.sessionUsd + results.push({ sessionId: s.sessionId, totalUsd: report.totals.sessionUsd, ok: report.ok, report }) + console.log(`run ${i + 1}/${runs}: ${usd(report.totals.sessionUsd)} USD${report.ok ? '' : ' (reconciliation failed)'}`) + } + const kept = o['drop-first'] ? results.slice(1) : results + if (!kept.length) { console.error('no run left to report'); process.exit(1) } + const agg = aggregate(kept.map((r) => r.report)) + const out = path.resolve(o.out ?? 'cost-bench') + fs.mkdirSync(out, { recursive: true }) + const runInfo = { dropped: !!o['drop-first'], pricesVersion: prices.version, runs: results.map(({ sessionId, totalUsd, ok }) => ({ sessionId, totalUsd, ok })) } + fs.writeFileSync(path.join(out, 'bench.json'), JSON.stringify({ workflow: map.workflow, ...runInfo, aggregate: agg }, null, 2) + '\n') + fs.writeFileSync(path.join(out, 'bench.md'), renderBenchMarkdown(map.workflow, agg, runInfo)) + console.log(`median ${usd(agg.total.median)} USD per run over ${agg.runs} run(s) -> ${out}/bench.md`) + process.exit(results.every((r) => r.ok) ? 0 : 1) +} + +if (process.argv[1] && fileURLToPath(import.meta.url) === path.resolve(process.argv[1])) main() diff --git a/.agents/skills/bpmn2agent-cost/scripts/cost-report.mjs b/.agents/skills/bpmn2agent-cost/scripts/cost-report.mjs new file mode 100644 index 0000000..9d903af --- /dev/null +++ b/.agents/skills/bpmn2agent-cost/scripts/cost-report.mjs @@ -0,0 +1,256 @@ +#!/usr/bin/env node +// Attributes the cost of one workflow run to BPMN elements, lanes and phases and reconciles the +// parts against the session total. Deterministic: tokens come from the transcripts, prices from +// prices.json, the element lookup from the generated -cost-map.json. No model involved. +// +// Usage: +// node cost-report.mjs --map --session [options] +// node cost-report.mjs --map --ledger [--session-id ] [options] +// Options: +// --claude-json result of `claude -p --output-format json`; its total_cost_usd is compared +// --prices default: prices.json next to this script +// --out default: // or ./cost// +// --json print cost.json to stdout as well +// +// Exit 0: reconciled (warnings allowed). Exit 1: a reconciliation check failed. Exit 2: bad usage. +// Attribution order per API request (first match wins): +// 1. subagent description starts with an element id of the map -> that element +// 2. subagent agent type is the orchestrator agent -> Orchestrierung +// 3. attributionSkill names a skill of the map -> its element, or the skill group +// 4. subagent agent type is a lane agent -> that lane (element unknown) +// 5. main thread -> Orchestrierung +// 6. anything else -> nicht zugeordnet (never spread) + +import fs from 'node:fs' +import path from 'node:path' +import { fileURLToPath } from 'node:url' +import { costOf, loadPrices, priceKey, readLedger, readSession, sumUsage } from './read-usage.mjs' + +const stripPlugin = (s) => (s ?? '').replace(/^[^:]+:/, '') + +export function attribute(req, map) { + const first = (req.description ?? '').split(/\s+/)[0].replace(/[:,]$/, '') + if (req.source === 'subagent' && map.elements[first]) return { type: 'element', id: first } + const agent = stripPlugin(req.agentType) + if (req.source === 'subagent' && map.orchestration.agentTypes.includes(agent)) return { type: 'orchestration', id: 'Orchestrierung' } + const skill = stripPlugin(req.attributionSkill) + if (skill && map.orchestration.skills.includes(skill)) return { type: 'orchestration', id: 'Orchestrierung' } + if (skill && map.skills[skill]) { + const ids = map.skills[skill] + return ids.length === 1 ? { type: 'element', id: ids[0] } : { type: 'skill', id: skill } + } + if (req.source === 'subagent' && map.agentTypes[agent]) return { type: 'lane', id: map.agentTypes[agent] } + if (req.source === 'main') return { type: 'orchestration', id: 'Orchestrierung' } + return { type: 'unassigned', id: 'nicht zugeordnet' } +} + +const MODEL_TOKEN_FIELDS = [ + ['input', 'inputTokens'], ['output', 'outputTokens'], ['cacheRead', 'cacheReadInputTokens'], +] + +export function buildReport({ requests, costState, map, prices, claudeTotal, sessionId, version, incomplete = 0 }) { + const rows = new Map() + const mcp = new Map() + const byModel = new Map() + let total = 0 + for (const r of requests) { + const cost = costOf(r.model, r.usage, prices) + total += cost + const a = attribute(r, map) + const key = `${a.type}:${a.id}` + if (!rows.has(key)) { + const el = a.type === 'element' ? map.elements[a.id] : null + const lane = el ? el.lane : a.type === 'lane' ? a.id : a.type === 'skill' + ? [...new Set(map.skills[a.id].map((i) => map.elements[i]?.lane))].filter(Boolean)[0] ?? null : null + rows.set(key, { + type: a.type, id: a.id, + label: el?.label ?? (a.type === 'lane' ? (map.lanes[a.id] ?? a.id) : a.type === 'skill' ? `Skill ${a.id}` : a.id), + elements: a.type === 'skill' ? map.skills[a.id] : a.type === 'element' ? [a.id] : [], + lane, phase: el?.phase ?? null, requests: 0, usage: [], costUsd: 0, + unassignedFrom: a.type === 'unassigned' ? new Set() : null, + }) + } + const row = rows.get(key) + row.requests++ + row.usage.push(r.usage) + row.costUsd += cost + row.unassignedFrom?.add(`${r.agentType ?? 'main'} | ${r.description ?? ''}`.slice(0, 80)) + if (r.attributionMcpServer) mcp.set(r.attributionMcpServer, (mcp.get(r.attributionMcpServer) ?? 0) + 1) + const m = priceKey(r.model, prices) + byModel.set(m, [...(byModel.get(m) ?? []), { usage: r.usage, cost }]) + } + const outRows = [...rows.values()].map((r) => ({ + ...r, usage: sumUsage(r.usage), unassignedFrom: r.unassignedFrom ? [...r.unassignedFrom].slice(0, 10) : undefined, + })).sort((a, b) => b.costUsd - a.costUsd) + + const rollup = (keyFn) => { + const m = new Map() + for (const r of outRows) { + const k = keyFn(r) + m.set(k, { requests: (m.get(k)?.requests ?? 0) + r.requests, costUsd: (m.get(k)?.costUsd ?? 0) + r.costUsd }) + } + return [...m.entries()].map(([name, v]) => ({ name, ...v })).sort((a, b) => b.costUsd - a.costUsd) + } + const laneName = (r) => r.type === 'orchestration' || r.type === 'unassigned' ? r.id : (r.lane ? (map.lanes[r.lane] ?? r.lane) : '(Lane unbekannt)') + const models = [...byModel.entries()].map(([model, list]) => ({ + model, requests: list.length, usage: sumUsage(list.map((x) => x.usage)), costUsd: list.reduce((s, x) => s + x.cost, 0), + })).sort((a, b) => b.costUsd - a.costUsd) + + // ---- reconciliation --------------------------------------------------------------------- + // cost-state is Claude Code's own running total of the PROCESS. It is the hard reference only when + // it matches the transcripts token for token (a single uninterrupted run, e.g. `claude -p`). For a + // resumed or cleared session it can hold less (earlier process) or lag behind (snapshot), which says + // nothing about our attribution, so that is a warning and never a failure. Failures are what this + // code can be blamed for: an unpriced model, a price table that does not reproduce costUSD on an + // exact match, a claude -p total that does not match. + const checks = [] + let aux = null + if (!costState) { + checks.push({ level: 'warn', text: 'Kein cost-state im Transkript: Summe der Teile kann nicht gegen die Session-Summe geprüft werden.' }) + } else { + aux = { costUsd: 0, models: [] } + // cost-state names carry context variants (claude-opus-5-5[1m]); sum them per priced model + const csBy = new Map() + for (const [model, v] of Object.entries(costState.modelUsage)) { + const key = priceKey(model, prices) ?? model + const cur = csBy.get(key) ?? { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, webSearch: 0, costUSD: 0, names: [] } + cur.input += v.inputTokens; cur.output += v.outputTokens; cur.cacheRead += v.cacheReadInputTokens + cur.cacheWrite += v.cacheCreationInputTokens; cur.webSearch += v.webSearchRequests ?? 0; cur.costUSD += v.costUSD + cur.names.push(model) + csBy.set(key, cur) + } + for (const [key, cs] of csBy) { + const mine = models.find((x) => x.model === key) + const have = mine?.usage ?? sumUsage([]) + const diff = { + input: cs.input - have.input, output: cs.output - have.output, cacheRead: cs.cacheRead - have.cacheRead, + cacheWrite: cs.cacheWrite - (have.cacheWrite5m + have.cacheWrite1h), + } + const lagging = Object.entries(diff).filter(([, d]) => d < 0).map(([k]) => k) + const tokenExact = Object.values(diff).every((d) => d === 0) && cs.webSearch === have.webSearch + if (lagging.length) { + checks.push({ + level: 'warn', + text: `${key}: cost-state steht hinter dem Transkript zurück (${lagging.join(', ')}). Die Session wurde fortgesetzt oder geleert, oder cost-state ist ein älterer Zwischenstand; die Transkriptsumme gilt.`, + }) + continue + } + if (tokenExact && mine) { + const drift = Math.abs(mine.costUsd - cs.costUSD) + if (drift > Math.max(0.005, cs.costUSD * 0.005)) { + checks.push({ level: 'fail', text: `${key}: Tokens stimmen exakt, aber der berechnete Preis (${mine.costUsd.toFixed(4)} USD) weicht von cost-state (${cs.costUSD.toFixed(4)} USD) ab. prices.json ${prices.version} ist veraltet oder falsch.` }) + } + } + const auxCost = cs.costUSD - (mine?.costUsd ?? 0) + if (!tokenExact || auxCost > 0.0001) aux.models.push({ model: key, tokens: diff, costUsd: auxCost, inTranscript: !!mine }) + aux.costUsd += Math.max(0, auxCost) + } + for (const m of models) { + if (!csBy.has(m.model)) checks.push({ level: 'warn', text: `${m.model}: im Transkript, aber nicht in cost-state (frühere Session oder anderer Prozess); die Transkriptsumme gilt.` }) + } + const share = (total + aux.costUsd) > 0 ? aux.costUsd / (total + aux.costUsd) : 0 + if (share > 0.01) { + const partial = aux.models.filter((x) => x.inTranscript) + checks.push({ + level: 'warn', + text: `${(share * 100).toFixed(1)} % der Kosten (${aux.costUsd.toFixed(4)} USD) stehen nur in cost-state. ` + + (partial.length + ? `Für ${partial.map((x) => x.model).join(', ')} fehlen Tokens im Transkript: die Session wurde vermutlich fortgesetzt oder geleert.` + : 'Das sind Hilfsaufrufe (z. B. WebSearch-Hilfsmodell), die kein Transkript haben.'), + }) + } + } + const grand = total + (aux?.costUsd ?? 0) + if (claudeTotal != null) { + const d = Math.abs(claudeTotal - grand) + if (d > Math.max(0.005, claudeTotal * 0.01)) { + checks.push({ level: 'fail', text: `total_cost_usd aus claude -p (${claudeTotal.toFixed(4)} USD) weicht um ${d.toFixed(4)} USD von der Summe aus Transkripten und Hilfsaufrufen (${grand.toFixed(4)} USD) ab.` }) + } + } + if (incomplete) checks.push({ level: 'warn', text: `${incomplete} Anfrage(n) ohne stop_reason (noch nicht abgeschlossen) wurden nicht mitgezählt.` }) + const unassigned = outRows.find((r) => r.type === 'unassigned') + if (unassigned && total > 0 && unassigned.costUsd / total > 0.05) { + checks.push({ level: 'warn', text: `${((unassigned.costUsd / total) * 100).toFixed(1)} % der Kosten sind keinem Element zugeordnet: Label-Konvention oder Kostenkarte prüfen.` }) + } + + return { + workflow: map.workflow, sessionId, claudeCodeVersion: version, pricesVersion: prices.version, + totals: { transcriptUsd: total, auxiliaryUsd: aux?.costUsd ?? null, sessionUsd: grand, requests: requests.length }, + byUnit: outRows, byLane: rollup(laneName), byPhase: rollup((r) => r.phase ?? '(ohne Phase)'), byModel: models, + auxiliary: aux, mcpCalls: Object.fromEntries(mcp), + checks, ok: !checks.some((c) => c.level === 'fail'), + } +} + +const usd = (n) => (n == null ? '–' : n.toFixed(4)) +const tok = (n) => n.toLocaleString('en-US') + +export function renderMarkdown(r) { + const L = [] + L.push(`# Laufkosten ${r.workflow}`, '') + L.push(`Session \`${r.sessionId}\` · Preise \`prices.json\` ${r.pricesVersion} · Claude Code ${r.claudeCodeVersion ?? 'unbekannt'}`, '') + L.push(`**Gesamt ${usd(r.totals.sessionUsd)} USD** (Transkripte ${usd(r.totals.transcriptUsd)} USD${r.totals.auxiliaryUsd ? `, Hilfsaufrufe/nicht im Transkript ${usd(r.totals.auxiliaryUsd)} USD` : ''}), ${r.totals.requests} API-Anfragen.`, '') + L.push('## Abgleich', '') + if (!r.checks.length) L.push('Alle Prüfungen bestanden: Tokens je Modell stimmen mit cost-state überein, der Preis reproduziert cost-state.', '') + for (const c of r.checks) L.push(`- ${c.level === 'fail' ? '**FEHLER**' : 'Hinweis'}: ${c.text}`) + if (r.checks.length) L.push('') + const table = (head, rows) => [head.join(' | ').replace(/^/, '| ').replace(/$/, ' |'), head.map(() => '---').join(' | ').replace(/^/, '| ').replace(/$/, ' |'), ...rows.map((x) => `| ${x.join(' | ')} |`), ''] + L.push('## Je Lane', '', ...table(['Lane', 'Anfragen', 'USD', 'Anteil'], r.byLane.map((x) => [x.name, x.requests, usd(x.costUsd), `${((x.costUsd / (r.totals.transcriptUsd || 1)) * 100).toFixed(1)} %`]))) + L.push('## Je Phase', '', ...table(['Phase', 'Anfragen', 'USD'], r.byPhase.map((x) => [x.name, x.requests, usd(x.costUsd)]))) + L.push('## Je Element', '', ...table(['Element', 'Label', 'Lane', 'Anfragen', 'Input', 'Output', 'Cache-Read', 'Cache-Write', 'USD'], + r.byUnit.map((x) => [x.type === 'element' ? `\`${x.id}\`` : x.type, x.label.replace(/\|/g, '/'), x.lane ?? '', x.requests, tok(x.usage.input), tok(x.usage.output), tok(x.usage.cacheRead), tok(x.usage.cacheWrite5m + x.usage.cacheWrite1h), usd(x.costUsd)]))) + const un = r.byUnit.find((x) => x.type === 'unassigned') + if (un?.unassignedFrom?.length) L.push('Nicht zugeordnet kommt von:', '', ...un.unassignedFrom.map((s) => `- ${s}`), '') + L.push('## Je Modell', '', ...table(['Modell', 'Anfragen', 'Input', 'Output', 'Cache-Read', 'Cache-Write 5 min', 'Cache-Write 1 h', 'USD'], + r.byModel.map((x) => [x.model, x.requests, tok(x.usage.input), tok(x.usage.output), tok(x.usage.cacheRead), tok(x.usage.cacheWrite5m), tok(x.usage.cacheWrite1h), usd(x.costUsd)]))) + if (Object.keys(r.mcpCalls).length) L.push('## MCP-Aufrufe (ohne Preis, außerhalb von Claude)', '', ...Object.entries(r.mcpCalls).map(([k, v]) => `- ${k}: ${v}`), '') + return L.join('\n') +} + +function parseArgs(argv) { + const o = {} + for (let i = 0; i < argv.length; i++) { + const a = argv[i] + if (a === '--json') o.json = true + else if (a.startsWith('--')) o[a.slice(2)] = argv[++i] + else o._ = [...(o._ ?? []), a] + } + return o +} + +function main() { + const o = parseArgs(process.argv.slice(2)) + if (!o.map || (!o.session && !o.ledger)) { + console.error('usage: cost-report.mjs --map (--session | --ledger ) [--claude-json f] [--out dir] [--json]') + process.exit(2) + } + const map = JSON.parse(fs.readFileSync(o.map, 'utf8')) + const prices = loadPrices(o.prices) + let requests, costState, sessionId, version = null, incomplete = 0, defaultOut + if (o.session) { + const s = readSession(o.session) + ;({ requests, costState, sessionId, version, incomplete } = s) + defaultOut = path.join('cost', sessionId) + } else { + const { requests: all, costStates } = readLedger(o.ledger) + sessionId = o['session-id'] ?? all.at(-1)?.sessionId + if (!sessionId) { console.error('ledger has no requests'); process.exit(2) } + requests = all.filter((r) => r.sessionId === sessionId) + costState = costStates.get(sessionId)?.costState ?? null + version = costStates.get(sessionId)?.version ?? null + defaultOut = path.join(path.dirname(o.ledger), sessionId) + } + const claudeTotal = o['claude-json'] ? JSON.parse(fs.readFileSync(o['claude-json'], 'utf8')).total_cost_usd : null + const report = buildReport({ requests, costState, map, prices, claudeTotal, sessionId, version, incomplete }) + const out = o.out ?? defaultOut + fs.mkdirSync(out, { recursive: true }) + fs.writeFileSync(path.join(out, 'cost.json'), JSON.stringify(report, null, 2) + '\n') + fs.writeFileSync(path.join(out, 'cost.md'), renderMarkdown(report)) + if (o.json) console.log(JSON.stringify(report, null, 2)) + console.log(`session ${sessionId}: ${usd(report.totals.sessionUsd)} USD, ${report.byUnit.length} cost units -> ${out}/cost.md`) + for (const c of report.checks) console.log(`${c.level === 'fail' ? 'FAIL' : 'warn'}: ${c.text}`) + console.log(report.ok ? 'reconciliation ok' : 'RECONCILIATION FAILED') + process.exit(report.ok ? 0 : 1) +} + +if (process.argv[1] && fileURLToPath(import.meta.url) === path.resolve(process.argv[1])) main() diff --git a/.agents/skills/bpmn2agent-cost/scripts/fixture.mjs b/.agents/skills/bpmn2agent-cost/scripts/fixture.mjs new file mode 100644 index 0000000..0e4889e --- /dev/null +++ b/.agents/skills/bpmn2agent-cost/scripts/fixture.mjs @@ -0,0 +1,88 @@ +// Builds a synthetic Claude Code session in the on-disk format the readers expect (see +// docs/plans/laufkosten/spike.md). Only metadata and usage, no message content, so the tests carry +// no real transcript text. Round numbers keep the expected costs checkable by hand. + +import fs from 'node:fs' +import path from 'node:path' + +export const OPUS_USAGE = { input: 1000, output: 2000, cacheRead: 100000, w5m: 10000, w1h: 5000 } +// opus-5-5: 1000*4 + 2000*20 + 100000*0.2 + 10000*5 + 5000*8 = 154000 / 1e6 +export const OPUS_COST = 0.154 +// haiku-4-5 aux call (only in cost-state): 10000 in * 1 + 1000 out * 5 + 2 web searches * 0.01 USD +export const HAUS_COST = 10000 / 1e6 + 5000 / 1e6 + 0.02 + +export const MAP = { + bpmn: { file: 'wf.bpmn', elements: ['C1', 'K2'] }, + workflow: 'wf', + pattern: 'orchestrator-agent', + generatorVersion: '0.1.0', + orchestration: { skills: ['wf'], agentTypes: ['wf-orchestrator'] }, + elements: { + C1: { label: 'Story einordnen', lane: 'po', phase: 'Eingang', kind: 'skill', skills: ['story-writing'] }, + K2: { label: 'Story prüfen', lane: 'qa', phase: 'Prüfung', kind: 'skill', skills: ['story-check'] }, + K3: { label: 'Story freigeben', lane: 'qa', phase: 'Prüfung', kind: 'skill', skills: ['story-check'] }, + }, + skills: { 'story-writing': ['C1'], 'story-check': ['K2', 'K3'] }, + agentTypes: { 'wf-po': 'po', 'wf-qa': 'qa' }, + lanes: { po: 'Product Owner', qa: 'QA' }, +} + +const u = (c, o = {}) => ({ + input_tokens: c.input, output_tokens: o.output ?? c.output, + cache_read_input_tokens: c.cacheRead, cache_creation_input_tokens: c.w5m + c.w1h, + cache_creation: { ephemeral_5m_input_tokens: c.w5m, ephemeral_1h_input_tokens: c.w1h }, +}) + +/** One API request = a thinking line with a streaming snapshot, then the final line. */ +function request(id, model, extra = {}, c = OPUS_USAGE) { + const base = { type: 'assistant', requestId: id, ...extra } + return [ + { ...base, message: { model, content: [{ type: 'thinking' }], stop_reason: null, usage: u(c, { output: 2 }) } }, + { ...base, message: { model, content: [{ type: 'text' }], stop_reason: 'end_turn', usage: u(c) } }, + ] +} + +const write = (file, lines) => { + fs.mkdirSync(path.dirname(file), { recursive: true }) + fs.writeFileSync(file, lines.map((l) => JSON.stringify(l)).join('\n') + '\n') +} + +/** + * @returns {{dir, transcript, map}} session `sess-1`: 2 main requests (orchestration skill, element C1), + * 3 subagent requests (element K2 by description, lane QA by agent type, unassigned) and a cost-state + * that also holds a haiku auxiliary call which no transcript has. + */ +export function makeFixture(dir, { dropSubagent = false, sessionId = 'sess-1' } = {}) { + const S = sessionId + const OPUS = 'claude-opus-5-5' + write(path.join(dir, `${S}.jsonl`), [ + { type: 'user', version: '2.1.283' }, + ...request('req_main_1', OPUS, { attributionSkill: 'wf' }), + ...request('req_main_2', OPUS, { attributionSkill: 'story-writing' }), + { type: 'assistant', requestId: 'req_syn', message: { model: '', usage: u({ input: 0, output: 0, cacheRead: 0, w5m: 0, w1h: 0 }), stop_reason: 'stop_sequence' } }, + { + type: 'cost-state', sessionId: S, totalCostUSD: (dropSubagent ? 2 : 5) * OPUS_COST + HAUS_COST, + modelUsage: { + [OPUS]: { + inputTokens: (dropSubagent ? 2 : 5) * 1000, outputTokens: (dropSubagent ? 2 : 5) * 2000, + cacheReadInputTokens: (dropSubagent ? 2 : 5) * 100000, cacheCreationInputTokens: (dropSubagent ? 2 : 5) * 15000, + webSearchRequests: 0, costUSD: (dropSubagent ? 2 : 5) * OPUS_COST, + }, + 'claude-haiku-4-5-20251001': { + inputTokens: 10000, outputTokens: 1000, cacheReadInputTokens: 0, cacheCreationInputTokens: 0, + webSearchRequests: 2, costUSD: HAUS_COST, + }, + }, + }, + ]) + const subs = [ + ['a1', { agentType: 'wf-qa', description: 'K2 Story prüfen' }], + ['a2', { agentType: 'wf-qa', description: 'Nachfrage klären' }], + ['a3', { agentType: 'general-purpose', description: 'irgendwas' }], + ] + for (const [id, meta] of dropSubagent ? [] : subs) { + write(path.join(dir, S, 'subagents', `agent-${id}.jsonl`), request(`req_${id}`, OPUS)) + fs.writeFileSync(path.join(dir, S, 'subagents', `agent-${id}.meta.json`), JSON.stringify({ ...meta, spawnDepth: 1 })) + } + return { dir, transcript: path.join(dir, `${S}.jsonl`), map: MAP } +} diff --git a/.agents/skills/bpmn2agent-cost/scripts/prices.json b/.agents/skills/bpmn2agent-cost/scripts/prices.json new file mode 100644 index 0000000..1a67725 --- /dev/null +++ b/.agents/skills/bpmn2agent-cost/scripts/prices.json @@ -0,0 +1,14 @@ +{ + "version": "2026-10-04", + "retrieved": "2026-10-04", + "source": "claude-api skill model table (cached 2026-09-25) plus cache multipliers; every entry below was checked against costUSD in Claude Code cost-state records (docs/plans/laufkosten/spike.md)", + "unit": "USD per 1M tokens", + "webSearchPerRequestUsd": 0.01, + "models": { + "claude-opus-5-5": { "input": 4, "output": 20, "cacheRead": 0.2, "cacheWrite5m": 5, "cacheWrite1h": 8 }, + "claude-opus-5": { "input": 5, "output": 25, "cacheRead": 0.5, "cacheWrite5m": 6.25, "cacheWrite1h": 10 }, + "claude-sonnet-5-5": { "input": 2, "output": 10, "cacheRead": 0.2, "cacheWrite5m": 2.5, "cacheWrite1h": 4 }, + "claude-sonnet-5": { "input": 2, "output": 10, "cacheRead": 0.2, "cacheWrite5m": 2.5, "cacheWrite1h": 4 }, + "claude-haiku-4-5": { "input": 1, "output": 5, "cacheRead": 0.1, "cacheWrite5m": 1.25, "cacheWrite1h": 2 } + } +} diff --git a/.agents/skills/bpmn2agent-cost/scripts/read-usage.mjs b/.agents/skills/bpmn2agent-cost/scripts/read-usage.mjs new file mode 100644 index 0000000..832d5a9 --- /dev/null +++ b/.agents/skills/bpmn2agent-cost/scripts/read-usage.mjs @@ -0,0 +1,177 @@ +#!/usr/bin/env node +// Reads token usage out of a Claude Code session: the main transcript plus every subagent +// transcript next to it. No model involved, Node built-ins only. +// +// Usage: node read-usage.mjs [--json] +// +// Layout it understands (internal Claude Code format, see docs/plans/laufkosten/spike.md): +// /.jsonl main thread, also holds the one `cost-state` entry +// //subagents/agent-.jsonl (+ .meta.json with agentType and description) +// +// One API request spans several lines (one per content block). Only the last one carries the final +// output_tokens, so each requestId keeps the line with the highest output_tokens. A request whose +// final line has no stop_reason is incomplete: it is counted and reported, never silently summed. +// A line that looks like a request but lacks requestId/usage/model is a format change: fail loudly. + +import fs from 'node:fs' +import path from 'node:path' +import { fileURLToPath } from 'node:url' + +const PRICES_PATH = path.join(path.dirname(fileURLToPath(import.meta.url)), 'prices.json') + +export function loadPrices(file = PRICES_PATH) { + return JSON.parse(fs.readFileSync(file, 'utf8')) +} + +/** claude-haiku-4-5-20251001 -> claude-haiku-4-5 (longest known prefix); null when unknown. */ +export function priceKey(model, prices) { + return Object.keys(prices.models).sort((a, b) => b.length - a.length).find((k) => model.startsWith(k)) ?? null +} + +function* jsonLines(file) { + const lines = fs.readFileSync(file, 'utf8').split('\n') + for (let i = 0; i < lines.length; i++) { + if (!lines[i].trim()) continue + try { + yield { line: i + 1, entry: JSON.parse(lines[i]) } + } catch { + throw new Error(`${file}:${i + 1}: not valid JSON (transcript format changed?)`) + } + } +} + +function toUsage(u, where) { + for (const k of ['input_tokens', 'output_tokens', 'cache_read_input_tokens', 'cache_creation_input_tokens']) { + if (typeof u[k] !== 'number') throw new Error(`${where}: usage.${k} missing (transcript format changed?)`) + } + const cc = u.cache_creation + const w1h = cc?.ephemeral_1h_input_tokens ?? 0 + return { + input: u.input_tokens, + output: u.output_tokens, + cacheRead: u.cache_read_input_tokens, + // without the TTL split every write is billed as 5 minutes + cacheWrite5m: cc ? (cc.ephemeral_5m_input_tokens ?? 0) : u.cache_creation_input_tokens, + cacheWrite1h: w1h, + webSearch: u.server_tool_use?.web_search_requests ?? 0, + } +} + +function readThread(file, meta, requests, stats) { + for (const { line, entry } of jsonLines(file)) { + if (entry.type !== 'assistant') continue + const where = `${file}:${line}` + const msg = entry.message + // client-generated lines (API error text, "No response requested") carry no requestId and no cost + if (msg?.model === '') { stats.synthetic++; continue } + if (!entry.requestId || !msg?.usage || !msg.model) { + throw new Error(`${where}: assistant entry without requestId/usage/model (transcript format changed?)`) + } + const usage = toUsage(msg.usage, where) + const cur = requests.get(entry.requestId) + if (!cur || usage.output >= cur.usage.output) { + requests.set(entry.requestId, { + requestId: entry.requestId, + source: meta.source, + agentId: meta.agentId, + agentType: meta.agentType, + description: meta.description, + parentAgentId: meta.parentAgentId, + attributionSkill: entry.attributionSkill ?? null, + attributionMcpServer: entry.attributionMcpServer ?? null, + attributionMcpTool: entry.attributionMcpTool ?? null, + model: msg.model, + stopReason: msg.stop_reason ?? null, + usage, + }) + } + } +} + +/** + * @param {string} transcript path to .jsonl (main thread) + * @returns {{sessionId, version, requests, incomplete, costState, synthetic, files}} + */ +export function readSession(transcript) { + const sessionId = path.basename(transcript, '.jsonl') + const requests = new Map() + const stats = { synthetic: 0 } + let costState = null + let version = null + for (const { entry } of jsonLines(transcript)) { + if (entry.type === 'cost-state') costState = entry + if (!version && entry.version) version = entry.version + } + readThread(transcript, { source: 'main' }, requests, stats) + + const subDir = path.join(path.dirname(transcript), sessionId, 'subagents') + let files = 1 + if (fs.existsSync(subDir)) { + for (const f of fs.readdirSync(subDir).filter((n) => /^agent-.*\.jsonl$/.test(n)).sort()) { + const agentId = f.slice('agent-'.length, -'.jsonl'.length) + const metaFile = path.join(subDir, `agent-${agentId}.meta.json`) + const m = fs.existsSync(metaFile) ? JSON.parse(fs.readFileSync(metaFile, 'utf8')) : {} + readThread(path.join(subDir, f), { + source: 'subagent', agentId, agentType: m.agentType ?? null, description: m.description ?? null, + parentAgentId: m.parentAgentId ?? null, + }, requests, stats) + files++ + } + } + const all = [...requests.values()] + return { + sessionId, + version, + requests: all.filter((r) => r.stopReason), + incomplete: all.filter((r) => !r.stopReason).length, + costState: costState && { + totalCostUSD: costState.totalCostUSD, + hasUnknownModelCost: !!costState.hasUnknownModelCost, + modelUsage: costState.modelUsage, + }, + synthetic: stats.synthetic, + files, + } +} + +/** Reads ledger.jsonl lines written by the generated cost-ledger hook. */ +export function readLedger(file) { + const requests = new Map() // keyed by requestId: a line written twice counts once + const costStates = new Map() + for (const { entry } of jsonLines(file)) { + if (entry.kind === 'cost-state') costStates.set(entry.sessionId, entry) + else if (entry.kind === 'request') requests.set(`${entry.sessionId}/${entry.requestId}`, entry) + } + return { requests: [...requests.values()], costStates } +} + +/** Cost of one usage record in USD; throws for a model without a price. */ +export function costOf(model, usage, prices) { + const key = priceKey(model, prices) + if (!key) throw new Error(`no price for model "${model}" in prices.json (version ${prices.version}); add it, do not estimate`) + const p = prices.models[key] + return ( + usage.input * p.input + usage.output * p.output + usage.cacheRead * p.cacheRead + + usage.cacheWrite5m * p.cacheWrite5m + usage.cacheWrite1h * p.cacheWrite1h + ) / 1e6 + usage.webSearch * (prices.webSearchPerRequestUsd ?? 0) +} + +export function sumUsage(list) { + const t = { input: 0, output: 0, cacheRead: 0, cacheWrite5m: 0, cacheWrite1h: 0, webSearch: 0 } + for (const u of list) for (const k of Object.keys(t)) t[k] += u[k] ?? 0 + return t +} + +const isMain = process.argv[1] && fileURLToPath(import.meta.url) === path.resolve(process.argv[1]) +if (isMain) { + const arg = process.argv[2] + if (!arg) { console.error('usage: read-usage.mjs [--json]'); process.exit(2) } + const s = readSession(arg) + if (process.argv.includes('--json')) console.log(JSON.stringify(s, null, 2)) + else { + const byModel = new Map() + for (const r of s.requests) byModel.set(r.model, [...(byModel.get(r.model) ?? []), r.usage]) + console.log(`session ${s.sessionId}: ${s.requests.length} requests from ${s.files} transcript(s), ${s.incomplete} incomplete`) + for (const [m, us] of byModel) console.log(` ${m}`, sumUsage(us)) + } +} diff --git a/.agents/skills/bpmn2agent-cost/scripts/test.mjs b/.agents/skills/bpmn2agent-cost/scripts/test.mjs new file mode 100644 index 0000000..11fba8c --- /dev/null +++ b/.agents/skills/bpmn2agent-cost/scripts/test.mjs @@ -0,0 +1,251 @@ +// node --test .agents/skills/bpmn2agent-cost/scripts/test.mjs +import assert from 'node:assert/strict' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import test from 'node:test' +import { fileURLToPath } from 'node:url' +import { buildReport, attribute } from './cost-report.mjs' +import { HAUS_COST, MAP, OPUS_COST, makeFixture } from './fixture.mjs' +import { costOf, loadPrices, readSession, sumUsage } from './read-usage.mjs' + +const here = path.dirname(fileURLToPath(import.meta.url)) +const prices = loadPrices() +const tmp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'cost-test-')) +const near = (a, b, msg) => assert.ok(Math.abs(a - b) < 1e-9, `${msg}: ${a} != ${b}`) + +test('one request spanning several lines is counted once with the final output tokens', () => { + const s = readSession(makeFixture(tmp()).transcript) + assert.equal(s.requests.length, 5) + assert.equal(s.incomplete, 0) + assert.equal(s.synthetic, 1) + const t = sumUsage(s.requests.map((r) => r.usage)) + assert.deepEqual([t.input, t.output, t.cacheRead, t.cacheWrite5m, t.cacheWrite1h], [5000, 10000, 500000, 50000, 25000]) +}) + +test('price table reproduces the hand-computed cost', () => { + const r = readSession(makeFixture(tmp()).transcript).requests[0] + near(costOf(r.model, r.usage, prices), OPUS_COST, 'opus request') +}) + +test('unknown model fails instead of being estimated', () => { + assert.throws(() => costOf('claude-unknown-9', { input: 1, output: 1, cacheRead: 0, cacheWrite5m: 0, cacheWrite1h: 0, webSearch: 0 }, prices), /no price for model/) +}) + +test('a transcript line without requestId is a format change and fails loudly', () => { + const d = tmp() + fs.writeFileSync(path.join(d, 's.jsonl'), JSON.stringify({ type: 'assistant', message: { model: 'claude-opus-5-5', usage: {} } }) + '\n') + assert.throws(() => readSession(path.join(d, 's.jsonl')), /without requestId/) +}) + +test('attribution order: description id, skill, lane, main, unassigned', () => { + const s = readSession(makeFixture(tmp()).transcript) + const by = Object.fromEntries(s.requests.map((r) => [r.requestId, attribute(r, MAP)])) + assert.deepEqual(by.req_main_1, { type: 'orchestration', id: 'Orchestrierung' }) + assert.deepEqual(by.req_main_2, { type: 'element', id: 'C1' }) + assert.deepEqual(by.req_a1, { type: 'element', id: 'K2' }) + assert.deepEqual(by.req_a2, { type: 'lane', id: 'qa' }) + assert.deepEqual(by.req_a3, { type: 'unassigned', id: 'nicht zugeordnet' }) +}) + +test('a skill shared by several elements is attributed to the skill, not guessed', () => { + const r = { source: 'main', attributionSkill: 'story-check', description: null, agentType: null } + assert.deepEqual(attribute(r, MAP), { type: 'skill', id: 'story-check' }) + assert.deepEqual(attribute({ ...r, attributionSkill: 'plugin:story-writing' }, MAP), { type: 'element', id: 'C1' }) +}) + +test('report reconciles: parts add up, the auxiliary call is its own bucket', () => { + const s = readSession(makeFixture(tmp()).transcript) + const rep = buildReport({ ...s, map: MAP, prices }) + assert.equal(rep.ok, true, JSON.stringify(rep.checks)) + near(rep.totals.transcriptUsd, 5 * OPUS_COST, 'transcript total') + near(rep.totals.auxiliaryUsd, HAUS_COST, 'auxiliary') + near(rep.totals.sessionUsd, 5 * OPUS_COST + HAUS_COST, 'session total') + near(rep.byUnit.reduce((n, r) => n + r.costUsd, 0), rep.totals.transcriptUsd, 'sum of units') + near(rep.byLane.reduce((n, r) => n + r.costUsd, 0), rep.totals.transcriptUsd, 'sum of lanes') +}) + +test('a stale price table is caught by the token-exact cost check', () => { + const s = readSession(makeFixture(tmp()).transcript) + const stale = structuredClone(prices) + stale.models['claude-opus-5-5'].output = 25 + const rep = buildReport({ ...s, map: MAP, prices: stale }) + assert.equal(rep.ok, false) + assert.match(rep.checks.find((c) => c.level === 'fail').text, /prices\.json/) +}) + +test('a missing subagent transcript (resumed session) is a warning with the reason', () => { + const s = readSession(makeFixture(tmp(), { dropSubagent: true }).transcript) + const full = readSession(makeFixture(tmp()).transcript) + // cost-state of the full session against only the main thread: tokens missing, not exceeding + const rep = buildReport({ ...full, requests: s.requests, map: MAP, prices }) + assert.equal(rep.ok, true) + assert.match(rep.checks.map((c) => c.text).join('\n'), /fortgesetzt oder geleert/) +}) + +test('cost-state behind the transcript (resumed session) warns and the transcript sum counts', () => { + const s = readSession(makeFixture(tmp(), { dropSubagent: true }).transcript) + const full = readSession(makeFixture(tmp()).transcript) + const rep = buildReport({ ...s, requests: full.requests, map: MAP, prices }) + assert.equal(rep.ok, true) + assert.match(rep.checks.map((c) => c.text).join('\n'), /steht hinter dem Transkript zurück/) + near(rep.totals.sessionUsd, 5 * OPUS_COST + HAUS_COST, 'transcript sum plus the auxiliary call, never the smaller cost-state') +}) + +test('cost-state keys with a context variant ([1m]) are summed per priced model', () => { + const s = readSession(makeFixture(tmp()).transcript) + const half = (v) => ({ ...v, inputTokens: v.inputTokens / 2, outputTokens: v.outputTokens / 2, cacheReadInputTokens: v.cacheReadInputTokens / 2, cacheCreationInputTokens: v.cacheCreationInputTokens / 2, costUSD: v.costUSD / 2 }) + const opus = s.costState.modelUsage['claude-opus-5-5'] + const split = { ...s.costState, modelUsage: { ...s.costState.modelUsage, 'claude-opus-5-5': half(opus), 'claude-opus-5-5[1m]': half(opus) } } + const rep = buildReport({ ...s, costState: split, map: MAP, prices }) + assert.equal(rep.ok, true, JSON.stringify(rep.checks)) + assert.ok(!rep.checks.some((c) => /zurück|nicht in cost-state/.test(c.text)), JSON.stringify(rep.checks)) // the two [1m]/plain halves add up exactly +}) + +test('client-generated synthetic lines without requestId are skipped, not a format error', () => { + const fx = makeFixture(tmp()) + fs.appendFileSync(fx.transcript, JSON.stringify({ type: 'assistant', isApiErrorMessage: true, message: { model: '', content: [], usage: {} } }) + '\n') + assert.equal(readSession(fx.transcript).synthetic, 2) +}) + +test('claude -p total is compared within 1 percent', () => { + const s = readSession(makeFixture(tmp()).transcript) + const total = 5 * OPUS_COST + HAUS_COST + assert.equal(buildReport({ ...s, map: MAP, prices, claudeTotal: total * 1.005 }).ok, true) + assert.equal(buildReport({ ...s, map: MAP, prices, claudeTotal: total * 1.05 }).ok, false) +}) + +// ---- the generated ledger hook --------------------------------------------------------------- +import { spawnSync } from 'node:child_process' +import { installCostLedger } from '../../bpmn2agent-generate/scripts/install-cost-ledger.mjs' +import { readLedger } from './read-usage.mjs' + +const SPEC = { + meta: { workflowName: 'wf', outputLayout: 'claude-dir', sourceBpmn: { path: 'wf.bpmn' }, generatorVersion: '0.1.0' }, + pattern: { chosen: 'orchestrator-agent' }, + roles: { po: { label: 'Product Owner', agentName: 'wf-po' }, qa: { label: 'QA', agentName: 'wf-qa' } }, + elements: { + C1: { kind: 'skill', label: 'Story einordnen', lane: 'po', generatedPaths: ['generated/wf/.claude/skills/story-writing/SKILL.md'] }, + K2: { kind: 'skill', label: 'Story prüfen', lane: 'qa', generatedPaths: ['generated/wf/.claude/skills/story-check/SKILL.md'] }, + K3: { kind: 'skill', label: 'Story freigeben', lane: 'qa', generatedPaths: ['generated/wf/.claude/skills/story-check/SKILL.md'] }, + Start: { kind: 'orchestrator', generatedPaths: ['generated/wf/.claude/skills/wf/SKILL.md'] }, + }, +} + +function runHook(project, transcript, payload = {}) { + const hook = path.join(project, '.claude', 'hooks', 'wf-cost-ledger.mjs') + return spawnSync('node', [hook], { input: JSON.stringify({ transcript_path: transcript, ...payload }), env: { ...process.env, CLAUDE_PROJECT_DIR: project }, encoding: 'utf8' }) +} + +test('install writes hook, map and settings; re-running changes nothing', () => { + const dir = tmp() + installCostLedger(SPEC, dir) + const first = fs.readFileSync(path.join(dir, '.claude', 'settings.json'), 'utf8') + installCostLedger(SPEC, dir) + assert.equal(fs.readFileSync(path.join(dir, '.claude', 'settings.json'), 'utf8'), first) + const settings = JSON.parse(first) + for (const ev of ['SubagentStop', 'Stop', 'SessionEnd']) assert.equal(settings.hooks[ev].length, 1) + const map = JSON.parse(fs.readFileSync(path.join(dir, '.claude', 'hooks', 'wf-cost-map.json'), 'utf8')) + assert.deepEqual(map.skills['story-check'], ['K2', 'K3']) + assert.deepEqual(map.orchestration.skills, ['wf']) + assert.equal(installCostLedger({ ...SPEC, meta: { ...SPEC.meta, outputLayout: undefined } }, tmp()).skipped, 'legacy output layout') +}) + +test('hook ledger matches the reader, is idempotent and records cost-state', () => { + const project = tmp() + installCostLedger(SPEC, project) + const fx = makeFixture(path.join(project, 'transcripts')) + const r1 = runHook(project, fx.transcript) + assert.equal(r1.status, 0, r1.stderr) + const ledger = path.join(project, '.claude', 'runs', 'wf', 'ledger.jsonl') + const lines1 = fs.readFileSync(ledger, 'utf8').trim().split('\n') + assert.equal(lines1.length, 6) // 5 requests + cost-state + // second event (and one handed the subagent's own transcript path) adds nothing + runHook(project, fx.transcript) + runHook(project, path.join(fx.dir, 'sess-1', 'subagents', 'agent-a1.jsonl')) + assert.equal(fs.readFileSync(ledger, 'utf8').trim().split('\n').length, 6) + + const fromLedger = readLedger(ledger) + const fromTranscript = readSession(fx.transcript) + const key = (rs) => rs.map((r) => [r.requestId, r.model, JSON.stringify(r.usage), r.agentType ?? null, r.description ?? null]).sort() + assert.deepEqual(key(fromLedger.requests), key(fromTranscript.requests)) + const rep = buildReport({ requests: fromLedger.requests, costState: fromLedger.costStates.get('sess-1').costState, map: MAP, prices, sessionId: 'sess-1' }) + assert.equal(rep.ok, true, JSON.stringify(rep.checks)) +}) + +test('hook never fails the run: bad payload and missing transcript exit 0', () => { + const project = tmp() + installCostLedger(SPEC, project) + const hook = path.join(project, '.claude', 'hooks', 'wf-cost-ledger.mjs') + assert.equal(spawnSync('node', [hook], { input: 'not json', encoding: 'utf8' }).status, 0) + assert.equal(runHook(project, '/nonexistent/x.jsonl').status, 0) +}) + +// ---- cost-bench ------------------------------------------------------------------------------ +import { aggregate, quantile, stats } from './cost-bench.mjs' + +test('stats: median, spread and IQR by linear interpolation', () => { + const s = stats([10, 1, 3, 2, 4]) + assert.deepEqual([s.n, s.median, s.min, s.max, s.iqr], [5, 3, 1, 10, 2]) + assert.equal(quantile([], 0.5), 0) +}) + +test('aggregate: a unit missing from a run counts as zero in that run', () => { + const mk = (cost, withK2) => ({ + totals: { sessionUsd: cost }, + byUnit: [{ type: 'element', id: 'C1', label: 'a', lane: 'po', costUsd: cost }, ...(withK2 ? [{ type: 'element', id: 'K2', label: 'b', lane: 'qa', costUsd: 1 }] : [])], + byLane: [{ name: 'po', costUsd: cost }], byPhase: [{ name: '(ohne Phase)', costUsd: cost }], + }) + const agg = aggregate([mk(1, true), mk(2, false), mk(3, true)]) + const k2 = agg.units.find((u) => u.id === 'K2') + assert.deepEqual(k2.perRun, [1, 0, 1]) + assert.equal(agg.total.median, 2) +}) + +function fakeClaude(dir) { + const f = path.join(dir, 'fake-claude.mjs') + fs.writeFileSync(f, ` +import fs from 'node:fs'; import path from 'node:path' +import { makeFixture, OPUS_COST, HAUS_COST } from ${JSON.stringify(path.join(here, 'fixture.mjs'))} +const c = process.env.FAKE_COUNTER; const n = (fs.existsSync(c) ? Number(fs.readFileSync(c, 'utf8')) : 0) + 1 +fs.writeFileSync(c, String(n)) +fs.writeFileSync(path.join(process.cwd(), 'ran.txt'), 'x') +makeFixture(path.join(process.env.CLAUDE_CONFIG_DIR, 'projects', 'p'), { sessionId: 'sess-' + n }) +console.log(JSON.stringify({ session_id: 'sess-' + n, total_cost_usd: 5 * OPUS_COST + HAUS_COST })) +`) + return f +} + +const bench = (args, env) => spawnSync('node', [path.join(here, 'cost-bench.mjs'), ...args], { encoding: 'utf8', env: { ...process.env, ...env } }) + +test('cost-bench runs the fake claude N times in fresh copies and reports the spread', () => { + const d = tmp() + const template = path.join(d, 'template'); fs.mkdirSync(template); fs.writeFileSync(path.join(template, 'CLAUDE.md'), 'x') + const mapFile = path.join(d, 'map.json'); fs.writeFileSync(mapFile, JSON.stringify({ ...MAP, humanCheckpoints: [] })) + const out = path.join(d, 'out'); const counter = path.join(d, 'counter') + const r = bench(['--map', mapFile, '--prompt', 'go', '--template', template, '--runs', '3', '--drop-first', '--confirm', '--claude-bin', fakeClaude(d), '--out', out], + { CLAUDE_CONFIG_DIR: path.join(d, 'cfg'), FAKE_COUNTER: counter }) + assert.equal(r.status, 0, r.stdout + r.stderr) + assert.equal(fs.readFileSync(counter, 'utf8'), '3') + assert.equal(fs.existsSync(path.join(template, 'ran.txt')), false) // the template itself stays untouched + const md = fs.readFileSync(path.join(out, 'bench.md'), 'utf8') + assert.match(md, /2 Läufe \(erster Lauf verworfen\)/) + const json = JSON.parse(fs.readFileSync(path.join(out, 'bench.json'), 'utf8')) + near(json.aggregate.total.median, 5 * OPUS_COST + HAUS_COST, 'median per run') + assert.equal(json.runs.length, 3) +}) + +test('cost-bench refuses human checkpoints and does not spend without --confirm', () => { + const d = tmp() + const mapFile = path.join(d, 'map.json'); const counter = path.join(d, 'counter') + const env = { CLAUDE_CONFIG_DIR: path.join(d, 'cfg'), FAKE_COUNTER: counter } + fs.writeFileSync(mapFile, JSON.stringify({ ...MAP, humanCheckpoints: [{ id: 'I3', label: 'freigeben' }] })) + const refused = bench(['--map', mapFile, '--prompt', 'go', '--confirm', '--claude-bin', fakeClaude(d)], env) + assert.equal(refused.status, 2) + assert.match(refused.stderr, /human checkpoints/) + fs.writeFileSync(mapFile, JSON.stringify({ ...MAP, humanCheckpoints: [] })) + const dry = bench(['--map', mapFile, '--prompt', 'go', '--claude-bin', fakeClaude(d)], env) + assert.equal(dry.status, 0) + assert.match(dry.stdout, /Re-run with --confirm/) + assert.equal(fs.existsSync(counter), false) +}) diff --git a/.agents/skills/bpmn2agent-design/references/pattern-rubric.md b/.agents/skills/bpmn2agent-design/references/pattern-rubric.md index 596f3ce..f7699e3 100644 --- a/.agents/skills/bpmn2agent-design/references/pattern-rubric.md +++ b/.agents/skills/bpmn2agent-design/references/pattern-rubric.md @@ -132,6 +132,12 @@ Two context-store rules apply on top of the table: Workflow-script phase resumes via its own journal. Log phase transitions as events so a resume can tell "mid-phase" from "between phases". +## Cost attribution + +Whatever the pattern, every dispatched step (`agent()` label, orchestrator `Agent` description) starts +with its BPMN element id. `bpmn2agent-cost` attributes a run's cost to elements by that prefix, so it +is a generate rule, not a design choice. + ## Explaining the choice to a non-technical business user Tie the explanation to the shape they drew, not to the pattern's name. Examples: diff --git a/.agents/skills/bpmn2agent-generate/SKILL.md b/.agents/skills/bpmn2agent-generate/SKILL.md index 5ef0571..6aa47d2 100644 --- a/.agents/skills/bpmn2agent-generate/SKILL.md +++ b/.agents/skills/bpmn2agent-generate/SKILL.md @@ -35,9 +35,11 @@ generated// agents/.md skills//SKILL.md (+ references/, assets/, scripts/ only where design put a script) hooks/.mjs (kind: hook, plus -write-guard / -memory-cap, step 6a) + hooks/-cost-ledger.mjs (always, step 6b) + -cost-map.json workflows/.workflow.mjs (workflow-script pattern only) settings.json (only with hooks or read-tool permissions: step 6) - (memory files are not generated: they appear in the user's project at run time) + (memory files and `.claude/runs//ledger.jsonl` are not generated: they appear in the user's + project at run time) README.md ← install + what was generated (meta.language) workflow-spec.yaml knowledge/ mapping/ ← review material, never copied ``` @@ -213,6 +215,26 @@ mapping report's "Context hooks" table): Smoke-test both with piped payloads before moving on (a guarded write, a read, a memory file at the cap and one line over); expectations are in the templates' header comments. +## 6b. Add run-cost tracking + +Always, for `claude-dir` output (the legacy layout is left alone). Run +`node ${CLAUDE_SKILL_DIR}/scripts/install-cost-ledger.mjs "$cacheDir" generated/` after +steps 5–7, once the skills and the orchestration file exist; re-run it after any spec change. It +writes, deterministically: + +- `.claude/hooks/-cost-ledger.mjs` from `assets/templates/hook-cost-ledger-template.mjs`: + on `SubagentStop`, `Stop` and `SessionEnd` it appends each finished API request once to + `.claude/runs//ledger.jsonl` (raw token counts, no prices) and never fails the run. Its + header lists no element (it belongs to the whole workflow) and claims no `generatedPaths` entry. +- `.claude/hooks/-cost-map.json` from the spec (`scripts/build-cost-map.mjs`): which skill + names, agent types and element ids in a transcript belong to which BPMN element, lane and phase. + It is data, not a script, and carries the `bpmn` trace as a JSON key. +- the three registrations in `.claude/settings.json`. + +Nothing is priced here: the lanecraft plugin's `bpmn2agent-cost` reads the ledger and the map after a +run. Smoke-test the hook with a piped payload (`{"transcript_path": …}`); the template header has the +expectation. + ## 7. Materialize the one top-level orchestration file Exactly one, by `pattern.chosen` (for `mixed`, one per phase): @@ -224,7 +246,9 @@ Exactly one, by `pattern.chosen` (for `mixed`, one per phase): from `args` and returns `blocked` when one is missing. One `phase()` per contiguous run of elements; `agent()` per `serviceTask`; `pipeline()` for the fan-out of a `parallelGateway`/`inclusiveGateway`/multi-instance marker unless the next stage needs all results - together (then `parallel()`); plain `if`/`while` for deterministic conditions and loop caps. + together (then `parallel()`); plain `if`/`while` for deterministic conditions and loop caps. Every + `agent()` gets `label: '