A destructive-action budget for AI agents' MCP tool calls. toolguard sits in the AI gateway, between a model proposing a tool call and the gateway dispatching it. It classifies every call by its blast radius, enforces a budget of irreversible actions across several scopes at once, blocks when any budget runs out, and records its decisions in a tamper-evident audit log. Getting going again needs a deliberate reset by two people.
Five irreversible actions per hour, then stop until a human releases you.
make install && make test && make demoNo database, no Docker, no credentials. The demo walks through the whole control in about a second.
An agent gets stuck in a loop after a prompt change. Nothing is technically failing, so nobody notices for days. If the loop only burns tokens, that's a bill. If the same agent can publish wiki pages, close tickets or send email, it's a thousand actions that cannot be taken back. Money can be refunded; a published page or a sent email cannot.
Gateways already ship static MCP permissions: allow and deny lists by key, team and tool. A static list can say this agent may publish pages. It cannot say five per hour, then stop until a person looks. That needs impact classification, a stateful budget over a time window, and an explicit recovery step, which is what toolguard adds.
From make demo: an agent publishing wiki pages in a loop, with a per-run limit of 3.
2. Classification catches what 'destructive' usually misses
publish_page deletes nothing. It is externally visible and cannot
be recalled, so it is budgeted as destructive. Run limit is 3.
ALLOW wiki/publish_page allowed_within_budget
actor=3/10 agent=3/40 agent_run=3/3 team=3/100
3. The loop is stopped
DENY wiki/publish_page budget_exhausted
actor=4/10 agent=4/40 agent_run=4/3 team=4/100
5. Unregistered tools are refused, not assumed harmless
DENY wiki/bulk_purge unknown_tool
7. Recovery needs two people and a reason
REFUSED the agent tries to unblock itself
REFUSED one operator, self-approved
RESET by alice, approved by bob (19ec50ff)
9. The evidence
hash chain valid True
raw arguments present False (hashes only)
tool call
├─ not in the registry? ─────────────► DENY (unknown blast radius)
├─ server not allowed for this agent? ► DENY
├─ effective impact = read ──────────► ALLOW (no budget used)
├─ effective impact = write ─────────► ALLOW (reversible and internal; audited)
└─ effective impact = destructive
├─ needs approval, none given ──────► REQUIRE_APPROVAL
├─ approval invalid, expired, reused ► DENY
├─ any budget scope over its limit ─► DENY (names the scope)
└─ otherwise ───────────────────────► ALLOW (slot reserved)
After dispatch, settle() records what happened. An unknown outcome, such as a timeout, counts as
executed, because the side effect may have landed. Where the tool records the idempotency key
toolguard sends, the slot can be released later on positive evidence that the action never happened.
The full request path, storage, concurrency and failure design is in docs/architecture.md.
The reasoning for each is in docs/decisions/. The four that matter most:
Enforce at execution, not at proposal. A model emitting a delete_page call has done nothing;
dispatching it has effect. Charging at proposal would spend budget on options the model discards, and
checking after dispatch is too late. So toolguard binds to LiteLLM's pre_mcp_call hook.
(ADR 0001)
"Destructive" is the wrong axis on its own. Publishing a page or sending an external email deletes nothing and still cannot be undone. The registry records reversibility and external visibility separately, and anything irreversible or externally visible is budgeted as destructive, however it was declared. (ADR 0004)
Several budget scopes at once; the most restrictive wins. One number cannot bound both "one run is looping" and "one person is hammering a shared agent". Every applicable scope is checked, and a denial names the one that ran out, which tells a responder what to reset. (ADR 0002)
| Scope | Bounds |
|---|---|
agent_run |
a runaway loop inside one execution |
actor |
one person's blast radius across runs |
agent |
a bad deployment, across all its runs |
team |
a last-resort ceiling |
tool |
one especially sharp tool |
A reset moves a watermark; it never deletes evidence. Prior actions stop counting, but the rows stay, so an investigator can still see what the agent did before it was unblocked. (ADR 0006)
guardrails:
- guardrail_name: mcp-action-guard
litellm_params:
guardrail: toolguard_guardrail.MCPActionGuard
mode: "pre_mcp_call"
default_on: trueLiteLLM loads toolguard_guardrail.py from the directory holding its config, so
examples/toolguard_guardrail.py ships beside
examples/litellm.yaml. The guardrail reads its configuration from the
environment: TOOLGUARD_REGISTRY and TOOLGUARD_POLICY are required, and
TOOLGUARD_DATABASE_URL should point at a shared Postgres. Without it, budgets fall back to SQLite
per gateway task, and a warning says so.
deploy/ has the gateway image with toolguard installed and a compose file for local runs.
from toolguard import Outcome, ToolGuard, load
from toolguard.store.postgres import PostgresControlStore
registry, policy = load(Path("tool_registry.yaml"), Path("policy.yaml"))
guard = ToolGuard(registry=registry, policy=policy, store=PostgresControlStore(dsn))
judgement = guard.decide(principal, call)
if judgement.allowed:
try:
dispatch(call)
guard.settle(judgement, Outcome.EXECUTED)
except TimeoutError:
guard.settle(judgement, Outcome.UNKNOWN) # still charged, conservativelyTwo versioned YAML files, see examples/:
tool_registry.yaml— every tool, withimpact,reversibleandexternal_side_effect.policy.yaml— per-agent budgets, server allowlists, fail behaviour and reset rules.
Both versions are written into every judgement, so a six-month-old decision can be explained by the
rules in force at the time. Two settings are deliberately impossible to express, because each would
silently defeat the control: unknown_tool: allow, and failing open on destructive calls.
| Dependency state | read | write | destructive |
|---|---|---|---|
| Store unavailable | allow | allow | deny |
| Audit write fails | allow | allow | deny, and release the reservation |
| Metrics backend down | allow | allow | allow |
Under contention, exactly limit calls get through: on Postgres, 500 attempts from 64 threads
against a limit of 25 admitted exactly 25. An uncontended decision takes about 2 ms
(benchmarks).
What it does not promise:
- At-most-once authorisation, not at-most-once execution. A retried call is charged once, but whether the MCP server already acted is up to the server honouring the idempotency key toolguard passes along. This is the most important limit to understand.
- Not a substitute for network controls. A gateway bug that bypasses hooks would bypass this too (BerriAI/litellm#25011), so restrict egress to MCP servers, pin the gateway version, and run the enforcement check on every upgrade.
- Not a prompt-injection detector. It assumes the model will be manipulated eventually, and bounds what that can do.
- Not a rate limiter. Exact windows with a row per action suit tens of actions per hour, not thousands per second.
Not built yet: a UI for granting approvals, a worker that reconciles timed-out calls, archiving sealed audit months to object storage, and a run identifier the platform can vouch for rather than take from the request. The threat model covers what each gap allows.
Python 3.11 or newer.
make check # lint, types and tests, as CI runs them
make postgres-up # local Postgres with pg_partman, migrated
make chaos # kill Postgres under load; assert nothing destructive is allowed
make integration # LiteLLM loads the guardrail inside the real gateway image
make help # everything elseCI also runs the Postgres suites, the load and chaos tests, the Terraform tests, and the LiteLLM integration tests inside the built gateway image. What each suite defends is in docs/testing.md.
src/toolguard/ the control: engine, registry, policy, audit, metrics,
Postgres and SQLite stores, LiteLLM adapter, demo
tests/ 222 tests with no infrastructure, plus Postgres and
LiteLLM suites that run where those exist
examples/ tool registry, policy and gateway configuration
docs/ architecture, 10 decision records, threat model,
failure modes, runbooks, test strategy
migrations/ schema, least-privilege grants, append-only triggers
deploy/ gateway image, compose file, Postgres with pg_partman
benchmarks/ load and chaos tests, with recorded results
infra/terraform/ runtime IAM and monitors, with tests
scripts/ partition setup and maintenance