A CLI / MCP so you can ask your agent to modify docs?

A CLI / MCP (first CLI, then MCP) so coding agents like Claude Code can interact with documents.

Happy to provide a PR with a POC.

System Information

BenchUp 0.6.6
Build: Release
Runtime: Browser (WebAssembly)
Browser / OS: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36
Platform: Mac
Window: 3320 × 1984 pixels
UI scale: 2.00 pixels/point
UI zoom: 100%
Theme: Light
Renderer: wgpu
Graphics backend: BrowserWebGpu
GPU: 
GPU type: Other
GPU vendor/device: 0000/0000
Driver: 
Driver details: 
Surface format: Bgra8Unorm

Definitely urgent! Would love to hear what you have in mind and since the app will never be centralized, we need to figure out a way to serve an URL from both the web and desktop versions.

Thanks! Let me check I understand "serve a URL from both the web and desktop versions" correctly: the agent should talk to the app that's running, with the document open (live edits, each one undoable), not just to .bup files on disk. Is that right?

If so, here's what I have in mind:

1. CLI on local files (the POC). A new crates/benchup-cli that depends only on benchup-core, plus gzip for .bup. It doesn't touch benchup-app.

benchup scene model.bup              # JSON: objects, dimensions, material, grain, hierarchy
benchup board model.bup --name "Side" --size 720x400x18 --at 0,0,0
benchup move|rotate|group|material|grain|erase ...
benchup undo model.bup

Each command is recorded in the document History, so it shows up as an undo step when you open the file. Output is JSON so agents can parse it. The operations would live in a shared module, so step 3 can reuse them.

2. benchup bridge: a URL that works for both web and desktop. A browser tab can't listen on a port, so I'd flip the connection. The CLI serves the URL on localhost (MCP/HTTP for the agent), and the app connects out to it over WebSocket:

agent ──MCP/HTTP──▶ benchup bridge (localhost)  ◀──WebSocket── BenchUp (web or desktop)

It's one protocol for both builds, and there's still no central server. The app already has a WebSocket client on native and wasm (realtime.rs).
Security: pairing with a one-time token shown in the app, an Origin check, and bound to 127.0.0.1 only. Chrome will ask for local network access when the hosted web version connects to localhost. Safari may be stricter, which we'd need to check.

3. In the app: an opt-in "Allow agent control" toggle that connects to the bridge. It applies incoming operations to the open document, each as one undo step, and respects the read-only state of shared projects.

For desktop alone, the app could serve localhost directly, but the bridge keeps web and desktop the same.

I'd start with (1) as a PR. Does that direction work for you, or did you have something else in mind for the URL part?

@furilo

Hi @furilo, your plan overlaps closely with something I've already written. I'd rather help with it than build a parallel version.

For #3 I wrote a tool layer for an LLM assistant: crates/benchup-app/src/assistant/tools.rs and scene.rs. It depends only on benchup-core and serde_json, with no egui and no app state, so it's roughly the shared operations module your step 1 describes:

27 operations:
- reading: scene, find, get object, get selection
- creating and changing geometry: create_board, move, rotate, scale, resize, push_pull
- structure and labels: group/ungroup, rename, tag, erase
- finishes: material, grain, paint
- annotations: add_joint, dimensions, text
- context: selection, enter/exit edit context
All lengths are in mm at the boundary and in metres inside the model.
A JSON Schema per tool, so tools/list needs no extra work.
Validation: arguments are parsed into typed structs and checked against limits. Every failure comes back as structured data ({ok:false, error, hint}) rather than a panic.
Respects the model's rules: locks, hidden entities, the current edit context, and the read-only state of shared projects.
About 500 lines of tests on plain Models.

Here's what I'd suggest. Happy to adjust any of it.

I open a small PR that only moves this layer into its own crate (e.g. benchup-tools). No behaviour changes and no UI.
Your CLI, MCP server and bridge build on top of it. I'm glad to take any piece you'd like help with. One example is moving .bup load/save out of project_files.rs into a crate the CLI can use.
Nothing app-side goes in until we've agreed on the shape.

One open question for the MCP side is undo granularity. Should each call be its own undo step, or should there be a begin/end batch so an agent's multi-step change undoes as one?

If you've already started on a shared module, I'll fold mine into yours instead. Just let me know which you'd prefer.

Add benchup-mcp, an MCP server for BenchUp projects

@furilo

An update: I went ahead and implemented this direction in #3, so there's something concrete to look at. It follows the three steps @furilo laid out above.

Shared operations and file mode.
- crates/benchup-tools is the shared layer. It has 27 tools (I wrote 28 earlier; that was a miscount), with schemas, validation and the read-only and lease checks, plus .bup encode/decode. It depends only on benchup-core.
- benchup-mcp file <project.bup> serves it over MCP on stdio. Each change is one history entry, and the file is saved atomically.
The bridge.
- benchup-mcp live listens on 127.0.0.1 only, and the app (web or desktop) connects out over WebSocket.
- Pairing is challenge-response with HMAC, so the token is never sent. There is an Origin allowlist, and calls have deadlines and cancellation.
In the app.
- Settings → Add-ons → Agent connection, off by default.
- Each agent call is one undo step labelled Agent: …, and shared-project read-only and lease rules apply.

The in-app MIRA chat is now a separate optional add-on on the same tool layer. Like the Agent connection, it is a Cargo feature with a run-time switch, so it can be left out entirely.

On the open question: it's one undo step per call for now. A begin/end batch would be an easy addition.

There's a user guide in docs/mcp.md on the branch, and three end-to-end tests that run the real bridge against the real app.

It's all yours to take, reshape, split or drop. @furilo, if you've already started on the CLI, I'm happy for yours to win and to rebase mine onto it. I'll leave the decision with you.

I'd start with (1) as a PR. Does that direction work for you, or did you have something else in mind for the URL part?

Yes, definitely! Sorry I wasn't clear about the URL part. I was actually referring to a bridge or protocol, just like you described.

You also bring up a really good point about document history. We could introduce a rule requiring the agent to synchronize every action with the CLI, while keeping a centralized record of all actions and changes. The CLI could then use that same record to provide context-aware autocompletion and suggestions, ensuring everything stays consistent.

I'm just thinking out loud here, but I think there's a lot of potential in this approach.

If you'd like to go ahead with this, that would be awesome!

@aarvee11

@aarvee11 let's move forward with the CLI first, then figure out how we can bring both sides together. @furilo's plan and proposed order of implementation make sense to me.

@furilo

I thought @furilo wanted to put a PR, while I have already added the CLI and MCP both in the PR which I have raised:
#3

@dannycoulombe

Thanks @dannycoulombe for confirming, and @aarvee11 for moving so fast. Sorry for the confusion. I hadn't started any code yet, and I'd rather build on your work than duplicate it.

I've read benchup-tools in #3, and it's exactly the shared operation layer I described in step 1: schemas, validation, structured errors, lock/lease checks, .bup encode/decode. No need to rewrite it.

My only concern is review size. #3 is ~14k lines and mixes MIRA, the add-on system, the bridge, the app UI, docs and CI. Danny asked for the CLI first, so I'd suggest splitting it into small PRs:

benchup-tools on its own (@aarvee11, since it's your code): the crate, plus project_files.rs using its encode/decode. I'd leave pairing.rs out for now, since it belongs to the bridge.

The CLI (I can take this one, on top of 1): a thin benchup binary for shells and coding agents:

benchup tools                               # list tools + JSON schemas
benchup scene model.bup                     # get_scene
benchup call model.bup create_board '{...}' # any tool; one undo step, atomic save
benchup call model.bup --batch ops.jsonl
benchup mcp model.bup                       # MCP over stdio (your file mode)

That way we'd have one binary instead of a separate benchup-cli and benchup-mcp.

Then the live bridge and the app-side "Agent connection", each in its own PR, reusing what's already in Add an MCP server and optional agent add-ons (Agent connection, Build with MIRA) #3.
MIRA on top of the MCP later, as you suggested.

@aarvee11, does that split work for you? If you'd rather open 2 yourself too, that's fine by me and I'll review/test it instead. Danny, let us know if you'd prefer a different order.

First item built with the test CLI!

I thought @furilo wanted to put a PR, while I have already added the CLI and MCP both in the PR which I have raised:

Oh my bad. I didn't check the PR in details as I thought it was the protocol only.

Definitely CLI, MCP is an old standard not really needed anymore. Current models are very proficient at navigating CLIs.

@aarvee11

Quick update: I'm building the CLI on top of benchup-tools from #3, following the split we discussed above. I'll send two PRs:

benchup-tools: @aarvee11's operation layer from Add an MCP server and optional agent add-ons (Agent connection, Build with MIRA) #3, unchanged and under their authorship. As before, it leaves out pairing.rs, which belongs to the bridge. @aarvee11, would you rather open this one yourself, or shall I open it with your commit?
benchup CLI, built on 1:
- tools, scene, get, call (any tool, or an all-or-nothing --batch as one undo step);
- undo/redo;
- mcp, the same tools over MCP on stdio;
- open, which hands the file to the desktop app.
It also includes fixes to the tool layer that came out of testing.

To find the rough edges, I ran 12 scenarios with coding agents that had only benchup --help to go on. The scenarios ranged from editing the example project to drawers with dovetails, back panels in a rabbet, dimensions and cut lists. Simple boxes work well. Joints, materials, components and drawn geometry were where the agents got stuck. I'm addressing that in the CLI PR, together with a short agent guide (benchup guide). Two findings are in the core rather than the CLI, so I opened #36 (erasing a board leaves its joints behind) and #37 (push_pull can create an inverted face).

The live bridge and the in-app Agent connection stay for later PRs, reusing what's in #3.

@furilo

Please pick up from my commit, once you are done with the cli, let's reconvene and take the MCP path ahead. On your lead please @furilo

@aarvee11

The two PRs are up: #40 (benchup-tools, @aarvee11's commit) and #41 (the benchup CLI).

How the CLI was tested, in case it is useful for the MCP and bridge work: 12 furniture tasks, run four times by coding agents. The agents knew only benchup --help, worked outside the repository (no source code to read) and could not ask questions, so they wrote down what they would have asked. After each round, I fixed the tools and the agent guide (docs/agent-guide.md, printed by benchup guide). Across the 12 tasks, the totals went from ~701 commands and 72 errors to ~245 commands and ~7 errors. The per-task table is in #41.

The tasks:

Lengthen the desk in Example.bup by 200 mm.
A two-bay bookcase: go from 5 to 4 shelves per bay, evenly spaced, keeping the joints.
A 1400×800 dining table, 750 mm high: 70 mm legs, mortise-and-tenon aprons, a 30 mm top.
A 500×450×700 chest of 3 identical drawers, with dovetailed fronts, all one component.
Add a 6 mm back in a rabbet and an 80 mm plinth set back 20 mm.
Remake the bookcase in 19 mm solid oak instead of 18 mm board.
A bedside table with curved side edges (40 mm deep arc) and a chamfered top.
A leaning ladder shelf: uprights at 15°, 4 shelves getting shallower towards the top.
Dimension the bookcase for the workshop and label it.
Organize the bookcase into modules, with one shelf component and tags.
Cut list and number of 2440×1220 sheets to buy.
A 1200 mm shelf in a 473 mm bay. It cannot fit; the agent should stop and ask.

What helped most:

telling agents which board is a in a joint (many joints came out reversed);
references between batch calls, so a change is one undo step;
check_model, run at the end of a --dry-run;
recipes with real numbers, such as the gap between joints in one board, tenon depth and drawer bottoms.

Four core limits came up. Two are in issues: #39 (full-thickness push/pull) and #36 (erased boards keep their joints). The other two are joints between a component instance and outside boards, and boolean precision with curves of many segments. The tools to add next are in #42.

Hi folks!
I am new to the conversation so if I repeat a decision already made, please ignore my recommendation. I create deeply embedded AI solutions and would like to share what I do in my apps.

Create a standardized canonical json model that represents a scene (created by AI for highest performance)
Add a view/edit json panel - this is used by AI to create/edit a scene using the canonical model. The canvas is tightly sync'ed to this json and vis-a-versa.
Add AI settings so users can
a) Enter their AI credentials/keys
b) Integrated with https://models.dev/models/ for the pick list
c) Define the AI pre-prompts and AI post-prompts and tools list
Add the AI chat panel with a link to the AI settings.
Tools handler to handle the LLM response
LLM prompt builder (assembles the pre-prompts, session memory, user's prompt, post-prompt, tools list if not a separate LLM feature)
LLM caller (define tools list if a separate feature, then send assembled prompt)
An LLM session memory global variable (or save to file if desktop version?)

If anyone needs help or wants to bounce ideas off me, please do not hesitate to ask. Or if you want me to code it :)

Scott
AI Enterprise Architect
VS Code using DeepSeek v4.1 Flash
Rust - C# - Blazor - Azure

@furilo

Hey guys!

Sorry it took me a while to get back to you and properly review your work. I've been prioritizing bug reports and strengthening the security of our CI pipeline lately.

First, I want you both to know how much I genuinely appreciate the time and effort you're putting into this important feature, and into shaping the overall vision of the software. All this momentum is new to me, and I'm still learning to adapt to it!

@furilo I merged both of your PRs and didn't find any obvious issues. I haven't gone through every command in detail, partly because I think we can refine things over time, but also because, with more PRs coming in, I believe it's important to get this part of the foundation in place, even if it's not quite complete yet.

@aarvee11 I guess we're now in a good position to adapt your PR to build on top of this, right? @furilo do you think we're ready to move on to the next step?

I'll definitely make some time this weekend to experiment more with the CLI and get a better feel for everything.

Thanks again to both of you for your help and involvement! 🫡🙏

@VR-Architect

On top of @VR-Architect comments, I'd say we probably need a clear documentation of all CLI commands for the AI to feed from.

EDIT: I just found out this file was generated: https://github.com/BenchUp3D/benchup/blob/main/docs/agent-guide.md

Do we have a shared work environment or issue to consolidate and collaborate a detailed AI feature requirements and the inventory of all tools?

BTW, I forgot to mention these additional items I also include in my solutions:

Use AI to create all use cases and test cases leveraging AI controlled playwright to conduct all regression testing via the UI. Then add these test to the project in a dedicated location (by module/feature) so all regression test are automated without access to the code or database. (Forces AI testing agent to use the UI)
Expose the create MCP endpoint to the internet with the standardized json format with optimized AI instructions so future AI models pickup them up and future LLM models will give their users an actual BenchUp working model. I am not sure if the GitHub hosting of the current app can expose MCP endpoints though?
ML.net model that handles user intent and routing. For example, if a user prompts "Enlarge the object by 20%" the ML.net can determine the tool = size, verb = grow, target = object, amount = 20%. The purpose of the ML.net is to handle synonyms and multiple languages. Though a separate model could be made by language if desired but becomes a packaging nightmare. I typically store all these records in a database and a separate program creates the ML.net model for me with each release. A file could also be used which would be better for git storage. The use of the local ML.net model has no LLM use costs as no LLM is hit. I call this the level 0 (L0) which is called when the user clicks the submit button in the AI chat. If there is no item scored in the model output then the flow falls to the L1 LLM call. So we would need the tool list with inputs and outputs of each tool documented.

Best Wishes...Scott

@aarvee11

Thanks Danny! Yes, I think we're ready for the next step. I've put the plan for the live bridge and in-app Agent connection in #55 so we can track it there. @aarvee11, let's coordinate who takes what in that issue.

Navigation Menu

Repository navigation

A CLI / MCP so you can ask your agent to modify docs? #13

System Information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A CLI / MCP so you can ask your agent to modify docs? #13

Description

System Information

Activity

dannycoulombe commented on Nov 9, 2026

furilo commented on Nov 9, 2026

aarvee11 commented on Nov 9, 2026

aarvee11 commented on Nov 10, 2026

dannycoulombe commented on Nov 10, 2026

dannycoulombe commented on Nov 10, 2026

aarvee11 commented on Nov 11, 2026

furilo commented on Nov 11, 2026

furilo commented on Nov 11, 2026

dannycoulombe commented on Nov 11, 2026

Maaxion commented on Nov 12, 2026

furilo commented on Nov 12, 2026

aarvee11 commented on Nov 12, 2026

furilo commented on Nov 12, 2026

VR-Architect commented on Nov 13, 2026

dannycoulombe commented on Nov 14, 2026

dannycoulombe commented on Nov 14, 2026

VR-Architect commented on Nov 14, 2026

furilo commented on Nov 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions