Repository navigation
computer: run several actions in one call - #2
Open
DroBov1Tya wants to merge 1 commit into
Open
DroBov1Tya wants to merge 1 commit into
DroBov1Tya wants to merge 1 commit into
Conversation
A known sequence (click a field, type, press enter, look) cost one model round trip per action. computer_actions runs up to 20 click, type_text, press_keys, scroll and screenshot steps in order in one call. A batch must never get less scrutiny than its steps one by one: - every step is checked before anything is asked or done; - a step whose tool is switched off in the mode, or that a before_tool hook denies (run per step with its own tool and arguments), refuses the whole batch; - one approval lists every step by its batch number, steps a session grant covers marked; mixed kinds get no session grant; the arbiter judges each step alone; - the steps travel structured in ApprovalRequest (protocol 11) and in the request text, so every client and chat platform shows each one; the terminal pages long batches and lets a yes count only after all steps were shown. A chat platform that would cut a long request gets no yes for it: it can only deny there, the app allows it. The single tools now build the same checked step as the batch, so the two cannot ask or act differently. Every click in a batch reads the picture taken before it. press_keys refuses control characters and shows keys by their normal names. Not verified: - a live batch on a real macOS desktop (AXPress clicks, typing, keys, screenshots inside a batch, coordinates after an inner screenshot); - the same on Linux X11 and Wayland, and a Linux build; - the approval drawn in the terminal, desktop and browser clients and in Telegram with a 20-step batch.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
computer_actions: up to 20 click, type_text, press_keys, scroll and screenshot steps in one call, so the model spends fewer round trips and tokens when it already knows where to act.A batch never gets less scrutiny than its steps one by one (two independent security reviews):
Not verified: a live batch on macOS or Linux desktops, a Linux build, how a 20-step approval looks in each client and in Telegram.