Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions changes/computer-actions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Added
Computer use: `computer_actions` runs up to 20 clicks, typing, keys, scrolls and
screenshots in one call, checked and asked for together before the first runs, with
every click read in the picture taken before the batch. Every client and chat platform
lists each step of the request; tool switches, plugin hooks and the arbiter judge each
step as if it were called alone. `press_keys` now refuses line breaks and control
characters in the keys and shows them by their normal names. A chat platform
gets a request whole or cannot allow it: one too long for its message can only be
denied there and is answered in the app.
39 changes: 39 additions & 0 deletions docs/computer-use.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,45 @@ the session does not allow it.
```

`type_text` and `press_keys` go to whatever has the keyboard focus and ask every time.
Keys are a key name with modifiers joined by `+`; only plain spaces may pad them, and
the request shows them by their normal names, such as `cmd+shift+t`.

## Several actions in one call

`computer_actions` runs up to 20 steps of `click`, `type_text`, `press_keys`, `scroll`
and `screenshot` in order, each with the arguments it takes alone, at most four
screenshots among them:

```json
{"steps": [
{"tool": "click", "arguments": {"x": 320, "y": 180}},
{"tool": "type_text", "arguments": {"text": "hello"}},
{"tool": "press_keys", "arguments": {"keys": "enter"}},
{"tool": "screenshot", "arguments": {}}
]}
```

Every step is checked before anything is asked or done; a wrong one names its number
and nothing runs. A step whose tool is switched off in the mode, or that a plugin's
`before_tool` hook denies, refuses the whole batch the same way.

The batch is approved before any step runs, in one request that lists every step with
its number, those a session grant covers marked "already allowed". So typing and keys
in a batch are approved before the user sees where the focus lands: approve one only
when the steps before it say where that is, or ask for the steps one by one. Steps of
one kind can be allowed for the session as that kind; mixed kinds are asked for this
time only, under a neutral heading. In arbiter mode each step is judged on its own,
and one no refuses the batch. So a batch is never asked for less than its steps one
by one. The terminal grows the request to show up to twelve rows of steps and pages
the rest; a yes there pages on until every step has been shown, and counts only a
moment after the last ones appear. The desktop scrolls a long list inside the request;
a chat platform such as Telegram gets every step in its message, and a request too
long for one message (over 3500 characters) is not offered for a yes there: it can
only be denied there, and is allowed in the app.

Every x, y is in the picture taken before the batch, even after a screenshot inside
it; after the batch the newest picture counts. The batch stops at the first failing
step and says which ran.

## Linux

Expand Down
6 changes: 6 additions & 0 deletions docs/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,12 @@ A hook reads one JSON object on stdin:
- `{"context": "…"}` adds a note the model reads: to the tool's result after a tool, or
to the thread for the other events.

A call that runs other tools inside it, such as `computer_actions`, is checked first
as itself and then once per step, with that step's `tool` and `arguments` and
`"within": {"tool": "computer_actions", "step": 2}`, so a hook for `type_text` sees
typing in a batch too. A deny from any step refuses the whole call before anything is
asked or done. `after_tool` runs once, for the batch, with every step's result in it.

Printing nothing allows the action. Hooks run in the order the plugins are named. A
`before_tool` hook that fails, prints something other than JSON or runs past its
timeout denies the call: a check that could not run has not allowed anything. Other
Expand Down
15 changes: 13 additions & 2 deletions projects/clients/browser/src/chat/approval.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ pub fn Approvals(chat: Chat) -> impl IntoView {
fn ApprovalCard(request: ApprovalRequest) -> impl IntoView {
let app = state();
let id = request.id;
let lines: Vec<String> = request.steps.iter().map(|step| step.line()).collect();
let session = request.session.is_some();
let answer = move |decision: Decision| {
run(api::approve(id, decision), move |_| {
// The core closes the request itself; every client sees that event.
Expand All @@ -31,10 +33,19 @@ fn ApprovalCard(request: ApprovalRequest) -> impl IntoView {
};
view! {
<div class="approval">
<div class="approval-text">{request.request.describe("The agent")}</div>
<div class="approval-text">{request.title("The agent")}</div>
// A batch lists every step, numbered: the yes covers all of them.
{(!lines.is_empty()).then(|| view! {
<div class="approval-steps">
{lines.into_iter().map(|line| view! { <div>{line}</div> }).collect_view()}
</div>
})}
<div class="approval-actions">
<button class="primary" on:click=move |_| answer(Decision::Once)>"Allow once"</button>
<button class="ghost" on:click=move |_| answer(Decision::Session)>"Allow for this chat"</button>
// Only what the core can keep for the session is offered, as the other clients do.
{session.then(|| view! {
<button class="ghost" on:click=move |_| answer(Decision::Session)>"Allow for this chat"</button>
})}
<button class="ghost danger" on:click=move |_| answer(Decision::Deny)>"Deny"</button>
</div>
</div>
Expand Down
1 change: 1 addition & 0 deletions projects/clients/browser/src/chat/data/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ fn an_approval_shows_until_it_is_closed() {
command: "rm x".into(),
},
session: None,
steps: Vec::new(),
};
chat.apply(ThreadEvent::Approval(request));
assert_eq!(chat.approvals.len(), 1);
Expand Down
1 change: 1 addition & 0 deletions projects/clients/browser/style.css
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ input:is([type=radio], [type=checkbox]) { accent-color: var(--accent); }
.card { margin: 10px auto; } .card img { max-width: 100%; border-radius: 10px; display: block; }
.approvals, .question { max-width: 820px; margin: 0 auto; width: calc(100% - 40px); }
.approval, .question { border: 1px solid var(--accent); border-radius: 12px; padding: 10px 14px; margin-bottom: 8px; background: var(--surface); }
.approval-steps { margin: 6px 0 0 12px; display: flex; flex-direction: column; gap: 2px; overflow-wrap: anywhere; }
.approval-actions, .row-actions { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; margin: 8px 0; }
.option {
display: grid; grid-template-columns: var(--s4) minmax(0, 1fr); align-items: start; gap: var(--s3);
Expand Down
1 change: 1 addition & 0 deletions projects/clients/desktop/src/dev/screenshot.rs
Original file line number Diff line number Diff line change
Expand Up @@ -730,6 +730,7 @@ fn act(
command: "cargo test --workspace".into(),
},
session: Some("cargo".into()),
steps: Vec::new(),
}),
};
if let Some(chat) = ws.shown.clone().and_then(|id| ws.chats.get_mut(&id)) {
Expand Down
20 changes: 20 additions & 0 deletions projects/clients/desktop/src/pages/chat/approval.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,27 @@ pub fn render(ws: &Workspace, cx: &mut Context<Workspace>) -> Div {
.on_click(cx.listener(move |ws, _, _, cx| ws.approve(id, Decision::Deny, cx))),
);
// A command is shown as code, in full: it came from the model and is about to run.
// A batch shows every step, one row each: the yes covers all of them.
let what = match &request.request {
// A long batch scrolls inside the card, so the buttons stay in view.
_ if !request.steps.is_empty() => {
let steps = request.steps.iter().fold(
div()
.id(SharedString::from(format!("steps-{id}")))
.max_h(px(240.))
.overflow_y_scroll()
.flex()
.flex_col()
.gap_1(),
|list, step| list.child(div().pl_3().child(step.line())),
);
div()
.flex()
.flex_col()
.gap_1()
.child(request.title("The agent"))
.child(steps)
}
Request::Command { command } => {
let mono = crate::ui::theme::prefs(cx)
.mono_font
Expand Down
2 changes: 2 additions & 0 deletions projects/clients/terminal/src/app/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ pub use panels::{
pub use panels::{Choice, Confirm, Step, Workshop, choices};
pub use state::{App, compact_number};
pub use tabs::{Tab, TabHit, TabMove};
#[cfg(test)]
pub(crate) use thread::approval::tests::batch as approval_batch;
pub use thread::approval::{ApprovalHit, Asked};
pub use thread::goal::{GoalButton, GoalHit, buttons as goal_buttons};
pub use thread::poll::{Poll, PollHit, PollTarget};
Expand Down
91 changes: 89 additions & 2 deletions projects/clients/terminal/src/app/thread/approval.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
//! everything else, the oldest first. Enter answers with the row under the cursor, Esc
//! denies; the core closes a request whose call is gone.

use std::time::{Duration, Instant};

use crossterm::event::{KeyCode, KeyEvent};
use dusk_protocol::tools::{ApprovalRequest, Decision, Request};

Expand All @@ -12,8 +14,19 @@ pub struct Asked {
/// The agent that asks; `None` for the chat itself.
pub agent: Option<String>,
pub cursor: usize,
/// The first row of a batch's steps in view, when they do not all fit.
pub step_scroll: usize,
/// Step rows in view on the last drawn frame.
pub step_page: usize,
/// When every step of a batch had first been on screen. Until then a yes pages on
/// instead of answering, so no step is allowed unseen.
pub steps_seen_at: Option<Instant>,
}

/// After the last steps of a batch first show, a yes waits this long, so a held or
/// repeated Enter that paged through does not also allow what it just showed.
pub const SETTLE: Duration = Duration::from_millis(600);

impl Asked {
/// The answers offered, in order; the session one only when the core can keep it.
pub fn choices(&self) -> Vec<(Decision, String)> {
Expand All @@ -34,10 +47,20 @@ impl Asked {
choices
}

/// A yes to a batch waits until every step has been on screen for [`SETTLE`]; a no
/// and a single request never wait.
pub fn answers_now(&self, decision: Decision, now: Instant) -> bool {
decision == Decision::Deny
|| self.request.steps.is_empty()
|| self
.steps_seen_at
.is_some_and(|at| now.saturating_duration_since(at) >= SETTLE)
}

/// What the tool asks for, in a line.
pub fn title(&self) -> String {
let who = self.agent.as_deref().unwrap_or("the agent");
self.request.request.describe(who)
self.request.title(who)
}
}

Expand All @@ -55,6 +78,9 @@ impl App {
request,
agent,
cursor: 0,
step_scroll: 0,
step_page: 0,
steps_seen_at: None,
});
}

Expand All @@ -70,6 +96,10 @@ impl App {
match key.code {
KeyCode::Up => asked.cursor = asked.cursor.saturating_sub(1),
KeyCode::Down => asked.cursor = (asked.cursor + 1).min(last),
KeyCode::PageDown => asked.step_scroll += asked.step_page.max(1),
KeyCode::PageUp => {
asked.step_scroll = asked.step_scroll.saturating_sub(asked.step_page.max(1))
}
KeyCode::Enter => return self.answer_approval(None),
KeyCode::Esc => return self.answer_approval(Some(last)),
_ => {}
Expand All @@ -82,12 +112,69 @@ impl App {
if self.approvals.is_empty() {
return Action::None;
}
let asked = self.approvals.remove(0);
let asked = &mut self.approvals[0];
let choices = asked.choices();
let at = choice.unwrap_or(asked.cursor).min(choices.len() - 1);
if !asked.answers_now(choices[at].0, Instant::now()) {
// Still unseen steps: show them. Just shown: wait, as if not pressed.
if asked.steps_seen_at.is_none() {
asked.step_scroll += asked.step_page.max(1);
}
return Action::None;
}
let asked = self.approvals.remove(0);
Action::Approve {
id: asked.request.id,
decision: choices[at].0,
}
}
}

#[cfg(test)]
pub(crate) mod tests {
use super::*;
use dusk_protocol::tools::ApprovalStep;

/// A batch of `count` keyboard steps, each typing `text`.
pub(crate) fn batch(count: usize, text: &str) -> Asked {
let steps: Vec<ApprovalStep> = (1..=count)
.map(|number| ApprovalStep {
number,
request: Request::Keyboard {
action: format!("type \"{text}\""),
},
allowed: false,
})
.collect();
Asked {
request: ApprovalRequest {
id: 1,
request: Request::Keyboard {
action: format!("{count} computer actions"),
},
session: None,
steps,
},
agent: None,
cursor: 0,
step_scroll: 0,
step_page: 0,
steps_seen_at: None,
}
}

#[test]
fn a_batch_is_allowed_only_a_while_after_every_step_was_shown() {
let now = Instant::now();
let mut asked = batch(20, "hi");
assert!(!asked.answers_now(Decision::Once, now));
assert!(!asked.answers_now(Decision::Session, now));
assert!(asked.answers_now(Decision::Deny, now));
asked.steps_seen_at = Some(now);
// A held Enter that showed the last page does not also allow it.
assert!(!asked.answers_now(Decision::Once, now + SETTLE / 2));
assert!(asked.answers_now(Decision::Deny, now));
assert!(asked.answers_now(Decision::Once, now + SETTLE));
assert!(batch(0, "").answers_now(Decision::Once, now));
}
}
Loading
Loading