Skip to content

Add agent workflow observations, MCP consent and isolated Chrome automation - #18

Merged
trevoedwards merged 11 commits into
mainfrom
pr/agent-workflow-features
Nov 15, 2026
Merged

trevoedwards merged 11 commits into
mainfrom
pr/agent-workflow-features

Conversation

@trevoedwards

Copy link
Copy Markdown
Collaborator

Agents can now inspect installed scripting terminology, request connection-scoped snapshot changes, and label app screenshots with existing accessibility refs. Action results can suggest exposed menu routes, while notification-assisted settling preserves signature checks and avoids adding a full polling interval after the quiet deadline.

The same branch adds structured MCP results, opt-in progress, optional per-connection human consent, and an explicit isolated Chrome backend for the existing web tools. Chrome uses a disposable automation profile with guarded DOM actions, policy enforcement, bounded transport queues, cancellation recovery and cleanup. Existing text/image responses and Safari defaults remain compatible. Diagnostic probes are excluded from real-use reports, and multiline details stay on one report row.

Validation

  • Final tip: 1,037 unit tests passed, including live headless Chrome tests; build, blocking lint and whitespace checks passed.
  • Final tip: full GUI suite passed (118 tests across 18 suites), with three skipped checks requiring runner Accessibility, person-approved Automation grants, or a configured Shortcut.
  • Agent evaluation: the initial run passed 29/31 tasks at 4d6fb02; after Xcode component setup, the two blocked Xcode tasks passed at d86f9e7. All 31 tasks have passing results across these runs, rather than one fresh final-tip run.
  • Final tip: 120 measured actions completed with no unsettled responses. The measurement exposed and verified a fix for notification quiet-time accounting and polling overshoot.
  • Temporary evaluation policy changes were restored to the original approved bytes; the GUI lease was released and measurement diagnostics disabled.

Scope and limits

Chrome requires an installed Google Chrome and supports plain article-text extraction and standard input/textarea editing; rich contenteditable editors, modified key chords and cross-frame actions remain unsupported. Tool definitions and server instructions use a 26,624-byte budget to accommodate the new schemas and options. This PR does not bump the version or publish a release.

GPT-6.1 Sol in cooperation with Trevor.

trevoedwards and others added 11 commits November 13, 2026 11:15
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Suggest exposed menu routes after button clicks, keep screenshot labels tied to verified AX refs, and retain polling and final signature checks when observing UI changes.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
… diffs

Keep action approval and retained screen baselines scoped to the connection, bound modern consent continuations to one call, and preserve existing text/image responses and cancellation behavior.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Use a private headless profile, current web policy, bounded CDP queues and guarded DOM targets. Verify typed values, reject unseen or changed UIDs, handle dialogs without blocking their answers, and close the session on cancellation or pause.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Describe supported routes, limits, connection scope and headless regression tests.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Wait for retired browsers to exit before final profile removal, bound retiring sessions, and prevent an abandoned command from closing the next session. Cover cancellation, timeout recovery and complete profile cleanup.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Poll with a stop flag so teardown is bounded even when shutdown does not wake accept. Rechecked both live Chrome regressions after the fixture-only change.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Avoid transient bridged strings during asynchronous remote registration and per-action observer churn. Include process birth identity when reusing an observer.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Avoid restarting quiet time when a later poll notices a notification, and sample at the quiet deadline rather than overshooting it. Cover delayed observation without relaxing signature verification.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@trevoedwards
trevoedwards merged commit 526fce7 into main Nov 15, 2026
1 check passed
@trevoedwards
trevoedwards deleted the pr/agent-workflow-features branch November 15, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant