Skip to content

Complete AGY preflight discovery and distinguish unsupported verification from account failure #42

Closed

Description

@anthonyandrei

Problem

The protocol-2 AGY adapter returns a model catalog but cannot establish eligibility for any model through its normal discovery path. It invokes agy models, then unconditionally attaches unknown access, entitlement, billing route, and usage observations. The selector correctly rejects those records, leaving the orchestrator to report that no workers ran.

This is an incomplete adapter integration following #40 and PR #41. A model list does not prove account access, but an adapter that always supplies unknown observations cannot satisfy the verification policy it now participates in. The observed rejection is not evidence of expired access or exhausted usage.

Verified reproduction

On 2026-09-06, the installed skill at commit b832e35 returned 12 models from its public AGY catalog operation. Every model had unknown access, entitlement, and usage. Passing that catalog to scripts/select-compatible-worker.ps1 for the researcher role with high effort returned exit code 4. Its rejection reasons included unverified access, missing account identifier, inactive/unestablished entitlement, and unestablished billing route. No worker was launched.

Reproduce through the public interfaces:

  1. Write a protocol-2 catalog request for role researcher, preference balanced, effort high, no required capabilities, and route default.
  2. Run scripts/agy-adapter.ps1 --operation catalog --request REQUEST.json and save its JSON output.
  3. Pass the catalog, repository-root policy, and request to scripts/select-compatible-worker.ps1 with an output path.
  4. Assert that models exist but selection fails because the adapter supplied unknown preflight observations. Use a fake AGY executable returning a fixed model list for the deterministic regression case.

The diagnosis checkout was at older commit 604b7c1, with protocol-1 code. Implement against current upstream containing #41. Do not revert to the old optimistic availability booleans or edit an installed skill copy as the source fix.

Intended outcome

Implement supported, bounded, read-only AGY account discovery behind the existing adapter contract. Populate verified observations only where the provider supplies evidence for the intended account and model. When the installed AGY version cannot supply a required observation, preserve unknown state and explain the missing interface precisely. The orchestrator must distinguish an unsupported verification path from a confirmed account failure.

Preserve the policy established in #40. This issue does not authorize bypassing account verification to make a worker launch.

User stories

  1. As a user with eligible AGY access, I want offload to verify my account and available allowance, so that it can dispatch authorized work when the provider exposes the required evidence.
  2. As a user whose AGY installation cannot report a required account fact, I want a specific explanation of what could not be checked, so that I can understand why work stays with the orchestrator.
  3. As a user with expired access, an incompatible billing route, or exhausted allowance, I want dispatch to stop with the observed reason, so that no unauthorized fallback runs.
  4. As a user explicitly choosing AGY with unknown usage, I want the existing usage exception applied only after the other checks pass, so that my choice does not override a known access or billing problem.
  5. As a maintainer, I want equivalent results in both supported shell families and repeatable regression tests, so that catalog changes cannot silently disable or improperly enable dispatch.

Implementation decisions

  • Keep provider commands, version checks, parsing, and evidence normalization in scripts/agy-adapter.ps1 and .sh. Reuse the protocol-2 preflight schema and existing selection/capacity machinery. No new provider or database is required.
  • Establish which installed AGY interfaces actually support headless, read-only access, entitlement, billing-route, and usage discovery. Record the tested version and interfaces. A documented interactive slash command is not proof of a supported headless query.
  • Use bounded probes and current, account-scoped observations. Carry a nonsecret account reference, model entitlement, billing route, observation source/time, and meaningful usage scopes/units as required by the existing contract. Do not equate model enumeration, CLI presence, or optimistic booleans with verified access or capacity.
  • Handle unsupported, missing, malformed, stale, and timed-out observations explicitly. Preserve each unknown reason through selection and the final handoff diagnostic. Distinguish an unqueried or unsupported fact from provider-confirmed denial or exhaustion.
  • Keep explicit-provider unknown-usage behavior unchanged. It cannot bypass access, entitlement, billing, capability checks, or known exhaustion. Automatic selection still requires fresh, sufficient capacity under the existing estimate and reservation rules.
  • Discovery must not launch a model request, alter accounts, renew subscriptions, purchase credits, consume resets, or activate a paid fallback. Redact provider responses before storing evidence; do not persist credentials or raw private account data.
  • Update adapter documentation with the supported version/interface matrix, unsupported cases, and the precise conditions under which AGY remains ineligible. If no supported interface exists for a mandatory fact, document that limitation as an unresolved dispatch blocker rather than inventing evidence or claiming live eligibility is fixed.

Acceptance and verification

Use the public adapter catalog and selector/launcher seams with fake vendor executables. Extend existing model-routing and worker-adapter suites rather than introducing internal mocking APIs. Run equivalent scenarios in PowerShell 7+ and Bash 3.2+.

  • A fixed model-list-only fixture reproduces the current rejection and yields a specific unsupported/unverified diagnostic, without claiming account denial or exhaustion.
  • Provider fixtures with verified access, active entitlement, an allowed billing route, and fresh sufficient capacity produce an eligible selection. Each normalized fact is traceable to the fixture response.
  • Unauthenticated, expired entitlement, wrong account/billing route, and known exhausted capacity remain ineligible. Cancelling renewal alone does not invalidate continuing entitlement.
  • Missing, malformed, stale, unsupported, or timed-out observations never become verified access or unlimited capacity.
  • Explicit provider selection with unknown usage works only after the other required checks pass and records uncertainty. Automatic selection rejects unknown usage.
  • Model discovery and preflight never invoke the worker launch operation or a billable prompt. Test fixtures record invocations so tests can assert this directly.
  • Both shell families preserve normalized outcomes and redact secrets from catalog, selection, and diagnostics. Existing routing, capacity, and dispatch regressions remain passing.
  • Validate available real read-only probes and report the tested AGY version. Separate live evidence from fixture coverage. If mandatory live observations remain unsupported, report that blocker explicitly; passing fixtures alone does not establish working live dispatch.

Out of scope

  • Relaxing the verified-access policy or adding an implicit trust/force-dispatch switch.
  • Implementing new providers or redesigning model ranking, capacity reservations, retries, or worker isolation.
  • Login automation, account switching, billing changes, quota purchases, or model requests used as probes.
  • Broad skillshare synchronization, installation cleanup, or changes to unrelated local work.
  • Implementing the fix as part of publishing this issue.

Activity

  1. added
    bugSomething isn't working
    ready-for-agentNo outstanding prerequisite issues; scoped and ready for implementation
    on Sep 6, 2026
  2. linked a pull request that will close this issueSelect compatible workers before offload #41on Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready-for-agentNo outstanding prerequisite issues; scoped and ready for implementation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions