Skip to content

feat(project): add LLM-as-a-judge and code-based evaluator TUI wizards - #2468

Merged
notgitika merged 4 commits into
refactorfrom
feat/project-add-evaluator-tui
Sep 30, 2026
Merged

notgitika merged 4 commits into
refactorfrom
feat/project-add-evaluator-tui

Conversation

@tejaskash

@tejaskash tejaskash commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

What. A bare agentcore add evaluator llm-as-a-judge or agentcore add evaluator code-based on a TTY opens a wizard. add evaluator joins the add menu as a submenu.

  • llm-as-a-judge: name, level, model (Bedrock or OpenResponses, each with its own model ID input), instructions, rating scale preset, review.
  • code-based: name, level, Lambda (scaffold a new one with a timeout step, or use an existing one by ARN), review.
  • Each handler and its wizard share one input builder, so both refuse the same input with the same message.
  • RouterScreen now keeps command names in one column when a description wraps.

Why. These were the evaluator entries in the add wizard design. The wizards ask only what has no default. Description, KMS key, tags and inline rating scales stay flag-only. Instruction placeholders are not checked locally, following #2124, which leaves that check to the service.

The Bedrock judge defaults to global.anthropic.claude-sonnet-4-6. The Evaluator service sends temperature: 0.0, and Claude Opus 4.7 and every later Claude model refuse it. I checked every active Anthropic profile in us-west-2.

llm-as-a-judge review

code-based review

Related Issue

Closes #

Documentation PR

Not applicable. command.md is regenerated for the two shorter descriptions.

Type of Change

  • New feature

Testing

  • bun test (3729 pass, 0 fail), typecheck, lint:check, format:check, build
  • Drove both wizards through the TUI harness in a real project, covering every step, each validation message and back navigation
  • Deployed both wizard-built evaluators to us-west-2. Both came back ACTIVE, then the stack was removed through the CLI

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xl PR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants