feat(project): add LLM-as-a-judge and code-based evaluator TUI wizards - #2468
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What. A bare
agentcore add evaluator llm-as-a-judgeoragentcore add evaluator code-basedon a TTY opens a wizard.add evaluatorjoins the add menu as a submenu.RouterScreennow keeps command names in one column when a description wraps.Why. These were the evaluator entries in the add wizard design. The wizards ask only what has no default. Description, KMS key, tags and inline rating scales stay flag-only. Instruction placeholders are not checked locally, following #2124, which leaves that check to the service.
The Bedrock judge defaults to
global.anthropic.claude-sonnet-4-6. The Evaluator service sendstemperature: 0.0, and Claude Opus 4.7 and every later Claude model refuse it. I checked every active Anthropic profile in us-west-2.Related Issue
Closes #
Documentation PR
Not applicable.
command.mdis regenerated for the two shorter descriptions.Type of Change
Testing
bun test(3729 pass, 0 fail),typecheck,lint:check,format:check,buildACTIVE, then the stack was removed through the CLI