Repository navigation
Phase 8 seed evals - #7
Merged
Merged
Conversation
… health - Triage suggestion card: per-field reasons, edit before apply, priority preview, reject, retry on failure; nothing changes until staff apply - Client mood card: sentiment, escalation risk, frustration trend with quoted evidence; internal only and never affects priority - Similar resolved issues with an on-demand suggested fix that cites its source tickets, plus helpful/not helpful feedback - Admin thread summary with key points citing comments or the description - High escalation risk card on the admin dashboard and engineer home - Client health page: weekly frustration trend per company (30/90/180 days) - Show category and team on the issue meta panel for staff - All AI panels are staff-only, hidden when AI_ENABLED is off, and degrade to a short message if the AI service is unavailable - Move the ITIL matrix to lib/priority and turn useIssuePermissions into the plain function getIssuePermissions in lib/ - Document the AI panels, hooks and design decisions in the README
- evals/: 70 labelled triage issues, 40 sentiment comments, a retrieval set across 3 companies and 24 prompt-injection attacks, with a runner that scores against baselines (client choice, product default, majority class), Cohen's kappa, recall@5/precision and injection outcomes; resumable, rescorable, and checked in CI without API calls - First real run (gemini-3.5-flash-lite): team 91%, priority 79% vs 63% for the client's own choice, sentiment kappa 0.79, recall@5 100%, 24/24 injection attacks blocked - Seed: 4 months of history (resolved issues with fixes, changing client tone, a long thread) appended so existing ticket numbers are unchanged - npm run ai:backfill queues triage and sentiment for seeded or restored data, spaced under the provider rate limit, never twice - Client health dates readings by when the client wrote them, not when they were assessed - Typecheck the seed in CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a labelled evaluation suite for the AI features, providing a comprehensive framework for measuring model quality against baselines and ensuring reproducibility. It updates documentation to reflect the addition, describes how to run and interpret the evaluation suite, and adds CI checks for data validity.
Evaluation suite integration and documentation:
evals/directory with a detailedREADME.mdexplaining what is measured, how to run the evaluation suite, latest results, and limitations. The suite covers triage, sentiment, retrieval, and prompt injection, with reproducible metrics and baselines.ai-service/README.mdto reference the evaluation suite, describe its location, and clarify that the suite measures model behaviour and accuracy on real tasks. [1] [2]evals/code.test_evals.py, which checks that evaluation data is well-formed and that metrics and offline runs work as expected.evals/__init__.pydescribing its purpose.