Repository navigation
Probeable spec: one generated version and the same test ladder as probeable code - #190
Merged
Merged
Conversation
…beable code Probe & Explain graded by generating five LLM code versions and showing the EiPL panel (versions bar, Abstraction card, View Analysis). The two probeable types now share one feedback: the ordered test ladder. Backend - variation_count is a handler property (5 by default, 1 for ProbeableSpecHandler); the generation prompt and the mock LLM honour it. The count-5 prompt is byte-identical to before, so EiPL and Prompt are unchanged. - Scoring is a handler policy (score_generated_code): EiPL keeps the version pass rate, spec uses the tests-passed ratio like probeable_code. - ProbeableSpecHandler.serialize_result emits the probeable_code shape (student_code, explanation, one truncated test_results entry with lockedCount, segmentation). Extraction and first-failure truncation live in one shared helper, handlers/ladder.py, used by both probeable types. - Submission history locks after the first failure for both probeable types and fills comprehension_level for spec. Frontend - One ProbeableFeedback.vue registered for both types, wrapping CodeSubmissionFeedback in ladder layout. Spec shows the explanation in the top panel instead of code, hides the debug button, and renders the Abstraction card under the ladder when segmentation is enabled. - ExplanationQuote.vue and AbstractionCard.vue are extracted from Feedback.vue and reused there, no duplicated markup. - ProblemSet.vue routes both probeable types through the probeable result branch; the old spec wrappers and their test are deleted. Tests: backend tests written first (19 failing, then passing); frontend unit suite 622; E2E spec tests rewritten to the ladder shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CoffeePoweredComputers
added
the
needs-senior-review
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Probe & Explain now grades by generating one LLM code version and shows the same ordered test ladder as Probe & Code. EiPL and Prompt are untouched: they keep five versions, the version pass rate, and their existing panel.
Backend
variation_countis a handler property (5 default, 1 forProbeableSpecHandler). The generation prompt and the dev mock LLM honour it. The count-5 prompt is byte-identical to before.score_generated_code. EiPL keeps the version pass rate; spec uses the tests-passed ratio likeprobeable_code. Note: the savedsubmission.scorewas already a tests-passed ratio for both; only the published progress message changes for spec.ProbeableSpecHandler.serialize_resultemits theprobeable_codeshape:student_code,explanation, one truncatedtest_resultsentry withlockedCount, andsegmentation. Extraction plus first-failure truncation now live in one shared helper,handlers/ladder.py, used by both probeable types.comprehension_levelfor spec. The "10/10 attempts for a 2-test problem" symptom is gone.Frontend
ProbeableFeedback.vueregistered for both types, wrappingCodeSubmissionFeedbackin ladder layout. Spec shows the explanation in the top panel instead of code, hides the debug button, and renders the Abstraction card under the ladder when segmentation is enabled.ExplanationQuote.vueandAbstractionCard.vueextracted fromFeedback.vueand reused there.ProblemSet.vueroutes both probeable types through the probeable result branch. The two old wrapper components are deleted.Verification
Noted, not changed
get_problem_configfor spec still sendsfeedback.show_variations: true; nothing reads it.🤖 Generated with Claude Code