Repository navigation
E2E coverage for every student probeable workflow, plus the six defects it exposed - #181
Merged
Merged
Conversation
…r helper, graded-wait helper, PR trigger Seed a new problem set e2e-probeable (new slugs only; get_or_create ignores changed defaults on seeded databases) with seven problems built for the student workflows: block mode with a budget of 3, two cooldown variants, a four-test clamp problem whose wrong solutions produce each feedback shape, visible- and hidden-signature oracles that divide by zero, and a spec problem with two tests. Soft deadline 30 days out. Probe budgets and history live in Redis per user with no reset, so e2e/helpers/users.ts creates a fresh enrolled student per test (mock Firebase accepts any email) and grants AI consent where a submission needs it. e2e/helpers/submission.ts waits on the real graded result (eager Celery grades inside the request) and surfaces a 400 as the toast text. selectors.ts gains probe and feedback groups. The e2e workflow only ran on pushes to master; it now also runs on pull requests with the same path filter. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Four new specs, one fresh student per test, run green twice in a row with two workers against a shared stack: - probing: first/repeat/duplicate probes, invalid types, history modal, reload, explore mode, hidden-signature sanitising, visible oracle error not charged - budget: block mode countdown to exhaustion and the per-user budget, cooldown exhaustion, refill after a submission without a reload, and the plural refill hint counting down - feedback: too short, syntax error with no attempt stored, grading in progress, all pass, early failure with later passes locked, singular lock, nothing locked when only the last fails, runtime error row, resubmit, switching attempts, reload restoring the truncated result, debugger modal - spec: probing, explanation graded by the mock LLM with untruncated rows, length bounds, two attempts submission-probeable.spec.ts loses its runtime skips, swallowed waits and accept-any-status checks, and its header no longer claims a Celery worker and Docker-in-Docker. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Pressing Enter on an already-probed input sent a new request and, in block or cooldown mode, spent a probe for a result the student already had. The Enter guard and executeProbe now honour the duplicate state. - After the probe that exhausted the budget the inputs stayed on a duplicate, so the button showed "Already probed" and the "0 left" badge, exhausted tooltip and hint never appeared. The duplicate branch now applies only while probes remain. - A cooldown refill earned by a submission was invisible until a reload: probe status was fetched only on mount. useProbeState re-fetches it when a submission finishes. - Switching to an earlier attempt rendered clamp([-3]) instead of clamp(-3): the call text formatter treated the positional-args list as one value. Extracted to functionCall.ts, following the backend's _format_function_call rules, with tests. - A 400 from /api/submit/ (syntax error) showed a generic "Error" toast because the submit catch branched on error.response, which the service layer never rethrows. submitError.ts reads status and message from either shape; the intended "Invalid Request" warning now shows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… from the student payload The container-pool and factory locks were gevent.lock.RLock. In native threads that each own a gevent hub (Django runserver with eager Celery, which is how e2e grades) a contended gevent lock never wakes the waiter, so concurrent submissions hung until the test timed out. threading.RLock is the cooperative lock under gunicorn and Celery gevent workers, where threading is monkey-patched, and a real lock elsewhere. Regression test contends the pool from four hub-owning threads. The problem-set payload sent every probeable problem's reference_solution (the hidden oracle) and the full function_signature even when the instructor hid it. The student service now nulls the signature when hidden and sends reference_solution only when the UI shows it: as the code to explain (show_reference_code) or for segment analysis. EiPL and spec-with-segmentation keep it; probeable_code, hidden-signature problems and spec-without-segmentation no longer get it. Partial fix for #159; GET /api/problems/<slug>/ still exposes both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #178 (merged as 95876b8); the branch is rebased onto master. Supersedes #180, which GitHub closed when the stacked base branch was deleted. Goal: every student-facing way of interacting with a probeable problem has a Playwright test that proves it, and the app passes them.
Coverage
39 tests across five specs under
e2e/student/, each test as a fresh enrolled student (probe budgets live in Redis with no reset), green twice in a row with--workers=2 --retries=0locally. Workflow numbering follows the enumerated list:submission-probeable.spec.tsrewritten without runtime skips, swallowed waits or accept-any-status checks.Supporting: seeded problem set
e2e-probeable(seven new slugs built for these shapes),e2e/helpers/users.ts(fresh student, enroll, consent, API as that user),e2e/helpers/submission.ts(waits on the real graded result; eager Celery grades inside the request), selector groups, and apull_requesttrigger one2e.yml, which previously ran only on pushes to master. This PR is the first time the e2e suite runs on a PR; expect the full suite, not just the new specs.App defects the tests exposed and this PR fixes
ProbePanel.vue,useProbeState.ts).clamp([-3])instead ofclamp(-3); formatter extracted tofunctionCall.tsfollowing the backend rules.error.response, which the service layer never rethrows;submitError.tshandles both shapes.gevent.lock.RLock, which never wakes a waiter in a hub-owning native thread (how e2e andrunservergrade). Swapped tothreading.RLock, which is the cooperative lock under gunicorn/Celery gevent workers anyway. Regression test contends the pool from four threads. This is the one change on a production path; please read it.Also, while verifying hidden signatures: the problem-set payload shipped
reference_solutionand the full signature for every probeable problem. The student service now withholds both unless the UI renders them (EiPL code-to-explain, spec segment analysis). Partial fix for #159 with integration tests per case;GET /api/problems/<slug>/still leaks and is noted on the issue.Found, not fixed here
settings/security.pyis never imported, soDOCKER_POOL_ENABLED=falsein the e2e workflow is a no-op and the pool is always on.probeable_specis not consent-gated; the spec tests grant consent up front so they stay valid once it is.str(e)from the server interpreter and differs between Python 3.11 (CI) and 3.14 (local), so A11 asserts a substring.Verification
pytest -m unit: 1315 passed. Probe integration tests: 63 passed. Vitest: 552 passed. ruff and eslint clean,i18n:checkclean.🤖 Generated with Claude Code