Repository navigation
Probeable fixes: lock tests after first failure, sandbox the oracle, probe count badge, seed + docs - #178
Merged
Conversation
…#165) Students saw every test case, hidden ones included, after a failed submission, so the test suite could stand in for probing. Per the Probeable Problems papers (Denny et al. ITiCSE 2025 §4.2; Tang et al. ACE 2026 §3.1.5) students now see passing tests up to the first failure, the first failing test, and only a count of the locked remainder. The rule is one pure helper, truncate_to_first_failure, applied in the two student-facing serialisation points: ProbeableCodeHandler.extract_test_results (live SSE result and the last-submission endpoint) and SubmissionHistoryView (per-variation and top-level rows). Pass/total counts still cover every test so the score and toast are unchanged. Stored TestExecution rows, the instructor and admin submission detail views, and the research export are untouched and still return all tests. probeable_spec goes through the EiPL path and is not changed here. Frontend renders "N more tests locked" via a new optional lockedCount prop on CodeSubmissionFeedback (default 0, so DebugFix feedback is unchanged). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The instructor's reference solution was exec'd inside the gevent web worker with student-chosen arguments and no timeout or memory limit, so one slow or memory-hungry call could stall a whole worker and its connections until gunicorn's 30 s timeout. Refute duplicated the same in-process oracle in _safe_execute. Both now run through oracle_sandbox.run_oracle_sandboxed: a stdlib-only child interpreter launched with -I -S by file path and an empty environment, which applies RLIMIT_CPU and RLIMIT_AS to itself before reading the request, in its own session so a timeout kills the whole group. Inputs over 10 KB are rejected before spawning, output is capped at 1 MB, and tuple/set/frozenset/non-string-key dict results are tagged on the wire so callers get the same Python types as before. Limits live in settings.PROBE_SANDBOX in base.py (settings/security.py is never imported) with matching defaults in the service. Existing student-facing messages are unchanged; timeouts, memory exhaustion and oversized inputs get their own messages and are not charged against the probe budget. The restricted builtins remain a guard against accidents, not a security boundary, as the module docstring says. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
useProbeState read probes_remaining, a key the API never sends (it sends
remaining), so every block- or cooldown-mode problem showed "0 left". The
backend status message, which carries the cooldown "submit N more times"
text, was fetched but never rendered.
The badge now reads remaining, the "{count} left" text is an i18n key in
en and es, and ProbePanel takes a probeStatusText prop that both inputs
pass and that renders only when probes are exhausted, so it adds
information the badge does not already show.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…docs (#167, #175) The sampler probeable problem's description told students which edge cases to probe and its visible "-> int" hinted at integer division, defeating the point of a probeable problem (Kumar et al. ACE 2026, Table 1). The description is now the one-line task and the return annotation is dropped; the hidden tests are unchanged. ADMIN_PANEL_GUIDE.md described probe modes and fields that do not exist (function/expression, cooldown_seconds, -1 for unlimited, specification). It now matches the real model: probe_mode block/cooldown/explore, max_probes, cooldown_attempts, cooldown_refill, show_function_signature, reference_solution, function_signature. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…s, refute infra failures - The feedback summary bar counted passes and failures from the truncated rows, so a submission that failed test 1 and passed the other 9 read "0 passing, 1 failing, 9 locked" while the score said 9/10. The bar now takes passedCount/totalCount from the entry (which count every test) and falls back to the visible rows for callers that do not supply them. - Singular "1 more test locked" in en and es. - oracle_sandbox: RLIMIT_CORE is 0 in the child so a CPU-limit SIGXCPU cannot dump core; a process-wide bounded semaphore caps concurrent children (PROBE_SANDBOX.MAX_CONCURRENT, default 2 per worker) and returns a Busy outcome after QUEUE_TIMEOUT_SECONDS instead of spawning unbounded. - settings.PROBE_SANDBOX is the single source of the limits; the duplicate defaults dict in probe_service is gone. - Refute _safe_execute reports error_type; submit() returns an error instead of persisting a 0-score attempt when the sandbox itself failed (SandboxError, Busy), so a valid counterexample is not graded wrong because of load or a fork failure. - Lazy logging for the hidden-oracle-error line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 15, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the bug-class findings from the probeable-vs-papers review (#164–#175). Design changes (#164, #166) and gaps (#169–#172) are deliberately not in here.
Closes #173, #174, #167, #175. Refs #165 (see below).
What changes
#165 — students see only up to the first failing test (
result_visibility.py,ProbeableCodeHandler.extract_test_results,SubmissionHistoryView,CodeSubmissionFeedback.vue)TestExecutionstorage, instructor/admin submission detail and the research export still return all tests (integration test asserts this).probeable_codeonly.probeable_specgoes through the EiPL handler for its live result and is not covered here, so Probeable: submission feedback reveals every test case incl. hidden ones; papers show only the first failure and lock the rest #165 stays open for that path.#174 — oracle runs in a bounded subprocess (
oracle_sandbox.py,ProbeService._execute_oracle, Refute_safe_execute)-I -Sby file path, empty environment, own session; it appliesRLIMIT_CPUandRLIMIT_ASto itself before reading the request. Wall timeout kills the whole process group. Inputs over 10 KB are rejected before spawning; output capped at 1 MB.settings.PROBE_SANDBOXinbase.py(settings/security.pyis never imported, which is pre-existing).#173 — probe badge showed "0 left" (
useProbeState.ts,ProbePanel.vue)remaining(the key the API sends) instead ofprobes_remaining; "{count} left" is now i18n'd (en/es); the backend exhausted-probe message is rendered via a newprobeStatusTextprop.vue-tscwould have caught the original key error but it crashes locally and CI runs it withcontinue-on-error. Not addressed here.#167 — sampler seed no longer names its own ambiguities;
-> intdropped from the visible signature. Hidden tests unchanged.#175 —
ADMIN_PANEL_GUIDE.mdprobeable section rewritten to the real fields and modes.Tests
test_result_visibility.py,test_probeable_result_visibility.py(history truncation, last-submission path, admin detail untruncated),test_oracle_sandbox.py(native types, TypeError/ValueError text preserved,while Truetimeout, memory bomb, oversized input not spawned, NaN, no__import__),test_probe_service_sandbox.py(timeout mapping in both reveal modes, timeout not recorded),ProbeableCodeFeedback.test.ts, probe-state label/hint cases.TestOracleErrors, refute handler tests and the probe/refute API tests pass unchanged. 324 focused backend tests and the full Vitest suite pass;ruff check/ruff format --check purplex/clean;yarn i18n:checkclean.Known limitations
HIDDEN_ORACLE_ERRORconvention.preexec_fnis not used (the child limits itself), so the gevent thread-safety concern does not apply.🤖 Generated with Claude Code