Probeable: submission feedback reveals every test case incl. hidden ones; papers show only the first failure and lock the rest #165

What the papers say

Denny et al. ITiCSE 2025 §4.2: "we show students the first such failing test case". Kumar et al. ACE 2026 §3.2: "a student's incorrect code submission will fail some first test case in sequence T ... Since we reveal this failing test case". Tang et al. ACE 2026 §3.1.5:

"the tool enables the viewing of all test cases passed until the first failure. Subsequent test cases are intentionally locked to discourage students from discovering remaining ambiguities from viewing test cases."

What Purplex does

• tasks/pipeline.py:1847 loads test cases with include_hidden=True and runs all of them.
• handlers/probeable_code/handler.py:133-174 extract_test_results returns inputs, expected and actual output for every test, with no is_hidden filtering.
• purplex/client/src/components/activities/feedback/CodeSubmissionFeedback.vue:164-227 renders every failing test expanded with "Expected:" / "Got:", and passing tests with call → expected.

So after one wrong submission a student sees every edge case the instructor hid, including inputs and expected outputs. The is_hidden flag has no effect on what the student sees.

Why it matters

The test suite becomes a free, complete oracle. Probing is unnecessary, and the "hidden" tests in seeds such as sampler-probeable-code-divide ([7,2]→3, [-7,2]→-4, [5,0]→0) are the exact ambiguities the problem is meant to make students discover.

Suggested direction

For probeable_code and probeable_spec results shown to students:

• Show passing tests up to the first failure, then the first failing test, then "N further tests locked".
• Never show inputs/expected for is_hidden=True tests; show only pass/fail counts.
• Keep full results in the instructor submission-detail view.

Needs a maintainer decision on whether this should be per-problem configurable.

🤖 Generated with Claude Code