Skip to content

test-audit: widen too-tight probabilistic margin in rarity prestige test - #819

Merged
stphung merged 2 commits into
mainfrom
claude/gifted-ptolemy-bx92g9
Oct 12, 2026
Merged

stphung merged 2 commits into
mainfrom
claude/gifted-ptolemy-bx92g9

Conversation

@stphung

@stphung stphung commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Summary

Ran the test-audit skill (3 parallel read-only agents across the full tests/*_tests/ scope) per the scheduled routine. One auto-fixable finding, several flagged for human review.

  • Auto-fixed: tests/item_tests/item_pipeline_test.rs::test_roll_rarity_prestige_bonus_shifts_toward_higher_tiers had a Monte Carlo threshold set at only ~1.7 sigma below the expected mean (the test's own comment noted a ~5% false-failure rate). Increased trials 1500 → 8000 and widened the margin to ~3.2 sigma (false-failure rate now <0.1%).
  • Flagged for human review (not auto-fixed, per skill rules):
    • HIGH: src/combat/enemy_generation.rs still calls rand::rng() directly (unlike items::drops, fixed in Fix unseeded RNG in item drop generation (test-audit) #692) — a production-code RNG-injection change, outside this skill's scope.
    • Several redundant/duplicate test pairs (soulforge discovery guard clause across 3 files, Deep pool-refresh 6h boundary across 2 files, 7 achievement-handler test pairs) — removing/consolidating tests needs human judgment.
    • The known ~40-60+ instance brute-force RNG-search loop backlog (tracked since 2026-07-02), plus a missing shared helpers.rs in tests/game_tick_tests/ — both previously flagged, still unresolved.

Full findings detail is in the history log entry added in this PR.

Verification

  • cargo test --test item_tests test_roll_rarity_prestige_bonus_shifts_toward_higher_tiers — passes
  • make check — fmt, clippy, full test suite (2908+3074+... tests), progression check, voyage simulator, and security audit all green
  • cargo test run 10x consecutively — 0 failures across all runs (flakiness verification per the skill's Phase 3)

Test plan

  • Targeted test passes with new margin
  • make check passes
  • 10x consecutive cargo test — 0 failures

🤖 Generated with Claude Code

https://claude.ai/code/session_01XAYN2B3RiSrxmVB6aNi8AL


Generated by Claude Code

test_roll_rarity_prestige_bonus_shifts_toward_higher_tiers set its
threshold at ~1.7 sigma below the expected mean, giving it a ~5% false
failure rate by the test's own comment. Increase trials 1500 -> 8000
and widen the margin to ~3.2 sigma to cut that down to <0.1%.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYN2B3RiSrxmVB6aNi8AL
@stphung
stphung marked this pull request as ready for review October 12, 2026 16:37
@stphung
stphung enabled auto-merge (squash) October 12, 2026 16:37
@stphung
stphung merged commit f7b0935 into main Oct 12, 2026
9 checks passed
@stphung
stphung deleted the claude/gifted-ptolemy-bx92g9 branch October 12, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants