Where: faircode/significance.py's significance_report - "significant": p_value < 0.05 (hardcoded), while confidence (default 0.95) only widens/narrows the returned ci_low/ci_high bounds.
The gap: a caller passing a stricter confidence (e.g. 0.99) expecting the significance verdict to tighten accordingly still gets exactly the same p < 0.05 threshold - confidence has zero effect on significant.
Repro (fully deterministic, no ML model fitting involved):
$ python3 -c "
import numpy as np
from faircode.significance import significance_report
rng = np.random.default_rng(0)
a = b = None
for _ in range(5):
a = rng.binomial(1, 0.55, size=60).astype(float)
b = rng.binomial(1, 0.35, size=60).astype(float)
r95 = significance_report(a, b, n_resamples=3000, n_permutations=3000, confidence=0.95, random_state=4)
r99 = significance_report(a, b, n_resamples=3000, n_permutations=3000, confidence=0.99, random_state=4)
print('p95:', r95['p_value'], 'sig95:', r95['significant'])
print('p99:', r99['p_value'], 'sig99:', r99['significant'])
"
p95: 0.03833333333333333 sig95: True
p99: 0.03833333333333333 sig99: True
p_value is identical (0.0383) in both calls, and significant is True in both - even though 0.0383 > 0.01, the threshold a 99%-confidence caller would reasonably expect.
Why it matters: significance_report is the public function every audit's unfair.py/fair.py uses directly. Anyone tightening confidence to get a more conservative significance verdict gets no actual change in behavior, silently.
Suggested fix: derive the significance threshold from confidence (e.g. p_value < (1 - confidence)), or explicitly document that significant is fixed at α=0.05 independent of the confidence parameter, so the two aren't implied to move together when they don't.
Where:
faircode/significance.py'ssignificance_report-"significant": p_value < 0.05(hardcoded), whileconfidence(default 0.95) only widens/narrows the returnedci_low/ci_highbounds.The gap: a caller passing a stricter
confidence(e.g.0.99) expecting the significance verdict to tighten accordingly still gets exactly the samep < 0.05threshold -confidencehas zero effect onsignificant.Repro (fully deterministic, no ML model fitting involved):
p_valueis identical (0.0383) in both calls, andsignificantisTruein both - even though0.0383 > 0.01, the threshold a 99%-confidence caller would reasonably expect.Why it matters:
significance_reportis the public function every audit'sunfair.py/fair.pyuses directly. Anyone tighteningconfidenceto get a more conservative significance verdict gets no actual change in behavior, silently.Suggested fix: derive the significance threshold from
confidence(e.g.p_value < (1 - confidence)), or explicitly document thatsignificantis fixed at α=0.05 independent of theconfidenceparameter, so the two aren't implied to move together when they don't.