Skip to content

significance_report's confidence parameter doesn't affect the significant flag - only the CI #548

Description

@yakew7

Where: faircode/significance.py's significance_report - "significant": p_value < 0.05 (hardcoded), while confidence (default 0.95) only widens/narrows the returned ci_low/ci_high bounds.

The gap: a caller passing a stricter confidence (e.g. 0.99) expecting the significance verdict to tighten accordingly still gets exactly the same p < 0.05 threshold - confidence has zero effect on significant.

Repro (fully deterministic, no ML model fitting involved):

$ python3 -c "
import numpy as np
from faircode.significance import significance_report
rng = np.random.default_rng(0)
a = b = None
for _ in range(5):
    a = rng.binomial(1, 0.55, size=60).astype(float)
    b = rng.binomial(1, 0.35, size=60).astype(float)
r95 = significance_report(a, b, n_resamples=3000, n_permutations=3000, confidence=0.95, random_state=4)
r99 = significance_report(a, b, n_resamples=3000, n_permutations=3000, confidence=0.99, random_state=4)
print('p95:', r95['p_value'], 'sig95:', r95['significant'])
print('p99:', r99['p_value'], 'sig99:', r99['significant'])
"
p95: 0.03833333333333333 sig95: True
p99: 0.03833333333333333 sig99: True

p_value is identical (0.0383) in both calls, and significant is True in both - even though 0.0383 > 0.01, the threshold a 99%-confidence caller would reasonably expect.

Why it matters: significance_report is the public function every audit's unfair.py/fair.py uses directly. Anyone tightening confidence to get a more conservative significance verdict gets no actual change in behavior, silently.

Suggested fix: derive the significance threshold from confidence (e.g. p_value < (1 - confidence)), or explicitly document that significant is fixed at α=0.05 independent of the confidence parameter, so the two aren't implied to move together when they don't.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions