Skip to content

COMPAS's "71%" reduction figure uses the wrong formula - every other audit uses relative reduction, this one uses raw percentage points #545

Description

@yakew7

Where: COMPAS/README.md and the root README.md's Results-at-a-Glance table, row 01 ("Reduction" column).

The gap: every other audit's "Reduction" column is a relative reduction, (before - after) / before. COMPAS's row alone uses the raw point difference instead.

Repro (pure arithmetic, no model re-run needed):

$ grep -n "86.77\|15.69\|71%" COMPAS/README.md
| Race | 86.77% | 15.69% | 71% |

$ python3 -c "print((86.77-15.69)/86.77*100)"
81.91771349544774

71.08 is the raw point gap (86.77 - 15.69 ≈ 71.08 ≈ "71%") - not the relative reduction, which is actually 81.9%. Every other row in the same table checks out against the relative-reduction formula:

$ python3 -c "
print('AI Fair Recruitment:', (4.51-0.12)/4.51*100)   # 97.3 - matches stated 97.3%
print('German Credit:', (7.16-1.89)/7.16*100)          # 73.6 - matches stated 73.6%
print('Insurance Age:', (7.93-3.18)/7.93*100)          # 59.9 - matches stated ~60%
print('Insurance Gender:', (5.44-1.54)/5.44*100)       # 71.7 - matches stated ~72%
print('Benefits Sex:', (18.00-8.52)/18.00*100)         # 52.7 - matches stated ~53%
"

Only COMPAS's row breaks the pattern the other 6 audits (14 rows including multi-attribute ones) all follow correctly, and it appears identically in both COMPAS/README.md and the root README.md, so it isn't a one-off transcription slip.

Why it matters: the table's own convention (relative % reduction, used everywhere else) is broken for the flagship first audit, understating COMPAS's actual mitigation effectiveness (real reduction is ~82%, not 71%).

Suggested fix: change "71%" to "82%" (or "81.9%") in both COMPAS/README.md and the root README.md Results-at-a-Glance table.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationgood first issueGood for newcomershelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions