Voice Consensus Calibration: Evaluating Artificial Intelligence Confidence Against Expert Perceptual Consensus in Auditory-Perceptual Voice Assessment

This repository contains the official experimental code, acoustic feature extraction routines, calibration algorithms, and replication scripts for the manuscript:

"Beyond Aggregate Clinical Labels: Evaluating Artificial Intelligence Confidence Against Expert Perceptual Consensus in Auditory-Perceptual Voice Assessment"
Target Journal: Journal of Voice (Elsevier, Ref: JVOICE-D-26-01793)
Funding: Deanship of Scientific Research, King Faisal University, Saudi Arabia (Grant No. KFU264715).

---

Table of Contents

1. Overview & Clinical Motivation
2. Mathematical Foundations
3. Repository Architecture
4. Installation & Prerequisites
5. End-to-End Replication Guide
6. Experimental Results Summary
7. Dataset Attribution & Ethical Compliance
8. License & Citation

---

1. Overview & Clinical Motivation

Auditory-perceptual assessment using the Consensus Auditory-Perceptual Evaluation of Voice (CAPE-V) represents the primary clinical reference standard for diagnosing dysphonia, planning interventions, and tracking recovery. However, expert clinician ratings inherently exhibit inter-rater dispersion, reflecting genuine clinical ambiguity in borderline voice pathologies and differing clinician perceptual anchor points.

Conventional machine-learning voice models routinely collapse multi-expert ratings into scalar arithmetic averages (e.g., mean Overall Severity), discarding listener variance. This practice creates a hazardous clinical failure mode: models trained on collapsed targets output high diagnostic certainty on recordings where expert human clinicians vigorously disagree.

This repository provides a complete, computationally reproducible framework to:

1. Extract 97 clinically interpretable acoustic descriptors (MFCCs, spectral slope/flatness, harmonic-to-noise ratio, pitch perturbation) using native scientific Python (scipy, soundfile).
2. Formulate Perceptual-Consensus Expected Calibration Error (P-ECE) to benchmark algorithmic confidence against expert clinician consensus agreement.
3. Screen High-Confidence / High-Disagreement (HCHD) discordance to protect against automation bias in clinical triage.
4. Evaluate Disagreement-Weighted Loss Functions (SVR-DW) to regularize learning on ambiguous recordings.
5. Stratify performance across the empirically validated Walden (2025) clinical severity bands (Normal: 0–15, Mild: 16–39, Moderate: 40–69, Severe: 70–100 VAS points).

---

2. Mathematical Foundations

2.1 Multi-Clinician Consensus and Empirical Dispersion

For a given patient recording $i$ evaluated by $m_i$ licensed voice-specialist clinicians:

• Consensus Rating ($C_i$): $$C_i = \frac{1}{m_i} \sum_{j=1}^{m_i} r_{ij}, \quad r_{ij} \in [0, 100]$$
• Inter-Rater Dispersion ($s_i$ and $D_i$): $$s_i = \sqrt{\frac{1}{m_i - 1} \sum_{j=1}^{m_i} (r_{ij} - C_i)^2}, \quad D_i = \frac{s_i}{\max(s)}$$
• Expert Clinician Agreement ($A_i$): $$A_i = 1 - D_i \in [0, 1]$$

2.2 Standard ECE vs. Perceptual-Consensus ECE (P-ECE)

• Domain-Normalized Model Confidence ($c^*$): For a $K$-class classifier ($K=4$) with maximum softmax/Platt posterior probability $c$: $$c^* = \frac{K \cdot c - 1}{K - 1} \in [0, 1]$$
• Standard Expected Calibration Error (ECE): $$\text{ECE} = \sum_{b=1}^B \frac{|B_b|}{N} \left| \text{acc}(B_b) - \text{conf}(B_b) \right|$$
• Perceptual-Consensus Expected Calibration Error (P-ECE): $$\text{P-ECE} = \sum_{b=1}^B \frac{|B_b|}{N} \left| \bar{A}(B_b) - \bar{c}^*(B_b) \right|$$

2.3 High-Confidence / High-Disagreement (HCHD) Risk Metric

$$\text{HCHD Rate} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}\left(c_i^* > \tau_{\text{conf}} ;\land; D_i > \tau_{\text{disagr}}\right)$$ Identifies cases where automated tools exhibit high confidence despite substantial clinician disagreement.

2.4 Disagreement-Weighted Loss Formulation (SVR-DW)

$$w_i = (1 - D_i)^\gamma = A_i^\gamma$$ $$\min_{w, b, \xi, \xi^} \frac{1}{2} |w|^2 + C \sum_{i=1}^N w_i (\xi_i + \xi_i^)$$ where $\gamma \ge 0$ controls the penalty assigned to clinician contestation.

---

3. Repository Architecture

voice-cosensus-calibration repository/
├── scripts/
│   ├── voice_ml_engine.py                # Calibration engine: P-ECE, ECE, HCHD, Platt scaling, BCa bootstrap
│   ├── extract_acoustic_features.py       # 97 acoustic feature extraction routines (MFCCs, spectral, HNR)
│   ├── run_experiments.py                # 5-fold cross-validation pipeline across all ML models
│   ├── generate_figures.py               # Publication figure replication script (Figures 1–8)
│   ├── analyze_ratings_demographics.py   # Multi-rater perceptual dispersion, ICC, and demographics
│   └── download_all_pvqd_audio.py        # Dataset acquisition utility for Mendeley Data
├── features/
│   └── pvqd_acoustic_features.csv        # Master extracted acoustic feature matrix (296 x 98)
├── results/
│   ├── results_metrics.csv               # Accuracy, Macro-F1, ECE, P-ECE, Brier, HCHD rate
│   ├── results_predictions.csv           # Out-of-fold predictions, true labels, dispersion, confidence
│   ├── results_disagreement.csv          # Patient-level consensus (C_i), dispersion (s_i), agreement (A_i)
│   ├── results_ablation.csv              # Disagreement-aware weighting gamma ablation (SVR-DW)
│   ├── results_per_class.csv             # Precision, recall, F1 across Walden (2025) severity bands
│   ├── results_task_comparison.csv       # Sustained vowels vs. connected speech vs. composite
│   ├── results_error_vs_disagreement.csv # Error vs. clinician disagreement correlation
│   └── results_human_agreement.csv       # Clinician inter-rater reliability benchmarks
├── figures/                              # High-resolution publication figures (300 DPI PNG & vector PDF)
│   ├── figure_1_pipeline.{png,pdf}
│   ├── figure_2_disagreement_distribution.{png,pdf}
│   ├── figure_3_error_vs_disagreement.{png,pdf}
│   ├── figure_4_confidence_vs_agreement.{png,pdf}
│   ├── figure_5_dual_calibration.{png,pdf}
│   ├── figure_6_hchd_quadrants.{png,pdf}
│   ├── figure_7_task_breakdown.{png,pdf}
│   └── figure_8_ablation.{png,pdf}
├── requirements.txt                      # Complete Python environment specifications
├── LICENSE                               # MIT Open-Source License
└── README.md                             # Comprehensive reproduction guide

---

4. Installation & Prerequisites

The codebase operates entirely on CPU without requiring external GPU hardware or monolithic deep learning frameworks.

Python Environment

• Python 3.10, 3.11, or 3.12 (compatible across Windows, Linux, and macOS).
• Install dependencies:

pip install -r requirements.txt

Core dependencies:

• numpy >= 1.22.0
• scipy >= 1.8.0
• pandas >= 1.4.0
• scikit-learn >= 1.1.0
• soundfile >= 0.10.0
• matplotlib >= 3.5.0
• seaborn >= 0.11.0

---

5. End-to-End Replication Guide

All replication commands can be executed directly from the repository root directory.

Step 1: Acoustic Feature Extraction

To re-extract the 97 acoustic descriptors from raw audio files:

python scripts/extract_acoustic_features.py

Note: The complete, pre-extracted feature matrix for all 296 recordings is already provided in features/pvqd_acoustic_features.csv.

Step 2: Run Full Experimental Suite

Execute 5-fold cross-validation across all regression and calibrated classification models:

python scripts/run_experiments.py

This script computes:

• Continuous regression metrics (MAE, RMSE, Pearson $r$, Spearman $\rho$, Lin's CCC)
• 4-class categorical triage metrics (Accuracy, Macro-F1, Weighted-F1)
• Standard ECE and Platt probability calibration
• Perceptual-Consensus Expected Calibration Error (P-ECE)
• High-Confidence/High-Disagreement (HCHD) discordance rates
• BCa bootstrap 95% confidence intervals (1,000 resamples)
• Disagreement-aware loss weighting ablation ($\gamma \in {0.0, 0.5, 1.0, 2.0}$)
• Outputs are written directly to results/.

Step 3: Replicate Publication Figures

Generate all 8 publication-quality figures (vector PDF and 300 DPI PNG):

python scripts/generate_figures.py

Figures are saved directly into figures/:

• figure_1_pipeline: End-to-end clinical voice AI architecture.
• figure_2_disagreement_distribution: Multi-rater consensus vs. empirical dispersion across Walden severity bands.
• figure_3_error_vs_disagreement: Algorithmic prediction errors as a function of clinician disagreement.
• figure_4_confidence_vs_agreement: Dual calibration curves exposing algorithmic overconfidence.
• figure_5_dual_calibration: Reliability diagrams across all ML architectures.
• figure_6_hchd_quadrants: Diagnostic safety quadrants and HCHD discordance screening.
• figure_7_task_breakdown: Task comparison (sustained vowels vs. connected speech sentences).
• figure_8_ablation: Disagreement-aware loss weighting ablation.

Step 4: Multi-Rater Reliability & Demographics

To compute inter-rater ICC, Pearson repeatability, and clinical demographic distributions:

python scripts/analyze_ratings_demographics.py

---

6. Experimental Results Summary

Primary Benchmark Results (Walden 2025 Severity Bands)

Model Architecture	Task	MAE (95% CI)	RMSE	Pearson $r$	Accuracy	Macro-F1	Standard ECE	Perceptual P-ECE	HCHD Rate
Support Vector Machine (RBF)	Continuous	13.76 [12.60, 14.93]	17.65	0.720	—	—	—	—	—
Random Forest	Dual	12.99 [11.83, 14.19]	16.71	0.748	53.4%	0.456	0.095	0.346 (3.6×)	7.1%
HistGradientBoosting	Dual	13.14 [11.96, 14.36]	16.92	0.739	52.0%	0.442	0.051	0.386 (7.6×)	9.1%
Linear SVC (Platt Scaled)	Categorical	—	—	—	49.3%	0.428	0.089	0.412 (4.6×)	11.5%

Key Clinical Finding: Standard ECE (0.051–0.095) suggests well-calibrated predictions. However, P-ECE reveals dramatic calibration divergence (0.346–0.412; up to 7.6-fold higher), exposing that conventional voice models express unwarranted diagnostic certainty on voices where human clinicians disagree.

---

7. Dataset Attribution & Ethical Compliance

• Dataset Origin: All primary acoustic recordings, patient demographic files, and multi-clinician perceptual ratings were collected, curated, and openly shared by Dr. Patrick R. Walden (Monmouth University) under a Creative Commons Attribution 4.0 International license (CC BY 4.0):
  • Walden PR. Perceptual Voice Qualities Database (PVQD): Database Characteristics. J Voice. 2022;36(6):875.e15–875.e23. doi:10.1016/j.jvoice.2020.09.014
  • Mendeley Data: doi:10.17632/9dz247gnyb.4
• Grant Support: Original data acquisition was funded by The Voice Foundation’s Advancing Scientific Voice Research Grant.
• Secondary Analysis Disclaimer: This investigation represents an independent secondary computational and methodological analysis. Dr. Patrick R. Walden, Monmouth University, and The Voice Foundation bear no responsibility for the analyses, interpretations, or conclusions presented herein.
• Institutional Funding: This secondary study was funded by the Deanship of Scientific Research, Vice Presidency for Graduate Studies and Scientific Research at King Faisal University, Saudi Arabia (Grant No. KFU264715).

---

8. License & Citation

This repository is distributed under the MIT License.

If you use this code, feature representations, or calibration framework in your research, please cite:

@article{iqbal2026beyond,
  title={Beyond Aggregate Clinical Labels: Evaluating Artificial Intelligence Confidence Against Expert Perceptual Consensus in Auditory-Perceptual Voice Assessment},
  author={Iqbal, Sajid and Ahmad, Muhammad Khubaib and Yasir, Kinza and Shadab, Muzammil and Iqbal, Abid and Asghar, Muhammad Nabeel},
  journal={Journal of Voice},
  year={2026},
  note={Under revision, Ref: JVOICE-D-26-01793}
}