annotation-quality

Group-safe research prototype for prioritising potentially inconsistent nucleus annotations for expert review.

Statistical validity checks for human-graded AI evaluations

Emotion architecture from Reddit comments: rater behavior, semantic clusters, and contradiction mapping in GoEmotions.

All-in-one quality control for LabelMe and YOLO datasets: find broken files, label mistakes, bad shapes and split leakage before training, then fix them in one desktop app with full undo.

QA for AI systems: LLM response evaluation harness (property checks, rubrics, flakiness detection, release gate) and an RLHF preference-data auditor (position and verbosity bias, per-annotator agreement, consistency). 43 tests, CI on 3 Python versions.

Modular visual evaluation and computer-vision framework for image QA, structured annotation, reliability analysis, and human pose data.

Code, logs and evaluation artifacts for "Shortcut Learning in a Public Grape Disease Dataset: Annotation Granularity as a Modulator, Not a Cause" — inconsistent annotation granularity induces a quantifiable shortcut, established by counterfactual retraining and a pre-registered negative result.

LabelBench by Bayanat Labs, a reproducible audit of automated Arabic data labeling. Measures how much of a labeling queue can be accepted at a declared quality floor, by taxonomy, task and Arabic dialect region. Code only; datasets download from their original sources.

Human-in-the-loop classification of 240 synthetic support tickets into 8 queues, with Label Studio evidence, ambiguous-case reasoning, reference-guided QA, and auditable revisions.

Job Match Assistant Chrome extension with job tracking and an A-Checker concept for SOP-based annotation quality assistance.

B.Sc. seminar project: annotation agreement analysis (Bennett's S, kappa, AC1), an annotation experiment on 216 tweets, and a 76-criterion COCO vs SA-1B dataset evaluation.

Production IAA engine — Krippendorff's Alpha, SBERT semantic agreement, Fleiss' Kappa, Shannon entropy, 3-layer collusion detection. Ran behind RawEval's 9-annotator workbench.

Reliability, rogue-rater, drift and leakage diagnostics for labelled data — on the raw GoEmotions ratings, 27 of 28 emotions fall below the 0.667 agreement floor.