Live Benchmark for long Horizon tool using Agentic Systems
-
Updated
Sep 16, 2026 - Python
Live Benchmark for long Horizon tool using Agentic Systems
A bilingual awesome list for refusal suppression research: benchmarks, papers, tools, models, and ecosystem updates.
中文网安防御问题误拒(over-refusal)评测基准:测安全模型是否误拒正当防御/安全教育问题,含真实危害对照组,可机器评分。补全 agent-safety-bench-zh 的另一半。
Open-source evaluation suite that stress-tests LLM over-refusal on legitimate requests involving defeated, unjust, or absurd rules. 5 defeat families × 19 authority types = 95 scenario types.
Türkçe LLM guardrail değerlendirme seti: bloklanması gereken prompt injection saldırıları + bloklanmaması gereken (over-refusal) masum istekler. Turkish LLM guardrail evaluation set.
XSTest-style over-refusal eval for Turkish LLM guards: 120 benign-but-scary pairs. Finding: ProtectAI over-refuses 59% of benign Turkish prompts vs 0.8% English (71x). Dataset + refusal scorer.
Measuring phrasing variance in over-refusal evaluation, and deriving how many phrasings an auditor needs per item.
Domain-expert-authored benchmark for LLM over-refusal on legitimate biology research queries.
RESEARCH DENIED 30: Fable 5 vs Kimi K3 on benign AI-safety research access
OR-Bench over-refusal evaluation for Inspect AI — LLM-judged refusal scoring, registered upstream in UK AISI's inspect_evals.
AI safety benchmark evaluating Google Gemini on harmful compliance and over-refusal, using 60 paired queries and a blind Anthropic Claude judge. Gemini: 0/30 harmful compliance while refusing only 7.
Ablation runner for LLM guardrails: compare guard variants on the same prompts, separate guard blocks from model refusals, and see attack success, over-blocking, latency and cost with confidence intervals. CLI, library and CI gate.
FABLE: reproducible false-positive refusal benchmark for Claude Fable 5 vs Kimi K3
IR-REFUSE: quantifying how defensive over-refusal compounds across an agentic incident-response workflow (Apart AI Incident Response Sprint)
MSc research project on selective activation steering for reducing unnecessary refusals in mixture-of-experts language models.
Measuring how often a model refuses benign professional questions, against a calibration set of genuinely harmful prompts that makes the number mean something
CI gate that fails when a model over-refuses benign prompts, scored across languages with a Turkish refusal-marker detector. pytest plugin + GitHub Action.
Benchmark measuring whether AI cyber safeguards separate authorized security work from materially identical unauthorized requests
Is safety repair language-fair? A pre-registered audit of multilingual refusal steering on Cohere's Tiny Aya, built on the Lexsi stack.
Evaluates LLM safety trade-offs: over-refusal versus harmful compliance, using XSTest and validated refusal classifiers.
To associate your repository with the over-refusal topic, visit your repo's landing page and select "manage topics."