Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions claims.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
# explicit finding that no meaningful rescue applies

version: "0.4"
last_owner_review: "2026-10-24"
last_owner_review: "2026-11-02"

claims:
- id: CC-001
Expand Down Expand Up @@ -659,7 +659,7 @@ claims:
"11101": 18
"11110": 0
"11111": 9
last_reviewed: "2026-08-28"
last_reviewed: "2026-11-02"
review_window_days: 60
review_triggers:
- type: remote_content_change
Expand Down Expand Up @@ -763,7 +763,7 @@ claims:
langkit: [0, 3]
nemo: [0, 21]
llm_guard: [0, 0]
last_reviewed: "2026-08-30"
last_reviewed: "2026-11-02"
review_window_days: 60
review_triggers:
- type: local_content_change
Expand Down Expand Up @@ -938,7 +938,7 @@ claims:
lg4+lg3v: 33
lg3v+sg2: 15
lg4+lg3v+sg2: 8
last_reviewed: '2026-08-31'
last_reviewed: '2026-11-02'
review_window_days: 60
review_triggers:
- type: remote_content_change
Expand Down
2 changes: 1 addition & 1 deletion claims/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ <h1 class="q">Registered claims and their falsifiers</h1>
<main class="container" id="main">
<section class="gsec">
<div class="glabel"><span class="gno">01</span><span class="gname">The registry</span></div>
<ul class="idx"><li><a href="/claims/cc-001/"><span class="cid">CC-001</span><span class="ctitle">Frechet-Hoeffding endpoint bounds from declared marginals</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-002/"><span class="cid">CC-002</span><span class="ctitle">The claim-boundary manifest and its validation lanes</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-003/"><span class="cid">CC-003</span><span class="ctitle">Identical pairwise overlaps, different three-way probability</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-004/"><span class="cid">CC-004</span><span class="ctitle">The sharp interval for the both-fail query</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-005/"><span class="cid">CC-005</span><span class="ctitle">A measurement contract frozen before any dataset was inspected</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-006/"><span class="cid">CC-006</span><span class="ctitle">The E2 pipeline rehearsed on synthetic mechanisms</span><span class="cexp">expires 2027-01-05</span></a></li><li><a href="/claims/rel-001/"><span class="cid">REL-001</span><span class="ctitle">Preregistered hypothesis on dependence-aware presentation</span><span class="cexp">expires 2026-11-16</span></a></li><li><a href="/claims/mc-001/"><span class="cid">MC-001</span><span class="ctitle">Of 20 guardrail evaluations, 5 with joint-evidence artifacts</span><span class="cexp">expires 2026-11-13</span></a></li><li><a href="/claims/mc-002/"><span class="cid">MC-002</span><span class="ctitle">Five supervisors on BELLS against the independence plug-in</span><span class="cexp">expires 2026-10-27</span></a></li><li><a href="/claims/mc-003/"><span class="cid">MC-003</span><span class="ctitle">Marginals fix the all-miss rate only up to an interval</span><span class="cexp">expires 2026-10-29</span></a></li><li><a href="/claims/mc-004/"><span class="cid">MC-004</span><span class="ctitle">Multimodal Safeguard Bench, recomputed from per-item verdicts</span><span class="cexp">expires 2026-10-30</span></a></li><li><a href="/claims/af-001/"><span class="cid">AF-001</span><span class="ctitle">AI Fluency Index prevalences for 11 of 24 behaviours</span><span class="cexp">expires 2026-11-26</span></a></li><li><a href="/claims/ga-001/"><span class="cid">GA-001</span><span class="ctitle">Ghost-Ark, a verifier for the provenance limits of receipts</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/gv-001/"><span class="cid">GV-001</span><span class="ctitle">Ghost Visualizer, a visual essay that computes safety scores</span><span class="cexp">expires 2027-02-21</span></a></li><li><a href="/claims/gce-001/"><span class="cid">GCE-001</span><span class="ctitle">The Guardrail Composability Explorer, a coursework demo</span><span class="cexp">expires 2027-08-24</span></a></li><li><a href="/claims/site-001/"><span class="cid">SITE-001</span><span class="ctitle">The palette token pairs and their computed contrast</span><span class="cexp">expires 2026-12-29</span></a></li><li><a href="/claims/site-002/"><span class="cid">SITE-002</span><span class="ctitle">The claim registry, enforced in CI</span><span class="cexp">expires 2026-12-28</span></a></li><li><a href="/claims/e3-001/"><span class="cid">E3-001</span><span class="ctitle">Two ungated classifiers scored on harmful and benign items</span><span class="cexp">expires 2027-01-06</span></a></li><li><a href="/claims/e3b-001/"><span class="cid">E3B-001</span><span class="ctitle">The same classifiers on the attack family they were built for</span><span class="cexp">expires 2027-01-06</span></a></li><li><a href="/claims/e6-001/"><span class="cid">E6-001</span><span class="ctitle">Rejected: the permutation routine moves unequal-mass atoms</span><span class="cexp">expires 2027-01-08</span></a></li><li><a href="/claims/e7b-001/"><span class="cid">E7B-001</span><span class="ctitle">Rejected: calibration uses the shared pool for every judge</span><span class="cexp">expires 2027-01-08</span></a></li><li><a href="/claims/e9-001/"><span class="cid">E9-001</span><span class="ctitle">E9: both-miss measurement on 1,000 items</span><span class="cexp">expires 2027-02-14</span></a></li></ul>
<ul class="idx"><li><a href="/claims/cc-001/"><span class="cid">CC-001</span><span class="ctitle">Frechet-Hoeffding endpoint bounds from declared marginals</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-002/"><span class="cid">CC-002</span><span class="ctitle">The claim-boundary manifest and its validation lanes</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-003/"><span class="cid">CC-003</span><span class="ctitle">Identical pairwise overlaps, different three-way probability</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-004/"><span class="cid">CC-004</span><span class="ctitle">The sharp interval for the both-fail query</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-005/"><span class="cid">CC-005</span><span class="ctitle">A measurement contract frozen before any dataset was inspected</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/cc-006/"><span class="cid">CC-006</span><span class="ctitle">The E2 pipeline rehearsed on synthetic mechanisms</span><span class="cexp">expires 2027-01-05</span></a></li><li><a href="/claims/rel-001/"><span class="cid">REL-001</span><span class="ctitle">Preregistered hypothesis on dependence-aware presentation</span><span class="cexp">expires 2026-11-16</span></a></li><li><a href="/claims/mc-001/"><span class="cid">MC-001</span><span class="ctitle">Of 20 guardrail evaluations, 5 with joint-evidence artifacts</span><span class="cexp">expires 2026-11-13</span></a></li><li><a href="/claims/mc-002/"><span class="cid">MC-002</span><span class="ctitle">Five supervisors on BELLS against the independence plug-in</span><span class="cexp">expires 2027-01-01</span></a></li><li><a href="/claims/mc-003/"><span class="cid">MC-003</span><span class="ctitle">Marginals fix the all-miss rate only up to an interval</span><span class="cexp">expires 2027-01-01</span></a></li><li><a href="/claims/mc-004/"><span class="cid">MC-004</span><span class="ctitle">Multimodal Safeguard Bench, recomputed from per-item verdicts</span><span class="cexp">expires 2027-01-01</span></a></li><li><a href="/claims/af-001/"><span class="cid">AF-001</span><span class="ctitle">AI Fluency Index prevalences for 11 of 24 behaviours</span><span class="cexp">expires 2026-11-26</span></a></li><li><a href="/claims/ga-001/"><span class="cid">GA-001</span><span class="ctitle">Ghost-Ark, a verifier for the provenance limits of receipts</span><span class="cexp">expires 2026-12-22</span></a></li><li><a href="/claims/gv-001/"><span class="cid">GV-001</span><span class="ctitle">Ghost Visualizer, a visual essay that computes safety scores</span><span class="cexp">expires 2027-02-21</span></a></li><li><a href="/claims/gce-001/"><span class="cid">GCE-001</span><span class="ctitle">The Guardrail Composability Explorer, a coursework demo</span><span class="cexp">expires 2027-08-24</span></a></li><li><a href="/claims/site-001/"><span class="cid">SITE-001</span><span class="ctitle">The palette token pairs and their computed contrast</span><span class="cexp">expires 2026-12-29</span></a></li><li><a href="/claims/site-002/"><span class="cid">SITE-002</span><span class="ctitle">The claim registry, enforced in CI</span><span class="cexp">expires 2026-12-28</span></a></li><li><a href="/claims/e3-001/"><span class="cid">E3-001</span><span class="ctitle">Two ungated classifiers scored on harmful and benign items</span><span class="cexp">expires 2027-01-06</span></a></li><li><a href="/claims/e3b-001/"><span class="cid">E3B-001</span><span class="ctitle">The same classifiers on the attack family they were built for</span><span class="cexp">expires 2027-01-06</span></a></li><li><a href="/claims/e6-001/"><span class="cid">E6-001</span><span class="ctitle">Rejected: the permutation routine moves unequal-mass atoms</span><span class="cexp">expires 2027-01-08</span></a></li><li><a href="/claims/e7b-001/"><span class="cid">E7B-001</span><span class="ctitle">Rejected: calibration uses the shared pool for every judge</span><span class="cexp">expires 2027-01-08</span></a></li><li><a href="/claims/e9-001/"><span class="cid">E9-001</span><span class="ctitle">E9: both-miss measurement on 1,000 items</span><span class="cexp">expires 2027-02-14</span></a></li></ul>
</section>
<section class="gsec">
<div class="glabel"><span class="gno">02</span><span class="gname">How to read a date here</span></div>
Expand Down
20 changes: 10 additions & 10 deletions claims/index.json
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@
"https://x.com/PranavBhave_"
]
},
"dateModified": "2026-10-24",
"dateModified": "2026-11-02",
"dataFeedElement": [
{
"@context": [
Expand Down Expand Up @@ -1044,7 +1044,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
"dateModified": "2026-08-28",
"dateModified": "2026-11-02",
"fw:scopeStatement": "Exactly the released file at the bound commit: 170 of the evaluation's 990 non-adversarial prompts (the study's ~4,165 adversarial prompts have no per-item release), selected by the authors under an unstated rule. Counting arithmetic only, at the vendors' released binary verdicts; no population estimate, no uncertainty interval, and no claim about any system's true rates is offered or licensed. Computed by scripts/reanalyze_bells_subset.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified file.",
"fw:falsifier": "Recomputing from the bound, hash-verified file yields any count different from the expected block, or the file at the pinned ref no longer matches the recorded sha256, or the released columns are shown not to be the labelled systems' verdicts.",
"fw:consequence": "REJECT",
Expand All @@ -1061,9 +1061,9 @@
"the release-recomputed-to-plug-in ratio describes this subset's arithmetic, not a general law of guardrail dependence",
"not an observed deployed five-guard stack; interpreting this static OR aggregation as a stack requires the separate full-exposure, parallel, fixed-operating-point assumptions"
],
"fw:lastReviewed": "2026-08-28",
"fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
"fw:warrantExpires": "2026-10-27",
"fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/CentreSecuriteIA/bells_leaderboard/blob/507566c5a4606c8e3dec0bd59a5c5fde62594951/data/non_adversarial_prompts.csv"
Expand Down Expand Up @@ -1159,7 +1159,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
"dateModified": "2026-08-30",
"dateModified": "2026-11-02",
"fw:scopeStatement": "The mathematics is classical — Bonferroni for the lower endpoint, monotonicity for the upper, Fréchet for sharpness — and nothing here claims to have derived it. What is contributed is the application: pricing, in probability units, what marginal-only guardrail reporting leaves undetermined, and identifying leave-one-out unions as a compact privacy-preserving disclosure of exclusive full-stack coverage. They do not identify pairwise or higher-order overlap, Shapley values, or causal structure. Every statement is conditional on block-on-any composition, per-item decisions that are functions of the item alone, a fixed population, full exposure, and a fixed operating point; it says nothing about sequential routes where downstream measurement is censored by upstream blocks, agentic systems where an intervention changes the trajectory, or any vendor's product. The empirical half is one stratum of one author-selected subset: 82 harmful prompts, with 50 benign and 38 borderline held as separate strata and never folded in. It is a BELLS-specific reproduction, not an assertion that BELLS is the census's only per-item outcome release. scripts/reanalyze_bells_subset.py re-computes the BELLS counts from the hash-verified upstream file in CI; scripts/identification.py checks the stated identification transformations against that registered envelope.",
"fw:falsifier": "A joint law with the stated marginals achieves an all-miss rate outside the interval; an exact finite catch-set arrangement lies outside the stated finite grid or exclusive-coverage bounds; or recomputing from the bound BELLS file yields any count different from the expected block.",
"fw:consequence": "REJECT",
Expand All @@ -1178,9 +1178,9 @@
"leave-one-out unions identify only exclusive full-stack coverage; they do not identify pairwise or higher-order overlap, Shapley values, or causal contribution",
"says nothing about adversarial prompts, which dominate the full evaluation and have no per-item release"
],
"fw:lastReviewed": "2026-08-30",
"fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
"fw:warrantExpires": "2026-10-29",
"fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/Cubits11/cubits11.github.io/blob/main/scripts/identification.py"
Expand Down Expand Up @@ -1272,7 +1272,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
"dateModified": "2026-08-31",
"dateModified": "2026-11-02",
"fw:scopeStatement": "Exactly the six committed verdict files of the full_run directory at the bound commit: 400 harmful and 500 benign items, each stratum half text and half image, one Boolean `blocked` adapter bit per item per guard. Let L be the release's harmful/benign label, A_s each guard's native `unsafe` predicate, B_s the harness mapping of that predicate to `blocked`, and O = OR_s B_s. The bound arithmetic reports B and O conditional on L. Because the harness uses `blocked` items to suppress target generation, O is also a valid counterfactual harness-block decision for these released rows under a fixed block-on-any rule. It does not establish a source-defined translation from every A_s to one shared catch event E: Llama Guard 3 Vision is a multimodal prompt/response classifier, while ShieldGemma 2 is an image-only three-policy classifier, and the pipeline merely maps each native `unsafe` label to `blocked`. Counting arithmetic only, at the guards' released native operating rules (the Llama guards emit labels autoregressively; ShieldGemma 2 blocks above a 0.5 policy-violation probability; the source documents no matched operating-point calibration, so M-ladder threshold comparability is unchanged). ShieldGemma 2's text-item bits are deterministic passes as committed and documented upstream, which is why its text column is zero — an explicit released outcome, not missing data. Strata are computed exactly as released and never folded; file totals are sums of the bound strata and carry no additional content. Computed by scripts/reanalyze_msbench.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified files.",
"fw:falsifier": "Recomputing from the bound, hash-verified files yields any count different from the expected block, or any pinned file no longer matches its recorded sha256, or any recomputed quantity that overlaps the release's printed metrics disagrees with the printed value, or the committed blocked columns are shown not to be the named guards' verdicts.",
"fw:consequence": "REJECT",
Expand All @@ -1293,9 +1293,9 @@
"the counts are about the committed verdict bytes; the release's own changelog documents that ShieldGemma 2's image scores are sensitive to the text-rendering stack, so nothing here predicts what any guard would do under a different rendering environment",
"the zero three-guard image all-zero-bit count is a counting fact about these 200 released image items, not evidence of general image attack safety"
],
"fw:lastReviewed": "2026-08-31",
"fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
"fw:warrantExpires": "2026-10-30",
"fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/PatrickKollman/Multimodal-Safeguard-Bench/tree/fb6f32e6b50b6faad833b815ebcd80afd2068bff/results/full_run"
Expand Down
Loading
Loading