diff --git a/claims/index.json b/claims/index.json
index c413bb2..9ab660a 100644
--- a/claims/index.json
+++ b/claims/index.json
@@ -76,7 +76,7 @@
"https://x.com/PranavBhave_"
]
},
- "dateModified": "2026-10-24",
+ "dateModified": "2026-11-02",
"dataFeedElement": [
{
"@context": [
@@ -1044,7 +1044,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
- "dateModified": "2026-08-28",
+ "dateModified": "2026-11-02",
"fw:scopeStatement": "Exactly the released file at the bound commit: 170 of the evaluation's 990 non-adversarial prompts (the study's ~4,165 adversarial prompts have no per-item release), selected by the authors under an unstated rule. Counting arithmetic only, at the vendors' released binary verdicts; no population estimate, no uncertainty interval, and no claim about any system's true rates is offered or licensed. Computed by scripts/reanalyze_bells_subset.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified file.",
"fw:falsifier": "Recomputing from the bound, hash-verified file yields any count different from the expected block, or the file at the pinned ref no longer matches the recorded sha256, or the released columns are shown not to be the labelled systems' verdicts.",
"fw:consequence": "REJECT",
@@ -1061,9 +1061,9 @@
"the release-recomputed-to-plug-in ratio describes this subset's arithmetic, not a general law of guardrail dependence",
"not an observed deployed five-guard stack; interpreting this static OR aggregation as a stack requires the separate full-exposure, parallel, fixed-operating-point assumptions"
],
- "fw:lastReviewed": "2026-08-28",
+ "fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
- "fw:warrantExpires": "2026-10-27",
+ "fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/CentreSecuriteIA/bells_leaderboard/blob/507566c5a4606c8e3dec0bd59a5c5fde62594951/data/non_adversarial_prompts.csv"
@@ -1159,7 +1159,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
- "dateModified": "2026-08-30",
+ "dateModified": "2026-11-02",
"fw:scopeStatement": "The mathematics is classical — Bonferroni for the lower endpoint, monotonicity for the upper, Fréchet for sharpness — and nothing here claims to have derived it. What is contributed is the application: pricing, in probability units, what marginal-only guardrail reporting leaves undetermined, and identifying leave-one-out unions as a compact privacy-preserving disclosure of exclusive full-stack coverage. They do not identify pairwise or higher-order overlap, Shapley values, or causal structure. Every statement is conditional on block-on-any composition, per-item decisions that are functions of the item alone, a fixed population, full exposure, and a fixed operating point; it says nothing about sequential routes where downstream measurement is censored by upstream blocks, agentic systems where an intervention changes the trajectory, or any vendor's product. The empirical half is one stratum of one author-selected subset: 82 harmful prompts, with 50 benign and 38 borderline held as separate strata and never folded in. It is a BELLS-specific reproduction, not an assertion that BELLS is the census's only per-item outcome release. scripts/reanalyze_bells_subset.py re-computes the BELLS counts from the hash-verified upstream file in CI; scripts/identification.py checks the stated identification transformations against that registered envelope.",
"fw:falsifier": "A joint law with the stated marginals achieves an all-miss rate outside the interval; an exact finite catch-set arrangement lies outside the stated finite grid or exclusive-coverage bounds; or recomputing from the bound BELLS file yields any count different from the expected block.",
"fw:consequence": "REJECT",
@@ -1178,9 +1178,9 @@
"leave-one-out unions identify only exclusive full-stack coverage; they do not identify pairwise or higher-order overlap, Shapley values, or causal contribution",
"says nothing about adversarial prompts, which dominate the full evaluation and have no per-item release"
],
- "fw:lastReviewed": "2026-08-30",
+ "fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
- "fw:warrantExpires": "2026-10-29",
+ "fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/Cubits11/cubits11.github.io/blob/main/scripts/identification.py"
@@ -1272,7 +1272,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
- "dateModified": "2026-08-31",
+ "dateModified": "2026-11-02",
"fw:scopeStatement": "Exactly the six committed verdict files of the full_run directory at the bound commit: 400 harmful and 500 benign items, each stratum half text and half image, one Boolean `blocked` adapter bit per item per guard. Let L be the release's harmful/benign label, A_s each guard's native `unsafe` predicate, B_s the harness mapping of that predicate to `blocked`, and O = OR_s B_s. The bound arithmetic reports B and O conditional on L. Because the harness uses `blocked` items to suppress target generation, O is also a valid counterfactual harness-block decision for these released rows under a fixed block-on-any rule. It does not establish a source-defined translation from every A_s to one shared catch event E: Llama Guard 3 Vision is a multimodal prompt/response classifier, while ShieldGemma 2 is an image-only three-policy classifier, and the pipeline merely maps each native `unsafe` label to `blocked`. Counting arithmetic only, at the guards' released native operating rules (the Llama guards emit labels autoregressively; ShieldGemma 2 blocks above a 0.5 policy-violation probability; the source documents no matched operating-point calibration, so M-ladder threshold comparability is unchanged). ShieldGemma 2's text-item bits are deterministic passes as committed and documented upstream, which is why its text column is zero — an explicit released outcome, not missing data. Strata are computed exactly as released and never folded; file totals are sums of the bound strata and carry no additional content. Computed by scripts/reanalyze_msbench.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified files.",
"fw:falsifier": "Recomputing from the bound, hash-verified files yields any count different from the expected block, or any pinned file no longer matches its recorded sha256, or any recomputed quantity that overlaps the release's printed metrics disagrees with the printed value, or the committed blocked columns are shown not to be the named guards' verdicts.",
"fw:consequence": "REJECT",
@@ -1293,9 +1293,9 @@
"the counts are about the committed verdict bytes; the release's own changelog documents that ShieldGemma 2's image scores are sensitive to the text-rendering stack, so nothing here predicts what any guard would do under a different rendering environment",
"the zero three-guard image all-zero-bit count is a counting fact about these 200 released image items, not evidence of general image attack safety"
],
- "fw:lastReviewed": "2026-08-31",
+ "fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
- "fw:warrantExpires": "2026-10-30",
+ "fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/PatrickKollman/Multimodal-Safeguard-Bench/tree/fb6f32e6b50b6faad833b815ebcd80afd2068bff/results/full_run"
diff --git a/claims/mc-002/index.html b/claims/mc-002/index.html
index 2f9a7ae..a04b646 100644
--- a/claims/mc-002/index.html
+++ b/claims/mc-002/index.html
@@ -112,7 +112,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
- "dateModified": "2026-08-28",
+ "dateModified": "2026-11-02",
"fw:scopeStatement": "Exactly the released file at the bound commit: 170 of the evaluation's 990 non-adversarial prompts (the study's ~4,165 adversarial prompts have no per-item release), selected by the authors under an unstated rule. Counting arithmetic only, at the vendors' released binary verdicts; no population estimate, no uncertainty interval, and no claim about any system's true rates is offered or licensed. Computed by scripts/reanalyze_bells_subset.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified file.",
"fw:falsifier": "Recomputing from the bound, hash-verified file yields any count different from the expected block, or the file at the pinned ref no longer matches the recorded sha256, or the released columns are shown not to be the labelled systems' verdicts.",
"fw:consequence": "REJECT",
@@ -129,9 +129,9 @@
"the release-recomputed-to-plug-in ratio describes this subset's arithmetic, not a general law of guardrail dependence",
"not an observed deployed five-guard stack; interpreting this static OR aggregation as a stack requires the separate full-exposure, parallel, fixed-operating-point assumptions"
],
- "fw:lastReviewed": "2026-08-28",
+ "fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
- "fw:warrantExpires": "2026-10-27",
+ "fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/CentreSecuriteIA/bells_leaderboard/blob/507566c5a4606c8e3dec0bd59a5c5fde62594951/data/non_adversarial_prompts.csv"
@@ -250,7 +250,7 @@
Five supervisors on BELLS against the independence plug-in
On the 170-prompt per-item subset publicly released with BELLS's 2025 misuse-detection evaluation (pinned commit below), the five specialized supervisors — Lakera Guard, Prompt Guard, LangKit, NeMo Guardrails, LLM Guard — have an OR-union of released binary verdict columns that flags 73 of the 82 prompts labelled harmful, leaving 9 all-miss (11.0%). The product of their individual miss rates is 3.5%; the release-recomputed all-miss rate is about 3.1× that independence plug-in on this file. The same OR-union flags 19 of the 50 benign prompts.
@@ -281,8 +281,8 @@
Five supervisors on BELLS against the independence plug-in
05Binding and freshness
Binding
non_adversarial_prompts.csv @ 507566c5 The upstream repository declares no license, so the file is cited and hash-verified (sha256 791dd4b0a168f2eb5831b308083a49 2e83200a9fa82585643c739023b03f57c3, asserted by the reproduction script), never redistributed from this record.
-
Reviewed
2026-08-28 · window 60 days
-
Expires
2026-10-27 — after this date the recorded review is overdue; this does not make the claim false
+
Reviewed
2026-11-02 · window 60 days
+
Expires
2027-01-01 — after this date the recorded review is overdue; this does not make the claim false
Triggers
executable fires when the released subset on the default branch changes
manual the authors release the full per-item dataset, state the subset's selection rule, or correct the released verdicts
diff --git a/claims/mc-003/index.html b/claims/mc-003/index.html
index 0526195..a74f551 100644
--- a/claims/mc-003/index.html
+++ b/claims/mc-003/index.html
@@ -112,7 +112,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
- "dateModified": "2026-08-30",
+ "dateModified": "2026-11-02",
"fw:scopeStatement": "The mathematics is classical — Bonferroni for the lower endpoint, monotonicity for the upper, Fréchet for sharpness — and nothing here claims to have derived it. What is contributed is the application: pricing, in probability units, what marginal-only guardrail reporting leaves undetermined, and identifying leave-one-out unions as a compact privacy-preserving disclosure of exclusive full-stack coverage. They do not identify pairwise or higher-order overlap, Shapley values, or causal structure. Every statement is conditional on block-on-any composition, per-item decisions that are functions of the item alone, a fixed population, full exposure, and a fixed operating point; it says nothing about sequential routes where downstream measurement is censored by upstream blocks, agentic systems where an intervention changes the trajectory, or any vendor's product. The empirical half is one stratum of one author-selected subset: 82 harmful prompts, with 50 benign and 38 borderline held as separate strata and never folded in. It is a BELLS-specific reproduction, not an assertion that BELLS is the census's only per-item outcome release. scripts/reanalyze_bells_subset.py re-computes the BELLS counts from the hash-verified upstream file in CI; scripts/identification.py checks the stated identification transformations against that registered envelope.",
"fw:falsifier": "A joint law with the stated marginals achieves an all-miss rate outside the interval; an exact finite catch-set arrangement lies outside the stated finite grid or exclusive-coverage bounds; or recomputing from the bound BELLS file yields any count different from the expected block.",
"fw:consequence": "REJECT",
@@ -131,9 +131,9 @@
"leave-one-out unions identify only exclusive full-stack coverage; they do not identify pairwise or higher-order overlap, Shapley values, or causal contribution",
"says nothing about adversarial prompts, which dominate the full evaluation and have no per-item release"
],
- "fw:lastReviewed": "2026-08-30",
+ "fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
- "fw:warrantExpires": "2026-10-29",
+ "fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/Cubits11/cubits11.github.io/blob/main/scripts/identification.py"
@@ -248,7 +248,7 @@
Marginals fix the all-miss rate only up to an interval
For k guardrails scored on a common item set at a common operating point under block-on-any composition, the rate at which every guard misses the same item is fixed by the published per-guard miss rates only up to the interval [max(0, Σp − (k−1)), min p], and every point of that interval is attained by some joint law with those marginals. Marginals prove non-degradation of a static OR composition relative to its best member, but, when that identified set is non-degenerate, they do not identify strictly positive incremental benefit. They also identify a positive lower bound on static benign union-flag rate when any member has a positive benign flag rate. On the BELLS 2025 partial per-item release — 82 prompts labelled harmful — the exact finite identified set is {0/82 … 12/82}; MC-002 recomputes 9/82 all-miss from the hash-verified released verdict file. Marginal catch counts plus the union identify only bounds on each guard's exclusive full-stack coverage; the registered leave-one-out unions identify the realized values: NeMo 18, Lakera 3, and LangKit, Prompt Guard, and LLM Guard 0 on this stratum.
@@ -279,8 +279,8 @@
Marginals fix the all-miss rate only up to an interval
05Binding and freshness
Binding
identification.py The file carries the analytic derivation in its docstring and checks explicit endpoint constructions numerically — the upper by nesting the events, the lower by laying the complements end to end around a circle. Its finite formulas are additionally exhaustively checked for n ≤ 4 and k ≤ 3.
-
Reviewed
2026-08-30 · window 60 days
-
Expires
2026-10-29 — after this date the recorded review is overdue; this does not make the claim false
+
Reviewed
2026-11-02 · window 60 days
+
Expires
2027-01-01 — after this date the recorded review is overdue; this does not make the claim false
Triggers
executable fires when the derivation or its tests change without a re-review; re-reviewed 2026-08-29 after removing a duplicate leave-one-out calculator and rechecking the registered outputs
manual a reader identifies an error in the stated Fréchet, finite-grid, or aggregate exclusive-coverage bounds, or a result is used outside the declared static full-exposure assumptions
diff --git a/claims/mc-004/index.html b/claims/mc-004/index.html
index 7b2936d..cabbd0c 100644
--- a/claims/mc-004/index.html
+++ b/claims/mc-004/index.html
@@ -112,7 +112,7 @@
"name": "Cubits11 claim registry",
"url": "https://cubits11.github.io/ledger/"
},
- "dateModified": "2026-08-31",
+ "dateModified": "2026-11-02",
"fw:scopeStatement": "Exactly the six committed verdict files of the full_run directory at the bound commit: 400 harmful and 500 benign items, each stratum half text and half image, one Boolean `blocked` adapter bit per item per guard. Let L be the release's harmful/benign label, A_s each guard's native `unsafe` predicate, B_s the harness mapping of that predicate to `blocked`, and O = OR_s B_s. The bound arithmetic reports B and O conditional on L. Because the harness uses `blocked` items to suppress target generation, O is also a valid counterfactual harness-block decision for these released rows under a fixed block-on-any rule. It does not establish a source-defined translation from every A_s to one shared catch event E: Llama Guard 3 Vision is a multimodal prompt/response classifier, while ShieldGemma 2 is an image-only three-policy classifier, and the pipeline merely maps each native `unsafe` label to `blocked`. Counting arithmetic only, at the guards' released native operating rules (the Llama guards emit labels autoregressively; ShieldGemma 2 blocks above a 0.5 policy-violation probability; the source documents no matched operating-point calibration, so M-ladder threshold comparability is unchanged). ShieldGemma 2's text-item bits are deterministic passes as committed and documented upstream, which is why its text column is zero — an explicit released outcome, not missing data. Strata are computed exactly as released and never folded; file totals are sums of the bound strata and carry no additional content. Computed by scripts/reanalyze_msbench.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified files.",
"fw:falsifier": "Recomputing from the bound, hash-verified files yields any count different from the expected block, or any pinned file no longer matches its recorded sha256, or any recomputed quantity that overlaps the release's printed metrics disagrees with the printed value, or the committed blocked columns are shown not to be the named guards' verdicts.",
"fw:consequence": "REJECT",
@@ -133,9 +133,9 @@
"the counts are about the committed verdict bytes; the release's own changelog documents that ShieldGemma 2's image scores are sensitive to the text-rendering stack, so nothing here predicts what any guard would do under a different rendering environment",
"the zero three-guard image all-zero-bit count is a counting fact about these 200 released image items, not evidence of general image attack safety"
],
- "fw:lastReviewed": "2026-08-31",
+ "fw:lastReviewed": "2026-11-02",
"fw:reviewWindow": "P60D",
- "fw:warrantExpires": "2026-10-30",
+ "fw:warrantExpires": "2027-01-01",
"fw:evidentialStatus": "supported_within_scope",
"fw:supportUrl": {
"@id": "https://github.com/PatrickKollman/Multimodal-Safeguard-Bench/tree/fb6f32e6b50b6faad833b815ebcd80afd2068bff/results/full_run"
@@ -254,7 +254,7 @@
Multimodal Safeguard Bench, recomputed from per-item verdicts
On the per-item guard-verdict files publicly released with Multimodal Safeguard Bench's full_run (pinned commit below; six verdict files and the release's two printed-metric files hash-verified), the three guards — Llama Guard 4, Llama Guard 3 Vision, ShieldGemma 2 — have harness-normalized native `unsafe` labels whose static Boolean OR is 1 on 192 of 200 harmful-labelled text items (8 all-zero rows, 4.0%) and all 200 harmful-labelled image items (0 all-zero rows), while the same OR is 1 on 32 of 250 benign-labelled text items (12.8%) and every one of 250 benign-labelled image items. Llama Guard 3 Vision alone has a 1-bit on all 250 released benign image items. The complete 2^3 native-label pattern table, leave-one-out bit ORs, and unique bit contributions for each released stratum are bound below. The release prints per-guard metrics and two two-guard compositions; it prints no three-guard bit OR, no all-zero-row count, no leave-one-out table, and no pattern decomposition — and every recomputed quantity that overlaps what it does print is asserted equal to the printed value.
@@ -285,8 +285,8 @@
Multimodal Safeguard Bench, recomputed from per-item verdicts
2026-10-30 — after this date the recorded review is overdue; this does not make the claim false
+
Reviewed
2026-11-02 · window 60 days
+
Expires
2027-01-01 — after this date the recorded review is overdue; this does not make the claim false
Triggers
executable fires when a bound verdict file on the default branch changes (6 bound sources)
executable fires when the release's printed per-guard metrics change
executable fires when the release's printed composition metrics change
executable fires when the guards' native adapter semantics change
executable fires when the harness label-to-block action changes
executable fires when the stated route or operating-point semantics change
manual the authors publish a corrected or extended full_run, print their own three-guard joint statistics, document a matched operating-point calibration, or publish a source-defined translation from all native predicates to one shared catch event
Supported within scope · Machine-generated, owner-executed
On the 170-prompt per-item subset publicly released with BELLS's 2025 misuse-detection evaluation (pinned commit below), the five specialized supervisors — Lakera Guard, Prompt Guard, LangKit, NeMo Guardrails, LLM Guard — have an OR-union of released binary verdict columns that flags 73 of the 82 prompts labelled harmful, leaving 9 all-miss (11.0%). The product of their individual miss rates is 3.5%; the release-recomputed all-miss rate is about 3.1× that independence plug-in on this file. The same OR-union flags 19 of the 50 benign prompts.
-
Scope
Exactly the released file at the bound commit: 170 of the evaluation's 990 non-adversarial prompts (the study's ~4,165 adversarial prompts have no per-item release), selected by the authors under an unstated rule. Counting arithmetic only, at the vendors' released binary verdicts; no population estimate, no uncertainty interval, and no claim about any system's true rates is offered or licensed. Computed by scripts/reanalyze_bells_subset.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified file.
Binding
non_adversarial_prompts.csv @ 507566c5 The upstream repository declares no license, so the file is cited and hash-verified (sha256 791dd4b0a168f2eb5831b308083a49 2e83200a9fa82585643c739023b03f57c3, asserted by the reproduction script), never redistributed from this record.
executable fires when the released subset on the default branch changes
manual the authors release the full per-item dataset, state the subset's selection rule, or correct the released verdicts
Falsifier
Recomputing from the bound, hash-verified file yields any count different from the expected block, or the file at the pinned ref no longer matches the recorded sha256, or the released columns are shown not to be the labelled systems' verdicts. CONSEQUENCE REJECT
Forbidden rescues
do not substitute a different subset, column set, or harm-level filter to preserve the numbers
do not fold borderline prompts into either denominator after seeing the results
do not recast this subset arithmetic as a population estimate, with or without an interval, if the primary counts are challenged
do not cite the release-recomputed-to-plug-in ratio without the selection caveat that scopes it
Non-claims
the subset is author-selected with an unstated rule — nothing here estimates any system's true rate, and no confidence interval is offered because the sampled population is undefined
not a ranking or endorsement of any vendor; the released verdicts are at unstated default configurations
says nothing about adversarial prompts, which dominate the full evaluation and have no per-item release
the release-recomputed-to-plug-in ratio describes this subset's arithmetic, not a general law of guardrail dependence
not an observed deployed five-guard stack; interpreting this static OR aggregation as a stack requires the separate full-exposure, parallel, fixed-operating-point assumptions
+
Scope
Exactly the released file at the bound commit: 170 of the evaluation's 990 non-adversarial prompts (the study's ~4,165 adversarial prompts have no per-item release), selected by the authors under an unstated rule. Counting arithmetic only, at the vendors' released binary verdicts; no population estimate, no uncertainty interval, and no claim about any system's true rates is offered or licensed. Computed by scripts/reanalyze_bells_subset.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified file.
Binding
non_adversarial_prompts.csv @ 507566c5 The upstream repository declares no license, so the file is cited and hash-verified (sha256 791dd4b0a168f2eb5831b308083a49 2e83200a9fa82585643c739023b03f57c3, asserted by the reproduction script), never redistributed from this record.
executable fires when the released subset on the default branch changes
manual the authors release the full per-item dataset, state the subset's selection rule, or correct the released verdicts
Falsifier
Recomputing from the bound, hash-verified file yields any count different from the expected block, or the file at the pinned ref no longer matches the recorded sha256, or the released columns are shown not to be the labelled systems' verdicts. CONSEQUENCE REJECT
Forbidden rescues
do not substitute a different subset, column set, or harm-level filter to preserve the numbers
do not fold borderline prompts into either denominator after seeing the results
do not recast this subset arithmetic as a population estimate, with or without an interval, if the primary counts are challenged
do not cite the release-recomputed-to-plug-in ratio without the selection caveat that scopes it
Non-claims
the subset is author-selected with an unstated rule — nothing here estimates any system's true rate, and no confidence interval is offered because the sampled population is undefined
not a ranking or endorsement of any vendor; the released verdicts are at unstated default configurations
says nothing about adversarial prompts, which dominate the full evaluation and have no per-item release
the release-recomputed-to-plug-in ratio describes this subset's arithmetic, not a general law of guardrail dependence
not an observed deployed five-guard stack; interpreting this static OR aggregation as a stack requires the separate full-exposure, parallel, fixed-operating-point assumptions
@@ -200,7 +200,7 @@
MC-003
Supported within scope · Machine-generated, owner-executed
For k guardrails scored on a common item set at a common operating point under block-on-any composition, the rate at which every guard misses the same item is fixed by the published per-guard miss rates only up to the interval [max(0, Σp − (k−1)), min p], and every point of that interval is attained by some joint law with those marginals. Marginals prove non-degradation of a static OR composition relative to its best member, but, when that identified set is non-degenerate, they do not identify strictly positive incremental benefit. They also identify a positive lower bound on static benign union-flag rate when any member has a positive benign flag rate. On the BELLS 2025 partial per-item release — 82 prompts labelled harmful — the exact finite identified set is {0/82 … 12/82}; MC-002 recomputes 9/82 all-miss from the hash-verified released verdict file. Marginal catch counts plus the union identify only bounds on each guard's exclusive full-stack coverage; the registered leave-one-out unions identify the realized values: NeMo 18, Lakera 3, and LangKit, Prompt Guard, and LLM Guard 0 on this stratum.
-
Scope
The mathematics is classical — Bonferroni for the lower endpoint, monotonicity for the upper, Fréchet for sharpness — and nothing here claims to have derived it. What is contributed is the application: pricing, in probability units, what marginal-only guardrail reporting leaves undetermined, and identifying leave-one-out unions as a compact privacy-preserving disclosure of exclusive full-stack coverage. They do not identify pairwise or higher-order overlap, Shapley values, or causal structure. Every statement is conditional on block-on-any composition, per-item decisions that are functions of the item alone, a fixed population, full exposure, and a fixed operating point; it says nothing about sequential routes where downstream measurement is censored by upstream blocks, agentic systems where an intervention changes the trajectory, or any vendor's product. The empirical half is one stratum of one author-selected subset: 82 harmful prompts, with 50 benign and 38 borderline held as separate strata and never folded in. It is a BELLS-specific reproduction, not an assertion that BELLS is the census's only per-item outcome release. scripts/reanalyze_bells_subset.py re-computes the BELLS counts from the hash-verified upstream file in CI; scripts/identification.py checks the stated identification transformations against that registered envelope.
Binding
identification.py The file carries the analytic derivation in its docstring and checks explicit endpoint constructions numerically — the upper by nesting the events, the lower by laying the complements end to end around a circle. Its finite formulas are additionally exhaustively checked for n ≤ 4 and k ≤ 3.
executable fires when the derivation or its tests change without a re-review; re-reviewed 2026-08-29 after removing a duplicate leave-one-out calculator and rechecking the registered outputs
manual a reader identifies an error in the stated Fréchet, finite-grid, or aggregate exclusive-coverage bounds, or a result is used outside the declared static full-exposure assumptions
Falsifier
A joint law with the stated marginals achieves an all-miss rate outside the interval; an exact finite catch-set arrangement lies outside the stated finite grid or exclusive-coverage bounds; or recomputing from the bound BELLS file yields any count different from the expected block. CONSEQUENCE REJECT
Forbidden rescues
do not change the composition rule after the fact to preserve the direction of the result
do not restate "identified only up to" as "estimated to be" — a bound is not an estimate
do not quote where a release-recomputed value sits inside the interval as a score or a percentage of a gap closed; the endpoints come from adversarial couplings with no detector-behavioural content
do not cite the zero-exclusive-coverage results as a vendor ranking, a causal attribution, or evidence about any product outside this stratum
do not fold the borderline stratum into either denominator to change any figure
Non-claims
no new mathematics is claimed; the bounds are Fréchet's and the lower endpoint is Bonferroni's
not a claim that guardrails in general fail together — this stratum cannot separate shared blind spots from prompt-difficulty heterogeneity
not a vendor evaluation; the released verdicts are at unstated default configurations, and one supervisor fires exactly once in the 170 released rows
the identified interval is what the marginals leave open, not a prediction about any deployed stack
leave-one-out unions identify only exclusive full-stack coverage; they do not identify pairwise or higher-order overlap, Shapley values, or causal contribution
says nothing about adversarial prompts, which dominate the full evaluation and have no per-item release
+
Scope
The mathematics is classical — Bonferroni for the lower endpoint, monotonicity for the upper, Fréchet for sharpness — and nothing here claims to have derived it. What is contributed is the application: pricing, in probability units, what marginal-only guardrail reporting leaves undetermined, and identifying leave-one-out unions as a compact privacy-preserving disclosure of exclusive full-stack coverage. They do not identify pairwise or higher-order overlap, Shapley values, or causal structure. Every statement is conditional on block-on-any composition, per-item decisions that are functions of the item alone, a fixed population, full exposure, and a fixed operating point; it says nothing about sequential routes where downstream measurement is censored by upstream blocks, agentic systems where an intervention changes the trajectory, or any vendor's product. The empirical half is one stratum of one author-selected subset: 82 harmful prompts, with 50 benign and 38 borderline held as separate strata and never folded in. It is a BELLS-specific reproduction, not an assertion that BELLS is the census's only per-item outcome release. scripts/reanalyze_bells_subset.py re-computes the BELLS counts from the hash-verified upstream file in CI; scripts/identification.py checks the stated identification transformations against that registered envelope.
Binding
identification.py The file carries the analytic derivation in its docstring and checks explicit endpoint constructions numerically — the upper by nesting the events, the lower by laying the complements end to end around a circle. Its finite formulas are additionally exhaustively checked for n ≤ 4 and k ≤ 3.
executable fires when the derivation or its tests change without a re-review; re-reviewed 2026-08-29 after removing a duplicate leave-one-out calculator and rechecking the registered outputs
manual a reader identifies an error in the stated Fréchet, finite-grid, or aggregate exclusive-coverage bounds, or a result is used outside the declared static full-exposure assumptions
Falsifier
A joint law with the stated marginals achieves an all-miss rate outside the interval; an exact finite catch-set arrangement lies outside the stated finite grid or exclusive-coverage bounds; or recomputing from the bound BELLS file yields any count different from the expected block. CONSEQUENCE REJECT
Forbidden rescues
do not change the composition rule after the fact to preserve the direction of the result
do not restate "identified only up to" as "estimated to be" — a bound is not an estimate
do not quote where a release-recomputed value sits inside the interval as a score or a percentage of a gap closed; the endpoints come from adversarial couplings with no detector-behavioural content
do not cite the zero-exclusive-coverage results as a vendor ranking, a causal attribution, or evidence about any product outside this stratum
do not fold the borderline stratum into either denominator to change any figure
Non-claims
no new mathematics is claimed; the bounds are Fréchet's and the lower endpoint is Bonferroni's
not a claim that guardrails in general fail together — this stratum cannot separate shared blind spots from prompt-difficulty heterogeneity
not a vendor evaluation; the released verdicts are at unstated default configurations, and one supervisor fires exactly once in the 170 released rows
the identified interval is what the marginals leave open, not a prediction about any deployed stack
leave-one-out unions identify only exclusive full-stack coverage; they do not identify pairwise or higher-order overlap, Shapley values, or causal contribution
says nothing about adversarial prompts, which dominate the full evaluation and have no per-item release
@@ -208,7 +208,7 @@
MC-004
Supported within scope · Machine-generated, owner-executed
On the per-item guard-verdict files publicly released with Multimodal Safeguard Bench's full_run (pinned commit below; six verdict files and the release's two printed-metric files hash-verified), the three guards — Llama Guard 4, Llama Guard 3 Vision, ShieldGemma 2 — have harness-normalized native `unsafe` labels whose static Boolean OR is 1 on 192 of 200 harmful-labelled text items (8 all-zero rows, 4.0%) and all 200 harmful-labelled image items (0 all-zero rows), while the same OR is 1 on 32 of 250 benign-labelled text items (12.8%) and every one of 250 benign-labelled image items. Llama Guard 3 Vision alone has a 1-bit on all 250 released benign image items. The complete 2^3 native-label pattern table, leave-one-out bit ORs, and unique bit contributions for each released stratum are bound below. The release prints per-guard metrics and two two-guard compositions; it prints no three-guard bit OR, no all-zero-row count, no leave-one-out table, and no pattern decomposition — and every recomputed quantity that overlaps what it does print is asserted equal to the printed value.
-
Scope
Exactly the six committed verdict files of the full_run directory at the bound commit: 400 harmful and 500 benign items, each stratum half text and half image, one Boolean `blocked` adapter bit per item per guard. Let L be the release's harmful/benign label, A_s each guard's native `unsafe` predicate, B_s the harness mapping of that predicate to `blocked`, and O = OR_s B_s. The bound arithmetic reports B and O conditional on L. Because the harness uses `blocked` items to suppress target generation, O is also a valid counterfactual harness-block decision for these released rows under a fixed block-on-any rule. It does not establish a source-defined translation from every A_s to one shared catch event E: Llama Guard 3 Vision is a multimodal prompt/response classifier, while ShieldGemma 2 is an image-only three-policy classifier, and the pipeline merely maps each native `unsafe` label to `blocked`. Counting arithmetic only, at the guards' released native operating rules (the Llama guards emit labels autoregressively; ShieldGemma 2 blocks above a 0.5 policy-violation probability; the source documents no matched operating-point calibration, so M-ladder threshold comparability is unchanged). ShieldGemma 2's text-item bits are deterministic passes as committed and documented upstream, which is why its text column is zero — an explicit released outcome, not missing data. Strata are computed exactly as released and never folded; file totals are sums of the bound strata and carry no additional content. Computed by scripts/reanalyze_msbench.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified files.
executable fires when a bound verdict file on the default branch changes (6 bound sources)
executable fires when the release's printed per-guard metrics change
executable fires when the release's printed composition metrics change
executable fires when the guards' native adapter semantics change
executable fires when the harness label-to-block action changes
executable fires when the stated route or operating-point semantics change
manual the authors publish a corrected or extended full_run, print their own three-guard joint statistics, document a matched operating-point calibration, or publish a source-defined translation from all native predicates to one shared catch event
Falsifier
Recomputing from the bound, hash-verified files yields any count different from the expected block, or any pinned file no longer matches its recorded sha256, or any recomputed quantity that overlaps the release's printed metrics disagrees with the printed value, or the committed blocked columns are shown not to be the named guards' verdicts. CONSEQUENCE REJECT
Forbidden rescues
do not fold text and image strata — or harmful and benign files — into pooled denominators to move any figure; the strata are bound exactly as released
do not substitute verdicts from the release's other run directories (adaptive_run, carrier sweeps, rendering probes) to preserve a number bound to full_run
do not reinterpret ShieldGemma 2's deterministic text passes as missing data to shrink a denominator after seeing the results
do not cite the benign-image 250/250 union without attributing it to Llama Guard 3 Vision's 250/250 column, and do not cite the harmful strata without the benign strata
do not call an OR of the harness-normalized native labels a shared-event catch union, stack safety result, or three-independent- guard finding unless a source-defined event translation is added
do not recast these file counts as population estimates, with or without an interval, if the primary counts are challenged
Non-claims
not a shared-event catch statistic: the common `blocked` bit is a harness normalization of distinct native predicates, and no source-defined translation to a common event E has been identified
not a population estimate — the items derive from HarmBench and XSTest under the release's own construction, and no interval is offered because no sampled population is defined
not a ranking, endorsement, or indictment of any guard or vendor; the verdicts are at native, unmatched operating rules
not an observed deployed three-guard stack; interpreting the OR aggregation as a stack requires the separate full-exposure, parallel, fixed-operating-point assumptions
says nothing about the release's carrier-prompt, adversarial-UAP, or cross-VLM runs, which are separate artifacts with their own contracts
the counts are about the committed verdict bytes; the release's own changelog documents that ShieldGemma 2's image scores are sensitive to the text-rendering stack, so nothing here predicts what any guard would do under a different rendering environment
the zero three-guard image all-zero-bit count is a counting fact about these 200 released image items, not evidence of general image attack safety
+
Scope
Exactly the six committed verdict files of the full_run directory at the bound commit: 400 harmful and 500 benign items, each stratum half text and half image, one Boolean `blocked` adapter bit per item per guard. Let L be the release's harmful/benign label, A_s each guard's native `unsafe` predicate, B_s the harness mapping of that predicate to `blocked`, and O = OR_s B_s. The bound arithmetic reports B and O conditional on L. Because the harness uses `blocked` items to suppress target generation, O is also a valid counterfactual harness-block decision for these released rows under a fixed block-on-any rule. It does not establish a source-defined translation from every A_s to one shared catch event E: Llama Guard 3 Vision is a multimodal prompt/response classifier, while ShieldGemma 2 is an image-only three-policy classifier, and the pipeline merely maps each native `unsafe` label to `blocked`. Counting arithmetic only, at the guards' released native operating rules (the Llama guards emit labels autoregressively; ShieldGemma 2 blocks above a 0.5 policy-violation probability; the source documents no matched operating-point calibration, so M-ladder threshold comparability is unchanged). ShieldGemma 2's text-item bits are deterministic passes as committed and documented upstream, which is why its text column is zero — an explicit released outcome, not missing data. Strata are computed exactly as released and never folded; file totals are sums of the bound strata and carry no additional content. Computed by scripts/reanalyze_msbench.py from scripts/mjgd_reference.py; the expected block below is what CI re-asserts against the downloaded, hash-verified files.
executable fires when a bound verdict file on the default branch changes (6 bound sources)
executable fires when the release's printed per-guard metrics change
executable fires when the release's printed composition metrics change
executable fires when the guards' native adapter semantics change
executable fires when the harness label-to-block action changes
executable fires when the stated route or operating-point semantics change
manual the authors publish a corrected or extended full_run, print their own three-guard joint statistics, document a matched operating-point calibration, or publish a source-defined translation from all native predicates to one shared catch event
Falsifier
Recomputing from the bound, hash-verified files yields any count different from the expected block, or any pinned file no longer matches its recorded sha256, or any recomputed quantity that overlaps the release's printed metrics disagrees with the printed value, or the committed blocked columns are shown not to be the named guards' verdicts. CONSEQUENCE REJECT
Forbidden rescues
do not fold text and image strata — or harmful and benign files — into pooled denominators to move any figure; the strata are bound exactly as released
do not substitute verdicts from the release's other run directories (adaptive_run, carrier sweeps, rendering probes) to preserve a number bound to full_run
do not reinterpret ShieldGemma 2's deterministic text passes as missing data to shrink a denominator after seeing the results
do not cite the benign-image 250/250 union without attributing it to Llama Guard 3 Vision's 250/250 column, and do not cite the harmful strata without the benign strata
do not call an OR of the harness-normalized native labels a shared-event catch union, stack safety result, or three-independent- guard finding unless a source-defined event translation is added
do not recast these file counts as population estimates, with or without an interval, if the primary counts are challenged
Non-claims
not a shared-event catch statistic: the common `blocked` bit is a harness normalization of distinct native predicates, and no source-defined translation to a common event E has been identified
not a population estimate — the items derive from HarmBench and XSTest under the release's own construction, and no interval is offered because no sampled population is defined
not a ranking, endorsement, or indictment of any guard or vendor; the verdicts are at native, unmatched operating rules
not an observed deployed three-guard stack; interpreting the OR aggregation as a stack requires the separate full-exposure, parallel, fixed-operating-point assumptions
says nothing about the release's carrier-prompt, adversarial-UAP, or cross-VLM runs, which are separate artifacts with their own contracts
the counts are about the committed verdict bytes; the release's own changelog documents that ShieldGemma 2's image scores are sensitive to the text-rendering stack, so nothing here predicts what any guard would do under a different rendering environment
the zero three-guard image all-zero-bit count is a counting fact about these 200 released image items, not evidence of general image attack safety
Last owner review: 2026-11-02. These review
events cannot be executed by CI, and the record says so instead of borrowing the
executable triggers' credibility:
CC-001 — kernel semantics change outside the bound module file
CC-003 — E2 produces empirical results — the synthetic scoping here must be restated
CC-004 — witness API change outside the bound module file
CC-005 — a conforming dataset is collected — E2's status must be restated
CC-006 — real E2 data is collected — this claim is superseded by an E2 result
REL-001 — a public protocol revision is bound, a real E2 result activates or prevents activation, or participant recruitment begins
MC-001 — a new qualifying evaluation is published or reported, an author corrects a row, or a benchmark publishes a previously absent joint statistic
MC-002 — the authors release the full per-item dataset, state the subset's selection rule, or correct the released verdicts
MC-003 — a reader identifies an error in the stated Fréchet, finite-grid, or aggregate exclusive-coverage bounds, or a result is used outside the declared static full-exposure assumptions
MC-004 — the authors publish a corrected or extended full_run, print their own three-guard joint statistics, document a matched operating-point calibration, or publish a source-defined translation from all native predicates to one shared catch event
AF-001 — the report publishes the artifact share, states which comparison group the differences are against, corrects a figure, or expands the analysis to a different sample
GV-001 — the hosted build is removed, moves from https://cubits11.github.io/ghost-visualizer/, or stops serving the bound revision's Evidence Mode — the scope sentence is then wrong
GCE-001 — GCE repository resumed or reframed
SITE-001 — site redesign
SITE-002 — workflow triggers or cadence change
E3-001 — a third pilot is run on a pool where both guards' miss rates are intermediate
E3B-001 — the contamination status of either guard's training corpus becomes checkable
E6-001 — Anthropic publishes the model's output at the Table 2 median vector, releases the 3,259 five-vectors, or ships a kernel whose digest differs from the pinned bytes
E7B-001 — a pool with intermediate marginals and more than 21 harmful items becomes available, or the upstream repository's scores change at a new commit
E9-001 — a frozen multi-pool or new-guard contract (E10) is executed, either pinned revision is withdrawn, or G2's training data is disclosed
Claim review dates are calculated from the registry's last review plus its review window. Recorded renewals retain the earlier deadline, calculated from the pinned registry revision. Other dated promises come from the publication list. Static snapshot: 2026-10-17; JavaScript updates labels against today's date in New York.
Disposition 2026-10-17: The owner renewed the elapsed review window on October 17. The earlier September 23 deadline remains visible. Renewal does not establish a completed reader study or change the hypothesis into a validated result. Registry transition: 592ed51.
Disposition 2026-10-17: Review recorded 24 days late. Failed historical checks, the now-published E9 result, the renewed REL-001 window and outstanding reader/accessibility work are distinguished.
Disposition 2026-10-17: The owner renewed the elapsed review window on October 17. The earlier September 23 deadline remains visible. Renewal does not establish a completed reader study or change the hypothesis into a validated result. Registry transition: 592ed51.
Disposition 2026-10-17: Review recorded 24 days late. Failed historical checks, the now-published E9 result, the renewed REL-001 window and outstanding reader/accessibility work are distinguished.
An elapsed review window requests review. It does not automatically refute the claim or change its evidential status.
Research pipeline
Published result — claim registration pending · observed 2026-10-17
E9 — 1,000-item scoped result
E9’s 1,000-item result merged in PR 26. A post-result claim registration, E9-001, is proposed in draft PR 31 and is not in the registry. The result is not a completion of E2 and does not establish deployment safety.
The result discloses a verdict rule outside readable preregistration prose, selected pools, omitted calibration uncertainty and no independent replication. Its interval excludes zero but does not lie wholly beyond the 0.05 effect-size threshold.