Skip to content

S7: Remove the inert low_bin/high_bin from the Ignorance genome (C-28b) #33

Description

@Polichinel

Epic: #26 · Wave 2 · Closes: register C-28(b) · Blocked by: #32 · Needs a decision before implementation

Background

Ignorance declares a three-parameter genome (metric_catalog.py:88-89):

"Ignorance": MetricSpec(function=calculate_ignorance_score_native,
                        genome=("bins", "low_bin", "high_bin")),

resolve_metric_params treats all three as required — it raises if any is missing from both the model overrides and the profile (metric_catalog.py:180-190). BASE_PROFILE duly supplies all three:

"Ignorance": {
    "bins": [0, 0.5, 2.5, 5.5, 10.5, 25.5, 50.5, 100.5, 250.5, 500.5, 1000.5],
    "low_bin": 0,
    "high_bin": 10000,
},

Two of the three do nothing. The kernel passes them to np.histogram_bin_edges:

edges = np.histogram_bin_edges(preds, bins=bins, range=(low_bin, high_bin))   # line 162

and NumPy ignores range= entirely when bins is a sequence. Verified 2026-08-02:

b = [0, 0.5, 2.5, 5.5, 10.5, 25.5, 50.5, 100.5, 250.5, 500.5, 1000.5]
np.array_equal(np.histogram_bin_edges(p, bins=b, range=(0, 10000)),
               np.histogram_bin_edges(p, bins=b, range=(0, 1)))
# True  ← range has no effect

So the genome contract advertises three tuning knobs, two of which are inert. A researcher who widens high_bin to fix the out-of-range crash from #32 would see no change at all — a false sense of control over the metric, and directly contrary to the epic doctrine's "no magic numbers, no defaults, no silent no-ops".

Note documentation/CICs/MetricCatalog.md states the invariant in one direction only — "All hyperparameters required by a metric function must be declared in the spec's genome tuple" — with no converse requirement that every declared param actually be used. That gap is what let this through.

⚠ Decision required before implementing

Two viable options. Decide and record the choice in this issue before writing code.

Option A — Drop low_bin/high_bin (recommended).
genome=("bins",); remove both keys from BASE_PROFILE and HYDRANET_UCDP_PROFILE if present. Explicit bin edges are the only form actually supported, and after #32 the edges are also what the error message reports. Simplest, and makes the contract honest.
Cost: a breaking change for any external profile supplying those keys — resolve_metric_params rejects unknown params (metric_catalog.py:169-174), so a stale profile will now raise. That is correct fail-loud behaviour, but it must be noted in #38 and governed by #28's deprecation policy.

Option B — Honour them.
Keep the genome and make the kernel use range only when bins is an integer count, raising if bins is a sequence and low_bin/high_bin are supplied (contradictory configuration). Preserves the tuning knob but adds a branch and a second supported bin-specification mode that nothing currently uses.

Recommendation: A. No caller uses integer bins; keeping a parameter alive to preserve an unused code path adds surface without value.

The work (assuming Option A)

  1. metric_catalog.py:88-89genome=("bins",).
  2. profiles/base.py:34-38 — remove low_bin and high_bin from the Ignorance entry.
  3. Check profiles/hydranet_ucdp.py — it spreads **BASE_PROFILE, so it inherits the change; confirm no local override.
  4. native_metric_calculators.py:142-173 — drop the low_bin/high_bin parameters from the signature and the np.histogram_bin_edges call. Keep the keyword-only * marker so ADR-042's no-defaults rule still holds.
  5. Consider adding the converse invariant to MetricCatalog.md (S13: Update Intent Contracts for changed behaviour (ADR-021) #39): every declared genome param must be consumed by its function. A test for this is in scope for S14: Restore a documentation-contract test guard and wire validate_docs.sh into CI #40 if cheap.

Acceptance criteria

  • The decision (A or B) is recorded in this issue with its rationale before code is written
  • Option A: Ignorance genome is ("bins",); low_bin/high_bin gone from all profiles and from the kernel signature
  • resolve_metric_params("Ignorance", {}, BASE_PROFILE) returns only the params the function actually consumes
  • A profile still supplying low_bin/high_bin fails loudly (existing unknown-param rejection) — assert this with a Red test rather than leaving it implicit
  • Ignorance values are numerically unchanged for in-range input — test_ignorance_known_bin_distribution passes unmodified
  • TestCatalogStructuralIntegrity passes — in particular test_base_profile_covers_all_implemented_genomes
  • No logging added (Level 0 exempt)
  • Full suite green, ruff clean

Validation

conda run --name views_pipeline pytest tests/test_metric_catalog.py -v
conda run --name views_pipeline pytest tests/test_metric_calculators.py -v -k "ignorance or Ignorance"
conda run --name views_pipeline pytest tests/ -q

conda run --name views_pipeline python - <<'PY'
from views_evaluation.evaluation.metric_catalog import METRIC_CATALOG, resolve_metric_params
from views_evaluation.profiles.base import BASE_PROFILE
print("genome:", METRIC_CATALOG["Ignorance"].genome)
print("resolved:", resolve_metric_params("Ignorance", {}, BASE_PROFILE))
try:
    resolve_metric_params("Ignorance", {"high_bin": 10000}, BASE_PROFILE)
    print("SILENT PASS - BUG: stale param accepted")
except ValueError as e:
    print("stale param rejected OK ->", str(e)[:80])
PY

Dependencies

Files

  • views_evaluation/evaluation/metric_catalog.py:88-89
  • views_evaluation/evaluation/native_metric_calculators.py:142-173
  • views_evaluation/profiles/base.py:34-38
  • views_evaluation/profiles/hydranet_ucdp.py (verify only)
  • tests/test_metric_catalog.py, tests/test_metric_calculators.py

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    implementationCode changeneeds-decisionBlocked on a human decision before work can startstoryA single, independently implementable and reviewable unit of an epic

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions