Skip to content

fix(benchmark): secure maintained offline reproduction without rewriting history (#89) - #106

Merged
phense merged 2 commits into
mainfrom
fix/maintained-reranker-security-89
Nov 4, 2026
Merged

phense merged 2 commits into
mainfrom
fix/maintained-reranker-security-89

Conversation

@phense

@phense phense commented Nov 4, 2026

Copy link
Copy Markdown
Owner

The archived reranker requirements triggered eight security alerts, and dependency bot PRs #85/#86 would rewrite the dependency identity of old measurements. This adds a separate maintained offline ONNX reproduction path, preserves every historical file, and documents all eight advisory dispositions.

The benchmark-only runner reuses the unchanged installed OnnxRanker, verifies/stages the three original ONNX model/tokenizer identities, requires hash-pinned runtime versions, and refuses Transformers, setuptools, Torch and custom-code v3.5 before loading. An owned child bounds native inference; exclusive output directories retain partial successes, failures and unattempted cases. No model download, production profile change, SQL change or data rewrite is introduced.

Release 0.6.32 includes the required metadata and source-version compatibility labels. The operator guide covers isolated wheel rebuild, retained failures, dependency upgrades and code-environment rollback.

Validation:

  • 118 targeted tests passed, covering native loader/failure behavior and existing reranker/evaluation contracts.
  • 54 release/compatibility tests passed, including populated archived-store upgrades, limited-owner operation and independent backup restores; 17 documentation tests passed.
  • Fresh noneditable wheel environment: 44 hash-verified runtime wheels + application, dependency check exit0, zero forbidden packages. Actual native smoke test uses an own synthetic ONNX graph, two cases, expected literal order, no pretrained model download/inference or quality claim.
  • Independent complete 30-file code/security review: 0 Critical/High/Medium/Low. Procedure review: 0 Important/Minor. Subsequent changes are evidence, glossary/status and archived specification only; all reviewed executable/lock/manifest inputs are hash-identical.
  • All 177 historical benchmark files and 39 SQL migrations unchanged. Built wheel runtime/assets/SQL match source. Offline lock/export and final local-link/spec-sync checks passed.

The full suite was not repeated for this benchmark-only addition and mechanical release labels. Raw observations, terminal results, package/source/model identities and independent reviews are published under docs/benchmarks/maintained-rerankers/verification/.

The historical four-model protocol remains failed (2,338 actual / 2,688 prescribed; 350 unattempted v3.5 observations). This does not claim v3.5 compatibility, pretrained-model quality/performance, or production deployment. Eight alerts attached to preserved archival requirements can remain visible; no advisory was dismissed or hidden. Supersede #85/#86 after this replacement merges rather than editing the frozen file.

Closes #89.

Next, separately: #83 measured BGE versus Jina v5 omni small/nano comparison using existing model-bound generation workflows; no production embedding cutover is authorized by this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1] Secure the maintained reranker benchmark reproduction path without rewriting frozen evidence

1 participant