Skip to content

Add genuine model-backed acceptance and clarify mocked integration coverage #35

Description

@cgfixit

Review follow-up, 2026-09-12. Rechecked origin at 98f267697b74.

Evidence and impact

Python 3.12.14 editable installation and 121 unit tests passed (89% coverage). CI run 34657207356 passed the unit matrix; its integration job was skipped.

The job is labeled “Integration Tests (full BERT pipeline),” but tests/integration/test_e2e.py patches SentenceTransformer and AutoTokenizer with mocks. Enabling this job alone does not prove weights/tokenizer compatibility or actual embedding inference. Its comment mentions a schedule, while the job condition permits workflow_dispatch or a commit-message marker.

This is a verification/claim gap, not a reproduced real-model crash.

Proposed implementation

  1. Keep the fast mocked pipeline tests and label them accurately. Separate model-backed checks with a clear opt-in marker/job.
  2. Choose the shipped supported model and record its identifier/revision, tokenizer revision, Python, torch and sentence-transformers versions. Permit a pre-populated local cache; make downloads explicit and bounded. Use synthetic/public sample text only.
  3. Run the real model loader and tokenizer without monkeypatches, call actual embedding/extraction paths, and check expected dimensions, finite values, nonempty meaningful output, correct offsets/provenance and repeat-run behavior. Test empty/oversized input according to the existing contract.
  4. Add failure-path coverage for unavailable/corrupt weights and offline cache misses. Report an explicit failed or unavailable real-model check rather than quietly switching to a mock.
  5. Keep a small hand-labeled extraction set separate from the installation smoke. Report precision/recall or a similarly defined task metric; record cold/warm latency and peak memory on a named machine. Do not treat successful inference as proof of semantic quality.
  6. Align workflow comments, trigger conditions and documentation. A manual acceptance job is sufficient initially; do not silently impose model downloads on every contributor run.

Acceptance criteria

  • Existing mocked suite remains green and accurately named.
  • One bounded run uses real weights and tokenizer and records reproducible environment details.
  • The run cannot pass if the real model never loads.
  • Model availability, inference correctness and task quality are reported separately.
  • No private input text or credentials appear in CI artifacts.

Implementation should start by refreshing origin and checking this finding against the new head. Preserve existing safety controls. This issue/comment records findings and proposed acceptance criteria; it does not claim the fixes or unperformed live checks are complete.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions