Skip to content

Strict mypy validation fails on unified benchmark report types #108

Closed

Description

@MatthiasLew

Problem

The required python scripts/dev.py --check workflow stopped at mypy with eight errors in scripts/unified_benchmark.py. Reproduced using the repository's isolated environment and locked tools against main commit 5c9f1b7bd4677d32b1650ebd62519044072e599a.

Reproduction

Run:

python scripts/dev.py --check

Or, within the prepared locked environment:

python -m mypy src tests scripts

Observed failures

  • len(idx.get("entries", [])) receives an object rather than a sized value.
  • The inferred nested report dictionary does not satisfy the declared return type.
  • Metadata and benchmark entries cannot be indexed/iterated through dict[str, object].
  • Two type-ignore comments are unused or do not cover the actual errors.

Observed result: eight errors in one file; the full development gate stops before later checks.

Expected: the complete strict mypy command passes without changing the benchmark JSON or console output format.

Proposed fix

Define TypedDict contracts for benchmark entries, metadata and the full report. Annotate construction, result lists and the function return value; express the existing sized-index expectation with an appropriate type cast; remove the ineffective ignores.

Acceptance criteria

  • python -m mypy src tests scripts passes.
  • Repository Ruff and git diff --check pass.
  • Existing benchmark JSON structure and table formatting remain unchanged.
  • No benchmark workload is required against real user repositories to validate this typing-only change.

Existing patch and evidence

Patch preserved in PR #106 (closed in favor of this issue at the maintainer's request), commit 2c1cc0d121dffc884542f74e5cf8ba6efcb30cb3.

With that patch, strict mypy passed for all 197 source files; repository Ruff and diff checks passed. PR CI, Docs and CodeQL subsequently passed. No production benchmark workload was run.

This issue replaces PR #106 as the work item; no patch has been merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions