Skip to content

docs: flag that the backend mirrors the tested Python versions - #150

Merged
fargito merged 1 commit into
masterfrom
cod-3506-runtime-matrix-pointer
Nov 6, 2026
Merged

fargito merged 1 commit into
masterfrom
cod-3506-runtime-matrix-pointer

Conversation

@fargito

@fargito fargito commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

CodSpeed keeps its own per-release copy of the Python versions each pytest-codspeed release is tested with, to warn on runs outside them. Flag it next to MIN_SUPPORTED_PYTHON_VERSION and MAX_SUPPORTED_PYTHON_VERSION so a change to the bounds reaches it.

Refs COD-3506

@codspeed

codspeed Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 5.79%

3 warnings, 1 note

⚠️ 6 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 3 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 12 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

ℹ️ 37 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 3 improved benchmarks
❌ 12 (👁 12) regressed benchmarks
✅ 262 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Memory test_multiprocessing_map[10000] 3.6 MB 3.1 MB +14.86%
⚡ Walltime test_fs_read[1000] 913.9 ns 847.9 ns +7.78%
⚡ Walltime test_combination_sum[candidates0-8] 11.5 µs 10.8 µs +6.11%
👁 Walltime test_solve_power_sum[13-2] 7 µs 7.7 µs -9.83%
👁 Walltime test_multiprocessing_map[100000] 370.5 ms 495.3 ms -25.18%
👁 Walltime test_sum_of_squares[sum_of_squares_sum_comprehension_power] 221.8 µs 237.6 µs -6.66%
👁 Walltime test_noop_pass 469.3 ns 520.2 ns -9.78%
👁 Walltime test_pipe_communication[100] 5.9 ms 6.3 ms -6.47%
👁 Walltime test_make_lowpass 5.5 µs 5.9 µs -6.01%
👁 Walltime test_multiprocessing_map[10000] 60.2 ms 64.3 ms -6.4%
👁 Walltime test_multiprocessing_map[1000] 54.9 ms 58.6 ms -6.25%
👁 Walltime test_tcp_connection[1.1.1.1-53] 823.5 µs 951 µs -13.41%
👁 Walltime test_pipe_communication[1000] 6 ms 6.5 ms -7.87%
👁 Memory test_multiprocessing_map[1000] 3.4 MB 3.6 MB -5.02%
👁 Memory test_multiprocessing_map[100000] 6.1 MB 6.5 MB -6.34%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-3506-runtime-matrix-pointer (15d46b4) with master (7d20221)

Open in CodSpeed

@fargito
fargito requested a review from not-matthias November 5, 2026 19:36
@fargito
fargito marked this pull request as ready for review November 5, 2026 19:37
@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] This comment-only change appears safe to merge.

Summary

Adds a reminder beside MIN_SUPPORTED_PYTHON_VERSION and MAX_SUPPORTED_PYTHON_VERSION to notify the team when the bounds change, since the CodSpeed app tracks tested Python versions per release.

  • The Python version note now includes the CodSpeed app.

Reviews (1) · Last reviewed commit: "docs: note that the CodSpeed app tracks ..." · Reviewed by Greptile

Comment thread src/pytest_codspeed/plugin.py Outdated
Comment on lines +43 to +44
# pyproject.toml, and the CI test matrix. The CodSpeed app keeps its own copy
# per release to flag untested runs: tell the team when they change.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could have something like this to make it stick out a bit more:

IMPORTANT: Keep in sync with the support matrix on the backend! 

CodSpeed keeps its own per-release copy of the Python versions each
`pytest-codspeed` release is tested with, to warn on runs outside them. Flag it
next to `MIN_SUPPORTED_PYTHON_VERSION` and `MAX_SUPPORTED_PYTHON_VERSION` so a
change to the bounds reaches it.

Refs COD-3506
@fargito
fargito force-pushed the cod-3506-runtime-matrix-pointer branch from 05099d6 to 15d46b4 Compare November 6, 2026 10:37
@fargito fargito changed the title docs: note that the CodSpeed app tracks the tested Python versions docs: flag that the backend mirrors the tested Python versions Nov 6, 2026
@fargito
fargito merged commit 15d46b4 into master Nov 6, 2026
61 checks passed
@fargito
fargito deleted the cod-3506-runtime-matrix-pointer branch November 6, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants