Repository navigation
fix(go-runner): count all timed segments of a b.Loop() iteration - #69
Conversation
[Medium impact] The change affects the benchmark rounds CodSpeed records for Go runs. Fix the new test’s missing Go-version check before merging.
|
In b.Loop() mode, each iteration is saved as one round. The round time was the time since the last StartTimer call, so the timed work before a StopTimer/StartTimer pair in the same iteration was dropped. A benchmark that times its work, stops the timer for a teardown, and restarts it before the next b.Loop() reported only the few hundred nanoseconds between that StartTimer and b.Loop(), while go test reported the real time per operation. Save the timed duration accumulated since the previous round instead. b.N rounds are unchanged: runN resets the timer before each round, so the delta is b.duration.
ab609d2 to
5a63029
Compare
Merging this PR will improve performance by 19.05%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [CodSpeedHQ/action](https://redirect.github.com/CodSpeedHQ/action) | action | patch | `v5.6.0` → `v5.6.3` | | [github/codeql-action](https://redirect.github.com/github/codeql-action) | action | patch | `v4.39.1` → `v4.39.2` | --- ### Release Notes <details> <summary>CodSpeedHQ/action</summary> ### [`v5.6.3`](https://redirect.github.com/CodSpeedHQ/action/releases/tag/v5.6.3) [Compare Source](https://redirect.github.com/CodSpeedHQ/action/compare/v5.6.1...v5.6.3) Bumps the CodSpeed runner to [v5.6.3](https://redirect.github.com/CodSpeedHQ/codspeed/releases/tag/v5.6.3), including [v5.6.2](https://redirect.github.com/CodSpeedHQ/codspeed/releases/tag/v5.6.2). ##### Highlights ##### Go benchmark times match `go test` `b.Loop()` benchmarks that stop and restart the timer inside an iteration now report all of their timed work. A round previously counted only the time since the most recent `StartTimer`, so results for those benchmarks will move. ##### Unsupported `go test` flags no longer fail the run `go test -p 1 ./... -bench .` used to strip `-p`, parse `1` as a package, and fail with `package 1 is not in std` after every benchmark had passed. Flags and their values are now dropped together. ##### Fixed randomness in CPU Simulation `getrandom()`, `/dev/urandom`, `AT_RANDOM` and hardware RNG instructions now return the same values on every CPU Simulation run, removing one source of run-to-run variance. ##### Fixes - Simulation: fix SIGILL on arm64 when programs read previously unsupported CPU ID registers or `TPIDRRO_EL0` ([#&open-telemetry#8203;59](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/59)). - Go: benchmarks that draw from the global `math/rand` now start from the same seed on every run, so their results no longer vary between runs ([#&open-telemetry#8203;596](https://redirect.github.com/CodSpeedHQ/codspeed/pull/596)). ##### 🚀 Features - Seed Go's global math/rand deterministically ([#&open-telemetry#8203;596](https://redirect.github.com/CodSpeedHQ/action/issues/596)) by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [#&open-telemetry#8203;596](https://redirect.github.com/CodSpeedHQ/runner/pull/596) - Always enable deterministic RNG ([#&open-telemetry#8203;600](https://redirect.github.com/CodSpeedHQ/action/issues/600)) by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [#&open-telemetry#8203;600](https://redirect.github.com/CodSpeedHQ/runner/pull/600) ##### 🐛 Bug Fixes - Bump go-runner to [1.4.0](https://redirect.github.com/CodSpeedHQ/codspeed-go/releases/tag/v1.4.0) ([#&open-telemetry#8203;605](https://redirect.github.com/CodSpeedHQ/action/issues/605)) by [@&open-telemetry#8203;adriencaccia](https://redirect.github.com/adriencaccia) in [#&open-telemetry#8203;605](https://redirect.github.com/CodSpeedHQ/runner/pull/605) - fix(cli): drop the value of unsupported go test flags by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [CodSpeedHQ/codspeed-go#67](https://redirect.github.com/CodSpeedHQ/codspeed-go/pull/67) - fix(go-runner): count all timed segments of a b.Loop() iteration by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [CodSpeedHQ/codspeed-go#69](https://redirect.github.com/CodSpeedHQ/codspeed-go/pull/69) ##### ⚙️ Internals - Bump pinned valgrind-codspeed to [3.26.0-0codspeed12](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/releases/tag/3.26.0-0codspeed12) ([#&open-telemetry#8203;604](https://redirect.github.com/CodSpeedHQ/action/issues/604)) by [@&open-telemetry#8203;adriencaccia](https://redirect.github.com/adriencaccia) in [#&open-telemetry#8203;604](https://redirect.github.com/CodSpeedHQ/runner/pull/604) - feat: add --deterministic-rng to make client randomness reproducible by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [CodSpeedHQ/valgrind-codspeed#54](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/54) - feat(vex): serve arm64 RNDR and RNDRRS with --deterministic-rng by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [CodSpeedHQ/valgrind-codspeed#56](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/56) - fix(vex): decode arm64 MRS of all EL0-visible ID registers by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [CodSpeedHQ/valgrind-codspeed#59](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/59) **Full Runner Changelog**: <https://github.com/CodSpeedHQ/codspeed/blob/main/CHANGELOG.md> #### What's Changed - chore: bump runner version to 5.6.2 by [@&open-telemetry#8203;github-actions](https://redirect.github.com/github-actions)\[bot] in [#&open-telemetry#8203;255](https://redirect.github.com/CodSpeedHQ/action/pull/255) - chore: bump runner version to 5.6.3 by [@&open-telemetry#8203;github-actions](https://redirect.github.com/github-actions)\[bot] in [#&open-telemetry#8203;256](https://redirect.github.com/CodSpeedHQ/action/pull/256) **Full Changelog**: <CodSpeedHQ/action@v5.6.1...v5.6.3> ### [`v5.6.1`](https://redirect.github.com/CodSpeedHQ/action/releases/tag/v5.6.1) [Compare Source](https://redirect.github.com/CodSpeedHQ/action/compare/v5.6.0...v5.6.1) Bumps the CodSpeed runner to [v5.6.1](https://redirect.github.com/CodSpeedHQ/codspeed/releases/tag/v5.6.1). ##### Fixes - Walltime: the runner no longer hangs after the benchmark exits when a Python process forks and leaves a corrupted perf jitdump ([#&open-telemetry#8203;599](https://redirect.github.com/CodSpeedHQ/codspeed/pull/599)) - Simulation: stop the libpython symlink warning from firing falsely ([#&open-telemetry#8203;594](https://redirect.github.com/CodSpeedHQ/codspeed/pull/594)) ##### 🐛 Bug Fixes - Stop the libpython symlink warning from firing falsely by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in [#&open-telemetry#8203;594](https://redirect.github.com/CodSpeedHQ/runner/pull/594) ##### 💼 Other - Bump submodule to not loop on corrupted jit dump by [@&open-telemetry#8203;GuillaumeLagrange](https://redirect.github.com/GuillaumeLagrange) in [#&open-telemetry#8203;599](https://redirect.github.com/CodSpeedHQ/runner/pull/599) **Full Runner Changelog**: <https://github.com/CodSpeedHQ/codspeed/blob/main/CHANGELOG.md> #### What's Changed - docs(examples): update the example workflows by [@&open-telemetry#8203;SuperMuel](https://redirect.github.com/SuperMuel) in [#&open-telemetry#8203;252](https://redirect.github.com/CodSpeedHQ/action/pull/252) - chore: bump runner version to 5.6.1 by [@&open-telemetry#8203;github-actions](https://redirect.github.com/github-actions)\[bot] in [#&open-telemetry#8203;254](https://redirect.github.com/CodSpeedHQ/action/pull/254) #### New Contributors - [@&open-telemetry#8203;SuperMuel](https://redirect.github.com/SuperMuel) made their first contribution in [#&open-telemetry#8203;252](https://redirect.github.com/CodSpeedHQ/action/pull/252) **Full Changelog**: <CodSpeedHQ/action@v5.6.0...v5.6.1> </details> <details> <summary>github/codeql-action</summary> ### [`v4.39.2`](https://redirect.github.com/github/codeql-action/releases/tag/v4.39.2) [Compare Source](https://redirect.github.com/github/codeql-action/compare/v4.39.1...v4.39.2) - Update default CodeQL bundle version to [2.27.4](https://redirect.github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.4). [#&open-telemetry#8203;4284](https://redirect.github.com/github/codeql-action/pull/4284) - *Upcoming breaking change*: CodeQL version 2.22.3 and earlier were discontinued on 16 October 2026 alongside GitHub Enterprise Server 3.18, and will be unsupported by the next minor release of the CodeQL Action. Added a deprecation warning for customers using these versions of CodeQL. [#&open-telemetry#8203;4266](https://redirect.github.com/github/codeql-action/pull/4266) - Starting with macOS 27 and Xcode 27, traced analyses are unsupported. See [this blog post for more information](https://github.blog/changelog/2026-10-19-codeql-changes-for-macos-27-xcode-27-and-swift-6-4/). The CodeQL Action will now try to identify if a workflow is about to perform a traced analysis on version 27 or above of macOS or Xcode. If so, the CodeQL Action will emit a corresponding warning that will also be visible on the [Tool Status Page](https://docs.github.com/en/code-security/how-tos/find-and-fix-code-vulnerabilities/manage-your-configuration/use-the-tools-status-page-for-code-scanning). This check will be rolled out following this release of the CodeQL Action. [#&open-telemetry#8203;4221](https://redirect.github.com/github/codeql-action/pull/4221) - The undocumented TRAP cache cleanup feature that could be enabled using the `CODEQL_ACTION_CLEANUP_TRAP_CACHES` environment variable has been removed, as announced in the release notes for CodeQL Action 4.35.2. The `analyze` Action now warns if this environment variable is set. If you used TRAP cache cleanup to limit Actions cache usage, we recommend disabling TRAP caching by passing the `trap-caching: false` input to the `init` Action. [#&open-telemetry#8203;4243](https://redirect.github.com/github/codeql-action/pull/4243) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "on tuesday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/open-telemetry/opentelemetry-collector). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMDIuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjIwNC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZWJvdCJdfQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
In
b.Loop()mode, each iteration is saved as one round. The round time was the time since the lastStartTimercall. If an iteration stops and restarts the timer, the round drops every timed segment before the lastStartTimer:The uploaded walltime results then only contain the few hundred nanoseconds between that
StartTimerand the nextb.Loop(). The ns/op thatgo testprints was correct because it usesb.duration, so the two numbers did not match.SaveMeasurementnow saves the timed duration accumulated since the previous round (b.duration - b.savedDuration).b.Nrounds are unchanged:runNresets the timer before each round, so the delta isb.duration.Local repro with 1 ms of timed work and untimed setup and teardown in each iteration:
go testns/opb.Loop()withStopTimer/StartTimerb.Loop()without timer callsBenchmarkWithTeardownInLoopModerninexample/covers this pattern.test_b_loop_round_includes_work_before_stop_timerchecks that its round includes the 1 ms of timed work. Without the fix, the test fails with a round of 11.8 µs.Closes COD-4085