Skip to content

fix(go-runner): count all timed segments of a b.Loop() iteration - #69

Merged
not-matthias merged 1 commit into
mainfrom
cod-4085-investigate-suspicious-go-walltime-results-with-timer
Nov 2, 2026
Merged

not-matthias merged 1 commit into
mainfrom
cod-4085-investigate-suspicious-go-walltime-results-with-timer

Conversation

@not-matthias

Copy link
Copy Markdown
Member

In b.Loop() mode, each iteration is saved as one round. The round time was the time since the last StartTimer call. If an iteration stops and restarts the timer, the round drops every timed segment before the last StartTimer:

for b.Loop() {
	work() // timed, but dropped from the round

	b.StopTimer()
	teardown()
	b.StartTimer() // the round starts here
}

The uploaded walltime results then only contain the few hundred nanoseconds between that StartTimer and the next b.Loop(). The ns/op that go test prints was correct because it uses b.duration, so the two numbers did not match.

SaveMeasurement now saves the timed duration accumulated since the previous round (b.duration - b.savedDuration). b.N rounds are unchanged: runN resets the timer before each round, so the delta is b.duration.

Local repro with 1 ms of timed work and untimed setup and teardown in each iteration:

Benchmark go test ns/op Uploaded median before Uploaded median after
b.Loop() with StopTimer/StartTimer 1,000,289 111 1,000,906
b.Loop() without timer calls 1,000,182 1,000,195 1,000,134

BenchmarkWithTeardownInLoopModern in example/ covers this pattern. test_b_loop_round_includes_work_before_stop_timer checks that its round includes the 1 ms of timed work. Without the fix, the test fails with a round of 11.8 µs.

Closes COD-4085

@not-matthias
not-matthias marked this pull request as ready for review November 2, 2026 10:09
@greptile-apps

greptile-apps Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The change affects the benchmark rounds CodSpeed records for Go runs.

Fix the new test’s missing Go-version check before merging.

Fix All in Claude CodeFindings

  1. P1 Go 1.24 CI fails ▶
Fix with agent prompt
### Issue 1
go-runner/src/integration_tests.rs:117
The new test runs an example that requires Go 1.25 without the `can_build_project` check used by `test_build_and_run`. The runner creates its overlay against the installed Go tree before running the example. If Go switches to 1.25 to satisfy `example/go.mod`, those paths no longer match, so the test gets no CodSpeed results and fails.

Add the same version check before calling `run_benchmarks`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR makes each b.Loop() round include all timed segments, rather than only the last segment.

  • Each b.Loop() round includes all of its timed work.

Reviews (1) · Last reviewed commit: "fix(go-runner): count all timed segments..." · Reviewed by Greptile

Comment thread go-runner/src/integration_tests.rs Resolved
In b.Loop() mode, each iteration is saved as one round. The round time
was the time since the last StartTimer call, so the timed work before a
StopTimer/StartTimer pair in the same iteration was dropped. A benchmark
that times its work, stops the timer for a teardown, and restarts it
before the next b.Loop() reported only the few hundred nanoseconds
between that StartTimer and b.Loop(), while go test reported the real
time per operation.

Save the timed duration accumulated since the previous round instead.
b.N rounds are unchanged: runN resets the timer before each round, so
the delta is b.duration.
@not-matthias
not-matthias force-pushed the cod-4085-investigate-suspicious-go-walltime-results-with-timer branch from ab609d2 to 5a63029 Compare November 2, 2026 10:15
@codspeed

codspeed Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 19.05%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚡ 1 improved benchmark
✅ 48 untouched benchmarks
🆕 1 new benchmark
⏩ 1 skipped benchmark1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ BenchmarkSleep100ns_Loop 300 ns 252 ns +19.05%
🆕 BenchmarkWithTeardownInLoopModern N/A 1.1 ms N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-4085-investigate-suspicious-go-walltime-results-with-timer (5a63029) with main (a5f6b7c)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

@not-matthias
not-matthias requested a review from lvaroqui November 2, 2026 10:18
@not-matthias
not-matthias merged commit 84ece6e into main Nov 2, 2026
23 checks passed
@not-matthias
not-matthias deleted the cod-4085-investigate-suspicious-go-walltime-results-with-timer branch November 2, 2026 17:42
gimsesu pushed a commit to gimsesu/opentelemetry-collector that referenced this pull request Nov 3, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [CodSpeedHQ/action](https://redirect.github.com/CodSpeedHQ/action) |
action | patch | `v5.6.0` → `v5.6.3` |
|
[github/codeql-action](https://redirect.github.com/github/codeql-action)
| action | patch | `v4.39.1` → `v4.39.2` |

---

### Release Notes

<details>
<summary>CodSpeedHQ/action</summary>

###
[`v5.6.3`](https://redirect.github.com/CodSpeedHQ/action/releases/tag/v5.6.3)

[Compare
Source](https://redirect.github.com/CodSpeedHQ/action/compare/v5.6.1...v5.6.3)

Bumps the CodSpeed runner to
[v5.6.3](https://redirect.github.com/CodSpeedHQ/codspeed/releases/tag/v5.6.3),
including
[v5.6.2](https://redirect.github.com/CodSpeedHQ/codspeed/releases/tag/v5.6.2).

##### Highlights

##### Go benchmark times match `go test`

`b.Loop()` benchmarks that stop and restart the timer inside an
iteration now report all of their timed work. A round previously counted
only the time since the most recent `StartTimer`, so results for those
benchmarks will move.

##### Unsupported `go test` flags no longer fail the run

`go test -p 1 ./... -bench .` used to strip `-p`, parse `1` as a
package, and fail with `package 1 is not in std` after every benchmark
had passed. Flags and their values are now dropped together.

##### Fixed randomness in CPU Simulation

`getrandom()`, `/dev/urandom`, `AT_RANDOM` and hardware RNG instructions
now return the same values on every CPU Simulation run, removing one
source of run-to-run variance.

##### Fixes

- Simulation: fix SIGILL on arm64 when programs read previously
unsupported CPU ID registers or `TPIDRRO_EL0`
([#&open-telemetry#8203;59](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/59)).
- Go: benchmarks that draw from the global `math/rand` now start from
the same seed on every run, so their results no longer vary between runs
([#&open-telemetry#8203;596](https://redirect.github.com/CodSpeedHQ/codspeed/pull/596)).

##### 🚀 Features

- Seed Go's global math/rand deterministically
([#&open-telemetry#8203;596](https://redirect.github.com/CodSpeedHQ/action/issues/596))
by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[#&open-telemetry#8203;596](https://redirect.github.com/CodSpeedHQ/runner/pull/596)
- Always enable deterministic RNG
([#&open-telemetry#8203;600](https://redirect.github.com/CodSpeedHQ/action/issues/600))
by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[#&open-telemetry#8203;600](https://redirect.github.com/CodSpeedHQ/runner/pull/600)

##### 🐛 Bug Fixes

- Bump go-runner to
[1.4.0](https://redirect.github.com/CodSpeedHQ/codspeed-go/releases/tag/v1.4.0)
([#&open-telemetry#8203;605](https://redirect.github.com/CodSpeedHQ/action/issues/605))
by [@&open-telemetry#8203;adriencaccia](https://redirect.github.com/adriencaccia) in
[#&open-telemetry#8203;605](https://redirect.github.com/CodSpeedHQ/runner/pull/605)
- fix(cli): drop the value of unsupported go test flags by
[@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[CodSpeedHQ/codspeed-go#67](https://redirect.github.com/CodSpeedHQ/codspeed-go/pull/67)
- fix(go-runner): count all timed segments of a b.Loop() iteration by
[@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[CodSpeedHQ/codspeed-go#69](https://redirect.github.com/CodSpeedHQ/codspeed-go/pull/69)

##### ⚙️ Internals

- Bump pinned valgrind-codspeed to
[3.26.0-0codspeed12](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/releases/tag/3.26.0-0codspeed12)
([#&open-telemetry#8203;604](https://redirect.github.com/CodSpeedHQ/action/issues/604))
by [@&open-telemetry#8203;adriencaccia](https://redirect.github.com/adriencaccia) in
[#&open-telemetry#8203;604](https://redirect.github.com/CodSpeedHQ/runner/pull/604)
- feat: add --deterministic-rng to make client randomness reproducible
by [@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[CodSpeedHQ/valgrind-codspeed#54](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/54)
- feat(vex): serve arm64 RNDR and RNDRRS with --deterministic-rng by
[@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[CodSpeedHQ/valgrind-codspeed#56](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/56)
- fix(vex): decode arm64 MRS of all EL0-visible ID registers by
[@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[CodSpeedHQ/valgrind-codspeed#59](https://redirect.github.com/CodSpeedHQ/valgrind-codspeed/pull/59)

**Full Runner Changelog**:
<https://github.com/CodSpeedHQ/codspeed/blob/main/CHANGELOG.md>

#### What's Changed

- chore: bump runner version to 5.6.2 by
[@&open-telemetry#8203;github-actions](https://redirect.github.com/github-actions)\[bot]
in [#&open-telemetry#8203;255](https://redirect.github.com/CodSpeedHQ/action/pull/255)
- chore: bump runner version to 5.6.3 by
[@&open-telemetry#8203;github-actions](https://redirect.github.com/github-actions)\[bot]
in [#&open-telemetry#8203;256](https://redirect.github.com/CodSpeedHQ/action/pull/256)

**Full Changelog**:
<CodSpeedHQ/action@v5.6.1...v5.6.3>

###
[`v5.6.1`](https://redirect.github.com/CodSpeedHQ/action/releases/tag/v5.6.1)

[Compare
Source](https://redirect.github.com/CodSpeedHQ/action/compare/v5.6.0...v5.6.1)

Bumps the CodSpeed runner to
[v5.6.1](https://redirect.github.com/CodSpeedHQ/codspeed/releases/tag/v5.6.1).

##### Fixes

- Walltime: the runner no longer hangs after the benchmark exits when a
Python process forks and leaves a corrupted perf jitdump
([#&open-telemetry#8203;599](https://redirect.github.com/CodSpeedHQ/codspeed/pull/599))
- Simulation: stop the libpython symlink warning from firing falsely
([#&open-telemetry#8203;594](https://redirect.github.com/CodSpeedHQ/codspeed/pull/594))

##### 🐛 Bug Fixes

- Stop the libpython symlink warning from firing falsely by
[@&open-telemetry#8203;not-matthias](https://redirect.github.com/not-matthias) in
[#&open-telemetry#8203;594](https://redirect.github.com/CodSpeedHQ/runner/pull/594)

##### 💼 Other

- Bump submodule to not loop on corrupted jit dump by
[@&open-telemetry#8203;GuillaumeLagrange](https://redirect.github.com/GuillaumeLagrange)
in [#&open-telemetry#8203;599](https://redirect.github.com/CodSpeedHQ/runner/pull/599)

**Full Runner Changelog**:
<https://github.com/CodSpeedHQ/codspeed/blob/main/CHANGELOG.md>

#### What's Changed

- docs(examples): update the example workflows by
[@&open-telemetry#8203;SuperMuel](https://redirect.github.com/SuperMuel) in
[#&open-telemetry#8203;252](https://redirect.github.com/CodSpeedHQ/action/pull/252)
- chore: bump runner version to 5.6.1 by
[@&open-telemetry#8203;github-actions](https://redirect.github.com/github-actions)\[bot]
in [#&open-telemetry#8203;254](https://redirect.github.com/CodSpeedHQ/action/pull/254)

#### New Contributors

- [@&open-telemetry#8203;SuperMuel](https://redirect.github.com/SuperMuel) made their
first contribution in
[#&open-telemetry#8203;252](https://redirect.github.com/CodSpeedHQ/action/pull/252)

**Full Changelog**:
<CodSpeedHQ/action@v5.6.0...v5.6.1>

</details>

<details>
<summary>github/codeql-action</summary>

###
[`v4.39.2`](https://redirect.github.com/github/codeql-action/releases/tag/v4.39.2)

[Compare
Source](https://redirect.github.com/github/codeql-action/compare/v4.39.1...v4.39.2)

- Update default CodeQL bundle version to
[2.27.4](https://redirect.github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.4).
[#&open-telemetry#8203;4284](https://redirect.github.com/github/codeql-action/pull/4284)
- *Upcoming breaking change*: CodeQL version 2.22.3 and earlier were
discontinued on 16 October 2026 alongside GitHub Enterprise Server 3.18,
and will be unsupported by the next minor release of the CodeQL Action.
Added a deprecation warning for customers using these versions of
CodeQL.
[#&open-telemetry#8203;4266](https://redirect.github.com/github/codeql-action/pull/4266)
- Starting with macOS 27 and Xcode 27, traced analyses are unsupported.
See [this blog post for more
information](https://github.blog/changelog/2026-10-19-codeql-changes-for-macos-27-xcode-27-and-swift-6-4/).
The CodeQL Action will now try to identify if a workflow is about to
perform a traced analysis on version 27 or above of macOS or Xcode. If
so, the CodeQL Action will emit a corresponding warning that will also
be visible on the [Tool Status
Page](https://docs.github.com/en/code-security/how-tos/find-and-fix-code-vulnerabilities/manage-your-configuration/use-the-tools-status-page-for-code-scanning).
This check will be rolled out following this release of the CodeQL
Action.
[#&open-telemetry#8203;4221](https://redirect.github.com/github/codeql-action/pull/4221)
- The undocumented TRAP cache cleanup feature that could be enabled
using the `CODEQL_ACTION_CLEANUP_TRAP_CACHES` environment variable has
been removed, as announced in the release notes for CodeQL Action
4.35.2. The `analyze` Action now warns if this environment variable is
set. If you used TRAP cache cleanup to limit Actions cache usage, we
recommend disabling TRAP caching by passing the `trap-caching: false`
input to the `init` Action.
[#&open-telemetry#8203;4243](https://redirect.github.com/github/codeql-action/pull/4243)

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - "on tuesday"
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/open-telemetry/opentelemetry-collector).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMDIuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjIwNC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZWJvdCJdfQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants