Skip to content

p2p: add message size, read error and concurrency metrics - #4680

Merged
KaloyanTanev merged 7 commits into
mainfrom
kalo/p2p-message-size-metrics
Sep 15, 2026
Merged

KaloyanTanev merged 7 commits into
mainfrom
kalo/p2p-message-size-metrics

Conversation

@KaloyanTanev

@KaloyanTanev KaloyanTanev commented Sep 3, 2026 •

Copy link
Copy Markdown
Collaborator

Add libp2p wire observability metrics, instrumented at the shared Send/SendReceive/RegisterHandler choke points, covering all runtime charon protocols (the DKG-ceremony-only dkg/sync protocol registers its own raw stream handler and is out of scope):

  • p2p_received_message_size_bytes{protocol, peer} and p2p_sent_message_size_bytes{protocol}: histograms of logical message sizes, with bucket edges at 8MiB/32MiB/128MiB so read-limit questions are exact bucket arithmetic.
  • p2p_message_read_errors_total{protocol, peer}: read failures excluding benign relay resets, including messages exceeding a protocol read limit.
  • p2p_inflight_requests{protocol, peer}, p2p_concurrent_requests{protocol, peer} and p2p_handler_duration_seconds{protocol}: inbound handling concurrency (the histogram observes queue depth at every arrival, catching bursts between scrapes) and processing time.

This provides the fleet data needed to pick the transport limits tracked in #4679 without touching honest traffic, and to validate the 32MB parsigex read limit proposed in #4637.

category: feature
ticket: #4679

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new inflight gauge update logic in observeHandlerStart can report incorrect values under concurrent handlers due to out-of-order Set calls.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds Prometheus observability for libp2p “wire-level” behavior at the shared p2p choke points (Send, SendReceive, RegisterHandler) to support selecting safe per-protocol transport limits (per #4679).

Changes:

  • Add histograms for logical protobuf message sizes (sent/received) and a counter for message read errors.
  • Add inbound handling concurrency/latency instrumentation (in-flight gauge, concurrent-at-arrival histogram, handler duration histogram).
  • Add internal tests covering the new metrics and update metrics documentation.
File summaries
File Description
p2p/sender.go Observes sent request/message sizes and received response sizes; increments read-error counter on response read failure.
p2p/receive.go Instruments inbound handler concurrency/duration; increments read-error counter on request read failures; observes received request and sent response sizes.
p2p/metrics.go Adds the new Prometheus metrics plus helper functions and inflight tracking.
p2p/metrics_internal_test.go Adds coverage validating size histograms, read-error counter behavior, and inflight/concurrency/duration metrics.
docs/metrics.md Documents the newly added p2p_* metrics.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread p2p/metrics.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are a few correctness/operational issues in the new metrics instrumentation (sent-size observation timing, inflight map growth, and misleading sent-size help/docs) that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

docs/metrics.md:119

  • This description says sent messages are not labelled by peer because they are broadcast identically, but sent_message_size_bytes is also recorded for per-peer responses written by handlers. Reword to avoid implying per-peer uniformity and instead note it’s omitted to reduce cardinality.
    p2p/metrics.go:115
  • sent_message_size_bytes is also observed for per-peer stream responses (see receive.go calling observeSentMessage on resp), so the current help text is misleading when it claims the lack of a peer label is because messages are broadcast identically to all peers. Consider rewording to reflect the actual reason (cardinality reduction) without implying per-peer uniformity.
  • Files reviewed: 5/5 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread p2p/metrics.go Resolved
Comment thread p2p/sender.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Relay read failures and the DKG sync protocol remain unobserved, leaving the advertised metrics incomplete.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

p2p/receive.go:57

  • This does not cover every Charon protocol as stated: dkg/sync/server.go:355 registers /charon/dkg/sync/1.0.0/ directly with SetStreamHandler, and its client opens streams directly at dkg/sync/client.go:231. Consequently that protocol contributes none of the new inbound concurrency/read/size metrics or outbound size metrics, leaving a gap in the fleet transport data. Instrument the sync protocol too, or move the observations to a layer shared by both handler implementations.
    p2p/receive.go:88
  • Relay resets are still excluded from this counter because the preceding IsRelayError branch returns first. Those are ReadMsg failures (and the outbound SendReceive path counts them), so p2p_message_read_errors_total currently has inconsistent and incomplete semantics. Increment the counter before returning from that branch as well.
    p2p/metrics_internal_test.go:138
  • This test sends only one request, so it never exercises the concurrent increment/decrement ordering or verifies that overlapping arrivals produce depths 1, 2, etc.—the central behavior of these new metrics. Please block the handler, issue multiple simultaneous requests, assert the peak gauge/histogram observations, then release them and assert the gauge returns to zero.
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@codecov

codecov Bot commented Sep 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.47%. Comparing base (7798cdd) to head (f3ae169).

Files with missing lines Patch % Lines
p2p/receive.go 77.77% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4680      +/-   ##
==========================================
+ Coverage   65.45%   65.47%   +0.01%     
==========================================
  Files         246      246              
  Lines       30020    30053      +33     
==========================================
+ Hits        19651    19678      +27     
- Misses      10368    10374       +6     
  Partials        1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@KaloyanTanev

Copy link
Copy Markdown
Collaborator Author

Addressed the suppressed comments in b0dfc5d: relay resets now excluded consistently on both read paths, and added a concurrency test asserting depths 1..3 and gauge reset. dkg/sync is deliberately out of scope (ceremony-only, raw stream handler) — PR body updated to say so.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Direct-stream resets are incorrectly excluded from the read-error metrics.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

p2p/receive.go:88

  • This skips metrics for resets on direct streams too: IsRelayError matches the transport-agnostic network.ErrReset and ErrResourceScopeClosed without verifying that the connection is relayed (p2p/ping.go:131-133). As a result, genuine direct-peer read failures never reach either new increment. Restrict the early return to relayed connections.
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread p2p/sender.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Structurally invalid but successfully read messages are omitted from both size and read-error telemetry.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

p2p/receive.go:108

  • This observation runs only after protonil.Check, so a protobuf that is read and decoded successfully but fails the required-field check is omitted from the size histogram. Since that case is not a read error either, malformed-but-readable traffic becomes invisible to both new metrics. Record the decoded size immediately after the read-error branches, before application-level validation.
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@KaloyanTanev

Copy link
Copy Markdown
Collaborator Author

Addressed in the latest commit — received size is now observed before protonil.Check, so malformed-but-readable messages appear in the size histogram. Deliberately not counting protonil failures as read errors: required-field mismatches fire between honest peers during rolling upgrades.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation and coverage align with the stated scope; only a minor timeout comment needs correction.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

p2p/metrics.go:142

  • This comment says the default receive timeout is about 10s, but defaultRcvTimeout is 5s in p2p/sender.go:28. Please keep the bucket rationale aligned with the actual default.
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@KaloyanTanev
KaloyanTanev force-pushed the kalo/p2p-message-size-metrics branch from 59ad34f to f3ae169 Compare September 15, 2026 08:54
@sonarqubecloud

Copy link
Copy Markdown

@KaloyanTanev
KaloyanTanev merged commit c6a6d42 into main Sep 15, 2026
11 checks passed
@KaloyanTanev
KaloyanTanev deleted the kalo/p2p-message-size-metrics branch September 15, 2026 10:37
KaloyanTanev added a commit that referenced this pull request Sep 22, 2026
* app/eth1wrap: avoid startup hang (#4694)

* app/eth1wrap: add regression test for ClientVersion reconnect deadlock

Reproduce charon#4689: ClientVersion can hang after RPC timeout when
reconnectCh is full, wedging WaitConnected / startup against a silent EL.

Co-authored-by: Cursor <cursoragent@cursor.com>

* app/eth1wrap: avoid startup hang when execution client is silent

Make maybeReconnect non-blocking and do not hold the client mutex across
RPC calls so WaitConnected can time out instead of deadlocking (charon#4689).

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

* app/eth1wrap: don't close reconnectCh on shutdown (#4699)

* core/parsigex: bind partial signature sender to share index (#4705)

* p2p: recover stream handler panics to avoid node crash (#4706)

Backport of #4706 adapted for the v1.11 release branch: drops the message_handler_panics_total metric, whose supporting infrastructure (#4680) is not present on this branch.

* build(deps): Bump the go-dependencies group with 7 updates (#4674)

Bumps the go-dependencies group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/showwin/speedtest-go](https://github.com/showwin/speedtest-go) | `1.8.1` | `1.8.2` |
| [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) | `1.45.0` | `1.46.0` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace](https://github.com/open-telemetry/opentelemetry-go) | `1.45.0` | `1.46.0` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) | `1.45.0` | `1.46.0` |
| [go.opentelemetry.io/otel/exporters/stdout/stdouttrace](https://github.com/open-telemetry/opentelemetry-go) | `1.45.0` | `1.46.0` |
| [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) | `1.45.0` | `1.46.0` |
| [go.opentelemetry.io/otel/trace](https://github.com/open-telemetry/opentelemetry-go) | `1.45.0` | `1.46.0` |

Updates `github.com/showwin/speedtest-go` from 1.8.1 to 1.8.2
- [Release notes](https://github.com/showwin/speedtest-go/releases)
- [Changelog](https://github.com/showwin/speedtest-go/blob/master/docs/release.md)
- [Commits](showwin/speedtest-go@v1.8.1...v1.8.2)

Updates `go.opentelemetry.io/otel` from 1.45.0 to 1.46.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.45.0...v1.46.0)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace` from 1.45.0 to 1.46.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.45.0...v1.46.0)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` from 1.45.0 to 1.46.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.45.0...v1.46.0)

Updates `go.opentelemetry.io/otel/exporters/stdout/stdouttrace` from 1.45.0 to 1.46.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.45.0...v1.46.0)

Updates `go.opentelemetry.io/otel/sdk` from 1.45.0 to 1.46.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.45.0...v1.46.0)

Updates `go.opentelemetry.io/otel/trace` from 1.45.0 to 1.46.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.45.0...v1.46.0)

---
updated-dependencies:
- dependency-name: github.com/showwin/speedtest-go
  dependency-version: 1.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel
  dependency-version: 1.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace
  dependency-version: 1.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  dependency-version: 1.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/stdout/stdouttrace
  dependency-version: 1.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/trace
  dependency-version: 1.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* core: perform sync committee duties for exited validators (#4712)

* core: perform sync committee duties for exited validators

* Add extra test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: b0a7 <127276179+b0a7@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
KaloyanTanev added a commit that referenced this pull request Nov 6, 2026
* app: fix flaky TestStartChecker (#4654)

(cherry picked from commit f0633d4)

* core/tracker: fix inclusion trim underflow at chain start (#4662)

* core/tracker: fix inclusion trim underflow at chain start

The inclusion checker's slot arithmetic underflows on fresh chains:
before genesis the wall-clock offset is negative, during the first
InclCheckLag slots the checked slot goes negative, and until wall slot
InclCheckLag+InclMissedLag the trim slot goes negative. The trim
underflow makes Trim(huge) delete every pending submission and report
each fresh duty as "duty not included on-chain", failing all early
proposals on every fresh-genesis network.

Skip ticks until a slot is old enough to check, and only trim once a
slot can actually be declared missed.

* core/tracker: check the genesis slot and deflake the regression test

The zero-valued checkedSlot skipped slot 0 as already-checked; start
from a MaxUint64 sentinel instead. Synchronize the chain-start
regression test on the first inclusion check rather than sleeping,
and pin it to the genesis slot so it covers both guards.

* core/tracker: pin attestation-inclusion feature off in regression test

The test wires only checkBlockFunc; disable the attestation_inclusion
feature explicitly so Run cannot take the nil checkBlockAndAttsFunc
path regardless of test ordering.

(cherry picked from commit ad04571)

* build(deps): Bump the docker-dependencies group across 2 directories with 1 update (#4661)

* build(deps): Bump the docker-dependencies group across 2 directories with 1 update

Bumps the docker-dependencies group with 1 update in the / directory: golang.
Bumps the docker-dependencies group with 1 update in the /testutil/promrated directory: golang.

Updates `golang` from 1.26.6-trixie to 1.27.0-trixie

Updates `golang` from 1.26.6-trixie to 1.27.0-trixie

Updates `golang` from 1.26.6-alpine to 1.27.0-alpine

Updates `golang` from 1.26.6-alpine to 1.27.0-alpine

---
updated-dependencies:
- dependency-name: golang
  dependency-version: 1.27.0-alpine
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: docker-dependencies
- dependency-name: golang
  dependency-version: 1.27.0-trixie
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: docker-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* bump go to 1.27

* bump cockroachdb/swiss for go1.27 support

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kalo <24719519+KaloyanTanev@users.noreply.github.com>
(cherry picked from commit a67667d)

* build(deps): Bump the go-dependencies group across 1 directory with 4 updates (#4687)

Bumps the go-dependencies group with 4 updates in the / directory: [github.com/prometheus/client_model](https://github.com/prometheus/client_model), [github.com/showwin/speedtest-go](https://github.com/showwin/speedtest-go), [golang.org/x/crypto](https://github.com/golang/crypto) and [github.com/vektra/mockery/v2](https://github.com/vektra/mockery).

Updates `github.com/prometheus/client_model` from 0.6.2 to 0.6.3
- [Release notes](https://github.com/prometheus/client_model/releases)
- [Commits](prometheus/client_model@v0.6.2...v0.6.3)

Updates `github.com/showwin/speedtest-go` from 1.8.2 to 1.8.3
- [Release notes](https://github.com/showwin/speedtest-go/releases)
- [Changelog](https://github.com/showwin/speedtest-go/blob/master/docs/release.md)
- [Commits](showwin/speedtest-go@v1.8.2...v1.8.3)

Updates `golang.org/x/crypto` from 0.55.0 to 0.56.0
- [Commits](golang/crypto@v0.55.0...v0.56.0)

Updates `github.com/vektra/mockery/v2` from 2.53.6 to 2.53.7
- [Release notes](https://github.com/vektra/mockery/releases)
- [Commits](vektra/mockery@v2.53.6...v2.53.7)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_model
  dependency-version: 0.6.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/showwin/speedtest-go
  dependency-version: 1.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: golang.org/x/crypto
  dependency-version: 0.56.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/vektra/mockery/v2
  dependency-version: 2.53.7
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 4eef0bd)

* app/stacksnipe: redact secret flag values from exported command lines (#4688)

The stack sniping collector read the full command line of every detected
validator client and exported it verbatim, both as the cli_parameters label
of the app_validator_stack_params gauge served on the monitoring endpoint,
and as a field of a debug log line.

Validator client command lines routinely carry secret material, for example
keystore passwords, the paths of files holding them, and keymanager bearer
tokens. Exporting them unmodified pushes that material into the metrics and
log planes, whose readers and retention are usually much broader than the
process table it came from.

Redact the values of flags whose names look sensitive (auth, jwt, key,
passphrase, password, secret, token) before the command line is logged or
handed to the metrics callback. Both the "--flag value" and the
"--flag=value" forms are handled, and flag names are preserved so the
telemetry stays useful. A /proc cmdline is NUL separated, so redaction
normally operates per argument; if the whole command line ever arrives as a
single blob it is split on whitespace first, so redaction fails safe rather
than passing the blob through untouched.

Also warn at startup when --proc-directory is set, and note the disclosure
tradeoff in the flag's help text, so the behaviour is not a surprise.

The feature remains disabled by default.

(cherry picked from commit d467312)

* build(deps): Bump the go-dependencies group with 2 updates (#4690)

Bumps the go-dependencies group with 2 updates: [golang.org/x/sync](https://github.com/golang/sync) and [golang.org/x/time](https://github.com/golang/time).

Updates `golang.org/x/sync` from 0.22.0 to 0.23.0
- [Commits](golang/sync@v0.22.0...v0.23.0)

Updates `golang.org/x/time` from 0.15.0 to 0.16.0
- [Commits](golang/time@v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-version: 0.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/time
  dependency-version: 0.16.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 67f4b59)

# Conflicts:
#	go.mod
#	go.sum

* dkg/pedersen: buffer channels to not drop early bundles (#4683)

* dkg/pedersen: buffer bundle channels to avoid dropping early bundles

* test response and justification bundle paths

(cherry picked from commit 7464e13)

* dkg: retry transient p2p send failures during ceremonies (#4684)

* p2p: include peer name in send errors

* dkg: retry transient p2p send failures during ceremonies

* p2p: address copilot review on send retries

* dkg: add frost round 1 send retry test

* dkg: apply 60s p2p send timeout to ceremony sends

* dkg: increase ceremony send retries to 5

* p2p: bound send retries by the send timeout budget

* p2p: bound stream creation by the send timeout budget

* p2p: cap each send attempt at its slice of the timeout budget

* p2p: bound each attempt's dial by its deadline slice

* p2p: give each SendReceive attempt the full timeout, clamp negative retries

* p2p: bound SendReceive by its send timeout without slicing the response wait

* p2p: make send timeout per-attempt instead of a shared budget

(cherry picked from commit c878ff0)

* p2p: add message size, read error and concurrency metrics (#4680)

* p2p: add message size and concurrency metrics

* p2p: update inflight gauge under lock

* p2p: cleanup idle inflight keys, observe sent after close write

* p2p: consistent relay error handling, concurrency test

* p2p: count direct-stream resets as read errors

* p2p: observe received size before proto validation

* p2p: named struct for inflight key

(cherry picked from commit c6a6d42)

* build(deps): Bump the go-dependencies group across 1 directory with 5 updates (#4698)

Bumps the go-dependencies group with 3 updates in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto), [golang.org/x/net](https://github.com/golang/net) and [golang.org/x/tools](https://github.com/golang/tools).

Updates `golang.org/x/crypto` from 0.56.0 to 0.57.0
- [Commits](golang/crypto@v0.56.0...v0.57.0)

Updates `golang.org/x/net` from 0.58.0 to 0.59.0
- [Commits](golang/net@v0.58.0...v0.59.0)

Updates `golang.org/x/term` from 0.45.0 to 0.46.0
- [Commits](golang/term@v0.45.0...v0.46.0)

Updates `golang.org/x/text` from 0.41.0 to 0.42.0
- [Release notes](https://github.com/golang/text/releases)
- [Commits](golang/text@v0.41.0...v0.42.0)

Updates `golang.org/x/tools` from 0.49.0 to 0.50.0
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](golang/tools@v0.49.0...v0.50.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.57.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.59.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/term
  dependency-version: 0.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/text
  dependency-version: 0.42.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/tools
  dependency-version: 0.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 6b1e607)

# Conflicts:
#	go.mod
#	go.sum

* cmd/exit_sign: warn on validators with pending deposits and no index (#4702)

(cherry picked from commit 1928fd0)

* *: keep //nolint:revive from nolintlint false autofix (#4703)

(cherry picked from commit 3f40e6c)

* build(deps): Bump github.com/ethereum/go-ethereum (#4730)

Bumps the go-dependencies group with 1 update: [github.com/ethereum/go-ethereum](https://github.com/ethereum/go-ethereum).

Updates `github.com/ethereum/go-ethereum` from 1.17.5 to 1.17.6
- [Release notes](https://github.com/ethereum/go-ethereum/releases)
- [Commits](ethereum/go-ethereum@v1.17.5...v1.17.6)

---
updated-dependencies:
- dependency-name: github.com/ethereum/go-ethereum
  dependency-version: 1.17.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit bcd3f41)

* build(deps): Bump the go-dependencies group with 7 updates (#4737)

Bumps the go-dependencies group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/ethereum/go-ethereum](https://github.com/ethereum/go-ethereum) | `1.17.6` | `1.17.7` |
| [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/exporters/stdout/stdouttrace](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |
| [go.opentelemetry.io/otel/trace](https://github.com/open-telemetry/opentelemetry-go) | `1.46.0` | `1.47.0` |

Updates `github.com/ethereum/go-ethereum` from 1.17.6 to 1.17.7
- [Release notes](https://github.com/ethereum/go-ethereum/releases)
- [Commits](ethereum/go-ethereum@v1.17.6...v1.17.7)

Updates `go.opentelemetry.io/otel` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/exporters/stdout/stdouttrace` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/sdk` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.46.0...v1.47.0)

Updates `go.opentelemetry.io/otel/trace` from 1.46.0 to 1.47.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.46.0...v1.47.0)

---
updated-dependencies:
- dependency-name: github.com/ethereum/go-ethereum
  dependency-version: 1.17.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/stdout/stdouttrace
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: go.opentelemetry.io/otel/trace
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit ce7f7b2)

* core/scheduler: fix flaky TestSchedulerWait clock rewind (#4726)

(cherry picked from commit a818241)

* p2p: label handler metrics from validated stream protocol (#4722)

* p2p: label handler metrics from validated stream protocol

* p2p: deflake metrics tests racing deferred observations
(cherry picked from commit eb506a6)

* p2p: restore message handler panics metric (#4706)

* build(deps): Bump golang.org/x/tools (#4742)

Bumps the go-dependencies group with 1 update: [golang.org/x/tools](https://github.com/golang/tools).

Updates `golang.org/x/tools` from 0.50.0 to 0.51.0
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](golang/tools@v0.50.0...v0.51.0)

---
updated-dependencies:
- dependency-name: golang.org/x/tools
  dependency-version: 0.51.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 7c0ff60)

* build(deps): Bump the go-dependencies group with 3 updates (#4770)

Bumps the go-dependencies group with 3 updates: [github.com/drand/kyber](https://github.com/drand/kyber), [github.com/drand/kyber-bls12381](https://github.com/drand/kyber-bls12381) and [golang.org/x/sync](https://github.com/golang/sync).

Updates `github.com/drand/kyber` from 1.3.2 to 1.3.3
- [Release notes](https://github.com/drand/kyber/releases)
- [Commits](drand/kyber@v1.3.2...v1.3.3)

Updates `github.com/drand/kyber-bls12381` from 0.3.4 to 0.3.5
- [Release notes](https://github.com/drand/kyber-bls12381/releases)
- [Commits](drand/kyber-bls12381@v0.3.4...v0.3.5)

Updates `golang.org/x/sync` from 0.23.0 to 0.24.0
- [Commits](golang/sync@v0.23.0...v0.24.0)

---
updated-dependencies:
- dependency-name: github.com/drand/kyber
  dependency-version: 1.3.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/drand/kyber-bls12381
  dependency-version: 0.3.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: golang.org/x/sync
  dependency-version: 0.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 3c480a1)

# Conflicts:
#	go.mod

* app/eth2wrap: bind eth2http service lifetime to caller ctx (#4772)

* app/eth2wrap: bind eth2http service lifetime to caller ctx

* app/retry: retry eth2http not active and not synced errors

(cherry picked from commit e76d179)

* build(deps): Bump the go-dependencies group across 1 directory with 7 updates (#4778)

Bumps the go-dependencies group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/ethereum/go-ethereum](https://github.com/ethereum/go-ethereum) | `1.17.7` | `1.17.8` |
| [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) | `1.24.1` | `1.25.0` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.57.0` | `0.58.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.60.0` | `0.61.0` |
| [golang.org/x/term](https://github.com/golang/term) | `0.46.0` | `0.47.0` |
| [golang.org/x/text](https://github.com/golang/text) | `0.42.0` | `0.43.0` |
| [golang.org/x/tools](https://github.com/golang/tools) | `0.51.0` | `0.52.0` |

Updates `github.com/ethereum/go-ethereum` from 1.17.7 to 1.17.8
- [Release notes](https://github.com/ethereum/go-ethereum/releases)
- [Commits](ethereum/go-ethereum@v1.17.7...v1.17.8)

Updates `github.com/prometheus/client_golang` from 1.24.1 to 1.25.0
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.25.0/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.24.1...v1.25.0)

Updates `golang.org/x/crypto` from 0.57.0 to 0.58.0
- [Commits](golang/crypto@v0.57.0...v0.58.0)

Updates `golang.org/x/net` from 0.60.0 to 0.61.0
- [Commits](golang/net@v0.60.0...v0.61.0)

Updates `golang.org/x/term` from 0.46.0 to 0.47.0
- [Commits](golang/term@v0.46.0...v0.47.0)

Updates `golang.org/x/text` from 0.42.0 to 0.43.0
- [Release notes](https://github.com/golang/text/releases)
- [Commits](golang/text@v0.42.0...v0.43.0)

Updates `golang.org/x/tools` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](golang/tools@v0.51.0...v0.52.0)

---
updated-dependencies:
- dependency-name: github.com/ethereum/go-ethereum
  dependency-version: 1.17.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/prometheus/client_golang
  dependency-version: 1.25.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/crypto
  dependency-version: 0.58.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.61.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/term
  dependency-version: 0.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/text
  dependency-version: 0.43.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/tools
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 6704e6a)

* build(deps): Bump the go-dependencies group across 1 directory with 2 updates (#4790)

* build(deps): Bump the go-dependencies group across 1 directory with 2 updates

Bumps the go-dependencies group with 2 updates in the / directory: [github.com/bufbuild/buf](https://github.com/bufbuild/buf) and [github.com/vektra/mockery/v2](https://github.com/vektra/mockery).

Updates `github.com/bufbuild/buf` from 1.72.0 to 1.74.0
- [Release notes](https://github.com/bufbuild/buf/releases)
- [Changelog](https://github.com/bufbuild/buf/blob/main/CHANGELOG.md)
- [Commits](bufbuild/buf@v1.72.0...v1.74.0)

Updates `github.com/vektra/mockery/v2` from 2.53.7 to 2.53.8
- [Release notes](https://github.com/vektra/mockery/releases)
- [Commits](vektra/mockery@v2.53.7...v2.53.8)

---
updated-dependencies:
- dependency-name: github.com/bufbuild/buf
  dependency-version: 1.74.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/vektra/mockery/v2
  dependency-version: 2.53.8
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* *: bump go-libp2p to v0.50.0 for quic-go v0.63

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kalo <24719519+KaloyanTanev@users.noreply.github.com>
(cherry picked from commit dd58d17)

# Conflicts:
#	go.mod
#	go.sum

* core/tracker: debug log sync selection proof inconsistency (#4801)
(cherry picked from commit 2b534eb)

* app/obolapi: send a User-Agent and type 404s on POST (#4808)

* app/obolapi: send a User-Agent and type 404s on POST

* app/obolapi: make WithUserAgent override the default

---------

Co-authored-by: kalo <24719519+KaloyanTanev@users.noreply.github.com>
(cherry picked from commit ae46490)

* cmd: convert deposit sign withdrawal addresses to creds (#4804)

* cmd: convert deposit sign withdrawal addresses to credentials

charon deposit sign hex-decoded --withdrawal-addresses straight into
DepositMessage.WithdrawalCredentials. A 20 byte address (as documented)
was zero-padded by SSZ hashing, so operators signed and the API stored
malformed deposits.

Addresses are now converted to 0x01 (or 0x02 for compounding clusters)
withdrawal credentials. Mixed case addresses must match their EIP-55
checksum. Full 32 byte credentials are still accepted if the prefix
matches the cluster's compounding setting. deposit fetch rejects
credentials from the API that are not 32 bytes.

* eth2util/deposit: reuse withdrawalCredsFromAddr for 32 byte creds

* cmd: test deposit sign zero address and compounding creds

---------

Co-authored-by: kalo <24719519+KaloyanTanev@users.noreply.github.com>
(cherry picked from commit e7d8587)

* cmd: validate addresses consistently across create and edit (#4805)

Validates fee recipient and withdrawal addresses the same way in `create cluster`, `create dkg` and `edit add-validators`, and stores them in EIP-55 checksummed form. Before this, `create cluster` without a definition file skipped the withdrawal address check entirely. A zero address on mainnet or gnosis is an error in `create dkg` and a warning elsewhere (where it's legitimate, e.g. splitting the keys of validators that are already active).

The "all-lowercase or all-uppercase, otherwise it must match EIP-55" rule now lives in one place, `eth2util.ValidateChecksumAddress`. It's used by these commands, by `feerecipient sign` (which wrongly rejected all-uppercase addresses) and by `deposit.WithdrawalCredsFromAddrOrCreds` (#4804).

category: bug
ticket: none
(cherry picked from commit fd8c6fa)

* build(deps): Bump the go-dependencies group across 1 directory with 4 updates (#4817)

Partial backport: only golang.org/x/net v0.62.0. go-builder-client, go-eth2-client and go-ethereum v1.17.10 (requires go-eth2-client v0.30.1 and dynamic-ssz v1.4.0) are gloas-related and left out.

(cherry picked from commit beea8ba)

* *: bump golangci-lint to v2.14.0 (#4741)

(cherry picked from commit 589fc22)

* app/log: silence revive package-naming in redact.go

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: aly <16789036+aly-obol@users.noreply.github.com>
Co-authored-by: Oisín Kyne <4981644+OisinKyne@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants