userspace-dp: publish per-zone traffic counters (#3651) - #4677
Merged
Merged
Conversation
Populate the deferred POPULATE half of #3643 for per-zone traffic. The eBPF-era dense zone_counters array was deleted in #1476; the HIDE half (#3643) made the read surfaces return ErrCounterNotPopulated ("not available") instead of a 500 / false alert / misleading 0, but nothing sourced the data. The Go populate hook (SetZoneCounterOffset) was wired but sourced only by tests, so show security zones Traffic statistics, REST /security/zones, and the Prometheus collector all rendered "not available". This sources per-zone ingress/egress packet+byte volume from the Rust AF_XDP dataplane per the design of record (docs/research/3643-dead-counters/plan.md §5A). Rust hot path (no per-packet hash, no per-packet atomic): a new userspace-dp/src/afxdp/zone_counters.rs builds a flat direct-index [u8;65536] zone-id to slot LUT (ZoneCounterSlotMap, 63 assignable slots plus overflow_active) at config apply. A forwarded packet resolves its ingress zone (from the shim meta) and egress zone (from the resolved egress ifindex) with two array reads and accumulates into a per-worker thread-local dense coalescer -- the same coalesce-then-fold technique as the policy/filter hit counters. The per-RX-batch flush folds the coalesced per-slot deltas into a coordinator-owned, zone-id-keyed ZoneCounterStore. Keying the store by the stable zone id (not by slot) removes the cold-path slot-reassignment zero-out entirely: accumulate and flush inside one loop iteration always use the same slot map, and the store cell is addressed by zone id. The store rides ForwardingState, carried forward from the previous state across applies so totals survive config commits, and reset only by the operator clear IPC. Accounting runs at the fast path (flow_cache_hit), the slow path (poll_descriptor forward-candidate), and the tunnel/next-table paths (record_forwarding_disposition ForwardCandidate, Hot-path only -- the Cold RPC-inject path's thread-local is never flushed). Wire: the helper pre-sums across workers into ONE ProcessStatus-level sparse per-zone block (zone_traffic_counters, layout version 1, nonzero rows only), tag-matched on both sides; protocol_wire_v1.json regenerated and the cross-language default-specimen + round-trip contract tests extended (Go zone_counters_status_test.go + Rust protocol/tests.rs). A new clear_zone_counters control IPC resets the helper store; the Go ClearZoneCounters override (+ ClearAllCounters wiring + LegacyDataPlaneAdapter route) sends it so an operator clear does not snap back on the next 1 s poll. Go: ProcessStatus gains ZoneCounterLayoutVersion / ZoneCounterOverflowActive / ZoneTrafficCounters; syncBPFCountersLocked mirrors each row into the bpfShim offset map via SetZoneCounterOffset (absolute overwrite, reset-safe on helper restart). Per-zone FLOOD counters remain deferred per the §5A recommended narrowing (the lower-value half; lean on the #3343 aggregate per-reason drop counters) -- SetFloodCounterOffset stays test-only and ReadFloodCounters keeps returning ErrCounterNotPopulated. Validation: RED-on-revert TestSyncBPFCountersPopulatesZoneCounters reverts to ErrCounterNotPopulated if the Go decode loop is removed; Rust zone_counters unit tests cover build/overflow/record+flush+snapshot/clear/ reconcile. FULL cargo test 3842/0; go test ./pkg/dataplane/... ./pkg/api/... ./pkg/grpcapi/... green. Docs: flipped the #3651 gap doc from deferred to shipped (traffic) and recorded the flood deferral. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015oARShYtiJJ2H4UB4nXGqi
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements the “POPULATE” half of #3643 / #3651 by sourcing per-zone ingress/egress packet+byte counters in the Rust userspace AF_XDP dataplane, publishing them over the userspace wire, and mirroring them into the Go bpfShim offset map so CLI/REST/Prometheus surfaces can show live per-zone traffic volume.
Changes:
- Add Rust hot-path per-zone traffic accounting (slot LUT + per-worker coalescer) with coordinator snapshot + clear IPC.
- Extend the userspace wire contract (new
zone_traffic_countersblock + status row type) and update fixtures/tests on both sides. - Add Go decode/mirroring and operator-clear wiring (
clear_zone_counters) including LegacyDataPlaneAdapter routing.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| userspace-dp/tests/fixtures/protocol_wire_v1.json | Updates wire fixture for new per-zone traffic fields/specimen. |
| userspace-dp/src/server/lifecycle.rs | Initializes ProcessStatus zone-counter fields. |
| userspace-dp/src/server/helpers.rs | Publishes zone counter block + layout version/overflow status on refresh. |
| userspace-dp/src/server/handlers/mod.rs | Adds clear_zone_counters control IPC handler. |
| userspace-dp/src/protocol/tests.rs | Adds round-trip/default-omit tests and default specimen for zone counter status. |
| userspace-dp/src/protocol/control.rs | Defines ZoneTrafficCounterStatus and extends ProcessStatus with zone counter fields. |
| userspace-dp/src/afxdp/zone_counters.rs | New module implementing slot map, coalescing, fold/store, snapshot, clear, reconcile + unit tests. |
| userspace-dp/src/afxdp/worker/loop_body/mod.rs | Flushes per-worker coalesced zone deltas once per RX batch. |
| userspace-dp/src/afxdp/types/forwarding.rs | Adds zone slot map + store to ForwardingState; adds egress_zone_id helper. |
| userspace-dp/src/afxdp/poll_descriptor/mod.rs | Records per-zone traffic on slow-path forward candidate (but see review comments). |
| userspace-dp/src/afxdp/poll_descriptor/flow_cache_hit.rs | Records per-zone traffic on fast-path flow-cache hits. |
| userspace-dp/src/afxdp/mod.rs | Wires in new zone_counters module. |
| userspace-dp/src/afxdp/forwarding_build/mod.rs | Builds zone slot map at apply; carries store forward across applies; reconciles removed zones. |
| userspace-dp/src/afxdp/disposition.rs | Records per-zone traffic for specific hot-path dispositions (tunnel/next-table/etc). |
| userspace-dp/src/afxdp/coordinator/mod.rs | Exposes zone counters snapshot/flags + clear method via coordinator. |
| pkg/dataplane/userspace/zonecounters.go | Adds userspace Manager override for ClearZoneCounters (+ helper IPC). |
| pkg/dataplane/userspace/zone_counters_status_test.go | Adds Go tests for mirroring behavior + JSON round-trip/omitempty behavior. |
| pkg/dataplane/userspace/protocol.go | Extends Go ProcessStatus + adds Go ZoneTrafficCounterStatus type. |
| pkg/dataplane/userspace/policycounters.go | Ensures ClearAllCounters also clears helper-side zone counter store. |
| pkg/dataplane/userspace/manager_ha.go | Mirrors helper zone counters into bpfShim offset map (but see review comments). |
| pkg/dataplane/userspace/legacy_dataplane.go | Routes ClearZoneCounters through userspace Manager override. |
| docs/userspace-dataplane-gaps.md | Updates gap doc: traffic populated, flood deferred. |
| docs/research/3643-dead-counters/plan.md | Updates design-of-record doc to reflect traffic shipped, flood deferred. |
| _Log.md | Logs the write/edit action for #3651. |
Comment on lines
+4011
to
+4021
| // #3651: per-zone traffic volume for this slow-path (first- | ||
| // packet / non-cacheable) forwarded packet, mirroring the | ||
| // flow-cache-hit fast path. | ||
| crate::afxdp::zone_counters::record_zone_traffic( | ||
| &worker_ctx.forwarding.zone_counter_slot_map, | ||
| meta.ingress_zone, | ||
| worker_ctx | ||
| .forwarding | ||
| .egress_zone_id(decision.resolution.egress_ifindex), | ||
| meta.pkt_len as u64, | ||
| ); |
Comment on lines
+839
to
+852
| // #3651: mirror the helper's pre-summed per-zone traffic totals into the | ||
| // bpfShim zone-counter offset map so Manager.ReadZoneCounters (and thus | ||
| // `show security zones` Traffic statistics, REST /security/zones, and the | ||
| // Prometheus collector) reports live per-zone volume instead of | ||
| // ErrCounterNotPopulated ("not available"). The helper reports cumulative | ||
| // totals keyed by the stable zone id; SetZoneCounterOffset stores them | ||
| // absolutely (overwrite), reset-safe on helper restart. | ||
| for i := range status.ZoneTrafficCounters { | ||
| z := &status.ZoneTrafficCounters[i] | ||
| m.bpfShim.SetZoneCounterOffset(z.ZoneID, | ||
| dataplane.CounterValue{Packets: z.IngressPackets, Bytes: z.IngressBytes}, | ||
| dataplane.CounterValue{Packets: z.EgressPackets, Bytes: z.EgressBytes}, | ||
| ) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #3651 — per-zone traffic counters are now published by the
userspace dataplane; per-zone flood counters remain deferred per the §5A
recommended narrowing. This is the POPULATE half of #3643 (the HIDE half
shipped earlier).
Before this change,
dataplane.ReadZoneCountersreturnedErrCounterNotPopulated("not available") on every surface because nothingsourced per-zone volume. Now
show security zonesTraffic statistics, REST/security/zones, and the Prometheus collector report live per-zoneingress/egress packet+byte volume. Design of record:
docs/research/3643-dead-counters/plan.md§5A.Rust (userspace-dp publish)
userspace-dp/src/afxdp/zone_counters.rs: a flat direct-index[u8;65536]zone-id → slot LUT (ZoneCounterSlotMap, 63 assignable slots +overflow_active) built at config apply. Hot path resolves the ingress zone(shim meta) and egress zone (resolved egress ifindex) with two array
reads and accumulates into a per-worker thread-local dense coalescer —
no per-packet hash, no per-packet atomic (same coalesce-then-fold technique
as the policy/filter hit counters).
flush_recorded_zone_countersfolds the coalesced deltas into acoordinator-owned, zone-id-keyed
ZoneCounterStore(ridesForwardingState, carried forward frompreviousacross applies so totalssurvive commits). Zone-id keying removes the cold-path slot-reassignment
zero-out entirely.
flow_cache_hit), slowpath (
poll_descriptorforward-candidate), and tunnel/next-table(
record_forwarding_dispositionForwardCandidate, Hot-path only).ProcessStatus-level sparse blockzone_traffic_counters(layout version 1, nonzero rows only). Newclear_zone_counterscontrol IPC resets the store.Go (decode + surfacing)
ProcessStatusgainsZoneCounterLayoutVersion/ZoneCounterOverflowActive/
ZoneTrafficCounters(JSON tags matched to Rust serde).syncBPFCountersLockedmirrors each row into the bpfShim offset map viaSetZoneCounterOffset(absolute overwrite, reset-safe on helper restart).zonecounters.go:ClearZoneCountersoverride (+ClearAllCounterswiring +
LegacyDataPlaneAdapterroute) sendsclear_zone_countersso anoperator clear does not snap back on the next 1 s poll.
Wire contract
Tag-matched both sides;
protocol_wire_v1.jsonregenerated; cross-languagedefault-specimen + round-trip contract tests extended (Go
zone_counters_status_test.go+ Rustprotocol/tests.rs).Deferred
Per-zone flood counters (§5A recommended narrowing — the lower-value half;
lean on the #3343 aggregate per-reason drop counters).
SetFloodCounterOffsetstays test-only;
ReadFloodCounterskeeps returningErrCounterNotPopulated.The issue stays open for that half.
Validation
TestSyncBPFCountersPopulatesZoneCountersreverts toErrCounterNotPopulatedif the Go decode loop is removed.zone_countersunit tests: build / wide-stable-hash-id / overflow /record+flush+snapshot / clear / reconcile.
go test ./pkg/dataplane/... ./pkg/api/... ./pkg/grpcapi/...green.recorded the flood deferral.
Not smoke-tested on the loss cluster in this run (no cluster access); the
hot-path increment is counter-only (cannot alter forwarding), and the wire
contract is pinned by the regenerated fixture + both-sides round-trip tests.
🤖 Generated with Claude Code
https://claude.ai/code/session_015oARShYtiJJ2H4UB4nXGqi