Skip to content

feat(mermin): vxlan and geneve tunnel flow support - #593

Open
kgrubb wants to merge 1 commit into
mainfrom
feat/vxlan-geneve-tunnel-support
Open

kgrubb wants to merge 1 commit into
mainfrom
feat/vxlan-geneve-tunnel-support

Conversation

@kgrubb

@kgrubb kgrubb commented Jun 9, 2026 •

Copy link
Copy Markdown
Contributor

This adds VXLAN and Geneve support by decapsulating in eBPF, so a tunneled flow is tracked and exported as its inner flow with tunnel.* metadata from the outer headers.

The first version parsed the tunnel payload in userspace on the first packet only. Since each outer direction (and the veth copy of the same flow) got its own map entry, it had to delete the extra entries to avoid duplicate spans, which lost those counts and made eBPF re-send a new flow event on almost every packet. TCP state, flags, and direction also came from the outer UDP flow. Keying the map on the inner 5-tuple in eBPF fixes all of that in one place.

Changes

  • decapsulate VXLAN (I flag set) and Geneve (Ethernet payload) in eBPF and key FLOW_STATS on the inner 5-tuple. Ports come from the parser config through a new TUNNEL_PORTS map.
  • store the outer addresses, ports, and VNI in FlowStats, oriented the same way as the span's source and destination
  • count inner frame bytes and read DSCP/ECN/TTL from the inner IP header
  • fall back to the outer UDP flow when the inner frame isn't IP (ARP, etc.)
  • populate tunnel.* span attributes from FlowStats, including flows first seen on a veth and decapsulated later
  • remove the userspace tunnel parser (mermin/src/packet) and the separate tunneled flow path in the producer
  • bump EBPF_MAP_SCHEMA_VERSION to 2 since FlowStats grows from 192 to 240 bytes (~5.2 MB max kernel memory at the default capacity, up from ~4.4 MB)
  • pin the CNI e2e ping pods to different nodes so traffic crosses the overlay. Flannel now watches eth0 with vxlan_port = 8472 and requires tunnel.type "vxlan" on the enriched span.
  • document the Flannel VXLAN port and update the interface guidance
  • fix the Cilium Kind install (pullPolicy, ipam.mode, rollout waits) in e2e setup

Notes

  • Inner protocols other than TCP, UDP, and ICMP still error out, same as plain traffic. WireGuard and GRE are untouched.
  • The verifier has only been exercised on kernel 7.0 (Docker Desktop) so far. CI covers the Kind runners.
  • fmt, clippy, and the eBPF and workspace tests pass locally in the builder image, including new eBPF tests for VXLAN, Geneve options, rejected payloads, disabled ports, and tunnel orientation. The flannel and kindnetd e2e runs passed locally on Kind. On flannel, each cross-node TCP connection showed up as one span per agent with real connection state and the outer node:8472 endpoints.

@kgrubb
kgrubb force-pushed the feat/vxlan-geneve-tunnel-support branch 4 times, most recently from 8dbfa8b to 2aacf0d Compare June 9, 2026 15:01
@kgrubb
kgrubb marked this pull request as draft June 9, 2026 15:04
@kgrubb
kgrubb force-pushed the feat/vxlan-geneve-tunnel-support branch from 2aacf0d to 01f146a Compare June 9, 2026 15:15
@kgrubb kgrubb self-assigned this Jun 9, 2026
@kgrubb

kgrubb commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

here's what the stdout logged spans look like:

		 ->  tunnel.type: String(Static("vxlan"))
		 ->  tunnel.network.interface.mac: String(RefCounted("2a:ad:87:d8:06:df"))
		 ->  tunnel.network.type: String(Static("ipv4"))
		 ->  tunnel.network.transport: String(Static("udp"))
		 ->  tunnel.source.address: String(RefCounted("172.18.0.3"))
		 ->  tunnel.source.port: I64(49951)
		 ->  tunnel.destination.address: String(RefCounted("172.18.0.4"))
		 ->  tunnel.destination.port: I64(8472)
		 ->  tunnel.id: I64(1)
		 ->  tunnel.type: String(Static("vxlan"))
		 ->  tunnel.network.interface.mac: String(RefCounted("2a:ad:87:d8:06:df"))
		 ->  tunnel.network.type: String(Static("ipv4"))
		 ->  tunnel.network.transport: String(Static("udp"))
		 ->  tunnel.source.address: String(RefCounted("172.18.0.3"))
		 ->  tunnel.source.port: I64(49951)
		 ->  tunnel.destination.address: String(RefCounted("172.18.0.4"))
		 ->  tunnel.destination.port: I64(8472)
		 ->  tunnel.id: I64(1)
spans 810
	Instrumentation Scope
		Name         : "mermin"
		Version  : "0.4.0"

	Name         : flow_ipv4_icmp
	TraceId      : 8233a8abbd5777594caef626e7998c5a
	SpanId       : c3d439b375e001bf
	TraceFlags   : TraceFlags(1)
	ParentSpanId : None (root span)
	Kind         : Client
	Start time   : 2026-06-09 15:28:16.751872
	End time     : 2026-06-09 15:28:22.708422
	Status       : Unset
	Attributes:
		 ->  flow.community_id: String(RefCounted("1:oa+LN637k5ND93Bx25MbNqZIN+g="))
		 ->  flow.direction: String(Static("forward"))
		 ->  network.type: String(Static("ipv4"))
		 ->  network.transport: String(Static("icmp"))
		 ->  source.address: String(RefCounted("10.244.0.4"))
		 ->  source.port: I64(0)
		 ->  destination.address: String(RefCounted("10.244.1.7"))
		 ->  destination.port: I64(0)
		 ->  flow.bytes.delta: I64(7448)
		 ->  flow.bytes.total: I64(7448)
		 ->  flow.packets.delta: I64(76)
		 ->  flow.packets.total: I64(76)
		 ->  flow.reverse.bytes.delta: I64(7448)
		 ->  flow.reverse.bytes.total: I64(7448)
		 ->  flow.reverse.packets.delta: I64(76)
		 ->  flow.reverse.packets.total: I64(76)
		 ->  flow.end_reason: String(Static("active timeout"))
		 ->  network.interface.index: I64(11)
		 ->  network.interface.name: String(RefCounted("eth0"))
		 ->  network.interface.mac: String(RefCounted("2a:ad:87:d8:06:df"))
		 ->  flow.ip.dscp.id: I64(0)
		 ->  flow.ip.dscp.name: String(Static("df"))
		 ->  flow.ip.ecn.id: I64(0)
		 ->  flow.ip.ecn.name: String(Static("non-ect"))
		 ->  flow.ip.ttl: I64(63)
		 ->  flow.reverse.ip.ttl: I64(64)
		 ->  flow.reverse.ip.dscp.id: I64(0)
		 ->  flow.reverse.ip.ecn.id: I64(0)
		 ->  flow.icmp.type.id: I64(0)
		 ->  flow.icmp.type.name: String(Static("echo_reply"))
		 ->  flow.icmp.code.id: I64(0)
		 ->  flow.icmp.code.name: String(Static(""))
		 ->  flow.reverse.icmp.type.id: I64(0)
		 ->  flow.reverse.icmp.type.name: String(Static("echo_reply"))
		 ->  flow.reverse.icmp.code.id: I64(0)
		 ->  flow.reverse.icmp.code.name: String(Static(""))
		 ->  tunnel.type: String(Static("vxlan"))
		 ->  tunnel.network.interface.mac: String(RefCounted("2a:ad:87:d8:06:df"))
		 ->  tunnel.network.type: String(Static("ipv4"))
		 ->  tunnel.network.transport: String(Static("udp"))
		 ->  tunnel.source.address: String(RefCounted("172.18.0.3"))
		 ->  tunnel.source.port: I64(49951)
		 ->  tunnel.destination.address: String(RefCounted("172.18.0.4"))
		 ->  tunnel.destination.port: I64(8472)
		 ->  tunnel.id: I64(1)
		 ->  source.k8s.namespace.name: String(RefCounted("default"))
		 ->  source.k8s.pod.name: String(RefCounted("cp-pod"))
		 ->  destination.k8s.namespace.name: String(RefCounted("default"))
		 ->  destination.k8s.pod.name: String(RefCounted("wk-pod"))

The 2 pods in this case are a basic alpine pod on the control plane and on the worker node.

@kgrubb
kgrubb marked this pull request as ready for review June 9, 2026 15:34
@kgrubb
kgrubb force-pushed the feat/vxlan-geneve-tunnel-support branch 2 times, most recently from 09c43a0 to ff14441 Compare June 9, 2026 16:16
Comment on lines -110 to -155
IpProto::Tcp => {
let (src_port, dst_port) = parse_l4_ports(data, offset)?;
Ok(ParsedPacket::Direct {
five_tuple: FiveTuple {
src_ip: outer_src_ip,
dst_ip: outer_dst_ip,
src_port,
dst_port,
protocol: IpProto::Tcp,
ip_version: if outer_src_ip.is_ipv4() { 4 } else { 6 },
},
l2_metadata: l2_meta,
ip_metadata: ip_meta,
})
}
IpProto::Icmp => {
let (type_code, _) = parse_icmp_type_code(data, offset)?;
Ok(ParsedPacket::Direct {
five_tuple: FiveTuple {
src_ip: outer_src_ip,
dst_ip: outer_dst_ip,
src_port: type_code,
dst_port: 0,
protocol: outer_protocol,
ip_version: if outer_src_ip.is_ipv4() { 4 } else { 6 },
},
l2_metadata: l2_meta,
ip_metadata: ip_meta,
})
}
IpProto::Gre => {
// TODO: GRE parsing
Err(ParseError::UnsupportedProtocol)
}
_ => Ok(ParsedPacket::Direct {
five_tuple: FiveTuple {
src_ip: outer_src_ip,
dst_ip: outer_dst_ip,
src_port: 0,
dst_port: 0,
protocol: outer_protocol,
ip_version: if outer_src_ip.is_ipv4() { 4 } else { 6 },
},
l2_metadata: l2_meta,
ip_metadata: ip_meta,
}),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I understand why the rest of this parsing is being removed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I removed this parsing because it only produced ParsedPacket::Direct, and direct flows never hit userspace parsing. eBPF already has the 5-tuple, ICMP fields, TCP flags, etc. in FlowStats.

My thought was to make the userspace parser’s job a lot more narrow. Basically, given a tunnel flow key and the UDP payload bytes, extract the inner 5-tuple and VNI.

If we do it this way, it would make the old full-packet parser redundant for direct traffic. And it'd make the old parsing incorrect for the buffer layout that eBPF sends for tunnels.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@veronica-m-ef also worth noting, the old parse_packet_deep path was dead code on main, and it expected a full ethernet frame while ebpf sends the buffer starting at the VXLAN header, so it couldn't have parsed tunnels anyway.

@kgrubb
kgrubb force-pushed the feat/vxlan-geneve-tunnel-support branch from ff14441 to 27ed7f9 Compare October 5, 2026 16:29
@kgrubb
kgrubb force-pushed the feat/vxlan-geneve-tunnel-support branch 2 times, most recently from fb6320a to 693bb59 Compare October 14, 2026 14:27
@kgrubb
kgrubb requested a review from veronica-m-ef October 14, 2026 16:15
@kgrubb
kgrubb force-pushed the feat/vxlan-geneve-tunnel-support branch from 693bb59 to f6700b1 Compare October 21, 2026 20:21
Keying the flow map on the inner 5-tuple in eBPF means both tunnel
directions and the veth copy of a flow share one FlowStats entry, so
userspace no longer has to dedupe by deleting map entries. It also gives
tunneled flows real TCP state, flags, timeouts, and inner byte counts,
and keeps distinct inner flows apart when they share an outer UDP port.

The outer headers and VNI are stored in FlowStats as tunnel metadata.
FlowStats grows to 240 bytes, so the map schema version moves to 2.
@kgrubb
kgrubb force-pushed the feat/vxlan-geneve-tunnel-support branch from f6700b1 to 0d3c6d2 Compare October 22, 2026 20:07
@kgrubb kgrubb changed the title feat(mermin): VXLAN and Geneve tunnel flow support feat(mermin): vxlan and geneve tunnel flow support Oct 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants