[nicolaskagami]

Context

In order to implement RFD 662 - Modular Router Configuration, the switch needs to support separate routing contexts. For context, RFD 404 - Boundary Tunnel Routing defines how we use tunneling to achieve external reachability. We're modularizing external reachability (for tunneled traffic), not implementing VRFs.

This PR focuses on separating these routing contexts while keeping strong backwards compatibility. We'll now support multiple TEP endpoints, each redirecting to a separate router configuration (endpoint selection is done in OPTE). A "default" routing context is kept for backwards compatibility, allowing legacy APIs to remain functional. Thus, it's still used for both underlay routing and the default router configuration (which we'll likely want to remove later).

Internal Changes

• Route lookup keys (route_index) are enriched with an exact match on routing_table_id, used to select the routing context (lpm -> (exact, lpm)).
• This routing_table_id lives entirely inside dendrite, associated with the router configuration uuid provided by the control plane.
• meta.routing_table_id (8 bits) is added to ingress metadata and defaults to 0.
• The claimv6 action now takes a routing_table_id and associates the registered address (TEP endpoint or other loopback) with it, enriching the packet metadata. Non-default tunnel endpoints use ids 1-255. Default endpoints and underlay use 0.
• Apparently adding 8 bits exact match to the keys made us to go to 18/20 stages (adding 1 to non-multicast). IPv4 LPM capacity goes to 8190/8153. IPV6 seems to be unaffected.

API Changes

(v14, VERSION_MODULAR_ROUTER)

Routers

Method	Path	Body / response
GET	/router	BTreeMap<Uuid, Ipv6Addr>
GET	/router/{router_id}	Ipv6Addr
PUT	/router/{router_id}	Ipv6Addr
DELETE	/router/{router_id}

Routes

Method	Path
GET	/router/{router_id}/route/{ip_family}
GET	/router/{router_id}/route/{ip_family}/{cidr}
POST	/router/{router_id}/route/{ip_family}
PUT	/router/{router_id}/route/{ip_family}
DELETE	/router/{router_id}/route/{ip_family}/{cidr}
DELETE	/router/{router_id}/route/{ip_family}/{cidr}/{port_id}/{link_id}/{tgt_ip}

Architectural decisions (in decreasing order of confidence)

• routing_table_id lives in dpd, not defined by the control plane (or derived from the uuid): This has many benefits, such as not introducing needless coupling, allowing switches to have different IDs and capacities in the long run and not limiting the total number of routing configurations to the minimum number any switch can support. For example, we could easily reassign the routing tables for different purposes in the future without breaking compatibility with the current API.
• Endpoints vs other loopbacks: It seems that we only use loopback addresses for endpoints at the moment, though the switch seems to support them. This PR is a first step in allowing us to separate our boundary services loopbacks from whatever else we might want to use loopbacks for. Endpoints must be unique to select the routing context.
• Default Router: This is the trick that allows backwards compatibility. The legacy API will implicitly use this routing context, both for the underlay routes as well as for the default router configuration. Every other router configuration goes elsewhere. We'll likely want to remove the default router in the future so we know underlay traffic routing is segregated. We can't do this now without re-routing every loopback address registration to it. /route/* returns both underlay and default router routes.
• Names: Naming things is really hard. I accept suggestions here.
  • The router_id name clashes with the Router ID concept. While it's identifying a (logical) router's context, it invites the wrong inference regarding its lifetime and purpose.
  • The default router could be called the system routing table or something else.
  • Depending on how we separate things, we could just talk about TunnelEndpoints instead of "routers".
• Locking and new structs: I put the new Routers inside RouteData so they share a lock because otherwise I'd have to hold an async lock through await points because, for some reason, RouteData is locked with an async lock. Loopbacks now also have a slight dependence on this data, which just means it needs to be locked in order. There are many improvements one can do to the overall structure depending on how we want to deal with endpoints x loopbacks. The current approach is trying to impose as few new architectural decisions as possible.

companion sidecar-lite PR: oxidecomputer/sidecar-lite#182

[cfzimmerman]

router.rs comment is most important imo. Lmk if you'd like to hop on a call to discuss. Nice work!

[cfzimmerman]

Reflecting on naming: Do we have a consistent definition of what a "router" is? In the context of Modular Router Configuration, this comment makes sense. But from a high level view, the word "router" feels overloaded. At least in these API docs, does it make sense to be more specific? For example, "in the given routing config."?

[nicolaskagami]

Yeah, this is something I brought up in the PR description and in discussions elsewhere. We can definitely think of these as "logical routers" (as the modules in "Modular Routing Configuration"), but I also agree with your sentiment and have called them "routing configurations" in other places, though that name can sound less "modular" than what we want to imply.

At this point I think I'm equally satisfied with both options.

[cfzimmerman]

Defer to my comment in dpd-api about duplication. If we stick with separate V4 and V6 APIs, thoughts on making this generic? So define RouteTargetPath<T> and then use RouteTargetPath<Ipv4Net> / RouteTargetPath<Ipv6Net>?

[cfzimmerman]

TOF report: dendrite#386

[WIP] Modular Router Configuration

	Target branch	PR
Branch	main	nsk/662-modular-router-config
Commit	6385c1ec3d0c	4584f4a7923e
Codegen	--stages 19 --multicast	--stages 19 --multicast
tof	ry/tof @ ed37578b	ry/tof @ ed37578b

Summary

From tof dump --summary, tof phv and pipe/logs/metrics.json

	main	nsk/662-modular-router-config	Δ
Last stage used, ingress	18	18	0
Last stage used, egress	5	5	0
PHV containers, ingress (of 280)	158	157	-1
PHV containers, egress (of 280)	44	44	0
PHV containers, total (of 280)	197	196	-1
MAU SRAMs	259	258	-1
MAU TCAMs	94	97	+3
MAU logical tables	134	134	0
Parser TCAM rows, ingress	104	104	0
Parser TCAM rows, egress	54	55	+1

Compiler resources

From pipe/logs/metrics.json

Resource	main	nsk/662-modular-router-config	Δ
Parser TCAM rows, ingress	104	104	0
Parser TCAM rows, egress	54	55	+1
MAU logical tables	134	134	0
MAU SRAMs	259	258	-1
MAU TCAMs	94	97	+3
MAU map RAMs	90	89	-1
Exact-match crossbar bytes	275	277	+2
Ternary-match crossbar bytes	169	171	+2
Action bus bytes	482	488	+6
Latency, ingress (cycles)	410	410	0
Latency, egress (cycles)	154	154	0
Power estimate, ingress (W)	14.93	14.86	-0.08
Power estimate, egress (W)	1.59	1.59	0
Deparser FDE entries, ingress	51	51	0
Deparser FDE entries, egress	42	42	0
Deparser POV bits, ingress	21	21	0
Deparser POV bits, egress	17	17	0
Compile time (s)	85.17	72.21	-12.96

PHV containers

Ingress

From tof phv -g ingress sidecar.bfa

Ingress PHV	main	nsk/662-modular-router-config	Δ
Normal word	30 / 48 (62.5%)	30 / 48 (62.5%)	0
Normal half	61 / 72 (84.7%)	61 / 72 (84.7%)	0
Normal byte	9 / 48 (18.8%)	8 / 48 (16.7%)	-1
Mocha word	15 / 16 (93.8%)	15 / 16 (93.8%)	0
Mocha half	23 / 24 (95.8%)	23 / 24 (95.8%)	0
Mocha byte	8 / 16 (50.0%)	8 / 16 (50.0%)	0
Dark word	2 / 16 (12.5%)	2 / 16 (12.5%)	0
Dark half	10 / 24 (41.7%)	10 / 24 (41.7%)	0
Dark byte	0 / 16 (0.0%)	0 / 16 (0.0%)	0
Total	158 / 280 (56.4%)	157 / 280 (56.1%)	-1

Egress

From tof phv -g egress sidecar.bfa

Egress PHV	main	nsk/662-modular-router-config	Δ
Normal word	12 / 48 (25.0%)	12 / 48 (25.0%)	0
Normal half	6 / 72 (8.3%)	6 / 72 (8.3%)	0
Normal byte	12 / 48 (25.0%)	12 / 48 (25.0%)	0
Mocha word	1 / 16 (6.2%)	1 / 16 (6.2%)	0
Mocha half	1 / 24 (4.2%)	1 / 24 (4.2%)	0
Mocha byte	9 / 16 (56.2%)	9 / 16 (56.2%)	0
Dark word	0 / 16 (0.0%)	0 / 16 (0.0%)	0
Dark half	2 / 24 (8.3%)	2 / 24 (8.3%)	0
Dark byte	1 / 16 (6.2%)	1 / 16 (6.2%)	0
Total	44 / 280 (15.7%)	44 / 280 (15.7%)	0

Total

From tof phv sidecar.bfa

Total PHV	main	nsk/662-modular-router-config	Δ
Normal word	41 / 48 (85.4%)	41 / 48 (85.4%)	0
Normal half	65 / 72 (90.3%)	65 / 72 (90.3%)	0
Normal byte	20 / 48 (41.7%)	19 / 48 (39.6%)	-1
Mocha word	16 / 16 (100.0%)	16 / 16 (100.0%)	0
Mocha half	24 / 24 (100.0%)	24 / 24 (100.0%)	0
Mocha byte	16 / 16 (100.0%)	16 / 16 (100.0%)	0
Dark word	2 / 16 (12.5%)	2 / 16 (12.5%)	0
Dark half	12 / 24 (50.0%)	12 / 24 (50.0%)	0
Dark byte	1 / 16 (6.2%)	1 / 16 (6.2%)	0
Total	197 / 280 (70.4%)	196 / 280 (70.0%)	-1

PHV fields

From tof vars

Container types: bf-p4c/specs/phv.h

Only in nsk/662-modular-router-config

Field	Gress	Containers
meta.routing_table_id	ingress	W

Only in main

None

Changed container kinds

Field	Gress	main	nsk/662-modular-router-config
hdr.geneve.$valid	ingress	B, W	H, W
hdr.geneve_opts.oxg_ext_tag.$valid	ingress	B, W	H, W
hdr.geneve_opts.oxg_mcast_tag.$valid	ingress	B, W	H, W
hdr.inner_eth.$valid	ingress	B, W	H, W
hdr.ipv4.$valid	ingress	B, W	H, W
hdr.udp.$valid	ingress	B, W	H, W
hdr.vlan.$valid	ingress	B	B, H

[cfzimmerman]

You can resolve this if not interested. Just wanted a quantitative look. The p4 diffs look pretty benign, so it surprised me that this pushed us past 17 into 18 stages sans multicast.