[Japabu]

Problem

P5 of issues/there-is-no-wifi.md; its defect is issues/netstack-drives-one-card.md. netstack opens only the first link a driver brings and refuses every other with OneLink. Its node is one interface with one DHCP client, built from that link's address. As a result nothing is served before a driver's hello, and a machine with no NIC driver waits 30 s (LINK_BOUND) and then has no netstack at all. The owner ruled that netstack runs every link a driver brings, wired first, with failover (#876, ruling 3).

Design

One stack over several interfaces. toyos-net-ip already holds several interfaces and routes over them under the strong host model. One Ip, one Tcp and one Udp stay as they are, so routing, listeners and bound sockets keep a single owner. Each layer above them gains a per-link part:

• link says its medium. A hello carries Wired or Wireless after the address; i219 and virtio-net say Wired. The two refusals are now Open (a link of this address is open and alive) and Full (the stack drives its most links). OneLink goes.

• ip ranks interfaces. Each interface is added with a rank, and arrival is its index. Every choice among eligible interfaces now goes by rank and then arrival, where it used to go by index alone: the default gateway, a tie between equally long connected prefixes, the multicast fallback and a link-local neighbour. netstack gives wired rank 0 and wireless rank 1.

• Failover and return come from that rule. A link that goes down leaves the lookup (up is already checked), so new flows go out on the next-ranked link with a route. When it comes back, it ranks first again. A lease outlives its link going down, as it does today.

• A flow bound to a lost address ends. Under the strong host model a TCP connection cannot move to another address. When an interface loses an address, because its lease ends, is replaced by another address, or is taken by another host, every connection from that address ends at once with AddressGone. No reset is sent, since nothing can leave from that address, and the client reads ADDRESS_GONE on its stream. Until now such a flow retransmitted until it timed out. A UDP socket bound to the address stays: each datagram it sends is refused by ip for want of a source, as before. A link that only goes down ends nothing: its address is still held.

• The shard holds several interfaces. Interfaces are added as links arrive, and each frame arrives tagged with its interface. One transmit opportunity serves every link with room:

  • the caller says how much room each link has;
  • each frame is handed out with its interface;
  • a flow whose route leaves by a link with no room waits on that link, as it waits on a next hop today;
  • ip's control queue hands out the oldest frame whose link has room, and bounds each interface's share, so a link that takes nothing holds back no other link's ARP, ICMP, probes or announcements;
  • a next hop is named by its link: TCP's reachability advice carries its connection's local address, and a UDP datagram waits on, and is failed by, its next hop on its own interface.

  The round stays one deficit round-robin over every flow, so fairness between flows does not depend on which link carries them.

• The node holds one link part per link. Each part has its own lease state, DHCP client (sending bound to its interface) and mDNS responder, all under one host name. A name lost to another host on one link is that link's alone: its responder probes for the same name again later, and the other links keep it. On one LAN two links share (a wired and a wireless link bridged, RFC 6762 §14), ip already refuses a datagram from any address its interfaces hold, so the machine never hears its own mDNS. A server's reply addressed to another link's MAC is dropped before this link's client sees it. An arriving datagram says which interface it came by, so a responder hears only its own link. The node is built before any link and serves at once; with no link, a connect is refused for want of a route. LINK_BOUND goes.

• Resolvers. A lookup asks the resolvers of the asked link's lease: of the links up and holding a lease, the first by rank and arrival. With none up, it keeps the link it asked while that link still holds its lease, since a down link keeps it; failing that, it takes the best-ranked held lease. One cache holds the answers. It is emptied whenever any link's state is written, a link goes down, or the asked link changes. A lookup ends with its lease's word, as it does now.

• netstack. It keeps up to MAX_LINKS sessions. A driver that ends and comes back with its address gets its interface back. inspect and the lease lines are per link.

• A flow on a down link. Under the strong host model it does not fail over. An established connection keeps retrying until its link returns. A SYN with no route ends unreachable, as on one link today.

Rejected: one shard per link. It would duplicate the routing choice above ip (longest prefix across shards, then rank) and fan out every wildcard listener and socket to every shard, including shards that arrive later.

What changes

Eight commits; the last three answer review r1 and the rebase onto main.

1. The lower layers.
   • lib/link: the hello's Medium, and the Open and Full refusals.
   • ip: Rank; the ordered lookup; Credit with a per-link control queue; NoRoom; a datagram's interface.
   • udp: the arrival interface; Origin; send_via.
   • tcp: lose_address and Failure::AddressGone.
   • The shard over several interfaces, with per-link credit and a wait on a full link.
2. The node.
   • A part per link: lease state, DHCP client and responder. The asked link.
   • ADDRESS_GONE carried through the SDK, std (AddrNotAvailable) and libc (EADDRNOTAVAIL).
3. netstack.
   • Up to eight sessions, one transmit opportunity across them, and per-link lease lines.
   • inspect under net.link.<n>: driver, medium, state, lease, and netstack's own sent count.
   • No LINK_BOUND, no OneLink.
   • osbench reads the per-link keys.
   • The two-NIC guest test and its profile.
   • The defect closed and the records updated.
4. virtio-net reports its link. The two-NIC test found QMP's set_link reaching nothing: the driver accepted no VIRTIO_NET_F_STATUS, so netstack took its link as up whatever QEMU said.
   • The driver now accepts the feature and reads status (§5.1.4) each pass. A change is a configuration change, which raises its one vector.
   • toyos-virtio's live device reads a 16-bit field of its structure.
   • The link word gains UpWithoutSpeed, its free fourth tag: a carrier with no speed.
5. The SDK. files.rs's count of stream refusals that cross the C interface moves from seven to eight.

Who changes behaviour:

• inspect's NIC keys moved under net.link.<n>, and osbench is their only reader.
• The lease line keeps its netstack: DHCP: lease prefix and names its link after the address.
• A machine with no NIC driver keeps its netstack. The iommu judge for an untranslated NIC now waits for the lease bound's report instead of netstack's exit.
• sshserver on a NIC-less machine now binds port 22, mints its host key and serves once a link brings an address. It used to leave saying the machine has no network.
• test_rs_netstack_answers is gone: it no longer ordered anything. i219_leaves_ulp_unarmed gates on the lease, which follows the card's bring-up.

Checks

Round 2, at d328d25 on origin/main 24a538c:

• cargo run -- --ci host: exit 0, 107 steps.
• Network guest tests, exit 0: the six from round 1, plus main's new a_client_killed_mid_connect_gives_its_place_back. That job read net.lease.held, now per link, and is updated.
• T14: the orchestrator's boot of ad04673 exited 0. Judged here, i219_carries_netstacks_link and i219_leaves_ulp_unarmed both pass.
• New host tests:
  • ip: advice landing on its own link's gateway where two links share one, and a stuck link holding back none of another's frames;
  • shard: a connection's progress confirming its own link's gateway, and one link's dead gateway dropping no datagram waiting on the other;
  • node: one LAN on two links, the name claimed on both and no reply refused, and the asked link checked whichever link goes down last;
  • dhcp: addressed_to.
• Negative controls, each red: the shard advising with Source::Any; ip looking advice up with Source::Any; failures dropping every interface's waiters; one shared queue bound; another link's reply reaching this client; asked falling back by rank.
• Evidence: https://github.com/ToyOSOrg/evidence/blob/fa19b45d44487b87131afe99dd5d910e6e98eb6b/p5-r2/README.md (round 1's index corrected at 9750fbe4).

Round 1's checks: https://github.com/ToyOSOrg/evidence/blob/9750fbe47a4dcb5849c364f5aa8599a3220471a3/p5-r1/README.md

Integration tests and where they run

traffic_leaves_by_the_second_link_while_the_first_is_down runs on QEMU x86-64, with the new Profile::HeadlessTwoNics: two virtio NICs on two user networks. Its claim is what host tests can't establish: two drivers' links, a NIC's carrier set from QMP and netstack's routes, across three processes and two devices. It checks order and counts only. Its datagrams go to TEST-NET-1, and the harness is the peer on the host's loopback, so it needs no public network. No metal machine has two NICs ToyOS drives; P5's T14 row, wired and wireless both leasing, is W6's.

The T14's existing I219 rows (i219_carries_netstacks_link, i219_leaves_ulp_unarmed) passed on one boot. The I219 now says a medium in its hello, and netstack's lines name their link.

Risks

• A flow on a link that goes down doesn't move: a strong-host-model connection waits for its link, and only new flows fail over.
• A link's interface lives for the boot. Up to eight addresses are held, and a ninth driver is refused Full.
• The answer cache is emptied whenever any link's lease state is written, which is wider than strictly needed.
• Two networks leasing one address: the second link never holds it (issues/two-links-leased-one-address-the-second-holds-none.md).

🤖 Generated with Claude Code

[Japabu]

Review of #1122 at ad04673 (P5), round 1

Architecture

The shape is right, and it is what the owner approved (#876 ruling 3). There is one Ip/Tcp/Udp over ranked interfaces. Rank lives in ip, the only place that chooses between interfaces. The node has one part per link, holding its lease, DHCP client and responder. Medium-to-rank policy sits in netstack. Credit is a real boundary: netstack knows ring room and the shard knows routes. Rejecting one shard per link is the right call. I checked these and they are sound:

• No NIC: the node is built before any link, LINK_BOUND is gone, and a connect is refused for want of a route.
• Bounds: MAX_LINKS is eight. Full goes only to a new address, and an ended link keeps its slot, as the PR documents.
• Answer cache: it is emptied on every write of set, every down link and every change of the asked link. That is complete, and wider than needed, as the PR says.
• AddressGone: it ends flows only on remove_address (expiry, NAK, a replaced address) or on ip's Lost. A link going down only ends nothing. AddrDuplicate keeps an address unique across interfaces, so keying lose_address by address is sound. EADDRNOTAVAIL and AddrNotAvailable are a defensible reading. POSIX lists no errno for a send or recv on a connection whose local address vanished.
• DHCP isolation: sends are bound to the interface, and receives go to the arrival link's client.

Net growth: production +463, tests +1008. Most of the test growth is mechanical signature updates.

The problem is that some per-link state is still keyed by an IPv4 address alone. With one interface that was unique. With several links it often is not: the same home gateway can be reached over wired and Wi-Fi, two consumer routers both use 192.168.1.1, and a wired and wireless link can be bridged onto one LAN.

Blockers

1. A next hop named by address alone reaches the wrong link.

   • TCP advice. At system/netstack/shard/src/lib.rs:428-429, TCP's Reachable and Reverify carry only the peer. Ip::advise (ip/src/config.rs:121-127) then looks it up with Source::Any, so the advice lands on the best-ranked link's next hop, not on the next hop of the link the connection uses. The common case is after the wired link returns: a flow still on wireless (strong host model) keeps confirming the wired gateway's entry. That masks the wired gateway's NUD failure detection. A wireless stall's Reverify pokes the wired gateway, and the wireless gateway is never hinted.
   • UDP failure. At shard/src/lib.rs:444-446, Event::Failed { iface, next_hop } drops its iface and calls udp.fail(next_hop) (udp/src/lib.rs:814). That drops, and reports unreachable, every datagram waiting on that address on any link. This includes the new NoRoom waiters, which are common when a wired ring is full under load. One link's dead gateway thus fails another link's traffic.
   • Fix: carry the local address in the TCP events and advise with Source::Bound, and key UDP waits by (iface, hop).
   • Cheapest test: a shard host test with two links and the same gateway address.

2. The shared control queue lets a link without room starve the others. Control has one 64-slot FIFO (ip/src/egress.rs:85). pop skips items whose link has no room, but they still occupy slots. A link that stays up with no ring room fills the queue, and then:

   • the other links' pushes (ARP replies and requests, ICMP) are dropped;
   • their holds (turns, ACD probes, announcements) sit in waiting, which refill only drains when something pops.

   So link B's address never verifies and its released datagrams never leave while A is stuck. A link that is up but not transmitting is the case failover exists for. This contradicts transmit's doc (egress.rs:245) and the PR's claim. rank.rs's test queues two items, so it cannot see the problem.

   • Fix: bound the queue per interface.
   • Test: the same ip test, run past CONTROL_QUEUE.

3. mDNS on a bridged wired and wireless LAN conflicts with itself. RFC 6762 §14 names this exact laptop case. Link A's probe arrives on link B from an on-subnet source with different rdata. B's responder takes it as another host's (node/src/name.rs:7-8, and toyos-mdns §8.2/§9). One link loses the name for good, re-probes every 60 s, and logs Lost. Each client also logs XidMismatch and ChaddrMismatch for the other link's server replies. The T14 row (W6) is this topology.

   • Fix: drop messages sourced from an address one of the node's links holds. Otherwise record it with an owner (W6) and an exit. Leaving it silent is not an option, because P5 says the per-link name is done.

4. asked()'s contract is false. node/src/lease.rs:139-141 and the PR say "with none up the lookups keep asking the one they asked". The code picks the lowest rank among held links. Take wired down, then wireless: wired is asked again, and the answer cache is flushed. node/tests/links.rs:295 passes only because it takes the links down in the other order. Make it sticky, or state what it does and rename the test.

5. Comments made false by serving before any link.

   • tests/toyos.rs:142-143 and :1797-1799, and tests/guest/rust-tests/src/bin/netstack_answers.rs:1-3, say netstack answers only after the I219's bring-up. It now answers at once, so i219_leaves_ulp_unarmed's gate on test_rs_netstack_answers no longer orders anything; only test_rs_lan_lease in the same arm does. Gate on what actually orders it, and keep netstack_answers only with a claim it still makes.
   • system/sshserver/src/main.rs:155-157 says a NIC-less machine "reports the network as the reason it is leaving". sshserver now binds and mints a key there. That is a behaviour change, and the PR's "Who changes behaviour" should list it.
   • tests/guest/rust-tests/src/bin/netstack_gone_mid_bind.rs:7-8: "On a NIC-less machine netstack prints its line and exits".

6. A doc comment is detached. At tests/toyos.rs:5770-5776, the new function was inserted between a_link_ends_under_a_held_stream_and_comes_back's doc comment and the function itself. That doc now reads as the first half of traffic_leaves_by_the_second_link_while_the_first_is_down's doc, and the old function has none.

Evidence

• Host gate: I accept it without a rerun at ad04673. All 107 steps ran at 44007af, and the only red was files.rs's count. ad04673 changes that one literal in a test assertion, and the red step's command exits 0 there (guest-libs.log). No other step can be affected by the change. The blockers above need code changes, so the next round needs a fresh run anyway.
• Guest test and controls: they check what they claim. The two-NIC test needs QMP carrier control across two drivers' processes, which a host test cannot provide, and its target is declared. by[first] == 0 is a real negative. All four controls go red. control-down goes red on the job's send_to panic, not on its count assertion, but that still exposes the defect.
• README revision: it is inconsistent. It says guest-4 ran on "44007afd5's tree", which is already on 49559dd, and also that guest-4 "ran before the rebase onto 49559dd". Name the revision that actually ran. The reasoning about the intervening main commits (fileserver cache, ax2xx files) holds.
• Owed before landing: the T14 run of i219_carries_netstacks_link and i219_leaves_ulp_unarmed (the I219's hello changed). This is the orchestrator's.

Suggestions

• shard/tests/links.rs:166 checks only that the address remains. To hold "a link going down ends nothing", also assert that a flow survives.
• The lease, DHCP and mDNS lines (netstack/src/main.rs:222, serve.rs) name a link by driver label. Two virtio-net links, as in this PR's profile, or two I219s, are indistinguishable. Add link.<n> or the MAC.
• One dhcp_log serves every client, so one link's suppressed count is attributed to another link's line.
• Two separate networks that lease the same address now reach AddrDuplicate. The second link's probe is refused and its client starts over indefinitely. Record it.
• MAX_PENDING_LINKS is 2 against 8 links. A third driver arriving at once is hung up and reconnects, with log lines.
• slirp forwards the job's TEST-NET-1 datagrams out of the host. restrict=on on the profile's netdevs would keep them in.
• node/tests/links.rs:67: let _ = ip; should be NetSlice::Ipv4(_).

SEND BACK

[Japabu]

Round 2: review r1

Blockers

1. Next hops named by their link (4de1180 after the rebase).
   • TCP's Reachable/Reverify now carry { local, remote }. The shard calls ip.advise(now, remote, Source::Bound(local), …), so the advice lands on the entry of the flow's own link.
   • Sent::Pending, Offer::Waits and udp.wake/udp.fail are keyed by (interface, hop), and so is the shard's Wait::Hop.
   • Tests, one LAN on two links with one gateway address:
     • ip: a_flows_advice_lands_on_its_own_links_gateway_where_two_links_share_one;
     • shard: a_flows_progress_confirms_its_own_links_gateway_where_two_links_share_one, an established connection on the wireless link whose data ACK confirms the wireless entry and leaves the wired one stale;
     • shard: a_gateway_that_fails_on_one_link_drops_no_datagram_waiting_on_the_other.
2. Per-link control queue. Control bounds each interface's share at CONTROL_QUEUE, and its echo replies at ECHO_REPLIES; refill admits what waits on each interface's own room. acd.rs's queue-full cases now fill the probing interface's own share, since a second interface no longer holds it back. New test: a_link_that_takes_nothing_holds_back_none_of_another_links_own_frames overfills a stuck link by eight; the other link's request still leaves, and its new address probes, announces and is assigned.
3. One LAN on two links.
   • mDNS self-conflict: refuted. ip already refuses any datagram whose source one of its interfaces owns, checked across all interfaces (ip/src/input.rs:90, IpOwnSource). A bridged link's copy of our own mDNS therefore never reaches a responder. name.rs now says so.
   • DHCP mismatch logs: fixed. toyos_dhcp::addressed_to reads a reply's chaddr, and the node drops a reply addressed to another link's MAC before this link's client sees it (node.dhcp-other-link).
   • Test: on_one_lan_by_two_links_the_name_is_claimed_on_both_and_no_reply_is_refused. Group and broadcast frames cross between the links, and the server broadcasts its replies. The name is claimed on both links, no Lost, no Dhcp event. Its premises are ip.own-source > 0 and node.dhcp-other-link > 0.
   • With the other-link drop removed, the test goes red.
4. asked() is sticky. It is chosen again on each state write and each link change. While no link is up it keeps the link it asked, falling back to the best-ranked held lease. The doc says so. with_no_link_up_lookups_ask_the_lease_they_asked now runs both orders, and reverting to the old rule turns it red.
5. Comments made false.
   • i219_leaves_ulp_unarmed gates on test_rs_lan_lease. A lease follows the card's bring-up, so it orders the lines. netstack_answers is deleted, since its remaining claim (a bind answered) is held elsewhere.
   • The BENCHCASE_ALONE, lan_lease, sshserver and netstack_gone_mid_bind comments are corrected.
   • The PR body now lists sshserver's change: a NIC-less machine binds port 22 and mints its host key.
6. The doc comment is back on a_link_ends_under_a_held_stream_and_comes_back.

Suggestions

• Taken:
  • each link's DHCP client has its own refusal log;
  • link lines say link <n> (<driver>), and the open line …'s link is open, as link <n>, keeping the substrings the tests read;
  • MAX_PENDING_LINKS = MAX_LINKS;
  • the same-address case is recorded in issues/two-links-leased-one-address-the-second-holds-none.md;
  • NetSlice::Ipv4(_).
• Declined: restrict=on. With it, the failover job's first datagram to TEST-NET-1 was refused NotConnected: the guest had no route off its networks, and the default route is what the test checks. I reverted it (a3a2e18). The datagrams go to a documentation address and nothing answers them.
• Not taken: a flow surviving a link down, at shard level. It needs an established connection on a down link. a_link_that_only_goes_down_keeps_its_address holds the invariant lose_address rests on.

README revision: this round's index names each run's revision exactly.

Checks at d328d25 (on main 24a538c): --ci host 107 steps green; the network guest tests exit 0, with main's new mid-connect test updated for the per-link lease key; the T14 boot judged, both I219 rows PASS; seven controls red. Evidence: https://github.com/ToyOSOrg/evidence/blob/fa19b45d44487b87131afe99dd5d910e6e98eb6b/p5-r2/README.md

[Japabu]

Round 3 at a478fe8, on main 24a538c: both required items fixed.

• i219_leaves_ulp_unarmed's comment says it reads the lines once its job has seen the lease, which follows bring-up.
• acd.rs: the helper is answering() (if0 holding ANSWERING); callers name h.if0; the comment is fixed; the two asserts check the leading frames are if0's ARP replies.
• Optional, taken: the mDNS bridge race noted in name.rs; "depends on no public network"; udp and shard headers say "the hop on its interface"; lease.rs line wrapped. The control count in my reply should read six.

Checks: cargo test of ip, udp, shard and node exit 0 (593); clippy on them clean; cargo check --tests clean; cargo run -- --ci host lint exit 0 (5 steps). Nothing here changes code the image runs.