Hole punching

Ten seconds of INITs into a wall

Here is the rule the design started with, and it reads sensibly: the initiator is whoever has a packet to send and no session. A has a staged ping. A sends HANDSHAKE_INIT to each of B's candidates. B's LAN address is unreachable. B's public address, 10.0.0.3:51820 in the lab, is behind a router that has never seen B talk to A's public address, so there is no row, and the router drops the packet. Every 500 ms, another INIT, another drop. B has no packet for A, so by the rule B is not an initiator, and B's router never gets an outbound packet toward A that would create the row. After ten seconds A gives up. The lab printed path none on both sides, and the ping timed out.

You already know why from the NAT traversal lesson: a hole is punched from the inside. Each NAT opens only for the destination its own host has sent to. For a packet from A to reach B, B must have sent to A first, and for B's packet to reach A, A must have sent to B. Both sides have to send. Which side "wants" the session is irrelevant to the routers.

The fix: every node probes every peer it knows

path.c runs a small state machine per peer, driven from tick() on every pass of the poll() loop. The rule became: if there is no usable session with a peer, probe it, staged packet or not. When PEER lines arrive from coordd, both nodes start probing each other within one tick, each one's INITs opening its own NAT toward the other, and whichever INIT arrives second finds a row waiting. The cost is one 117-byte datagram per candidate per 500 ms for ten seconds, then one every five seconds, for every peer the node cannot reach. For a mesh of tens of nodes that is less traffic than a single idle SSH session.

The elided lines are the relay fallback, in src/path.c and in lesson 10. Three things to notice. The burst goes to every candidate: the LAN address and the public one from lesson 08, at the same instant. Whichever is reachable will answer, and a wrong one costs a dropped datagram. The INIT is the real handshake message from lesson 06, not a special probe packet: the first thing that gets through is also the first half of a key exchange, so there is no second round trip to "connect" after the hole is open. And because both sides probe, both sides may send INITs at once; that is the simultaneous initiation case lesson 06 settled with the smaller-key-initiates tie-break. You already paid for this.

The first authenticated reply wins

What makes a candidate the endpoint? Not that you sent to it. Not that something came back from it. That an authenticated packet came from it: a RESP whose box opened under the peer's static key, or a DATA packet whose tag verified. The UDP handler in meshd.c calls path_on_authenticated only after handshake_handle_init, handshake_handle_resp or session_decrypt succeeded, and passes the from address the kernel reported.

path_on_authenticated does two jobs with one assignment. During punching it selects: the first candidate to produce an authenticated packet becomes endpoint, and path_send uses only that from then on. After that it roams: if an authenticated packet arrives from an address that is not the current endpoint, the endpoint moves. A laptop that walks from wifi to a phone hotspot gets a new public ip:port; its first keepalive or data packet from the new address arrives at the peer, decrypts, and the peer's next packet goes to the new place. No signaling, no reconnect. This is exactly how WireGuard behaves, and it is why a WireGuard tunnel survives a network change without the user noticing.

Keeping the hole open, and noticing when it closes

A NAT row that carries no traffic is aged out, often after 30 s. If both sides are quiet, both rows vanish, and the next real packet starts from zero. So tick_established sends an empty DATA packet whenever nothing has been sent for KEEPALIVE_MS, 25 s. It is a real session packet with a real counter, so the receiver authenticates it, calls path_on_authenticated, and then writes nothing to tun0 because the plaintext is zero bytes. It refreshes both NAT rows and both last_rx_ms stamps for the price of 29 bytes.

The converse: if nothing authenticated has arrived for DEAD_PATH_MS, 90 s, the path is declared dead. Three keepalive intervals and change. drop_path zeroes both sessions, clears endpoint_ok, and the next tick is back in tick_punching with the 500 ms burst, because whatever broke the path may have changed the addresses too. Nothing here consults the peer; it is a local decision based on local clocks, which is the only kind a node can make about a path.

You saw this sequence in the NAT traversal lesson with a "punch" string as the probe. Step through it again; now the probe is a 117-byte HANDSHAKE_INIT, the packet that counts as "connected" is an authenticated RESP, and the 25-second keepalive is an empty DATA packet.

Timers without a timer thread

None of this uses sleep, alarm or a second thread. path_next_deadline computes, for one peer, the earliest moment it will need attention: the next burst, the next keepalive, the dead-path cutoff, or the rekey. next_timeout in meshd.c takes the minimum over every peer and the node's own timers and hands it to poll() as the timeout. The loop wakes exactly when something is due or when an fd is readable, whichever is first, then runs tick() for everyone. This is the event-loop lesson applied with a dozen timers instead of one.

Run it

The lab from test/netns-up.sh: node A in its own namespace behind natA, node B behind natB, both NATs masquerading onto a bridge in the cloud namespace where coordd listens on 10.0.0.1. A and B cannot see each other's LAN addresses at all. On a Linux host run it as root; on a Mac, in the Docker image from test/Dockerfile. Leave --relay off this time: there is no relay yet, and you want to see the direct path win on its own.

Read the status line slowly. candidates lists B's LAN address and B's public address; endpoint is the public one, because the LAN one never answered. path direct means no third machine is carrying this traffic. session 6s with no_session 0 says the handshake finished before you typed ping: the probing started the moment the PEER line arrived, not when traffic did. Type the ping within that first second and the first echo request is staged instead, and no_session reads 1. A's log says responder: both probed, B's INIT found A's row first, and A answered. Run it again and it may say initiator. Either way, it took under a second, through two routers that each refuse unsolicited packets.

Now set --random-fully on natB's masquerade rule, as test/run.sh does in section 3, and run it again. The public port B reports to coordd is not the port natB uses toward A, A's INITs go to a port with no row, and path stays none. That is a symmetric NAT, and it is the next lesson.