Ten seconds of INITs and not one answer
Change one word in the lab. On natB, replace the masquerade rule with the same
rule plus --random-fully, restart B, and ping it from A. Nothing. Run
meshd status on A and the peer line says path none,
session none. The candidates are there, two of them, exactly as coordd
sent them. tcpdump on the cloud bridge shows A's INITs leaving
natA toward 10.0.0.3:41200 every 500 ms, and B's INITs
leaving natB toward A, from a port nobody told A about.
You saw why in the NAT traversal lesson. --random-fully makes
natB pick a fresh public port for every destination. The port STUN reported
was the port natB opened toward coordd's UDP socket. For packets to
natA it opens a different one. So A's INIT arrives at natB on a
port whose only row points at coordd; the lab NAT treats it as a new inbound flow and
drops it. B's INIT arrives at natA from a port A never sent to;
natA has a row for 10.0.0.3:41200, not for this one, and drops
it too. Both sides keep punching a wall.
No amount of timing fixes this. What fixes it is a destination both nodes can reach, passing bytes between them: the relay. It is the one place in the course where traffic touches a machine that is not a peer, so the design question is what that machine gets to see.
[7][dst key][ciphertext]; it cannot decrypt and cannot forge.
Every packet that reaches it was already a complete, encrypted meshd packet. The relay
adds an envelope, reads the address on the envelope, and copies the contents.
Three packet types, one 33-byte header
wire.h gives the relay three types. All three start the same way: one type
byte, then a 32-byte public key. RELAY_HDR_LEN is 33.
| type | name | layout | who sends it |
|---|---|---|---|
| 9 | RELAY_REG | [9][my_pub 32] | node → relay, every 25 s |
| 7 | RELAY_SEND | [7][dst_pub 32][inner packet] | node → relay |
| 8 | RELAY_RECV | [8][src_pub 32][inner packet] | relay → node |
The inner packet is whatever the node would have sent directly: a 117-byte INIT, a RESP,
or a DATA packet with its 13-byte header, ciphertext and tag. The relay does not parse it.
It does not know which of the three it is carrying. It copies len − 33 bytes
from one socket address to another.
Registration is the part that makes the NATs cooperate. Both nodes send a REG to the
relay first. That packet is outbound, so each NAT creates a row for
relay_ip:7002. Everything the relay later sends back comes from exactly that
address and port, so it matches the row and gets through. The relay's reply never has to
punch anything, because the node punched for it.
relayd.c
The whole server is one UDP socket, one table and a loop. The table maps a public key to
the endpoint that key last registered from, with a timestamp. An entry older than
REG_TTL_MS, 75 seconds, is treated as gone: three missed REGs and the node
is unreachable through the relay until it registers again.
Read handle_send twice. It looks the sender up by the address the
packet came from, and the destination by the key in bytes 1–32. If either lookup
fails the packet is dropped silently: an unregistered sender gets no service, and a peer
whose registration expired is asleep as far as the relay is concerned. Then it rewrites the
header. The outgoing packet says [8][src_pub] so the receiver knows which peer
is talking, which matters because the UDP source address of a relayed packet is always
the relay. The inner bytes are copied unchanged, untouched, unread.
Look at the Makefile: RELAYD = src/relayd.o src/util.o. The relay is not even
linked against crypto.o. The only two copies of a session key live in two
struct sessions on two other machines.
[9][B's key] from their own address and the relay will believe it.
They still cannot read or forge B's traffic: the next INIT from A, forwarded to the
wrong place, fails to open because only B holds B's private key. But they can
black-hole B until B's next REG. Tailscale's DERP makes the client prove it holds the
key before it will forward for that key. This repo does not; it is in the gap list in
lesson 12.
The node side: path.c falls back
Lesson 09 gave each peer a punching phase: INIT to every candidate, every 500 ms, with
punching_since_ms marking when it started. The relay fallback is three
additions to that state machine, all in tick_punching.
After ten seconds without an authenticated answer, via_relay flips to true and
last_probe_ms is zeroed so the next tick sends immediately. Two things change
at once. send_init_burst still sends the INIT to every direct candidate, and
now also wraps one copy in RELAY_SEND. And the burst slows from every 500 ms to
every 5 s: the fast phase was for racing the NAT timers, and that race is over.
path_send is where every finished packet goes, DATA and INIT alike. Its first
branch is the relay. While via_relay is set, a DATA packet is encrypted
exactly as before by session_encrypt, and only then handed to
send_via_relay, which prepends 33 bytes. The encryption code never learns that
a relay exists.
Registration lives in meshd.c's timer code: cmd_up calls
path_relay_register once before the loop, and tick calls it again
whenever RELAY_REG_MS, 25 s, has passed. 25 against the relay's 75 means two
lost REGs in a row still leave the node reachable. Receiving is one case in
handle_packet:
One recursive call. Strip 33 bytes, set via_relay = true, and dispatch the
inner packet through the same switch that handles direct traffic. The INIT,
RESP and DATA cases do not have a relay branch. The handshake code from lesson 06 and the
session code from lesson 05 run unchanged. Two guards keep this honest: the outer packet
must really come from the configured relay address, and a RELAY_RECV inside a RELAY_RECV
is refused (!via_relay), so nobody can nest envelopes to spoof the relay.
Step through what the two nodes and the relay see. The relay card in the middle only ever shows a type byte, a key and noise.
Switching back
The relay is slower and it costs someone bandwidth, so it must never win when a direct
path exists. That rule is written in path_on_authenticated, the same function
that handles roaming:
A relayed packet can turn the relay on only if no direct endpoint is known. A direct
packet turns it off unconditionally and records the address it came from. Because the
burst after the 10 s timeout keeps going to every candidate as well as through the relay,
a peer whose NAT eventually cooperates will answer directly, and that RESP, arriving
with via_relay == false, flips the path back. The log line says
(leaving relay). The next DATA packet goes straight to the peer with no
33-byte envelope.
path_tick: with a usable session it runs tick_established, which
sends keepalives and rekeys through path_send, and path_send
obeys via_relay. The direct candidates are tried again only when the path is
rebuilt: after 90 s of silence drop_path clears via_relay and the
next burst goes direct for ten seconds first, or when the peer's own burst reaches us
directly. Tailscale probes direct paths continuously while on DERP and upgrades
mid-session. Adding that here is one timer in tick_established that calls
send_init_burst; the switch-back logic above already handles the answer.
What DERP adds
Tailscale's relay is called DERP, Detour Encrypted Routing Protocol, and it is this program with four more decisions. It runs over TCP with TLS on port 443, not UDP, so it works from networks that block everything else; the cost is head-of-line blocking, which is why it is a fallback. Clients authenticate to it with their node key, which closes the hole in the danger callout above. There are many relays grouped into regions, and each client picks its "home" region by latency; a packet for a peer homed elsewhere is forwarded relay to relay, so any two nodes on earth are at most two relay hops apart. And the first handshake packet always goes via DERP while direct probing runs in parallel, so a connection is usable in one round trip and upgrades to direct a moment later. Same 33-byte idea: a key names the destination, the relay never holds a session key.
Run it
The easiest way to see the fallback is section 3 of the lab test, which turns
natB symmetric and restarts B on a new port. From the project root, on Linux as
root, or inside the lab container on any host:
To do the same by hand, with the lab up and coordd, relayd and both nodes running as in
lesson 09 (every node started with --relay 10.0.0.1:7002):
path via_relay with no endpoint field: status.c prints
the endpoint only when endpoint_ok, and the relay never sets it.
no_session 1 is the first ping's packet, staged for ten seconds while the direct
burst failed. In meshd-A.log, in order: no direct answer in 10 s, trying
relay, reachable via relay, established (initiator, via relay).
tcpdump -i br0 -nn udp port 7002 in the cloud namespace shows every packet
starting with 07 or 08 and 32 bytes of key; nothing after byte 33
repeats.
session_encrypt runs before path_send
picks a destination, so the relay, like the NAT, like the ISP, only ever handles bytes
that were already sealed. Any component you add later with the same property is safe to
add.
When both NATs refuse, both nodes send to a third address they punched themselves, and that address forwards by public key: it reads 33 bytes and copies the rest, so it can route your packets without ever being able to read them.