Shipping it

Ping prints nothing. Now what?

Every failure you have seen so far happened in a terminal where you could read the daemon's stderr. A node you actually use does not run in a terminal. It starts at boot, survives you closing the lid, and when ping 100.64.0.2 hangs you need to know, in one command, which of eight things went wrong: no overlay address, coordd unreachable, no peer entry, no candidates, punching, relayed, session expired, or a packet dropped by the source check. All eight are fields in struct node and struct peer. They are in the daemon's memory and nowhere else. So the daemon has to hand them out.

meshd status: a unix socket that dumps the tables

status.c is forty lines. In cmd_up, after coordd has answered, status_listen binds a SOCK_STREAM unix socket at /run/meshd.sock, or wherever $MESHD_SOCK points, and the fd goes into the poll set as the fourth entry. A unix socket is an fd like any other, which is why the loop needed no new mechanism. When it becomes readable, one connection is accepted, the tables are printed into it with fprintf, and it is closed:

meshd status is the client half: connect, copy to stdout until read returns 0, exit. No protocol, no parsing; the daemon's fclose is the EOF the client waits for. tailscale status and wg show have the same shape, plus a JSON mode for scripts.

Read the output in order and it answers the eight questions. control disconnected: coordd is unreachable and control_connect retries every 5 s. peers 0: nobody was introduced, or an ACL hid them. path none with candidates listed: punching is in progress or failed with no relay. session 175s: a rekey due at 120 s did not happen. last_rx 80s: the path dies in ten more seconds. The drops line names which of the four rejection points in meshd.c is counting.

Logging

util.c has one function, vlog, behind log_info, log_warn, log_err and die. It writes to stderr: a wall-clock timestamp with milliseconds, a level, the message, a newline. Nothing else. Under systemd, stderr is the journal, which adds the unit name and a stable cursor, so the daemon does not need a log file, rotation or a config option for either.

Every log line in meshd marks a state change: an endpoint learned, a peer added or removed, a session established and how, a path dropped, a fallback to the relay. Per-packet events are counters in n->drops instead; at a thousand packets a second, a log line per packet is a denial of service you run against yourself.

A systemd unit

Install the binary and the key, then write this file. It is complete and it works.

/etc/mesh/env holds three lines, MESH_COORD=vpn.example.com:7000, MESH_AUTHKEY=…, MESH_RELAY=vpn.example.com:7002, with mode 0600, so the secret is not in a file every user can read. Restart=on-failure covers the case where coordd is down at boot: control_connect fails, die exits 1, systemd tries again in three seconds, forever. RuntimeDirectory creates /run/mesh at start and removes it at stop, and the Environment line puts the status socket inside it. For meshd status to find it, either export MESHD_SOCK in your shell or run MESHD_SOCK=/run/mesh/meshd.sock meshd status.

The unit runs as root. tun_open needs CAP_NET_ADMIN to create the interface, and so does every ip command system() runs. You could add User=mesh and AmbientCapabilities=CAP_NET_ADMIN; the capability is inherited by the ip child, so that part works, but the /dev/net/tun device node must also be readable by that user. Tailscale runs tailscaled as root for the same reasons.

Three machines

The lab put coordd, relayd and two nodes on one host. The real layout is one public host and two nodes anywhere. The public host needs three open ports: TCP 7000 for the control connection, UDP 7001 for STUN, UDP 7002 for the relay. meshd assumes the STUN port is the coordd TCP port plus one, so keep them adjacent. Nothing needs to be open on either node, which was the whole point of lessons 08 through 10.

coordd and relayd want units of their own, the same shape as above with a WorkingDirectory for nodes.db. The first laptop gets 100.64.0.1, the second 100.64.0.2, and each keeps its address across reboots because nodes.db remembers the key.

The whole journey

You have now written every function a ping touches. Step through the viz and read the function names alongside it. Sixteen hops; your code is nine of them, the kernel is the rest.

1. ping writes 64 bytes to an ICMP socket; the kernel prepends a 20-byte IPv4 header. 84 bytes.
2. Kernel routing: 100.64.0.0/10 dev tun0, installed by tun_configure, beats the default route.
3. on_tun_readable: one read on the tun fd returns the 84 bytes; ipv4_valid checks version and length.
4. lookup_by_overlay_ip(ipv4_dst(pkt)): longest-prefix match over every peer's allowed[]. Cryptokey routing, outbound half.
5. path_send_data → session_encrypt: 13-byte header with the receiver's index and the counter, then ciphertext and tag. 113 bytes.
6. path_send → udp_send: the kernel wraps it in UDP and IPv4 and routes the outer destination out eth0.
7. NAT-A rewrites the source address and port using the row A's own earlier packets created.
8. The internet carries two public endpoints, a size, a time and a type byte. Nothing else is readable.
9. NAT-B rewrites the destination back to B's LAN address, through the row B punched toward A.
10. B's kernel strips the outer headers; on_udp_readable gets the same 113 bytes A produced.
11. handle_packet: one switch on byte 0. Type 3.
12. lookup_by_rx_index: B's own index from the header names the session, and so the peer and the rx key.
13. session_decrypt: the counter must be new under the replay window; the tag must verify before a byte of plaintext exists.
14. peer_allows_src(p, ipv4_src(plain)): the inner source must be one of the addresses coordd assigned to this key. Cryptokey routing, inbound half.
15. write(n->tun_fd, plain, plen): the kernel believes a packet arrived on tun0.
16. The kernel delivers it to the ICMP code, which builds the echo reply. Fifteen hops back, and ping prints a line.

That list is also the checklist for the question that opened this lesson: steps 2 and 15 are ip route, steps 4, 12, 13 and 14 are the drop counters, steps 7 and 9 are the path and candidates fields. No hop is unobservable from a shell on one of the two machines.

What this is not

Each lesson named its simplification as it made it. Here they are in one place, with what WireGuard and Tailscale do instead, so that when you read their code you recognize what each piece is for.

None of these are mysteries now. Each is a function you would add next to one you already wrote: cookies next to handshake_handle_init, the packet filter next to peer_allows_src, netlink in place of set_route, key expiry as one more column in nodes.db. The architecture does not change. That is what it means to have built the real thing small.

Reading list

• Jason A. Donenfeld, WireGuard: Next Generation Kernel Network Tunnel (NDSS 2017). Twenty pages. You will recognize every section: cryptokey routing, the handshake, the counter nonce, the sliding window, cookies, roaming.
• David Anderson, How NAT traversal works (Tailscale, 2020). Lessons 08 through 10 of this course in long form, including the cases this repo does not handle: hairpinning, NAT64, and why birthday attacks on symmetric NATs sometimes work.
• Trevor Perrin, The Noise Protocol Framework. Read section 7 for the handshake patterns and find IK; then reread handshake.c and list what it lacks.
• RFC 1918 (private address space) and RFC 6598 (100.64.0.0/10, shared address space). Why the overlay lives where it does, and why it never collides with your office LAN.
• RFC 4787, NAT behavioral requirements for UDP. The vocabulary (endpoint-independent mapping, address-and-port-dependent filtering) behind the four NAT kinds you met in the networking track.
• The project repo. Read src/meshd.c top to bottom one more time. It should take ten minutes now.

Run it

The last run is the one on real machines, which is the sequence in the three-machines section above. The checks are the lab's checks, done by hand on two laptops behind two different NATs:

Six log lines, one per state change, 1.3 seconds from start to an established direct session through two NATs neither of you controls. The 38 ms is the real distance between the two laptops; the relay at 192.0.2.99 was configured and never used. tcpdump -i any udp port 51820 on either side shows type-3 packets and nothing readable after byte 13.

That is the course. You started with a program that read a packet from a file descriptor and sent it in the clear, and you finished with three programs that find each other through NATs, prove who they are with keys, encrypt everything, route by prefix, say no by policy, and tell you what they are doing when asked. Every one of those properties is a function you can point to. The next time something says "secure" or "mesh" or "zero-config", you know which functions to go looking for.