Subnet routes and ACLs

A printer that is not on the mesh

B's office has a printer at 192.168.1.20. It does not run meshd. It will never run meshd. From A, ping 100.64.0.2 works, but ping 192.168.1.20 gets nothing, and meshd status on A shows no drop counter moving. meshd never saw the packet.

Run ip route get 192.168.1.20 on A. It says via 10.0.1.1 dev eth0: the default route. A's kernel has one route into tun0, 100.64.0.0/10, and 192.168.1.20 is not in it. So the packet left through the real interface toward natA, which has no idea what 192.168.1.0/24 is either. From the routing lesson: the kernel picked the longest prefix that matched, and the only match was 0.0.0.0/0. Suppose you fix that with ip route add 192.168.1.0/24 dev tun0. Now on_tun_readable reads the packet, calls lookup_by_overlay_ip with 192.168.1.20, finds no peer whose allowed[] contains it, and drops.no_peer goes up by one. Same problem, one layer down.

Two tables, two longest-prefix matches, and both of them need a new row. The design decision is who adds the rows. Not the student at a shell on every node: the node that owns the subnet announces it, the coordination server passes the announcement on, and every other node updates both tables on its own.

You stepped through the kernel's table in the routing lesson. Step through it again with this in mind: the peer table in peers.c runs the same algorithm over the same kind of rows, except that a row resolves to a public key instead of an interface.

--advertise and the ROUTES line

meshd up --advertise 192.168.1.0/24 stores the string in n->advertise. That is the only thing the flag does at startup. control_connect, right after it sends ENDPOINTS, sends it to coordd:

coordd does not parse the CIDRs. It keeps the string in c->routes and appends it to the last field of this node's PEER line, after the /32 that is always first. The line A receives about B changes from PEER Zr8p… 100.64.0.2 10.0.2.2:51820,10.0.0.3:51820 100.64.0.2/32 to the same thing ending in 100.64.0.2/32,192.168.1.0/24. Then broadcast_lists re-sends the full list to everyone, as it does for any change. Simple beats clever: the node side needs no "route added" message, because it already knows how to apply a complete PEER line.

allowed[] becomes a prefix table

In lesson 03, allowed[] held exactly one entry per peer, the overlay /32, and lookup_by_overlay_ip was a longest-prefix match over a table where every prefix was 32. That was the design waiting for this lesson. Here is what apply_peer does with the last field now:

Each CIDR in the field goes into allowed[], up to MAX_ALLOWED, eight. Every entry with a prefix shorter than 32 also becomes a kernel route on tun0. The /32 does not need one; it is already covered by the 100.64.0.0/10 route tun_configure installed. The code says ip route replace, not add: a PEER line is re-applied on every broadcast, and add would fail with EEXIST the second time. It is still system() running the ip binary, as in lesson 02; a product would speak netlink.

Now both tables have the row. A packet for 192.168.1.20 matches 192.168.1.0/24 dev tun0 in the kernel, arrives in on_tun_readable, and lookup_by_overlay_ip finds B's entry with best_prefix = 24. It is encrypted for B's session like any other packet. The inner destination is a LAN address; the outer destination is B's endpoint. On the way back, B's inbound check peer_allows_src runs against A's allowed[], and on A, a reply with inner source 192.168.1.20 passes because B's allowed[] now contains the /24. The access-control story from lesson 03 did not change. The table it reads just grew.

When a peer disappears, remove_peer walks allowed[1..] and calls set_route with add = false for each, so the kernel route goes away with the key that owned it.

The router node

The receiving side is done. The advertising node needs two things from the kernel that a laptop does not have switched on. When B's meshd writes A's packet into tun0, the destination is 192.168.1.20, not B. A kernel receiving a packet that is not for itself discards it, unless ip_forward is on. And the printer's reply is addressed to 100.64.0.1, which the printer has no route for; it sends the reply to its default gateway, which is not B, and the packet is lost. The usual fix is to masquerade: rewrite the source of forwarded packets to B's own LAN address, so the printer answers B, and the NAT table on B maps the reply back to 100.64.0.1. You built exactly this table in the NAT lesson; now you are the router.

In the lab the advertised address is B's own second interface, lan0 with 192.168.1.1/24, so B answers the ping itself and neither command is needed. On a real office network with a real printer, both are.

acl.conf: saying no on the control plane

You now have a node that can reach a whole office subnet. The next question from anyone who runs this for other people is: how do I stop node C from reaching it? The data plane already has an answer, and it is the one you have been using since lesson 03. A node can only send to peers in its table, and only accept from keys in its table. So the control plane does not need to teach the data plane anything new. It needs to tell each node less.

coordd --acl acl.conf reads a file of allow lines, each naming two public keys or *:

acl_allows is symmetric by construction: it tests the rule both ways round, so if A may see B then B may see A. There is no "deny" verb. The absence of a matching allow is the deny, and with the file active, a node with no rule is told about nobody. Without --acl at all, acl_active stays false and the server behaves as in lesson 07. The filter runs in send_peer_list, once per recipient: a PEER line is sent only if the pair is allowed. That is the entire enforcement mechanism.

Reloading is a signal, not a restart. SIGHUP sets a flag; the poll loop checks it, calls acl_load and then broadcast_lists. Every node receives a fresh full list ending in PEERS_END, and finish_batch in control.c removes any peer that was not in it, kernel routes included. A node that was just hidden from you vanishes from your table within one round trip, without your meshd knowing an ACL exists.

Why the peer table is enough

Follow a packet from A to a hidden B through the code. Outbound, on_tun_readable calls lookup_by_overlay_ip, which returns NULL because no entry has 100.64.0.2/32; drops.no_peer increments and nothing leaves the socket. There is no session to encrypt with and no endpoint to send to, so there is nothing to leak. Inbound, if a stale or malicious B sends an INIT anyway, handshake_handle_init looks up the static key in the first 33 bytes with lookup_by_pubkey and returns "not our peer". A DATA packet carrying an old index fails in lookup_by_rx_index because finish_batch zeroed the session with the entry. Both count as bad_auth. The data plane needs no filter rules, because the data plane was never willing to talk to a key it had not been introduced to.

Tailscale adds one more layer that this repo does not have. Its ACL grammar is not allow key key but src → dst:port with protocols, users and tags, and it is directional: A may open an SSH connection to B without B being able to open one to A. Hiding a peer cannot express "port 22 only", so the control plane also compiles a per-node packet filter, a list of allowed inner (src, dst, proto, port) tuples that the node checks after decryption, right where peer_allows_src runs here. The peer table handles who; the packet filter handles what. You have the hook for it: it would be one more test next to peer_allows_src in handle_packet.

Run it

Sections 4 and 5 of test/run.sh do both halves. By hand, with the lab up and both nodes running as in lesson 09, give B a second interface and restart it with the flag:

Two rows into tun0; the second was installed by B's announcement passing through coordd, not by you. 192.168.1.1:51820 also appeared in B's candidates, because endpoints_csv lists every non-loopback interface. A probed it with the others and got no answer, which is correct.

Now the ACL. coordd in the lab was started with --acl /tmp/mesh-lab/acl.conf, a file that does not exist yet, so it is currently allowing everyone. Write a rule that allows A to see only C, a key that is not online, and signal coordd:

peers 0, and ip route on A no longer lists 192.168.1.0/24: remove_peer took it with the entry. After the second SIGHUP the full list comes back, the handshake runs from scratch, and the ping answers within seconds. meshd-A.log shows control: peer 100.64.0.2 removed, then control: peer 100.64.0.2 added (3 candidates, 2 allowed). meshd never saw a rule; it saw a shorter list, then a longer one.